I've seen this happen a lot when teams start splitting a monolith into smaller, installable libraries. You want to keep a consistent naming convention—like company.utils—but you want different teams to maintain company.utils.network and company.utils.database in separate repositories. You set it up, you install both, and then... one of them just vanishes.

The disappearing sub-package problem

Imagine you have two separate folders (perhaps installed via pip into your site-packages) that are supposed to contribute to the same acme namespace. Your file structure looks like this:

# Project A (Installed)
acme/
└── __init__.py
    └── auth/
        └── __init__.py
        └── session.py

# Project B (Installed)
acme/
    └── telemetry/
        └── __init__.py
        └── metrics.py

You try to run this code:

from acme.auth import session
from acme.telemetry import metrics

print("Success!")

You'll get a ModuleNotFoundError: No module named 'acme.telemetry'. This is infuriating because you can literally see the telemetry folder sitting right there in your environment. You might spend an hour checking your PYTHONPATH or restarting your IDE, thinking it's a caching issue. It isn't.

Stripping the init files to unlock the namespace

The culprit is that __init__.py file inside the first acme/ directory. In Python, if a directory contains an __init__.py, Python treats it as a regular package. Once it finds a regular package with that name, it stops looking elsewhere in your system paths for other directories with the same name.

To fix this, we need to turn acme into a Namespace Package. In modern Python (3.3+), this is incredibly simple: you just delete the __init__.py from the namespace directory.

Here is the corrected structure:

# Project A
acme/
└── auth/
    └── __init__.py
    └── session.py

# Project B
acme/
    └── telemetry/
        └── __init__.py
        └── metrics.py

By removing acme/__init__.py, you're telling Python: "This isn't a single, self-contained package. It's a namespace. Please scan all available paths in sys.path and merge everything you find under the acme name." Now, when you import acme.telemetry, Python doesn't stop at Project A; it keeps searching and finds the module in Project B.

How Python actually finds these pieces

I find it helpful to think of regular packages as "closed" and namespace packages as "open." A regular package is a single unit of distribution. A namespace package is more like a shared folder that multiple different distributions can drop files into.

• Regular Packages: Defined by the presence of __init__.py. They are fast to resolve because Python stops searching the moment it finds the directory.
• Namespace Packages: Defined by the absence of __init__.py. Python uses a special importer that collects all matching directories across your entire environment and creates a virtual package from them.

One word of caution: you can still have __init__.py files in the sub-packages (like auth/ or telemetry/). Those are still regular packages; it's only the shared parent (the namespace) that must remain "empty" of an init file.

📋 Practical Task

Splitting a Cloud Toolkit into Distributed Namespace Packages

You are building a cloud toolkit where the core logic and the AWS-specific extensions are maintained in two different repositories. Currently, the AWS extensions are not being imported because the core toolkit is acting as a regular package.

Your Goal: Modify the file structure and code to ensure both cloud_tool.core and cloud_tool.aws can be imported simultaneously.

Current Broken State:

# Directory: /libs/core_pkg/
cloud_tool/
├── __init__.py
└── core/
    └── __init__.py
    └── engine.py (contains: def run(): print("Core engine running"))

# Directory: /libs/aws_pkg/
cloud_tool/
└── aws/
    └── __init__.py
    └── s3.py (contains: def upload(): print("Uploading to S3"))

Task:

1. Identify which file must be deleted to convert cloud_tool into a namespace package.
2. Write a Python script that adds both /libs/core_pkg/ and /libs/aws_pkg/ to sys.path.
3. Import both cloud_tool.core.engine and cloud_tool.aws.s3 and call their respective functions to verify the namespace is merged.