skip to content

Why can a package missing its __init__.py import from the source tree but vanish from the installed wheel?

level: seniorimportance: should knowfreq 30%

answer

  1. It works here, not after install
  2. The checkout is on the search path
  3. Two definitions of the word package
  4. Discovery looks for __init__.py, the interpreter does not
  5. Namespace package standing in for the real one

basics

~20 s

Running from the checkout puts the source directory on sys.path, so the bare directory imports as a PEP 420 namespace package. Build backends discover packages by looking for init.py, so it never enters the wheel.

solid answer

~40 s

Two systems disagree about what a package is. The **import system** since Python 3.3 will happily import a directory with no `__init__.py` as a namespace package, and running anything from the repository root puts that directory on `sys.path` — so locally everything works. The **build backend's automatic package discovery** looks for directories containing `__init__.py` and skips the rest unless you configure namespace-aware discovery, so those modules never enter the wheel. The result is a clean local run and a `ModuleNotFoundError` in the environment that installed the artifact. The same missing file produces sibling symptoms: `unittest` discovery, which dropped namespace-package support in Python 3.11, silently contributes zero tests from a `tests/` directory without `__init__.py`, and `pkg.__file__` is `None`, so any data-file lookup built on `__file__` raises `TypeError`.

code

python · 18 lines
python
import pathlib
import sys

def missing_init(source_root):
    root = pathlib.Path(source_root)
    return sorted(
        str(d.relative_to(root))
        for d in root.rglob("*")
        if d.is_dir()
        and any(d.glob("*.py"))
        and not (d / "__init__.py").exists()
    )

if __name__ == "__main__":
    offenders = missing_init(sys.argv[1] if len(sys.argv) > 1 else ".")
    for name in offenders:
        print("no __init__.py:", name)
    sys.exit(1 if offenders else 0)

go deeper

for a junior

Remember the shape of the bug: code that imports fine in your checkout can be missing from the built package. Adding the empty init.py is usually the fix, and running the code outside the source directory is how you notice.

for a middle

Explain both halves — sys.path includes the repository root locally, and package discovery keys on init.py — and name the runtime check (file is None) that distinguishes a namespace package from a regular one.

for a senior

Diagnose it from symptoms that do not name the cause: a suite that collects nothing, a data-file lookup failing with TypeError, a wheel that is quietly short a module. Then close the loop with an installed-artifact test in CI.

for a principal

Decide the standard that makes this class extinct: src layout, explicit package declaration instead of auto-discovery, an allow-list for real namespace portions, and a release gate that installs the artifact into a clean environment before anything ships.

### The scenario A log-ingest pipeline gains a submodule for a locale-dependent timestamp format. It is committed inside a package directory whose `__init__.py` was lost in a rename. Every local run works: developers run tests from the repository root, the root is on `sys.path`, and PEP 420 imports the bare directory as a namespace package. The wheel built at the end of the three-week release train does not contain the module at all, and the first environment to install it fails with `ModuleNotFoundError`. Nothing in the local signal predicted it. ### Why the two sides disagree The import system's rule is permissive by design: if a `sys.path` walk finds only bare directories with the requested name, it builds a namespace package from them. That rule has no opinion about whether you *meant* it. Packaging's rule is conservative by design. Automatic package discovery in the common build backends enumerates directories that contain `__init__.py`, because that is the only unambiguous marker that a directory is a package rather than a data folder, a fixtures tree, or a stray checkout. Namespace-aware discovery exists but must be asked for, and the directories you want must be declared or matched by a pattern. So the file that the interpreter treats as optional is the file the build treats as the definition. Add the environment difference: locally, `sys.path` contains the source root; after install, it does not. The source tree is the only reason the local import worked, so the moment the code runs anywhere else, the namespace package that stood in for the real one is simply not there. ### The sibling symptoms The same missing file shows up in ways that do not obviously point at it: - **Test discovery goes quiet.** `unittest` removed namespace-package support from discovery in Python 3.11. Run discovery from the project root with a `tests/` directory that has no `__init__.py` and it collects nothing, exiting with status 5 and "NO TESTS RAN". Point the start directory *at* `tests` with a different top-level directory and you get `ImportError: Start directory is not importable`. A pipeline that treats a green-ish exit or a small test count as normal never notices the whole suite stopped running. - **Data files stop resolving.** A namespace package's `__file__` is `None`, so the venerable `os.path.dirname(__file__)` trick raises `TypeError` instead of returning a directory. `importlib.resources.files()` is the correct replacement and works for both kinds of package. - **Two trees merge instead of shadowing.** A stale copy of the same directory elsewhere on `sys.path` does not shadow the new one; both become portions of one namespace, and submodules resolve from whichever came first. Debugging that looks like the code on disk is not the code running. - **Editable and non-editable installs disagree**, because they place different things on `sys.path` and therefore expose different portions. ### Confirming the diagnosis Start at runtime in the failing environment: `python -c "import ingest; print(ingest.__file__, list(ingest.__path__))"`. A `__file__` of `None` says namespace package, and an empty or surprising `__path__` says which directories the interpreter actually found. `importlib.util.find_spec` answers the same question without executing anything. Then list the wheel's contents and compare with the source tree — the module you are missing will simply not be there. ### Preventing the next one Three controls, cheapest first: 1. **A repository lint.** Walk the source tree and fail on any directory that contains `.py` files but no `__init__.py` unless it is on an explicit allow-list of deliberate namespace portions. This is a dozen lines and it catches the mistake at the commit that makes it. 2. **Test the installed artifact, not the checkout.** Build the wheel, install it into a clean virtual environment, and run at least an import smoke test — ideally the whole suite — from a working directory that is *not* the source root. A `src/` layout gets you most of this for free, because the source root is no longer importable by accident. 3. **Make discovery explicit.** Declare packages in the build configuration rather than relying on automatic discovery, so adding a package is a visible diff and a forgotten one is a build failure rather than a silent omission. And treat "no tests were collected" as a hard CI failure. The missing-`__init__.py` bug is nearly always found by whichever check refuses to pass quietly.

  • How would you confirm in the failing environment that a namespace package is what you have?
    Import it and print `pkg.__file__` and `list(pkg.__path__)`: `__file__` of `None` means a namespace package, and the path list shows which directories the interpreter actually merged. `importlib.util.find_spec("pkg")` gives the same answer without running the package, which matters when importing has side effects.
  • Why does a src/ layout reduce this class of bug?
    With the code under `src/`, the repository root that lands on `sys.path` does not contain the package, so a checkout-only import cannot succeed. You are forced to install the project — editable or not — before anything imports, which means local runs exercise the same layout the wheel produces instead of a parallel one that only exists in the checkout.
  • When is a directory of .py files without __init__.py legitimate rather than a bug?
    When it is a deliberate namespace portion shared across distributions, or when it is not meant to be imported at all — a scripts directory, generated output, or fixture data. The point of the lint is not to ban the pattern but to force it onto an explicit allow-list, so every such directory is a decision someone made rather than a file someone forgot.

saying these in an interview costs you the question

  • Blames the install and reinstalls instead of reading __file__
  • Assumes the build ships whatever is in the source tree
  • Says a missing __init__.py always raises ImportError
  • Treats zero collected tests as a passing run
  • Uses os.path.dirname(__file__) for package data
  • Tests only from the repository root, never the installed wheel

context

resolving links…