src Layout vs Flat Layout

## The one entry that causes all of this `import indexer` asks the import machinery to search `sys.path` in order and use the first directory that yields a match. Everything about flat-layout shadowing follows from *which entry is first*. Before your first line runs, CPython prepends one path entry: - `python script.py` — the directory containing `script.py`. - `python -m package.module` — the current working directory. - `python -c "..."` and the interactive prompt — the current working directory. Only then come the environment's `PYTHONPATH` entries, the standard library, and finally site-packages. So in a flat-layout repository, where the package directory sits at the top level next to `pyproject.toml`, running *anything* from that root puts the working copy ahead of every installed distribution in the environment. ```python import sys import importlib.util print(sys.path[0]) print(importlib.util.find_spec("json")) ``` ## What the shadow hides The shadow is not itself a bug — it is what makes a checkout convenient. The problem is what it conceals. **A stale install.** You installed the project weeks ago, have since renamed a function, and every local run keeps working because none of them touched site-packages. The moment a real deployment imports the installed copy, the rename is missing. **An incomplete install.** You added `indexer/locale_formats.py`, but the build never picked it up — a subpackage without `__init__.py`, an include rule that does not match, a backend that needs the module listed. Every local run imports it from the working copy and passes. The installed wheel does not contain it at all. **Two copies of the same code.** `sys.modules` is keyed by module *name*, so within one process only one `indexer` wins; but different processes, different working directories or a subprocess launched from elsewhere can each win differently. A test job that runs from the root and a production entry point that runs from `/` are then executing genuinely different code with the same version number. **A name collision that is not yours.** A top-level `logging.py`, `types.py` or `queue.py` at the repository root shadows a standard-library module for anything started from that directory, producing errors far from their cause. This is the same mechanism, applied to a module you never meant to publish. ## How to see which copy you actually got The diagnosis is short and worth having at your fingertips: - print the module's `__file__` — the definitive answer to "where did this come from"; - print `sys.path[0]` — the entry doing the shadowing; - use `importlib.util.find_spec("indexer")` without importing, to see what *would* be imported; - compare with `importlib.metadata.version("indexer")`, which reads installed distribution metadata rather than the import path — a mismatch between the two is the signature of the problem. ## Turning it off Since **Python 3.11** you can ask CPython not to prepend that entry at all: ```console python -c "import sys; print(sys.path[0])" python -P -c "import sys; print(sys.path[0])" ``` `-P` suppresses the potentially-unsafe prepended path; the `PYTHONSAFEPATH` environment variable does the same for every interpreter started in that environment, and `-I` (isolated mode) implies it. That is a useful hardening flag for entry points, but as a development discipline it is fragile: it protects only the commands you remember to pass it to. ## Why "just install it first" is not the answer The reflex fix is to install the project into the environment and assume the installed copy now wins. It does not: installing changes what is in site-packages, and site-packages is still last. As long as the package sits at the repository root and you launch from there, the working copy is found first no matter how many times you reinstall. Nothing about the install order, the tool used, or the environment changes the search order — only the layout, an explicit flag, or launching from somewhere else does. That is also the reason the problem is so persistent in practice. Every individual developer's experience is that "it works", because for them it genuinely does; the divergence is only ever visible to whoever imports the artefact without a checkout underneath them. ## Why the layout is the durable fix A src layout makes the shadow impossible rather than optional. The repository root contains `src/`, `tests/` and configuration — no importable package — so the prepended entry matches nothing, and `import indexer` resolves through the environment's installed distribution however you launch the interpreter. You cannot forget a flag, and nobody has to remember a convention. That is the actual argument for src layout: not tidiness, but that it converts "the developer must remember to test the installed artefact" into "the developer cannot avoid it". The natural next question is whether an editable install gives the shadow back. Broadly, it points imports at your working tree on purpose, so a file you forgot to declare can still import cleanly; the strong check remains a plain install of a built wheel into a clean environment before the test run. The mechanics of how an editable install wires that up are their own topic.