How can a stale `.pyc` in `__pycache__` make a patched module keep running old code?

Compiling Python to bytecode is not free, so CPython caches the result. The cache is almost always invisible — until the day it is not, and a module you just patched keeps running the code you replaced. ## Where the cache lives and what is in it Importing `payroll_import.py` writes `__pycache__/payroll_import.cpython-314.pyc`. The directory and the tag in the filename come from PEP 3147: the tag is `sys.implementation.cache_tag`, so builds from different interpreters and different implementations coexist instead of clobbering each other. `importlib.util.cache_from_source(path)` computes that filename for you, which is the fastest way to see which file the loader will consult. The file begins with a 16-byte header: * **bytes 0–3** — a magic number, available as `importlib.util.MAGIC_NUMBER`. It changes whenever the bytecode format changes, which is at least once per minor release. A mismatch means "recompile"; for a source-less deployment it means `ImportError`. * **bytes 4–7** — a flags word. Bit 0 selects hash-based validation, bit 1 says the hash must actually be checked. * **bytes 8–15** — in the default *timestamp* mode, the source's modification time (whole seconds) and its size; in *hash* mode, an 8-byte hash of the source bytes. Then comes the marshalled code object. There is no machine code anywhere in the file. ## The failure In timestamp mode the loader compares the recorded mtime and size against the source file it found. If both match it loads the cached bytecode without ever reading the source. That check is a heuristic, and it is defeated by ordinary tooling: * a checkout, rsync or image build that restores or normalises timestamps; * a patch applied in place whose replacement happens to be the same size, within the same second, or with the mtime deliberately preserved; * clock skew between a build machine and a runtime, or a container layer whose mtimes are pinned for reproducibility; * a `__pycache__` directory baked into an image, then overlaid with newer sources whose timestamps look older. The symptom is maddening because the source on disk is unambiguously correct. A payroll CSV importer keeps producing the old floating-point rounding after the fix was deployed; a print statement you added never appears. The tell is that `payroll_import.__file__` points at the source you edited while the behaviour matches the previous release — at which point the question is not "what is wrong with my code" but "which artefact is the interpreter actually executing". ## Diagnosing and fixing Delete `__pycache__` (or touch the sources) and re-run; if the behaviour changes, you have your answer. Then fix it at the deployment level rather than by ritual cleaning: * Build the cache explicitly at image-build time with `python -m compileall -f --invalidation-mode checked-hash <dir>`. Checked-hash files record a hash of the source and re-verify it on every import, so a same-size, same-mtime edit is caught. The cost is reading and hashing the source, which is small next to compiling it. * Use `unchecked-hash` only when the image is immutable and produced by a trusted build step; it stores the hash but never validates, trading safety for a slightly faster import. * Or turn the cache off for the runtime with `-B` or `PYTHONDONTWRITEBYTECODE=1`, having pre-compiled at build time. This is common for read-only container filesystems, where the process would otherwise fail to write and silently recompile every module on every start — measurable for a large dependency tree and for short-lived CLI processes. * `PYTHONPYCACHEPREFIX` (or `-X pycache_prefix`) relocates the whole cache tree out of the source directory, which keeps a read-only or bind-mounted source clean. Two details round out the mental model. The optimisation level is part of the filename, not the header: running under `-O` writes `payroll_import.cpython-314.opt-1.pyc`, and `-OO` writes `.opt-2`, so an optimised build and a normal one never read each other's cache. And a cached file sitting alone inside `__pycache__` is not importable at all — the loader only considers it when the matching source is present. A source-less deployment has to put the file where the `.py` would have been, in the legacy pre-PEP-3147 layout, which is worth knowing before someone proposes shipping bytecode only. ## The related trap Bytecode is version-specific, and the magic number is what enforces it: a file compiled by 3.13 is simply not loadable by 3.14, and the cache tag in the name keeps them apart on disk anyway. That is fine when the source is present — the interpreter recompiles — and fatal when it is not. Shipping `.pyc` files without sources as a form of obfuscation therefore buys very little (they decompile readily) while guaranteeing that a Python upgrade breaks the deployment outright.