Why do loop-registered callbacks in a 17-stage genomics pipeline all read one stale value?

This is late binding meeting deferred execution, and the pipeline's cache is the amplifier, not the cause. ### The shape of the bug Somewhere in the registration code there is a loop like this: ```python stages = ["gene", "exon", "cds"] # ...17 of them in the real pipeline checks = [] for stage in stages: checks.append(lambda record: validate(record, cache[stage])) ``` Each `lambda` creates a function object but does not run its body. The name `stage` inside the body is a free variable, resolved **when the callback fires**, in the scope that was enclosing the lambda. By the time anything fires, the loop has finished and `stage` holds `"cds"`. All seventeen callbacks therefore read the same cache entry — the last stage's — and the pipeline reports one stale value everywhere it should have reported seventeen different ones. Nothing raises. Every callback is registered, every callback is called, every callback returns something. They simply all agree. ### Why it survives so long in production * **Single-item paths hide it.** A pipeline run with one stage is correct. So is a unit test that registers one handler and asserts on it. The bug needs two or more registrations *and* a deferred call to appear. * **The cache makes it plausible.** Once a memoized result is stored under the wrong stage, the wrong answer becomes consistent and reproducible, which reads like a data problem rather than a code problem. Teams chase invalidation logic for days. * **The last stage is often the least surprising.** If the final stage is the broadest or most permissive check, every record passes and the pipeline just gets quietly less strict. ### Confirming it in minutes You do not need heavy tooling. Call two registered callbacks that must differ and compare their results — identical output for different stages is the tell. Or have each callback report the value it resolves at call time: ```python print([c("sample") for c in checks]) # all seventeen agree ``` If they all report the same stage, the callbacks are reading one shared variable. A second confirmation: build the registrations in a loop of length one and watch the bug disappear. ### The fix Bind the value at registration time so the callable carries its own: ```python from functools import partial def make_check(stage): def check(record): return validate(record, cache[stage]) return check checks = [make_check(stage) for stage in stages] ``` or `partial(validate_stage, stage)` over a plain named function. The default-parameter trick (`lambda record, stage=stage: ...`) also works and is fine for a throwaway lambda, but on a registered callback it adds a public parameter that a dispatcher could fill, so it is the weakest of the three here. Then invalidate the cache: whatever it memoized while the bug was live is keyed correctly but computed from the wrong stage, so the entries are wrong rather than merely stale, and a normal TTL expiry will not clear them on the timescale you need. ### Making it structurally impossible The senior move is not the one-line fix, it is removing the shape. Give the registration API a parameter for the per-iteration value: ```python def register(stage, handler): # handler receives the stage ... ``` so the loop body has nothing to close over. Then the loop reads `register(stage, validate_stage)` and the eager binding is a property of the API rather than a discipline anyone has to remember. Add one test that registers **two** handlers and asserts they differ — that single test would have caught seventeen callbacks agreeing. ### Where else this shape appears Anywhere a function is built in a loop and invoked later: signal and event handlers, dispatch tables built from a list of command names, retry or timing wrappers applied per item, callbacks handed to a scheduler, per-column validators, and partially-applied functions stored in a registry. The common factor is deferred execution, not the container. If a function defined inside a loop body mentions the loop variable and is not called in that same iteration, treat it as a defect until proven otherwise — that is the whole review heuristic, and it is cheap to apply. ### Distributing the work does not change the cause If the seventeen stage callbacks are later spread across worker processes, the failure mode changes shape but not origin: a lambda closure cannot be pickled, so it fails at submission instead of quietly reading the wrong stage. On Python 3.14 the default `multiprocessing` start method on Unix other than macOS is `forkserver` (macOS and Windows use `spawn`), so you get that failure by default rather than inheriting parent state through `fork`. Useful to know when the symptom moves — but the fix is still eager binding.