Python worker threads all rebuild the same cache on a 45-second cold start — what race is this and how do you fix it?

### The symptom and the shape of the code A triage service starts, four worker threads begin pulling tickets, and the first four tickets each take 45 seconds instead of one. Memory peaks at four times the expected size. Occasionally a worker classifies a ticket against a cache that is missing half its entries, and once in a while a thread dies with `RuntimeError: dictionary changed size during iteration`. The code behind this is almost always some variant of: ```python _index = None def get_index(): global _index if _index is None: # check _index = build_index() # ...45 seconds... then act return _index ``` ### Why it races The test and the assignment are separate operations, and everything between them is a window. Here the window is the entire build: 45 seconds. Every thread that calls `get_index()` during those 45 seconds sees `None`, because nothing was published yet, and starts its own build. The GIL does not help — it serializes bytecode, and the threads are not fighting over a single instruction, they are all sitting inside a long-running function. This is the same family as `if key not in d: d[key] = ...`, just with a window wide enough that the race is not occasional but guaranteed. The variant that fills a shared structure in place is worse: ```python _index = {} def get_index(): if not _index: for k, v in load_rows(): _index[k] = v # readers can see this half-done return _index ``` Now a second thread finds `_index` truthy after the very first insert and returns a cache with one entry. Correctness bugs of that kind are far harder to diagnose than the duplicated work, because the service does not slow down — it just produces wrong answers for a few seconds after every restart. ### Diagnosing it Log the build, with the thread name, at both ends: ```python import logging, threading logging.info("building index on %s", threading.current_thread().name) ``` If the same start line appears four times per process launch, you have your answer. Timing the first N requests per restart shows the same thing from the outside: a burst of very slow requests at exactly the width of the build, on every worker, once per deploy. Memory tells the third part of the story — four concurrent builds hold four intermediate copies. ### Fix 1: serialize and re-check ```python import threading _index = None _index_lock = threading.Lock() def get_index(): global _index if _index is None: # fast path, no lock with _index_lock: if _index is None: # re-check: someone may have built it _index = build_index() return _index ``` The inner re-check is the whole point: a thread that waited on the lock must not build again. The outer check is only an optimisation for the common case where the cache already exists. Note what the callers pay — the second, third and fourth threads block for the remainder of the 45 seconds. That is usually right, since they had no useful work without the cache, but it is a decision, not a default. ### Fix 2: publish by rebinding, never by mutating Build into a fresh local dict and assign the finished object to the module-level name in one store. Readers hold a reference to whichever complete object existed when they looked, and no reader ever observes a partial structure: ```python def refresh(): global _index new_index = {k: v for k, v in load_rows()} _index = new_index # single publication step ``` Readers should copy the reference into a local variable once (`index = _index`) and use the local, so a refresh partway through their work cannot swap the object under them. ### Fix 3: build eagerly, or serve stale while one thread builds If a 45-second stall on the first request is unacceptable, move the cost: build the cache during startup, before the pool starts accepting work, so the process is either not ready or fully warm. Readiness checks exist precisely for this. Where stale data is acceptable, keep the previous cache in place, let a single builder thread produce the replacement, and have readers continue on the old object until the new one is published — a `threading.Event` (or simply a "build in progress" flag guarded by the lock) prevents a second builder from starting. Where callers have their own deadline — say five seconds — do not let them block for 45. Use `lock.acquire(timeout=...)` and, on timeout, serve stale data or fail fast with a clear message. Blocking a caller past its own deadline converts a throughput problem into a queue of doomed requests. ### What about `functools.lru_cache`? It is a reasonable memoisation tool and its internal bookkeeping is thread-safe, but it does **not** collapse concurrent misses: several threads calling it for the same missing key can each run the wrapped function. It is therefore not a stampede fix. If you want dedupe, you still need the lock-and-re-check pattern, or a single owner thread fed through a `queue.Queue` that returns the built object to everyone who asked. ### The general lesson Check-then-act races are ranked by the width of their window. A two-bytecode window makes a flaky test; a 45-second window makes a guaranteed production incident. When you see a test whose corresponding action is expensive, assume every concurrent caller will take the branch, and design for that.