What goes wrong with a hand-rolled decorator that caches results in a plain dict?
answer
- The dict outlives every call
- Nothing evicts and nothing expires
- Strong references pin keys and values
- Only successful results belong in it
- Check-then-insert is not atomic
basics
~20 sThe dict is created once at decoration time and never evicts, so it grows for the life of the process, pins whatever it holds, caches failures and stale values, breaks on unhashable arguments, and races under concurrent threads.
solid answer
~50 sA cache dict closed over by a decorator is per-decorated-function process-lifetime state, and that is the source of every failure mode. It never evicts, so memory climbs until the process is restarted, and it keeps every cached value - and every argument used as a key - reachable, so nothing is collected. It has no notion of freshness, so a value cached once is served forever; if the wrapper swallows an exception and stores a sentinel, that failure is cached too and the retry never happens. Unhashable arguments raise `TypeError` at the key lookup. And `if k not in cache: cache[k] = f(k)` is a check-then-act sequence that two threads can interleave, so the expensive call runs twice. Prefer `functools.lru_cache` with a bound, or an explicit cache object with eviction, a TTL and a clear path.
code
python · 23 linesdef memoize(func):
cache = {}
def wrapper(batch_id):
if batch_id not in cache:
try:
cache[batch_id] = func(batch_id)
except Exception:
cache[batch_id] = None # a swallowed failure is now permanent
return cache[batch_id]
wrapper.cache = cache
return wrapper
attempts = []
@memoize
def total_for(batch_id):
attempts.append(batch_id)
if len(attempts) == 1:
raise TimeoutError("ledger unavailable")
return 1_250
print(total_for("b1")) # None - the failure was cached
print(total_for("b1")) # None - and the retry never happensgo deeper
Know that a dict created in the decorator body is shared by every call and never empties itself, so it grows for as long as the process runs.
Explain the mechanics: one dict per decorated function created at decoration time, strong references to keys and values, TypeError on unhashable arguments, and no expiry of any kind.
Diagnose it in a running service - correlate entry count with resident memory, spot a cached failure or a stale value, and fix it with a bound, an eviction rule and a lock around the check-then-insert.
Decide the policy: which caches a system is allowed to keep in process, what bound and freshness guarantee each carries, and when the answer is a shared cache with real semantics rather than a decorator hiding a dict.
## The scenario Take a nightly payment reconciliation job that turned into a long-running service. Someone wrapped the per-batch total lookup in a hand-written memoizing decorator; the dashboard shows an 83% cache-hit rate and everyone is pleased. Then: - resident memory climbs across the week, - and a batch that failed once keeps reporting a total of `None`. Both symptoms come from one fact: the cache dict is created **once**, in the decorator body, and lives as long as the wrapper - which for a module-level function means as long as the process. ## What goes wrong ### Unbounded growth Nothing evicts. Every distinct argument tuple adds an entry, and entries are never removed, so the dict is a monotonic function of the number of distinct inputs the process has seen. A cache with a naturally small key space is fine; one keyed by batch id, customer id or timestamp is a slow leak that a short-lived process hides and a long-running one exposes. The hit rate says nothing about this - 83% hits with an ever-growing key space is exactly the shape of the problem, because the 17% misses are all new keys being retained forever. ### Retention, not just size A dict holds **strong references** to both keys and values. Cached objects are never collected, and anything they reference transitively stays alive too. If the arguments are objects - a batch record, a session, an instance - the cache pins them. Caching a method by including `self` in the key is the canonical version of this: the cache keeps every instance alive for the process lifetime. ### No freshness A plain dict has no expiry. The first answer computed for a key is the answer forever, which is wrong whenever the underlying data can change. A reconciliation total that was correct at 02:00 is served at 18:00 without anyone deciding that was acceptable. Caching is a correctness decision as much as a performance one, and a dict quietly makes it for you. ### Caching failures The nastiest variant is a **swallowed exception**. If the wrapper does `try: cache[k] = f(k) except Exception: cache[k] = None`, the failure is now permanent: the retry path can never run, because the key is present. Even without a sentinel, caching a partially-built or default value on the error path poisons that key. The rule is that only a successful result belongs in the cache, and an exception should propagate - or at minimum be recorded somewhere with its own, short lifetime. ### Key construction The cache is keyed by whatever you build from the arguments. - Using the argument tuple directly raises `TypeError: unhashable type` the first time someone passes a `list` or a `dict`. - Ignoring keyword arguments makes `f(1)` and `f(x=1)` collide onto the same entry with different call semantics. - Ignoring types makes `1` and `True` collide, since they hash and compare equal. ### Concurrency `if key not in cache: cache[key] = func(key)` is **check-then-act**. Two threads can both miss and both run the expensive call, and if that call has side effects - writing a settlement record, charging something - the duplicate is not merely wasteful. The individual dict operations are internally safe, but the *sequence* is not, and no amount of global interpreter locking makes a multi-statement sequence atomic. This gets sharper on the free-threaded build, officially supported since Python 3.14 (PEP 779), where threads genuinely run in parallel and there is no coarse serialization to hide the window at all. The fix is either of: - a `threading.Lock` around the check-and-insert, - or accepting duplicate computation and using `dict.setdefault` so only one value wins. Holding a lock *around the expensive call itself* serialises every miss, which is usually worse than the duplicate work - so decide deliberately. ## What to do instead Use the stdlib's `functools.lru_cache` with a real `maxsize`, which gives you bounded size, eviction and `cache_clear()` for free. When you need a TTL, per-entry invalidation or shared state across processes, build an explicit cache object with those semantics rather than hiding it in a closure - a decorator's job is to apply a policy, not to be one. Whatever you choose, give the state: - a name, - a bound, - an eviction rule - and a way to clear it, and expose its size so an operator can see it. Then, when memory climbs, the cache is a place you can look rather than a fact you have to rediscover.
- Why is a high hit rate not evidence that the cache is healthy?Hit rate measures reuse, not size, freshness or correctness. A cache serving 83% hits can still be growing without bound on the misses, serving values that went stale hours ago, and holding a cached failure for one key. The metrics that would have caught those are entry count over time, entry age at hit, and the miss path's error rate.
- Two threads miss on the same key at once. What are your options, and what does each cost?Hold a lock across the whole miss, which guarantees one computation but serialises all misses. Lock only the check and the insert and let both threads compute, then `setdefault` so one value wins - cheap, but the underlying call runs twice, which is unacceptable if it has side effects. Or give each key its own lock or in-flight future, which is correct and concurrent but is real machinery you now maintain.
- How would you make such a cache visible to whoever is on call?Give the state a name and expose it: attach the mapping and a `clear()` to the wrapper, report entry count and, if bounded, evictions, and log at a threshold rather than silently. The point is that memory growth becomes a place someone can look, instead of something they have to infer from resident-set size.
It is a filing cabinet with no shredder and no dates on the folders: everything ever filed is still in there, the first note about a case is what you keep reading, and two clerks can open the same empty drawer at once.
saying these in an interview costs you the question
- Assumes the cache is cleared when the process goes idle
- Caches `None` or an exception sentinel and never retries
- Believes the GIL makes check-then-insert atomic
- Uses raw argument tuples as keys and hits unhashable types
- Cites a high hit rate as proof the cache is correct
- Hand-rolls a memoizer where a bounded stdlib one would do