skip to content

A WeakValueDictionary cache of parsed chat transcripts never hits, leaving a 45-second cold start every request — how do you diagnose and fix it?

level: seniorimportance: should knowfreq 30%

answer

  1. Nothing else was holding the object
  2. The mapping never promised to retain
  3. Flat memory can mean zero hits
  4. Measure eviction, do not infer it
  5. Add a bounded strong tier beside it

basics

~20 s

The mapping holds values weakly, so the entry deletes itself once the handler returns and drops its reference. Confirm with hit counters and weakref.finalize callbacks, then add a bounded strong tier retaining the last N transcripts.

solid answer

~50 s

The cache has no retention policy: a `weakref.WeakValueDictionary` extends no object's lifetime, so the parsed transcript dies when the handler that created it returns, and every request pays the 45-second parse again. Confirm it rather than assume — count hits and misses explicitly, log `len(cache)` at insert and at the next lookup, and register `weakref.finalize(obj, record, key)` to see the exact moment each entry dies. Flat memory alone proves nothing, because a never-retaining cache and a healthy one look identical from process metrics. The fix is a second tier, not a different weak container: an LRU of the last N transcripts (`functools.lru_cache`, or a `collections.OrderedDict` you reorder) supplies the hit rate and the memory ceiling, while the weak mapping alongside it guarantees that any key still alive in flight resolves to the same object instead of being parsed twice.

code

python · 25 lines
python
import weakref


class Transcript:
    def __init__(self, key):
        self.key = key


cache = weakref.WeakValueDictionary()


def load(key):
    hit = cache.get(key)
    if hit is not None:
        return hit
    fresh = Transcript(key)      # stands in for the 45-second parse
    cache[key] = fresh
    return fresh


a = load("room-7")               # caller holds it -> entry survives
print(len(cache), load("room-7") is a)

load("room-8")                   # nobody holds it -> gone immediately
print(len(cache), cache.get("room-8"))

go deeper

for a junior

Recall the core fact behind the symptom: a weak mapping never keeps its values alive, so if nothing else holds the object the entry is gone before the next request. That is the whole explanation for a zero hit rate.

for a middle

Walk through the mechanics and the measurement: hit and miss counters, len() at insert versus next lookup, and a weakref.finalize callback that records the eviction moment. Know that the fix is a bounded strong tier, not a different weak container.

for a senior

Demonstrate the production instincts: distinguish a never-retaining cache from a leaking one when both show flat or calm metrics, refuse to trust cross-host timestamps for causal ordering, and size the strong tier from measured object size against the cost of a miss.

for a principal

Own the design statement: retention policy and its memory ceiling must live in one identifiable place, and a weak mapping is explicitly not that place. Argue the N, the failure mode when N is wrong, and who is accountable for the object lifetimes the rest of the system assumes.

### Read the symptom literally A `weakref.WeakValueDictionary` holds its values weakly, which means it extends no object's lifetime. In a request-shaped archiver the parsed transcript is created inside the handler, inserted into the mapping, used, and then the handler returns — dropping the last strong reference. The entry deletes itself before the next request arrives. So the cache is not "cold"; it is empty by construction, and the 45-second parse is paid every single time. The zero hit rate is not a bug in the mapping, it is the mapping behaving exactly as documented against a design that expected retention. The tell is that `len(cache)` is `1` immediately after insertion and `0` at the top of the next request, with no eviction code anywhere in the repository. ### Confirming it rather than guessing Three cheap measurements settle it in minutes: 1. **Instrument insertion and lookup.** Count `hits` and `misses` explicitly and log `len(cache)` at both points. Flat memory plus zero hits is the signature; flat memory alone proves nothing, because a leaking cache and a never-retaining one both look calm from the process metrics. 2. **Watch the eviction moment.** Register `weakref.finalize(transcript, record, key)` when you insert. The callback fires when the object is actually collected, so you see whether entries die at the end of the handler (nothing else holds them — this diagnosis) or never die at all (the opposite bug: something is pinning them). 3. **Ask who is holding it.** `weakref.getweakrefcount(obj)` and `sys.getrefcount(obj)` — remembering that the latter counts the temporary argument reference too — tell you whether any strong holder exists at the moment you look. A warning about the second measurement: if you correlate eviction logs against timestamps taken on more than one host, clock skew will make finalizer events appear to precede the insertions that caused them, and you will spend an afternoon chasing a reordering that never happened. Sequence the events inside one process with `time.monotonic` and a counter, and only then compare against wall-clock logs. ### The fix is a second tier, not a different weak container Swapping in a `WeakKeyDictionary` or a `WeakSet` changes nothing — they are weak in a different direction, not less weak. What the system actually lacks is an owner. The standard shape is two layers with distinct jobs: * a **bounded strong tier** that decides retention: an LRU of the N most recently used transcripts (`functools.lru_cache` over the parse function, a `collections.OrderedDict` you move keys within, or a `collections.deque(maxlen=N)` if crude recency is enough). This is what produces a hit rate, and N is what bounds memory; * the **weak mapping** kept alongside for identity: while a transcript is alive anywhere — held by the strong tier, or by an in-flight request that outlived its eviction — every lookup for that key returns the *same* object rather than parsing a second copy. Insert into both; look in the weak mapping first, fall back to the strong tier, then parse. Sizing N is now an ordinary capacity decision: measured object size times N against the memory you are willing to spend, with the 45-second parse cost telling you what a miss is worth. ### Second-order effects to mention * **Cycles delay eviction.** A transcript that participates in a reference cycle — a parent document and child message objects pointing back — will not be collected when the last external reference drops; it waits for the cycle collector. The weak entry then lingers, and memory falls in steps rather than smoothly. That is not a reason to abandon weak references, but it does mean eviction timing is not a hard real-time property. * **Check-then-act is a race.** `if key in cache: return cache[key]` can miss between the two statements, and under threads it will. Use `obj = cache.get(key)` and keep the returned strong reference for the rest of the block. * **Iteration.** On 3.14 the weak mappings remove entries as soon as the referent dies and iterate over a snapshot of their internal table, so a collection during a loop does not raise; through 3.13 the implementation deferred removals with an internal guard while an iteration was in progress. Either way, the safe habit is not to hold conclusions across an iteration: an object yielded to you is alive because *you* now hold it, and one you merely counted may already be gone. * **Know what you are measuring.** After the fix, memory should rise to a plateau proportional to N and stay there. If it keeps climbing, the strong tier is unbounded somewhere; if the hit rate is still zero, the strong tier is not being populated on the path you think it is. ### The one-sentence version for the interviewer "A `WeakValueDictionary` deduplicates objects that are alive; it does not keep anything alive, so a cache built only from one has no retention policy at all — pair it with a bounded strong tier that owns the last N entries."

  • Eviction timestamps in the log appear before the insertions that caused them. What now?
    Suspect the clocks before the code. Entries logged from different hosts, or wall-clock timestamps taken across a skewed pair of machines, will reorder causally related events and send you hunting a reordering that never happened. Sequence the events inside one process with `time.monotonic` and a counter, establish the true order there, and only then correlate against wall-clock lines from elsewhere.
  • You add the strong tier and the hit rate improves but memory climbs steadily. What is wrong?
    The strong tier is unbounded on some path — an LRU whose eviction never runs, a list that only appends, or a second structure holding the same objects. Weak references cannot rescue you here: the objects have a live strong holder, so nothing evicts them. Find the holder rather than tuning the weak mapping; the weak layer is a follower and reports lifetimes, it never shortens them.
  • Why not just make the mapping a plain dict and be done?
    Because that trades a zero hit rate for an unbounded one. A plain dict cache of parsed transcripts grows with the number of distinct rooms ever seen and never returns the memory. The two-tier shape keeps both properties you want: a fixed ceiling of N retained objects, and a guarantee that a key still alive in flight is never parsed into a second copy while the first is still around.
  • Entries stop disappearing after a refactor that added parent pointers. Why?
    The transcripts now form reference cycles — parent to children and back — so dropping the last external reference no longer brings any reference count to zero. The group survives until the cycle collector reclaims it, so weak entries clear late and memory falls in steps rather than smoothly. Eviction from a weak mapping is prompt for acyclic objects and merely eventual for cyclic ones.

saying these in an interview costs you the question

  • Blames the weak mapping instead of the missing retention policy
  • Swaps in WeakKeyDictionary or WeakSet as the fix
  • Reads flat memory as evidence the cache is working
  • Trusts cross-host wall-clock ordering of eviction events
  • Replaces it with an unbounded plain dict and calls it done
  • Expects weak entries to clear instantly even inside reference cycles

context

resolving links…