A pickle.Pickler reused for millions of genome records makes memory grow without bound — what is holding them?

### The symptom A genome-annotation pipeline writes one long stream: a helper opens the output file once, constructs a `pickle.Pickler` over it, and calls `dump()` once per annotated record. Resident memory rises smoothly with the number of records processed, is never released, and a `gc.collect()` changes nothing. The records themselves are dropped by the loop as soon as they are written, so on a first reading nothing should be retaining them. That shape — linear growth, tracking work done rather than concurrency, immune to garbage collection — says *something holds strong references*, not *reference cycles* and not *fragmentation*. Reference cycles would be reclaimed by the collector; the growth would also plateau if the culprit were an interpreter arena that could be reused. ### What the memo is and why it retains `pickle` guarantees that a shared or recursive object graph round-trips with its sharing intact: if the same sub-object appears twice, the stream carries it once and refers back to it, and a self-referential structure terminates rather than recursing forever. The mechanism is the **memo**, a table keyed on `id()` of each object the pickler has already emitted, mapping to the memo slot and the object. The object has to be in that table, not just its id. Ids in CPython are addresses, and an address is only unique while the object is alive; if the pickler let a memoized object be freed, a later object could be allocated at the same address and be mistakenly written as a back-reference to the wrong value. Holding a strong reference is the only correct implementation. The scope of that table is the part teams get wrong. Reading the API — `Pickler.dump(obj)` — it is natural to assume the memo is per call. It is per **Pickler**. That is deliberate: reusing one Pickler across many `dump()` calls is how you get sharing *between* records in a single stream. It is also why a Pickler that lives for the whole run accumulates every object of the whole run. ```python pickler = pickle.Pickler(open("out.pkl", "wb"), protocol=5) for record in records: # each record stays alive in the memo pickler.dump(record) ``` The unpickling side has the mirror-image problem: a long-lived `pickle.Unpickler` reading the same stream keeps its own memo, so a streaming reader that loads a million records holds a million of them even if the caller discards each after use. ### Confirming it rather than guessing The diagnosis that separates this from an application-level cache is a retention test rather than an allocation test. Take `tracemalloc` snapshots at two points far apart and diff them: the growth localizes to the code that builds records, which is misleading on its own — it tells you where they were allocated, not who keeps them. Then prove retention directly: hold `weakref.ref` objects to a sample of records, drop your own references, run `gc.collect()`, and check how many are still alive. If they survive while your code holds nothing, walk referrers with `gc.get_referrers` and the chain ends at the pickler's memo. Calling `clear_memo()` and re-checking the weakrefs is the decisive experiment — the count drops to zero. ### Fixes and their trade-offs * **Fresh Pickler per record** (or `pickle.dumps` per record, writing each blob with a length prefix). Simplest, and the natural shape for a record-oriented file you want to read incrementally. Every record becomes independently loadable, which is usually a feature. * **`Pickler.clear_memo()` between records, or every N records.** Keeps one Pickler and one stream, and bounds the memo to a window. Note that clearing loses back-references, so shared sub-objects are re-emitted and the output file grows; and you must clear at a record boundary, never mid-object. * **Do not stream a million objects through pickle at all.** If the artefact is a table of annotations, a record format with an explicit schema costs less memory, is readable by other tools, and does not make your class names part of the file format. The organizational fix matters too. Where an eleven-person team shares one serialization helper, a comment on the shared writer — "the memo is per Pickler; clear it or make a new one per record" — prevents the same bug being reintroduced by whoever next adds an output path, which is exactly how it usually returns.