pymalloc and Arena Behaviour

### Separate the three claims before touching the code "Memory grows" can mean three different things, and the fix differs for each: 1. **Live Python objects are accumulating** — a genuine leak: a cache, a module-level list, a logging handler holding records, a closure capturing batch state. 2. **Objects are freed but pymalloc keeps the arenas** — allocator retention/fragmentation. Nothing is leaking; RSS simply ratchets to the high-water mark. 3. **Non-Python memory is growing** — a C extension's own allocations, thread stacks, or the system allocator's heap, none of which pymalloc reports on. The whole diagnosis is about telling these apart with evidence rather than guessing. ### Step 1 — is the block count flat? Instrument the worker to log `sys.getallocatedblocks()` at the same point in every batch — right after the batch is dropped, so you are comparing like with like. * Count returns to roughly the same baseline each cycle, while RSS keeps rising to a plateau → case 2. The objects genuinely are being deallocated. * Count climbs batch over batch → case 1. Something is holding Python objects. * Count flat *and* RSS rising without bound → case 3; the growth is outside the Python heap. This one number is the cheapest, most decisive measurement available, and it costs nothing in production. ### Step 2 — if it is a real leak, name the objects Start `tracemalloc` (optionally with `tracemalloc.start(25)` for deeper frames), take a snapshot at the end of an early batch and another many batches later, and diff them: ```python import tracemalloc tracemalloc.start() snap1 = tracemalloc.take_snapshot() # ... run several sync batches ... snap2 = tracemalloc.take_snapshot() for stat in snap2.compare_to(snap1, "lineno")[:10]: print(stat) ``` The diff ranks allocation sites by growth, which usually names the offending line directly. Pair it with `gc.get_objects()` counted by `type` if you want a type histogram rather than an allocation site. ### Step 3 — if it is fragmentation, confirm the shape `sys._debugmallocstats()` prints "# arenas allocated current" and "# bytes in allocated blocks". Fragmentation has an unmistakable signature: a large arena count multiplied by 1 MiB against a small live-bytes figure. A tiny fraction of survivors is enough to cause it, because an arena is returned to the OS only when every pool inside it is empty: ```python import gc, sys records = [{"sku": i} for i in range(200_000)] survivors = records[::1000] # keep 0.1% — an index of the batch del records gc.collect() sys._debugmallocstats() # arenas still allocated: the survivors pin them ``` In a run of exactly that shape the arena count went from 4 to 47 while the batch was live, stayed at 47 with only 200 survivors held, and dropped back to 5 once the survivors were dropped too — 0.1% of the objects pinning ~47 MB. ### Fixing case 2 * **Lower the high-water mark.** Stream the sync in smaller pages so the peak is never reached; an inventory feed rarely has to be materialised whole. * **Do not interleave survivors with the spike.** If each batch keeps a small index, build the batch, copy the survivors into a fresh structure, and drop the originals — the copies land in fresh pools instead of being sprinkled across every arena the batch touched. * **Hold bulk data above the 512-byte line.** One `bytes`/`bytearray`/`array.array` buffer bypasses pymalloc and is released as a unit rather than pinning arenas. * **Recycle the worker.** Restarting after N batches, or doing the heavy phase in a child process, returns the address space unconditionally. Weigh it against startup cost: a 45-second cold start means you recycle on a schedule you control, with a replacement warm before the old one exits — never reactively at the moment memory looks bad. And say plainly what does *not* help: `gc.collect()` frees cycles but never shrinks the process, and `del` on an already-dropped name changes nothing. Claiming a fix without a before/after block count is how these investigations go around twice.