A webhook receiver's Python worker keeps dying at exit 137 after traffic bursts, though its average memory looks fine. How do you diagnose it?

## Start by proving what killed it Exit 137 only says SIGKILL. Before optimising anything, check: - the cgroup's `memory.events` for an incrementing `oom_kill` count, - the kernel ring buffer for the line naming the killed process, - and the runtime's recorded exit reason. If none of them show an OOM event, you are chasing the wrong failure -- a shutdown escalation from SIGTERM produces the identical number. ## Measure the peak, not the average A memory limit is a ceiling on **instantaneous resident set size**, and the kill is instantaneous too. Dashboards built from a scrape every fifteen or thirty seconds smooth exactly the spike that killed you: a burst that lasts two seconds is very likely never sampled at all. Two better instruments exist. - The cgroup exposes a **peak counter** that records the high-water mark since it was reset, which survives the death of the process. - And inside the process, a small **watchdog thread** that samples resident size several times a second and records a running maximum -- emitting it periodically -- gives you the shape of the spike rather than a single number. The crucial property is that both are max-based, not sample-based. ## Resident size is not the Python heap The number the kernel enforces against includes far more than your objects: - the interpreter and its extension modules, - thread stacks, - the allocator's own pools and arenas, - and, in cgroup v2, the page cache your process generated by reading and writing files. Cache is reclaimable, so it usually causes reclaim rather than a kill, but it does inflate the counter and confuses naive comparisons against a heap measurement. So when the object graph you can account for is a few hundred megabytes and resident size is well above that, the gap is normal, not necessarily a bug. ## Freed memory is not necessarily returned Deleting a large structure drops it from Python's view immediately, but the pages may stay with the process. Small objects are freed into the allocator's pools and only handed back to the OS when an entire arena becomes empty, which fragmentation can prevent indefinitely; large allocations that were mapped separately usually do go back. The practical consequence is a **ratchet**: after each burst, resident size settles at a plateau higher than before, so the second and third bursts start closer to the ceiling than the first. That is why a service can survive its first spike and die on an identical one an hour later. Distinguishing this from a real leak is a matter of shape -- a **leak** keeps climbing across quiet periods too, whereas **retention** rises to a plateau and then stops. ## Attribute the peak to concurrency Peak footprint is approximately **bytes retained per in-flight unit of work times the number of units in flight**. In a receiver that accepts a webhook and fans out to a large dependency graph -- say seventeen downstream services per event -- the request payload, the parsed object graph and every partial response are all reachable until the slowest of those calls returns. The ordering assumption is the trap: code written as though responses come back in the order they were issued, releasing buffers as it goes, actually holds *all* of them until the last one lands. One dependency going from 50 ms to 5 s therefore multiplies the number of concurrently live payloads by a hundred without any change in request rate, which is exactly the profile of a service that is fine for weeks and then dies during someone else's incident. ## Fixes, in order of leverage 1. Bound in-flight work explicitly with a semaphore or a small fixed pool, so peak is a design parameter rather than an emergent property of dependency latency. 2. Stream instead of buffering: iterate over the body, spool oversized payloads to a temporary file, and never hold a whole response when a chunk will do. 3. Reject payloads above a size you chose, at the edge, before allocating. 4. Set aggressive timeouts on downstream calls so buffers cannot be held hostage by a slow dependency. 5. Right-size worker counts: each worker process pays its own resident baseline, and multiplying processes multiplies that baseline against a shared limit. ## A 3.14 note on workers The default `multiprocessing` start method on Unix other than macOS is now `forkserver` rather than `fork` (macOS and Windows use `spawn`, and `fork` must be requested explicitly). - With `fork`, children shared the parent's pages copy-on-write, so a warm parent made children look cheap until reference-count writes gradually unshared those pages. - With `forkserver` each child starts from a small clean server process and imports what it needs, so per-child memory is more predictable but no longer subsidised by the parent. If a service was tuned for worker counts under the old default, its resident footprint can change on upgrade.