Your log-shipping service's memory climbs steadily for hours and then dies; how do you show it is a rate mismatch, not a leak?

## Two shapes of growth Both a leak and an unbounded hold look the same from the outside: memory rises steadily, the process dies, a restart buys another few hours. They are different problems and the diagnosis has to separate them before anyone reaches for a fix. | | Object leak | Rate mismatch in a hold | |---|---|---| | What is retained | Objects nothing will use again | Items still waiting to be processed | | Why they are retained | An owner keeps a reference by mistake | Retention is the configured behaviour | | Grows with | Operations performed on a path | Elapsed time, at arrival minus drain rate | | Effect of reducing input | Growth slows but never reverses | Depth falls as the queue drains | | Correct repair | Release the reference | Bound the stage and change a rate | The distinguishing property is **reversibility**. A queue's contents are supposed to leave. A leak's are not. ## The drain test The single most decisive experiment: reduce the input rate below the drain rate and watch memory for a few minutes. 1. Cut the reader's rate, pause it, or route a fraction of the input elsewhere. 2. Watch retained memory and, if it is exported, the hold's depth. 3. A mismatch shows a plateau followed by a decline as the archival writer clears the backlog. The decline rate is the writer's spare capacity, and it is a second measurement of the same number. 4. A leak shows a plateau at best. Nothing comes back, because nothing is waiting to be consumed. If you can run this test, you rarely need anything else. The reason to know the other signals is that on many systems you cannot throttle production input on demand. ## Confirming with rates, not with memory size Memory is the last observable in this chain and the least informative. The rates are the first: - Export **lines read per second** at the source and **lines written per second** at the archival writer. - Plot them together. A leak leaves the two curves on top of each other; a mismatch shows a persistent gap. - Integrate the gap over the incident window. At a deficit of 500 lines per second over four hours that is 7.2 million lines. If the retained item count is near that number, the retained memory is explained entirely by the backlog and there is nothing else to find. This is also the evidence that survives a restart. Growth that resumes at the **same slope** every time, from a clean process, in proportion to the rate gap, is structural. A leak triggered by a rare input would restart with a different slope depending on whether that input recurred. ## Where the items are held Inspecting what dominates retained memory adds a third, independent signal. A mismatch shows **one owner holding many equal-shaped items of the pipeline's own type**, in a queue-shaped structure, all of them reachable by design. A leak more often shows an owner that has no business retaining anything — a registry, a cache with no eviction, a listener list that only ever grows — and the retained objects are frequently unrelated to the stage that allocated them. Be careful with one trap: item type alone proves nothing, because a leak of pipeline items is entirely possible. It is the combination of an intended owner, a rate gap that accounts for the count, and reversibility under the drain test that closes the case. ## The stall case A writer that is fast on average can still ratchet. Suppose the archival writer stalls for **10 minutes each hour** while the reader runs at 12,000 lines per second. Over the hour, 43.2 million lines arrive and must all be written in the remaining 3,000 seconds, so the writer needs `43,200,000 / 3,000 = 14,400 lines per second` of real capacity to end each hour empty. At 12,000 per second — the reader's own rate, which sounds sufficient — it ends every hour with a residue, and the residue compounds. "The average is fine" is therefore not a defence; the question is whether the backlog built during each stall is cleared before the next one starts. ## What the fix is and is not Raising the memory limit moves the failure later and makes it worse, because a larger backlog also means a larger delivery age and more data lost when it finally dies. The repair has two parts: 1. **Bound the retaining stage** so the mismatch produces an early, explicit signal at the stage boundary instead of an allocation failure hours later. What should happen when that bound is reached is a separate design decision with its own trade-offs. 2. **Close the rate gap**: make the archival writer faster, reduce what is read, or accept that some items will not be kept. And the durable part of the fix is the instrumentation: depth and delivery age per retaining stage, alerted on a depth that never returns to baseline. That signal turns this four-hour mystery into a two-minute triage the next time it happens.