A Go ingestion service's live heap climbs under steady input — how do you find the unbounded backlog?

## Step 1 — is the live heap really growing? Rising RSS alone proves nothing: Go's collector is not compacting and does not immediately return memory to the OS, so a service with a bursty allocation profile can hold a large resident set with a small live set. The cheap discriminator is `GODEBUG=gctrace=1`, which makes the runtime print one line per collection to standard error, including the heap size at the end of the cycle. Under constant input, a healthy service shows those end-of-cycle sizes oscillating around a stable level. A backlog shows them stepping upwards, cycle after cycle, forever. `runtime/metrics` gives the same picture programmatically if you would rather not restart with a GODEBUG setting. While you are there, note the collection frequency. A growing live set makes the collector run more often over a bigger graph, so CPU spent in GC climbs alongside — which is why these incidents usually present first as latency, not as memory. ## Step 2 — what holds the bytes? Take a heap profile and read it in `inuse_space` mode, which reports memory currently held rather than cumulative allocation: ``` go tool pprof -inuse_space http://localhost:6060/debug/pprof/heap ``` You are looking for a single allocation site that dominates and grows between two profiles taken minutes apart. A backlog has a recognisable shape: the retaining site is on the *intake* path — the decode loop, the batching helper, the enqueue wrapper — and the retained objects are the event or job type itself. Taking two profiles and diffing them makes the growth unambiguous. ## Step 3 — backlog or leak? The two look identical on a graph of heap over time, and they are fixed differently, so this distinction is the whole diagnosis. A **leak** retains objects whose work is finished: completed requests kept in a map nobody deletes from, a cache with no eviction, a slice of results appended to forever. Its size tracks *total traffic served*. A **backlog** retains work that has not happened yet. Its size tracks *how far behind the consumers are* — arrival rate minus drain rate, integrated over time. Two more tells: it disappears the moment input pauses, because the consumers catch up; and its growth rate changes with consumer speed, not with traffic volume. ## Step 4 — the goroutine profile decides it This is the step that turns a guess into a conclusion. Fetch the goroutine profile (`/debug/pprof/goroutine?debug=2` gives full stacks) and look at where the producers are. In a correctly bounded pipeline under overload, the producing goroutines are parked in a channel send. That is backpressure working: the stall is being transmitted. If instead every producer is running happily, the heap is growing, and the consumers are the slow part, then something between them is absorbing the rate gap — an unbounded staging slice, a per-key map of pending items, a channel created with an enormous capacity, or a goroutine spawned per item so that the queue has become "the scheduler's run queue". The goroutine count itself is worth reading: a steadily climbing count under steady input means the unbounded thing is goroutines rather than a slice, which is the same failure wearing a different costume. ## Step 5 — fix it where the pressure should have been felt Bound the intake, so that the producer's send blocks and the stall reaches the source. Compute the ceiling you are choosing: capacity times item size, plus the workers' in-flight items. Then decide what happens when it is full — wait, wait with a deadline, or shed with a counter — because that is now a real code path rather than an implicit "grow". ## What not to do Raising `GOMEMLIMIT` is the reflex, and it is the wrong lever here. It is a *soft* limit: as the live heap approaches it the collector runs more and more aggressively to stay under it, and since a growing backlog cannot be collected, the service degrades into a collector-bound crawl instead of dying quickly. The runtime does cap GC's CPU share so the process does not livelock entirely, but you have converted a clear memory failure into a murky latency failure. Lowering `GOGC` has the same character. Both are legitimate tuning knobs for a service whose live set is stable and too large; neither bounds anything that is genuinely unbounded. Adding more workers is worth testing but is not a fix either: if the consumers are limited by a downstream dependency, more of them just move the queue somewhere else. ## Verifying the fix Reproduce with a producer deliberately faster than the consumer, and watch for the heap to plateau instead of climb, the goroutine profile to show parked senders, and either the enqueue block time or the shed counter to become non-zero. All three changing together is the proof that the pressure is now being transmitted rather than stored.