After lowering GOMEMLIMIT, a Go service stops crashing but pins CPU and stalls — what is happening?

## What you actually changed Lowering the memory ceiling changes one thing: the number the pacer aims at. It does not change how much data the program keeps reachable. If an aggregation service holds a window's worth of state in one large map, that map is the same size before and after the change — the collector simply has less room in which to work around it. As long as the live set is comfortably below the ceiling, the difference between them is the space that garbage can fill before a cycle must run. Halve the ceiling and you halve that space. Bring the ceiling close to the live set and the space nearly vanishes: a cycle finishes, the program allocates a small amount, the total is at the target again, and another cycle starts. That is the thrash. ## Why the process does not simply die Because the ceiling is soft. Nothing in the runtime refuses an allocation or terminates the program for being over target. The pacer's only lever is *how hard the collector works*, and when that lever runs out, the runtime does the pragmatic thing: it lets the program exceed the ceiling. Specifically, a **GC CPU limiter** caps collection at roughly half the available CPU. Without it, a collector chasing an unreachable target could take essentially all the CPU and the program would make no progress at all — a true death spiral. With it, the program keeps a share of the machine and slowly proceeds, while memory drifts above the target. So the observable state is a process that is alive, responds to health checks slowly or not at all, burns CPU constantly, and serves almost nothing. From an availability standpoint this is often *worse* than the crash it replaced, because a dead process gets replaced and a catatonic one keeps taking traffic. ## Recognising it The signature is unmistakable once you look for it: - `GODEBUG=gctrace=1` prints one line per cycle: the lines arrive continuously, with almost no gap, and the heap goal is pinned at or just under the ceiling rather than tracking live data; - the fraction of CPU attributed to garbage collection climbs and then plateaus near the limiter's cap; - the runtime exposes, through `runtime/metrics`, both the cumulative count of completed cycles and the last cycle in which the CPU limiter was engaged — a limiter that has been enabled recently is a direct confirmation rather than an inference; - application latency degrades in proportion to the CPU the collector took, not in proportion to load. The cleanest way to see all of it is the experiment this failure deserves: a **soak test held near the ceiling**. Size the workload so the live set approaches the number, hold it there for minutes rather than seconds, and move the ceiling between runs — from code with `debug.SetMemoryLimit`, so a run needs no rebuild. You will see a clear knee: above some value, cycles are spaced and throughput is flat; below it, cycles merge and throughput falls off a cliff. That knee is a property of the service's live set, and it tells you exactly how much room the working set actually needs. ## The fixes, in order of honesty **Reduce the live set.** This is the only fix that changes the underlying fact. Bound the aggregation window, evict or spill completed groups, use a more compact representation for the accumulated state, or partition the work across more instances so each holds less. Everything else moves the pain around. **Give it more room.** Raise the ceiling — and, if that puts the process over its allowance, raise the allowance. That is a capacity decision with a cost attached, but it is a legitimate answer: some workloads simply need the memory. **Choose the other failure.** Remove the ceiling, accept that the process will occasionally be killed, and let it be restarted. This is a real option, not a defeat, when a restarted instance is more useful than a degraded one — but it is a policy decision about what the service owes its callers, not a tuning trick. **Not a fix:** forcing extra collections, or lowering the ceiling further. The collector is already running as often as it can, and the problem is that it cannot find anything to reclaim. ## The lesson to carry A memory ceiling is a genuinely useful tool, but it converts an out-of-memory failure into a performance failure. That conversion is only an improvement while there is real garbage for the collector to find. Once the live set approaches the ceiling, the ceiling has nothing left to give, and continuing to tune it is choosing between two shapes of the same underlying shortage.