Why can raising a worker's configured maximum heap, inside an unchanged container ceiling, make the worker die sooner?

## Permission, not reservation, and not a prediction A configured maximum heap answers one question: at what point must the runtime stop growing the heap and reclaim instead? It does not state how much memory the program needs, and it does not reserve that memory for the program's exclusive benefit. It is an upper bound on a behaviour. The practical consequence is that the maximum influences the *typical* footprint, not only the worst case. A runtime given more room reaches the point where it must reclaim later, so: - more allocation happens between reclamations; - more objects that have already become unreachable are still occupying space at any instant you sample; - the heap the process actually holds settles at a higher level, even though the set of objects the program genuinely needs has not changed. That last bullet is the one that surprises people. The live set is a property of the program. The footprint is a property of the program *and* the setting. ## Why the ceiling does not care about your intent The party enforcing the container ceiling counts what the process holds. It has no view of which of those bytes are live objects, which are unreclaimed garbage, and which are stacks or buffers. Raising the maximum heap therefore moves the process closer to the ceiling without moving any indicator that a reader of the heap numbers would notice — heap occupancy as a fraction of the maximum may even look healthier than before, because the denominator grew. | | Before the change | After raising the maximum | |---|---|---| | Live set | unchanged | unchanged | | Typical unreclaimed garbage resident | lower | higher | | Heap occupancy as a fraction of the maximum | higher, looks worse | lower, looks better | | Whole-process footprint against the ceiling | lower | higher | | Margin left for stacks, metadata and buffers | as budgeted | eroded | The row that matters is the last one. The non-heap lines did not shrink to make room; they simply have less room, and they will claim it anyway the next time concurrency rises. ## The sequence that produces the surprise 1. A worker occasionally reports an allocation failure under load. 2. Someone raises the configured maximum heap, reasoning that more headroom means fewer failures. Within the heap alone, that reasoning is correct. 3. The allocation failures stop, which confirms the fix in everyone's mind. 4. Weeks later, under a traffic peak that fills the thread pool and the buffer pools, the worker begins disappearing and restarting with no memory error recorded anywhere. 5. The visible heap numbers look better than they did before, so the heap is the last place anyone looks. Step 3 is real and is why the change survives review. The defect is that step 2 spent budget that belonged to steps outside the heap. ## What to do instead - **Re-run the subtraction.** A change to the heap maximum is a change to the footprint budget, so it must be checked against the ceiling and the non-heap lines, not judged on heap metrics alone. - **Find out which limit was actually being hit.** Reported allocation failures point inside the heap; silent restarts point at the ceiling. They call for opposite changes, and mistaking one for the other is how a service oscillates between the two failure modes. - **If the heap genuinely needs more room, take it from a named line.** Cut the thread-pool maximum, bound a buffer pool, or move data out of the process — and record what was traded. - **If nothing can be released, the ceiling itself is the decision**, and it is a capacity decision rather than a configuration one. ## The nuance worth stating out loud Runtimes differ in how eagerly they take memory up to the maximum and how readily they give it back after a peak: some hold their high-water mark, others shrink when the live set falls. Because that behaviour differs and is often configurable, a budget must not depend on it. Assume the process may hold whatever the maximum permits, size the rest of the budget around that assumption, and treat any memory the runtime hands back as a bonus rather than a line you have already spent.