Sizing Against a Limit

## Two numbers, two accountants A managed runtime's **maximum heap** is an instruction to one component of the process: the object heap may grow to at most that many bytes before the collector must reclaim space rather than ask for more. The runtime enforces it from the inside, at a moment when the program is between operations. If an allocation still cannot be satisfied after collection, the runtime raises a failure the program can see, log and sometimes recover from. A **container memory ceiling** is an instruction to whoever granted the memory: this process may hold at most that many bytes in total, whatever it is holding them for. It is enforced from the outside, and its enforcement action is to stop the process. Those two numbers count different sets of bytes. The heap setting bounds one region; the ceiling bounds the whole process. Every byte the process holds that is not an object on the managed heap sits in the gap between them, and the gap is where services die. ## What the ceiling charges that the heap setting does not | Budget line | Inside the heap setting? | What drives it | |---|---|---| | Live objects, plus the room collection needs to work in | Yes | live set and allocation rate | | One call stack per thread | No | thread count times per-stack size | | Runtime metadata: loaded type information, compiled-code caches, symbol tables | No | code size and dependency count | | Buffers held outside the managed heap: network, compression, serialization, mapped files | No | concurrency and message size | | The native allocator's caches, arenas and fragmentation overhead | No | thread count and allocation size mix | | The program image and shared libraries mapped into the process | No | fixed per process | Only the first row responds to the heap setting, and note what is *inside* it: the space a moving collector needs to copy live objects into is heap space, counted against the heap maximum, whether or not it is occupied at the instant you look. Everything below the first row is charged to the ceiling in full, and most of those lines are unbounded by default — nothing in a typical configuration states how many threads may exist, how large a pool of reusable buffers may grow, or how much metadata a lazily initialised dependency will add on first use. ## Why the gap widens exactly when you are busiest - Every thread the worker adds brings its own call stack, and usually one or more per-request buffers with it. - Pools of buffers outside the managed heap grow to the high-water mark of concurrency and then stay at it, because a pool that shrinks defeats its own purpose. - The native allocator keeps caches and arenas and returns memory to the platform lazily, so the process tends to keep its peak rather than its current need. - Metadata grows as code paths are reached for the first time, so a worker that has served only health checks has not yet paid for the paths that matter. A measurement taken on an idle worker is therefore the least useful measurement available. A footprint budget is only trustworthy if it was taken at the concurrency and the code coverage the service actually reaches in production. ## The two failures look different, and the difference is the diagnosis 1. **Heap exhaustion.** The managed heap reaches its configured maximum, collection cannot free enough, and the runtime reports an allocation failure from inside the process, with a stack trace. The process usually lives long enough to write it down. 2. **Ceiling exhaustion.** The process is stopped between two instructions. There is no in-process record of any kind: no allocation failure, no shutdown handler, no flush of in-flight work. From inside, the only evidence is that the log stops mid-sentence. The tell is a **restart with no memory error recorded anywhere in the process's own output**. A worker that reports allocation failures is over-filling its heap and may want a larger heap. A worker that disappears silently and restarts is over-filling its ceiling, and raising the heap maximum is the one change that reliably makes it worse. ## Where the accounting usually goes wrong - Sizing from a development run, where thread pools never reach their maximum and buffer pools never fill. - Treating the heap maximum as a promise about the process rather than about one region. - Forgetting that the fixed lines — metadata, mapped code, the allocator's own structures — are paid once per process and therefore multiply when you run more, smaller processes. - Leaving no explicit margin, so every measurement error lands on the one line you cannot control. ## What this means for sizing The maximum heap is not a number chosen from what the program would like. It is a remainder: measure the non-heap lines at peak concurrency, subtract them and an explicit margin from the ceiling, and give the heap what is left. Then sanity-check that what is left is still a healthy multiple of the live set measured after a collection. If it is not, the thing to change is the live set or the concurrency, not the setting — because the heap setting is the only line you can move with a configuration change, and it is also the line that silently absorbs the error in every other line.