A worker is killed outright while the engine's own numbers show its budget well below full. What explains that?

## Three numbers, and the one that killed you - **The engine's accounted budget.** What the engine hands out to operators and to kept results, and what its per-unit report describes. It is a real number and it is not the whole story. - **The process's actual footprint.** What the machine sees the worker process holding. It is the accounted budget plus **off-budget overhead**: memory the process really uses that the engine's accounting never counted. - **The limit the platform enforces.** A ceiling the platform, not the engine, applies to the whole process, and enforces by killing it outright rather than by refusing one allocation. The scenario in the question is the gap between the first and the second being charged against the third. The engine was telling the truth about the region it manages, and the region it manages was never the thing being measured at the ceiling. ## Why the kill looks nothing like an allocation failure | | allocation the engine refused | the platform killing the process | |---|---|---| | who noticed | the engine's own allocator | the platform, outside the engine | | what you see | an error attributed to a unit of work | a worker that stops existing, with no engine message | | usual first response | spill to disk attached to the worker, or fail that unit | the unit is rescheduled on another worker | | does the engine's report show it | yes, in that run's numbers | no - the last numbers it wrote predate the kill | | retry behaviour | may well succeed after a replan or a spill | reproduces, because the same footprint recurs | The abruptness is diagnostic. A killed process cannot flush its own final numbers, which is why the evidence is a gap in the record rather than an entry in it. ## What lives in the gap - **The language runtime itself** - its own structures, thread stacks and internal buffers, which exist before your first record. - **Native and network buffers** allocated outside the engine's accounted regions for reading, writing and moving bytes between workers. - **User-supplied function allocations.** The largest and most common cause: a lookup table loaded once per unit of work, a parser that buffers a whole document, a native library with its own allocator, or records held in a closure that never lets go. The engine sees the function as a black box and cannot account for what it allocates. - **Concurrency multiplying all of the above.** Several units of work run at once inside one worker, and each of them may carry its own copy of the uncounted allocations, so the gap is roughly per unit rather than per worker. ## What actually helps 1. **Decide which number moved before changing anything.** If the engine's report shows the accounted budget nowhere near full, the accounted budget is not your problem and enlarging it is not your fix. 2. **Leave the process room the engine will never ask for.** The gap has to fit under the ceiling alongside the accounted budget. Enlarging the accounted budget without enlarging the ceiling makes the situation strictly worse, because the uncounted region now has less room. 3. **Look inside the user-supplied code first.** Anything allocated per record, per unit of work, or loaded eagerly at start-up is uncounted by construction. A per-record allocation that is not released is the classic finding. 4. **Count the concurrent units.** If each concurrent unit carries an uncounted buffer, the footprint scales with concurrency even though the accounted budget does not visibly move. 5. **Watch for repeated kills of the same shape.** A single kill can be a noisy neighbour on the machine; the same unit dying on three different workers is a property of the work, not of a machine. ## What varies between engines This is one of the places where engines of this class diverge most. Some ask you to declare an allowance for the uncounted region explicitly alongside the accounted budget, so the two are sized together; others expect the operator simply to leave room, and will happily let you configure a budget that cannot fit under the ceiling. The reported peak also means different things: usually it is the peak of what the engine accounted for, not the process's resident footprint, so two engines' peak figures are not comparable and neither is a statement about the ceiling. And on engines holding records as ordinary language objects rather than as packed bytes the engine manages, the same records occupy several times more, which widens the gap for identical data. ## The line to hold in an interview Name the three numbers, say which one was exceeded, and say why the evidence for it is the *absence* of an engine-side error. A candidate who reaches for a bigger budget before saying which number moved has answered a different question - and has usually made the real one worse. What the ceiling is, who granted it and how the platform applies it is the platform's subject, not the engine's; what the engine owes you is an honest account of the region it manages and room left for the region it does not.