Your platform team's standard answer to a single unit running out of room is to raise every worker's memory — what does that policy cost?

## What the policy actually buys A single unit of work fails because one thing did not fit in one process. Raising every worker's memory does make that thing fit, if the increase is larger than the shortfall. The question a lead has to answer is what was bought and at what price. What was bought is **one multiple**. If the failure was a group that must be held whole, the group's size is a property of the data — how many records carry that key, and how large each is. Doubling the budget survives a doubling of the group. It does not survive the following quarter, and it does not survive the next job that hits the same shape with a different key. Purchasing headroom against data growth is a losing race that has to be re-run every time it is lost. ## What it costs, and where The cost is not paid by the job that failed. It is paid by the fleet: - **Every worker process of every job** holds the larger footprint for the whole of its run, whether its working set needed the room or not. Multiply by process count and by hours to get the real number. - **Parallelism falls where the machine total is fixed.** A machine with a fixed amount of memory divides into fewer, larger worker processes. Fewer processes means fewer units of work in flight, and jobs that were perfectly healthy take longer — a slowdown attributed to the cluster rather than to the change that caused it. - **Reclamation pauses lengthen** where the runtime holds records as ordinary language objects, each carrying the runtime's own per-object bookkeeping and each reclaimed automatically: a larger region takes longer to sweep, and the worker's own work stops while it happens. Engines that keep records as a packed byte layout they allocate and interpret themselves are much less exposed to this, so the size of the effect depends on which representation your platform is standardised on. - **Scheduling gets harder.** In a shared pool, a job asking for larger processes waits for machines that can host them; bigger indivisible requests pack worse. Who grants that share is the resources subject, but the queueing is a real consequence of this decision. - **The defect is hidden.** The job whose step must hold a whole group is now passing, so nobody reformulates it, and the reformulation is the thing that would have made it durable. | Lever | Blast radius | Buys you | |---|---|---| | Reformulate the failing step | One job, one team | A fix that survives data growth | | Scoped override for that job | One job, on request | Time, with the defect still recorded | | Fewer units per worker | The jobs that opt in | Room per unit, at a cost in parallelism | | Fleet-wide default raise | Every job, every run | One multiple, charged to everyone | ## When raising the default is the right call It is not always wrong, and a lead who refuses it on principle is as unhelpful as one who reaches for it first. It is defensible when: 1. **Measurement says the shape is wrong for the median job**, not for the loudest one. If peak footprint across most jobs sits near the ceiling and spilling is routine and costly, the fleet is genuinely undersized and every job is paying in time for memory it should have had. 2. **The machines have memory that nothing can use.** If the per-machine total is not the binding constraint — cores are — then larger processes cost nothing in parallelism and the objection above does not apply. 3. **The engineering cost of the fix genuinely exceeds the memory cost**, which does happen for a job with weeks of life left in it. Then make it explicit and temporary rather than permanent and silent. ## The better default, and the thing to institutionalise The useful distinction is blast radius. A scoped override that applies to one job, carries a written reason and has an expiry date, keeps the cost proportional to the problem and keeps the defect visible. A fleet-wide raise is a decision about every job the organisation runs, and should be justified by a distribution rather than by an incident. Three things worth owning at this level: - **A record of every override and why.** Without it, the fleet default creeps upward by accumulation, one incident at a time, and nobody can say which increases are still needed. - **A supported way to express the operations that keep failing.** If teams keep writing whole-group possession because the alternative is awkward, the platform is the reason the failure recurs. - **A standard diagnosis path**, so that the failing unit's own numbers are read before anything is bought. The tell of a weak organisation here is not that it raises memory; it is that it raises memory without ever having looked at what the failing unit contained.