Capacity plans usually size a fleet so peak load lands well below 100% of its capacity — often somewhere near 60%. Why leave that much unused, and what do you gain and lose by raising the target?

## Headroom is a purchase, not an accident Idle capacity looks like waste on a bill, and someone will eventually ask why the servers are only 60% busy. The answer is that the gap is doing four jobs, each of which has a failure attached if you remove it. **Redundancy.** If the service must keep serving peak after losing a zone or a cell, the arithmetic already caps utilization: with three equal zones you cannot exceed about two-thirds, because the survivors must absorb the third. For most multi-zone services this is the binding constraint and everything else is a smaller adjustment on top of it. **Forecast error.** The demand number is an estimate. Headroom is what turns a 20% miss into an ordinary week rather than an incident. **Time to acquire capacity.** Even fully elastic systems react rather than anticipate: a metric window, a provisioning call, instance boot, dependency registration, cache warm-up and JIT warm-up all cost minutes. A promotional spike or a push notification lands in seconds. Headroom is what serves traffic during the gap between demand arriving and capacity arriving. **Transient self-inflicted load.** A rolling update takes instances out of service while adding load to the rest. A failover moves a shard's traffic onto peers with cold caches. A blip causes clients to retry, briefly multiplying request volume. All of these consume headroom precisely when you least want to be near the limit. And separately from all four: response time rises well before a resource is fully saturated, so the practical operating ceiling is below 100% regardless of what the plan says. ## What raising the target buys The cost relationship is linear and easy to state. Fleet size is roughly `peak / (per_unit_capacity x target)`. So: ``` target 50% -> 2.00x peak purchased target 60% -> 1.67x peak purchased target 75% -> 1.33x peak purchased target 90% -> 1.11x peak purchased ``` Moving from 60% to 75% removes about 20% of the fleet — real money at scale. Moving from 60% to 90% removes a third. What you give up is proportionate: at 90% across three zones you cannot lose a zone, you cannot absorb a forecast miss, and a rolling deploy alone may push you into saturation. ## Choosing the number Start from the redundancy ceiling, because it is usually the strictest. Then subtract for forecast uncertainty (larger for a volatile or launch-heavy service, smaller for a steady one) and for how fast you can actually add capacity. Then check the number against measured behaviour — the point where response time starts climbing for this specific service — and take the minimum. Different tiers deserve different targets. A latency-critical user-facing tier and an asynchronous batch tier that can be late do not need the same cushion, and applying one company-wide number to both overspends on one and underprotects the other. ## Measure per resource and per domain "Utilization" is not one number. The binding resource is whichever one saturates first, and it is often not CPU: a connection pool, memory, file descriptors, a downstream quota, disk throughput, or a lock. A fleet reported at 55% CPU while its database connection pool is at 95% has no headroom at all, and the CPU graph will not say so. Granularity matters just as much. A 60% hourly average can contain minutes at 100%. Compute utilization against the peak interval you actually need to survive, not against a comfortable average. And measure per failure domain. Aggregate utilization across zones hides one zone at 90% when traffic is unevenly distributed, which is exactly the zone whose loss you were planning for. ## Making the headroom cheaper The honest way to reduce the cost of headroom without reducing the headroom is to fill it with work that can vanish instantly: batch jobs, reprocessing, model training, anything preemptible and sized so it can be evicted in seconds when foreground demand arrives. You are still holding the capacity; you are just not holding it idle. The discipline required is that the filler work must genuinely yield immediately — filler that takes ten minutes to drain is not filler, it is a second workload competing during your incident.