Your payments ledger fails over into its standby zone and half its replacement machines will not launch — why is capacity scarcest exactly then?

## The assumption hiding in most failover plans A standby zone is usually described as `we run in two zones and fail over to the other one`. Read closely, that sentence contains a purchase nobody made: it assumes that at the moment of the failure, machines of the right shape will be free in the surviving zone and will launch on request. Capacity that has not been launched is not a resource. It is a forecast about another company's inventory, made for the one hour in which the forecast is least likely to hold. ## Why demand correlates with the failure The event that removed a zone is a **shared cause**, and the reactions to it are alike for structural reasons: - **The same trigger.** One physical or platform-wide event starts everybody's recovery inside the same few minutes. - **The same destination.** Two-zone and three-zone designs are the norm, so the surviving zones of that region are where everyone goes. There is nowhere else to go. - **The same shapes.** Fleets are built from a small set of popular families and sizes, so the demand lands on the same pools. - **Automation rather than people.** Health checks and orchestrators react in seconds, compressing everyone's demand into one window instead of spreading it over an hour. - **Amplification.** Failed launches are retried, often aggressively, so the offered demand is larger than the real demand and arrives faster. The consequence is blunt: the pool your plan depends on is drained by the plans of every other tenant, and regional capacity in aggregate is no comfort, because you cannot launch in a zone that is not the one you are failing into. ## Three grades of standby capacity | What sits in the standby zone | What it costs while nothing is wrong | What it is worth at failover | |---|---|---| | A plan to launch | nothing | whatever happens to be free that minute | | An allocation held for your account | close to the price of running those machines | a launch drawn from stock no other tenant may take | | Machines already running | the price of machines doing little | capacity that needs no launch at all | Only the lower two rows are insensitive to correlated demand. The top row is what most organisations actually have, and it works most of the time — which is exactly why it is rarely questioned before the hour in which it does not. ## Sizing the part you pre-pay for Nobody holds a duplicate fleet. The tractable question is how much of the fleet must exist immediately for the service to be serving at all, and only that much is held. Take a service running ninety machines spread evenly across three zones, thirty per zone, with peak demand needing all ninety. Lose a zone and sixty survive. If the service is acceptable at two thirds of peak throughput, sixty is already enough and nothing needs holding. If it is only acceptable above roughly eighty-five percent of peak, it needs about seventy-seven machines, so the gap is seventeen — held as, say, twenty split between the two surviving zones rather than a standby thirty. Those figures are illustrative and assume each machine contributes equally and that traffic can actually be steered onto the survivors. The method is the point: hold the **minimum serving set** and launch the remainder best-effort. ## What to do with everything above the held line 1. **Degrade deliberately.** Decide in advance which features, tenants or request classes are dropped first, so a partly-capacitated service is a designed state rather than a random one. 2. **Widen the fallback list for the remainder**, with entries genuinely unlike each other: another generation, a differently weighted family, the less fashionable surviving zone. 3. **Retry spaced, not tight.** Backoff with jitter places more machines than a loop, because the loop competes with every other tenant's loop and can be throttled. 4. **Prefer smaller units for the overflow**, since they fit the fragments that whole large shapes cannot use. ## Rehearsals are optimistic by construction A failover drill run on a quiet afternoon proves the automation works, the data is where it should be and traffic steers correctly. It cannot prove the pool will be there, because the variable that matters — how many other tenants want the same shapes in the same zone in the same minutes — is absent from the drill and supplied only by the real event. Treat drill timings as a lower bound on the good path, and rehearse the degraded path explicitly: force the launches to fail and confirm the service still serves on what it holds. How the data stays consistent across zones, and how traffic is steered after the loss, are their own decisions. This one is narrower and comes first: whether the machines you intend to run will exist when you ask for them.