Every machine in a reclaimable training fleet disappeared inside one minute - why does that happen, and what spreads the risk?

## A capacity pool is a failure domain Providers do not hold one undifferentiated heap of machines. Capacity is organised by **shape** - a family weighted toward processors, memory, local disk or accelerators, at a given size - and by **place**, meaning a zone inside a region. The combination is a pool, and reclamation decisions are made against a pool: when guaranteed demand for that shape in that zone rises, the provider takes back the reclaimable machines of that shape in that zone. This is the mental model the scenario depends on. Reclamation is **not** a random per-machine event with a low probability that you can multiply out across a fleet. It is a **correlated** event whose unit is the pool. A hundred identical machines in one zone is one bet, made a hundred times, on the same outcome. ## Why the whole fleet goes at once - The trigger is **external to your workload** - someone else's demand, a large launch, a regional peak. Nothing about your job attracts or repels it. - The provider reclaims from the pool it needs, and a fleet built from one shape in one zone lives entirely inside that pool. - The demand that caused the reclamation is usually **correlated demand**: everyone reclaimed at the same moment is now looking for the same kind of machine at the same moment, which is why a wave is also the worst time to go shopping for replacements. ## What spreading actually means Diversity has to be in the dimensions the pool is defined by, or it is decoration: 1. **Several shapes.** Different families and sizes, so demand for one does not empty the fleet. This is the dimension teams skip, because it is the one that requires work. 2. **Several zones.** Different places inside the region, so a localised surge does not reach the whole fleet. Cheap to do, and it reduces correlation without removing it - a shortage of one scarce shape can span the region. 3. **A placement layer that is free to choose.** If the work distributor can only place units on one shape, having others available changes nothing. 4. **A mix with guaranteed capacity underneath**, so a total reclamation of the reclaimable half still leaves something running. | A fleet built from | Behaves as | On a reclamation wave | |---|---|---| | One shape, one zone | A single failure domain | Empties completely, in minutes | | Several shapes, one zone | Partly independent | Loses the shape in demand | | Several shapes, several zones | Several domains | Loses a slice; the rest keeps running | ## What diversity costs The interview is looking for the second half of the answer, because spreading is not free: - **Validation.** Every shape is another configuration the workload must be proven on - memory headroom, accelerator availability, throughput per machine. - **Uneven throughput.** A mixed fleet does not produce work at a uniform rate, so anything that assumes equal machines - static sharding of work, per-machine batch sizes - has to become dynamic. - **Synchronous work suffers most.** A job whose members exchange state at every step advances at the pace of its slowest member, so a slower shape slows everyone, and losing one member stalls the rest until it is replaced. That is an argument for making the work **asynchronous or elastic**, not an argument against diversity. - **Operational surface.** More shapes means more configuration to keep current and more ways for one of them to be quietly broken. ## Where spreading does not help Be honest about the limits. If the workload can only run on one scarce shape - it needs a specific accelerator, or a memory size only one family offers - diversity across shapes is not available, and the answer becomes a larger guaranteed baseline instead. If every unit of work is so long that a restart costs hours, spreading reduces how often you are hit but not what each hit costs; that is a checkpointing problem, not a placement one. And if the fleet's coordinating component sits in the pool too, spreading the workers is beside the point: the wave still stops everything. ## What interviewers listen for The telling moment is whether the candidate treats reclamation as independent random failure or as a correlated event with an external trigger. Everything else follows from that: a candidate holding the correlated model reaches for shape and zone diversity, a floor of guaranteed capacity and elastic work distribution, and they price the diversity honestly rather than presenting it as free.