How would you set the split between guaranteed and reclaimable machines for a fleet, and which workloads would you bar from the reclaimable half?
answer
- size for the total-loss case
- a floor, not an average
- keep the recovery path guaranteed
- the saving is paid in engineering
- drill it, do not assume it
basics
~20 sSize the guaranteed baseline at the demand that must still be served when every reclaimable machine is taken back at once, and put resumable, deadline-tolerant work on the surge. Keep coordinators, sole copies of state and the recovery path itself off the reclaimable half.
solid answer
~50 sThe split is a risk decision, not an arithmetic one. The **baseline** is guaranteed capacity sized for the total-loss case: the work that has a deadline, holds state that cannot be rebuilt, or whose failure stalls everything else. The **surge** is reclaimable capacity carrying work that is resumable, elastic and tolerant of finishing later - and it is cheaper because the provider kept the right to take it back, which is the opposite risk from a term commitment, where you carry the obligation. Bar from the surge anything whose loss is not merely redone work: the component that hands out and restarts units, the only copy of any state, work with a deadline shorter than a restart, and anything whose rerun is not idempotent. Then validate the split by measuring what a full reclamation wave actually does to the deadline, rather than assuming replacements arrive.
go deeper
Recall that fleets often mix capacity that cannot be taken away with cheaper capacity that can, and that the two carry different kinds of work.
Explain the sorting rule: resumable and deadline-tolerant work on the reclaimable half, state and coordination on the guaranteed floor, with the floor sized for the reclaimable half disappearing.
Demonstrate the operating side - measured reclamation rates at the hour that matters, a drill that removes the surge, and the long tail the mix adds to completion times.
Own the posture: what the floor guarantees, which workloads are barred from the surge and why, what the saving costs a team in engineering, and when the bet is re-examined.
## What the split is really deciding A mixed fleet is two different bets running side by side. The guaranteed half is capacity nobody can take from you, paid for accordingly. The reclaimable half is cheaper **because the provider retained an option on it** - and that is the opposite risk from a term commitment, where you took on the obligation and are exposed if your demand shrinks. Mixing them is deliberate: you keep a floor you control and put elastic work on capacity someone else may recall. So the question is not "what percentage" but **"what must still be true after the reclaimable half is gone"**. Answer that and the percentage falls out. ## Sizing the baseline The baseline is a floor, sized for the worst case rather than the average: - Start from the work that has a **hard deadline** - a result due at a fixed hour, a pipeline something else waits on. - Add the work that holds **state that cannot be rebuilt cheaply**, or whose rerun cost is measured in hours. - Add the **control components** without which nothing recovers. - Sanity-check it against the demand curve: an average-sized baseline is the classic error, because a reclamation wave does not politely arrive during a trough. An average-sized baseline covers the average and fails the deadline; a peak-sized baseline pays for the peak twice and gives up the saving the mix exists to collect. The defensible number is between them, and it is defended by naming what is on it. ## What may never sit on the reclaimable half 1. **The coordinator.** Whatever hands out work, tracks leases and restarts lost units. Losing it converts a routine reclamation into a stalled fleet, because the machinery that recovers everything else is the thing that went. 2. **Any sole copy of state.** The work source, the store the checkpoints go to, anything whose loss is not "redo some work" but "lose the work". 3. **Deadline-bound work with no slack.** If a restart plus re-run exceeds the time remaining, the surge cannot hold it however cheap it is. 4. **Non-idempotent work.** Anything whose second run is not safe - unguarded external effects, appends that duplicate. 5. **Synchronous work that cannot shrink.** A job that halts unless every member is present turns one reclamation into a fleet-wide stall; make it elastic first, or keep it on the baseline. | Workload trait | Belongs on | |---|---| | Resumable, checkpointed, deadline slack | Reclaimable surge | | Cheap to repeat, independent units | Reclaimable surge | | Holds the only copy of state | Guaranteed baseline | | Recovers or coordinates everything else | Guaranteed baseline | | Deadline shorter than a restart | Guaranteed baseline | ## What the saving costs to collect The discount is real and it is not free. Collecting it requires checkpointing that is frequent and restorable, leases and idempotent restarts, placement across several shapes and zones, and a validation burden for each shape. It also lengthens the tail: a job on reclaimable capacity finishes later on average, because some of its runs are redone. A fleet that cannot afford a longer tail should not be on the surge regardless of price. This is also where the decision stops being technical. The saving lands on one budget and the engineering lands on a team's roadmap, so a split decided without that conversation is usually reverted the first time a deadline slips. ## How to review the decision rather than assume it - **Measure the reclamation rate for the pools you use, at the hours that matter.** A daily deadline cares about the rate at that hour, not the monthly mean. - **Drill it.** Take the reclaimable half away deliberately and see whether the deadline still holds on the baseline. Any answer that has not been drilled is a hypothesis. - **Watch the tail, not the mean.** The cost of the surge shows up as long-tail completion times, which averages hide. - **Revisit when the workload changes shape.** A job that becomes synchronous, or acquires a tighter deadline, has quietly moved itself off the surge. ## What interviewers listen for They are testing whether a candidate can defend a number with an argument rather than a habit. The strong answer names the total-loss case as the sizing rule, names the components barred from the surge and why, prices the engineering the saving requires, and describes the drill that verifies it. The weak answer is "put everything on it, it is cheaper" - or its mirror, refusing reclaimable capacity entirely for work that is resumable by construction.
- Your fleet hits a daily deadline at a fixed hour. How does that change the split?It makes the reclamation rate at that hour the number that matters, not the monthly average, and it caps how much of the deadline path may sit on the surge - work must have slack for at least one restart before the cut-off. Many teams run the surge freely earlier in the window and shift onto the baseline as the deadline approaches.
- A team argues the baseline is waste and wants it trimmed to nothing. What is the counter-argument?The baseline is not idle capacity, it is the part of the fleet that survives a wave. Trimmed to zero, a single correlated reclamation leaves nothing running, nothing coordinating and nothing to restart from. Ask what result must still be produced during a wave; if the answer is anything at all, that is the size of the floor.
- How would you decide whether a new workload may run on the reclaimable half?Three questions: is a restart idempotent, is the state durable outside the machine, and is there slack between its worst restart cost and its deadline. Three yeses put it on the surge. Any no puts it on the baseline until the gap is closed, because the mix is a property of the workload, not a placement preference.
saying these in an interview costs you the question
- Puts the whole fleet on reclaimable capacity because it is cheaper
- Treats the guaranteed baseline as waste to be trimmed away
- Runs the component that restarts lost work on reclaimable capacity
- Assumes the saving needs no engineering to collect
- Sizes the baseline from average demand rather than the total-loss case