A provider can reclaim a batch job's machine with minutes of warning - what must fit inside that notice, and what does it not promise?

## What reclaimable capacity actually is A provider sells the same machines under several postures. **Metered on-demand capacity** is yours until you release it. A **term commitment** is capacity you promised to keep paying for, so the risk sits with you. **Reclaimable capacity** is the third posture: the machine is yours until the provider wants it back, and it is priced lower because the *provider* kept an option, not because you signed anything. That difference in who holds the risk is the whole design consequence - a commitment hurts when your demand shrinks, reclaimable capacity hurts when someone else's demand grows. Because the provider holds that right, it needs a way to exercise it, and the reclaim notice is that mechanism. ## The notice, precisely - It is delivered **to the machine**, as a signal software running there can observe - not to a human, and not to your work queue or your scheduler. - It is **short and fixed**: minutes rather than hours. Providers publish a length; the lengths differ, and some capacity classes carry a longer one than others. - It starts a countdown you **cannot extend, defer or decline**. Nothing you send back changes the outcome. - It is **best-effort**. A machine can also disappear with no notice at all, because ordinary hardware and network failures do not announce themselves. ## What has to fit inside the window In order, because the order is what makes the work recoverable: 1. **Stop claiming new work.** Anything claimed after the notice is work you are certain not to finish. 2. **Checkpoint the unit in progress** to durable shared storage that outlives this machine. The machine's own local disk goes away with it, so a checkpoint written there is gone exactly when it is needed. 3. **Hand the unit back** - release the lease or return it to the queue so another worker can claim it immediately instead of waiting for a lease to expire. 4. **Deregister** from whatever was routing work or requests here, so nothing new arrives during the last seconds. 5. **Exit.** Whatever is unfinished at the deadline is cut off wherever it happens to be. | The notice gives you | The notice does not give you | |---|---| | A bounded window to make work resumable | Time to finish the unit in progress | | A defined point to stop accepting work | An extension, a deferral or a veto | | A chance to release leases and held state | Any guarantee that it will arrive | ## Size the work unit to the window The practical move is to make the **unit of work smaller than what you can wrap up inside the window**, and to checkpoint often enough that the redo left over is bounded and small. If one indivisible step runs longer than the notice, the notice buys that step nothing: the step is cut off mid-way whatever your handler does, and the only recovery is the last checkpoint. Teams discover this when a job that checkpoints every hour meets a warning measured in minutes and loses the hour every time. Two related constraints follow. First, the drain itself must be fast: if writing state takes longer than the window, shrink what a checkpoint contains or write it incrementally. Second, the handler must be **safe to run twice and safe to interrupt**, because a notice can arrive while a scheduled checkpoint is already being written. ## Why the notice is an optimisation, not the design If the notice were a guarantee, you could build around it. It is not, so the design has to survive its absence: - Work is handed out under a **lease that expires**, so a silent disappearance eventually returns the unit. - A restart is **idempotent**, so a unit that runs twice does no harm. - Progress lives in **durable shared storage**, never only in memory or on the machine's disk. Given those three, the notice makes recovery *faster and cheaper*: less redone work, and no waiting for a lease to expire before another worker picks the unit up. Without them, the notice only shrinks the loss on the machines that happened to get one. ## What interviewers listen for A strong answer separates three things that candidates routinely merge: what the notice *is* (a short, one-way, best-effort warning), what it is *for* (making the work resumable elsewhere), and what the design must do *anyway* (tolerate loss with no warning). A weak answer treats reclamation as a rare failure to be handled by a shutdown hook, which is exactly backwards: on reclaimable capacity, interruption is normal operation.