On a shared pool, the share policy takes capacity back from a job already running — what does that job lose?

## Two different actors, one English word Capacity can be taken away from a running job by two completely different parties, and conflating them produces the wrong remedy every time. - **The share policy reclaims capacity from a running job.** This happens inside your organisation, to honour a share someone else is owed. It is a policy you chose and can change, and it is the subject here. - **The provider reclaims a discounted machine.** That is an infrastructure bargain — a cheap machine the provider may take back at short notice for reasons that have nothing to do with your job — and it is a different decision with a different warning and a different remedy. ## What actually leaves with the capacity When worker processes are taken from a running job, the job loses exactly three things: - **The pieces of work in flight on them.** Anything part-way through is abandoned; a piece is the unit that has to be attempted again. - **Whatever those processes were still holding for others.** Intermediate results produced earlier and not yet consumed are the costly loss, because they represent work the job had already finished and counted. - **Anything cached or written locally there.** A result deliberately kept in a worker's memory to be reused later is gone, and the next use of it pays full price. The job as a whole does **not** end. The single **coordinating process** — the process that plans the pieces, hands them out and tracks what finished — is a different process on a different footing, and losing worker processes is not losing it. Whether the job then reattempts just the lost pieces or restarts more broadly is a property of the engine's recovery model rather than of the share policy, and the policy has no say in it. ## The bill depends on what the engine keeps where This is where an answer built from one engine goes wrong, because the three common designs in this class pay in different currencies. | Engine design | What the reclaimed capacity was holding | What the job pays | |---|---|---| | Finite job that materialises intermediate output locally for later fetch | completed output other workers had not fetched | redoing finished work, sometimes a whole earlier round of it | | Record-at-a-time runtime with per-key state | live state and records in flight | rolling affected work back to its last saved recovery point and replaying from there | | Two-phase disk-to-disk model writing each phase out durably | only the current attempt | reattempting that unit, with earlier phases still readable | One mitigation is worth naming because it changes the arithmetic of the first row: a separate process that keeps serving a finished worker's intermediate output after that worker is gone. Where such a thing exists, losing the worker process no longer by itself loses the output, and reclaiming becomes far cheaper. Where it does not, taking capacity back late in a long job can cost more than the machines were worth. ## How policies soften it Policies sit on a spectrum, and knowing where yours sits is most of the operational answer: 1. **Never reclaim.** An owed share is honoured only as running jobs release capacity. Simple and lossless, but the guarantee may take hours to materialise — or never, behind a continuous job. 2. **Reclaim only at a boundary between rounds of work**, when no piece is half-finished. The cheapest point to take capacity, at the price of waiting for that moment to arrive. 3. **Warn, then take.** The job is told capacity is going and given an interval to stop cleanly, save what it must and hand the processes back; anything still running when the interval expires is killed anyway. 4. **Take immediately.** Lowest latency for the owed share, highest destroyed work. ## Making a job cheap to reclaim from The properties that make reclaiming survivable are the same ones that make any capacity loss survivable, and a team that runs on a reclaiming pool should design for them deliberately: - Keep the unit of work small, so an abandoned piece is a small loss. - Prefer work whose intermediate output survives the process that made it, or accept that a late reclaim can cascade. - Save recovery points often enough that a rollback is cheap, and understand that this costs throughput continuously in exchange for a cheaper worst case. - Know which of your jobs must never be reclaimed from, and put those in a group whose share is not lent out. The judgment an interviewer is listening for is that reclaiming trades a **latency** improvement for somebody else against **wasted machine-hours** for you, and that the trade is only sound when the work destroyed is smaller than the wait it saves.