A machine holding a finished producing step's written buckets is lost while later steps run — what does recovering that cost?

## Two kinds of output, and only one of them is precious A job produces two very different classes of bytes. - **Final output** goes to the **shared store** — durable storage every worker can reach — and is normally replicated there, because losing it means losing the answer. - **Intermediate output** is what a redistribution leaves behind: one bucket per destination, written by a producing unit to the local disks of the machine it ran on. It exists only so the collecting step can pick up its share, and it is normally held in a single copy. That single copy is a deliberate trade, not an oversight. Replicating every intermediate byte would add a full extra network crossing and extra storage to every movement in every job, to insure against a machine loss that usually does not happen. Most designs decide that making the bytes again on the rare occasion is cheaper than copying them always. ## Why the bill lands on the producing work When the machine holding a share disappears, a collector asking for that share gets nothing, and there is no second copy to ask for instead. Each producer's bucket is *unique*: it holds the records that this producer made for this destination, and no other producer has them. What the engine still has is the description of how that piece was produced. So the repair is to **run the producing work again** — read that step's own input, redo its computation, re-route its records, re-hold its buckets — and only then can the waiting collector be served. Three consequences follow: 1. The expensive part is the **recomputation**, not the transfer. A transfer is seconds; the producing work can be minutes of processor time over a full piece of input. 2. If the lost step's input was itself the output of an earlier movement, the earlier shares may be needed again too, and the question repeats one level up. A loss late in a deep job can therefore unwind a long way. 3. The **later a loss happens, the more it costs**, because more producing work is resting on those disks by then. ## Surviving the worker is not the same as surviving the machine Two distinct lifetimes are easy to conflate: the worker process that produced the bytes, and the machine whose disks hold them. | arrangement | survives the producing worker exiting | survives the machine going away | what it costs | |---|---|---|---| | the producing worker serves its own output | no — the worker must stay alive | no | idle producing capacity cannot be released | | a separate process on the machine keeps serving the output | yes | no | one more process per machine to run and operate | | the output is written somewhere remote or replicated | yes | yes | every intermediate byte crosses the network at least once more | | nothing is stored; records are pushed as produced | there is no stored share at all | there is no stored share at all | recovery is an entirely different mechanism | The second row is the one candidates most often get wrong in both directions. A process that outlives the producing worker is genuinely valuable — it lets a job hand back workers it no longer needs while their output is still being collected — but it does nothing whatsoever about a lost machine. The disks are still that machine's disks. ## What varies between engines, and how to say it This is a place where a confident sentence about how it works is usually a sentence about one product: - whether intermediate output is replicated or offloaded at all, and whether that is a choice the platform makes or the job does; - whether a separate serving process exists, so capacity can be reclaimed mid-job; - whether losing a share triggers automatic recomputation or simply fails the job; - and, in a runtime that pushes each record downstream as it is made, there is **no written bucket to lose**. What a machine loss takes there is the records in flight and whatever the consumer had accumulated, and the job returns to a recorded point rather than re-collecting a share — a different recovery subject entirely. The safe formulation in an interview is to state the mechanism and then name the variation: *the share is on one machine's local disks, so its loss costs the producing work again, on engines that can recompute it from what they remember — and there is no share at all where the hand-off stores nothing.* ## Where this stops This explains **why** a loss is expensive and **what** gets redone in principle. Exactly which unit is retried, how many attempts it gets, and where a retry resumes from are questions about failure granularity and belong with recovery, not with the mechanics of the exchange.