A worker dies mid-piece in a 400-piece finite job: what does the engine re-run, and what must be true for that to be correct?

## Three different things called "the unit" A distributed job has at least three units, and retry granularity is about exactly one of them. - **The unit of parallelism** — one piece of the split input, which decides how much of the cluster works at once. - **The unit of retry** — the smallest thing the engine will re-run independently on another machine after a failure. That is the subject here. - **The unit of recovery** — the region of the job that has to rewind together to a *recovery point*: a durable copy of everything the job would otherwise lose, written while the job keeps running, so a restart can resume from it instead of from the beginning of the input. It is often much larger than either of the others. In a finite job that keeps nothing between records, the first two coincide: one piece of input, one unit of work, one re-run. Treating that coincidence as a law is the standard wrong answer, because it stops holding the moment the job accumulates state between records or keeps finished results in worker memory. ## What the engine actually does The **coordinating process** — the single process that holds the job's plan and tracks which units are done — stops hearing from the machine. (How it concludes the machine is gone is failure detection, a theory subject owned elsewhere.) It then: 1. marks every unit that machine was running as failed, and increments those units' attempt counters; 2. re-queues them as fresh attempts, placed wherever the cluster has capacity — **not necessarily on a different machine**, though several engines will steer away from a host that has failed repeatedly; 3. leaves alone every finished unit whose result is still readable from somewhere other than the dead machine. Step 3 is where designs diverge and where the whole blast-radius argument lives: an engine that materialises each step's output to storage outside any one machine has lost only the work in flight, while an engine that holds finished results in worker memory or on a local disk has also lost completed work and must re-run whatever produced it. ## The three preconditions for a per-unit re-run | precondition | what it means | what breaks without it | |---|---|---| | **Re-readable input** | the unit's slice of input can be read again, unchanged, from a position the engine recorded | there is nothing to re-run the unit against; what makes a source re-readable is a subject of its own | | **Output determined by that input** | the computation is a function of the records, not of a mutable external store or the current clock | the second attempt legitimately produces a different answer, so the retry silently changes the result instead of repairing it | | **Abandonable partial output** | whatever the failed attempt half-wrote can be thrown away | the surviving fragment plus a complete re-run means some records land twice | The third is usually arranged by **staged output promoted in a single step**: each attempt writes to a private location, and one final act makes the whole result visible at once, so an abandoned attempt leaves files nobody ever promotes. Where the destination offers no such staging — rows written one at a time into a live table, a call made to an outside system — the partial effect is already visible, and making a repeat harmless at the destination is a separate subject from re-running the unit. ## What the re-run costs - The unit's compute from the beginning: an attempt resumes nothing, because there is no saved partial progress inside a unit to restart from. - Re-reading the unit's input, which may mean pulling it across the network again. - Whatever warm local cache that attempt had built. - Wall clock for everything waiting behind it. If the step ends at a synchronisation point where every producing unit must finish before any consuming unit starts, one late re-run holds up the entire step. That last item is why a small retry unit is not automatically a cheap one. Cutting the input into more, smaller pieces makes each individual re-run cheaper, but a re-run that starts at the worst moment still costs a full round trip through everything queued behind it. ## What this is not - **Not the layer above re-running the whole job.** A scheduling system that runs a failed job again tomorrow morning is a different graph with its own rules, owned elsewhere; the engine's per-unit retry is exhausted entirely inside one run. - **Not the platform restarting the process.** Something that restarts a crashed process or replaces a machine hands the job a worker back; it restores none of the work, which the engine still has to re-run. - **Not a unit that is merely slow.** A worker that is behind has not failed, and the remedy for that has a different name and lives in a different part of this tree.