Surviving Failure

Recovery is reprocessing: a restart re-reads input the job had already handled and writes it a second time. The destination remembers nothing of the first attempt, so a plain append adds those rows again.

The engine re-runs only the failed unit of work — the smallest piece it hands one worker — on whichever machine has capacity. That is correct only when the piece can be recomputed from re-readable input and its partial output is discarded.

Recovery is reprocessing: a restart re-reads input it already consumed. So the source must retain those records and let a reader resume from a durably recorded position — a source that forgets each record after handing it over makes the lost work unrecoverable.

A marker injected at the sources travels with the records; each worker saves its own part as the marker reaches it and then passes it on, so every part covers the same prefix of the input.

A safe write key is computed only from fields the input record already carries, so a re-run derives the same key and the destination replaces the earlier row. Keys drawn from randomness, wall-clock time, attempt identity or a destination-assigned sequence all differ on the second run.

Advancing the read position first turns a crash into permanent silent loss, because the restart resumes past output that was never written. Advancing it last turns the same crash into a duplicate. Committing output and position together removes the window where only one of them happened.

Identical failure on every attempt, across machines, points at a deterministic cause: the records that unit holds, or the code path they take. Retrying cannot fix it — only changing the input, the code or the unit's tolerance can.

Stop reading new input, let the records already inside reach the destination durably, advance the recorded read position to match, then write a restart point and exit. A hard kill loses no input but guarantees repeated work.

The ordering chooses which failure you get. Advancing the recorded read position first means a crash in the gap loses records nobody will ever reprocess. Writing output durably first means a crash re-does covered work — repetition, which the destination can be made to absorb.

Whether the job carries anything between records. A job that keeps nothing is stopped and resubmitted, provided its recorded read position is durable. A job holding accumulated results must write a restart point before it exits, or those results are gone.

Because the seam between two little jobs is already a moment with nothing in flight: all work for the completed piece has finished and none has started for the next, so the picture can simply be written there.

Files must be immutable once visible, become visible in one atomic step, and survive until the job can no longer need them. The recorded position is then a set of file identities, and each identity must change whenever the content does.

Two options: hold back the input that already delivered its marker until the other arrives, paying a pause that spreads upstream; or save at once and write the still-arriving records into the picture, paying size and a slower resume.

Weigh what one capture costs — the slowdown while workers save and the bytes pushed to shared storage — against the input a crash makes you reprocess, which is about half an interval on average.

Three moves remain: publish staged output in one visible act so abandoned attempts never appear, stamp every row with an identifier derived from the input record and collapse copies at read time, or record intent before an effect that cannot be retracted. The fourth honest answer is to accept duplicates and say so.

A high attempt cap converts a broken job into a merely slow one. The extra four hours are retry time: some failure recurs, is paid for on every attempt, and never surfaces because the final status is green.

Blast radius follows where the lost results lived. Durable outside the machine: only the in-flight unit re-runs. On its local disk or in its memory: the producing units re-run too. Held as accumulated state: a whole connected region rewinds to its last recovery point.

Different moment, different lifetime, different purpose. Automatic saves happen on a timer while the job runs and may be discarded when it stops; a restart point taken on purpose is written once after the drain, for a build not yet deployed.

Put a durable, re-readable copy in front of the feed and have the job read that instead. Recoverability then starts at the copy, so the unrecoverable segment moves to the trivial hop that writes it — it shrinks, it does not vanish.

The strong promise names an effect at one named destination, not a delivery across a pipeline, so it has to be decided hop by hop. Inventory each effect by whether it can be retracted and who already consumed it, then price the next notch in freshness, storage and coupling.

Five spans: draining, writing the restart point, the deploy and getting machines, reading the saved copy back, and catching up the input that piled up. It must fit inside the source's retention and whatever freshness the consumers were promised.