Why must a job's recorded read position advance only after the output it covers is durable, and what does committing both as one unit buy?

## Two durable facts and a gap between them Every job that can resume has two things it must make durable: what it produced, and how far into the input it got. The second is the **recorded read position** — the marker saying how far into the input the job has durably progressed, only ever advanced once the work it covers is safely written. The two are written by different acts, at different moments, often to different systems, and a machine can die in between. That gap is the entire subject. ## The two orderings, and why only one survives | Order | Crash in the gap | What anyone can see | Repairable? | |---|---|---|---| | Advance the read position, then write the output | That span's output never lands, and the restart resumes past it | Nothing. The rows simply are not there | Only by finding the hole and reprocessing that span by hand | | Write the output, then advance the read position | The restart re-reads that span and writes it again | A duplicate at the destination | Yes — by keying the write, or collapsing copies at read time | The asymmetry is not about probability; both windows are the same few milliseconds wide. It is about detectability. A duplicate announces itself in a count, a sum or a uniqueness violation. A hole announces nothing at all, and the job that made it reports success. So the rule is stated as an ordering constraint: **the recorded read position is advanced only after the output it covers is durable.** Then the residual failure is a repeat, and the writing side's remaining job is to make the repeat leave no extra trace. ## Closing the gap: committing both as one unit **Committing output and read position as one unit** means arranging that the write becomes visible and the position advances together, so no crash can leave one done and the other not. In practice it shows up in three shapes: 1. **One transactional destination.** The output rows and a small row holding the position are written in a single transaction against the same system. Either both are visible or neither is. This is the cleanest version and it requires the destination to be able to store the position at all. 2. **Staged output promoted in a single step, with the position inside the promotion.** Each unit writes to a private location; one final act makes the whole result visible and records the position in the same act — a pointer write, a manifest, a single small marker. A failed attempt's partial output was never visible, and a crash before the final act leaves nothing to undo. 3. **A write-ahead record of intent.** The job durably records what it is about to do before doing it, so a restart can tell whether the effect already happened and finish or skip accordingly. This is the fallback when the destination cannot participate in anything. ## When one unit is simply not available If the destination is one system and the position lives in another — the job's own recovery point store, a separate coordination store, the source's own bookkeeping — then no single act can cover both, and claiming otherwise is the most common overreach in this area. What you actually have is: - the ordering rule, which bounds the damage to a repeat; - a **deterministically chosen write key**, so the repeat replaces rather than adds; - or a deterministic identifier per row and deduplication pushed onto every reader. Runtimes differ a great deal in how much of this they do for you. Some ship destination integrations that arrange the coupling and expose it as a setting; others leave the entire construction to the author; and in the oldest batch model the equivalent is the promotion of staged output, where "which input pieces were processed" *is* the position and the publish is the commit. Do not assume the runtime in front of you belongs to whichever family you learned first. ## The interview shape Expect to be handed a job that writes rows and then records where it got to, and asked what a crash between the two costs. The strong answer names both orderings, says which failure each produces, states the rule, and then says what remains: even with perfect ordering a duplicate is still possible, so something at the destination has to absorb it. The weak answer says "we commit atomically" without noticing that two separate systems were named in the question.