After a multi-step data pipeline fails midway, when is restarting only the failed steps unsafe?

## What restart-from-failed actually promises When a run fails at step 4 of 6, the attractive option is to resume at step 4 and leave steps 1–3 alone. Every orchestrator offers some version of this, and for well-built pipelines it is the right default: it saves the expensive extract, it shortens recovery, and it keeps the deadline in reach. But it is an *assertion*, not a convenience. You are asserting that the world is in exactly the state a fresh run at step 4 expects. ## Failure mode 1: the failed step's partial writes A step that dies partway has usually already done something. If it appends rows, the restarted attempt appends them again and the target silently double-counts — the worst kind of failure, because the run then goes green and the number is wrong. Restart is only safe here if the step's write is **all-or-nothing or self-replacing**: write to a scratch location and swap at the end, overwrite the whole target window for the run, or merge on a key so re-applying is a no-op. If the step is a blind append into a table with no natural key and no window boundary, restarting it is a data-corruption decision, and the correct recovery is to delete the run's partial output first, or to rerun from a step that rebuilds the target wholesale. ```sql -- restartable: the step replaces exactly its own window delete from analytics.orders where order_date = :run_date; insert into analytics.orders select * from staging.orders where order_date = :run_date; ``` ## Failure mode 2: the moving source This is the subtle one and the reason seniors get asked it. Suppose step 2 read a mutable source table at 02:00 and landed an extract. The run failed at step 4. You restart at 14:00. Steps 4–6 now run against the 02:00 extract, but if any of them also read the source directly — a lookup, a dimension join, a late reconciliation — they see the 14:00 state. The output is a blend of two points in time that never coexisted: an order that did not exist at 02:00 joined to a customer record that changed at 11:00, or worse, orders from the extract whose keys are missing from a dimension the restarted step read later. Nothing errors. The result is simply not a coherent snapshot of anything. The test: **does any step downstream of the restart point read state that has changed since the completed steps ran?** If yes, either pin that read to the same snapshot (a version, a timestamp bound, a frozen extract) or rerun from the step that established the snapshot. ## Failure mode 3: the vanished scratch space Many pipelines write intermediate results to per-run temp locations, and many platforms clean those on failure, on a lifecycle policy, or when the worker went away. A restart at step 4 that expects step 3's temp output to still exist fails immediately in the best case, and reads a stale leftover from an earlier run in the worst. ## Failure mode 4: the failure was a symptom Step 4 crashed on a null it should never have seen. Step 2 succeeded — but it succeeded at loading a truncated file. Restarting at step 4 either fails again or, after you "fix" step 4 to tolerate nulls, produces confidently wrong output from bad input. Before choosing a restart point, establish that everything upstream of it is not merely *green* but *right*. Green is a claim about exit codes. ## Failure mode 5: parameters recomputed at restart If a step derives its target window from the wall clock at execution time rather than from the run's own pinned parameters, restarting it twelve hours later processes a different window than the completed steps did. The pipeline must carry its window as run-scoped parameters that every attempt and every restart resolves identically. ## When a full rerun is the right call Rerun the whole thing when the source is mutable and you need one coherent snapshot, when the failed step is not self-replacing and you cannot cleanly remove its partial output, when you have changed transformation logic that earlier steps also embody, and whenever the cost of being subtly wrong exceeds the compute cost of doing it again. For most warehouse pipelines the compute is cheap relative to a wrong published number; the real constraint on a full rerun is usually the *deadline*, not the money. ## Designing so the question stops being interesting The mature answer is to make restart-from-failed unconditionally safe: - every step writes atomically — scratch then swap, or a single window-replacing statement; - every step is scoped to a window derived from run parameters, not from the clock at execution time; - source reads inside a run are pinned to one snapshot (an immutable landed extract, a version, a timestamp predicate); - intermediate outputs are durable and addressed by the run's parameters, so a restart can find them; - steps validate their inputs, so an upstream truncation fails at step 2 rather than corrupting step 5. When those hold, restart-from-failed is a cost optimisation with no correctness argument attached — which is exactly where you want the decision to be at 03:00.