In a multi-stage batch pipeline (e.g. extract -> validate -> transform -> load), a crash happens partway through processing a large batch after some records have already reached the load stage. What are the main strategies for making the pipeline recoverable, and what does each guarantee?

## The question a restart has to answer Recoverability in a multi-stage pipeline is fundamentally about answering one question precisely after a crash: which records have and haven't had their effects durably applied at each stage, so a restart can pick up correctly rather than guessing. This is a genuinely hard problem in Pipes and Filters specifically because the whole point of the style is that filters are decoupled and don't share transaction context — which is exactly what makes recovering a consistent state across stages non-trivial once a crash happens mid-flight. | Tool | What it addresses | |---|---| | **Checkpointing** | the blast radius of a crash — the window of work since the last checkpoint | | **Idempotency** | correctness under reprocessing | | **Staged or atomic-publish patterns** | visibility of partial work rather than reprocessing correctness | ## The three tools 1. **Checkpointing.** The first tool is checkpointing: each stage (or the pipeline driver) durably records its own progress marker — the offset, ID, or timestamp of the last unit it successfully finished processing — separately from the actual data processing. On restart, the pipeline reads the checkpoint and resumes from that point instead of restarting from the very first record. This bounds the 'blast radius' of a crash to the window of work since the last checkpoint, rather than the entire batch; a pipeline checkpointing every 10,000 records only has to reprocess up to 10,000 records after a crash, not the full multi-million-record batch. The trade-off is checkpoint frequency versus overhead: checkpointing after every single record gives the tightest recovery window but adds a durable-write cost to every unit of work, while infrequent checkpointing is cheap but widens the window of potential reprocessing (or, if the checkpoint mechanism itself isn't crash-safe, potential loss). 2. **Idempotency.** The second, more powerful tool is idempotency: designing each filter's write-side effect so that applying it more than once produces the same result as applying it once. The canonical technique is upserting by a natural or business key rather than blind appending — reprocessing the same record a second time after a crash just overwrites the same row with the same values instead of creating a duplicate. Idempotency matters enormously because it changes the delivery guarantee you actually need: without it, you need **exactly-once** delivery (each record's effect applied exactly one time, ever) to avoid duplicated financial transactions, double-counted metrics, or duplicate emails — and exactly-once delivery across independent, decoupled filters communicating through pipes, especially across process or network boundaries, is expensive to guarantee and sometimes provably impossible without additional coordination. With idempotency in place, you only need the much cheaper **at-least-once** guarantee (a record's effect is applied one or more times, but reprocessing is harmless) — 'at-least-once delivery plus idempotent effects' is functionally equivalent to exactly-once from an outside observer's point of view, and it's the strategy most real distributed pipelines actually use, because building true exactly-once coordination between independent filters is disproportionately expensive relative to just making the write idempotent. 3. **Staged or atomic-publish patterns.** The third tool addresses visibility of partial work rather than reprocessing correctness: staged or atomic-publish patterns prevent downstream consumers of the pipeline's final output from ever seeing a half-completed batch. Rather than the load stage writing rows directly into the production table as it goes, it writes into a staging table or a new partition, and only atomically swaps/publishes that staging area into the visible production location once the entire batch has completed successfully. If a crash happens mid-batch, the staging area is simply discarded and the batch is restarted (potentially using checkpointing to avoid redoing the earlier stages), but the production table never showed a half-loaded, inconsistent state to anyone reading it. This is the pipeline-level analog of a database transaction's atomicity guarantee, achieved without a literal cross-stage database transaction, by controlling what's externally visible rather than trying to make the underlying writes themselves transactional across independent filters. ## What it looks like when none of this is in place In production, the failure mode when none of these are in place looks like this: - a crash mid-batch leaves the load stage with some rows written and some not; - an operator, under pressure, restarts the whole pipeline from scratch; - because the load stage isn't idempotent, the restart duplicates every row that had already been loaded before the crash, and nobody notices until a downstream report shows inflated totals days later. The more insidious version happens when checkpointing exists but is itself not crash-safe — e.g. the checkpoint is updated in memory or in a non-durable location and is lost along with everything else on crash, giving a false sense of safety. ## A concrete example A concrete real-world instance is how large-scale streaming ETL frameworks like Apache Flink implement fault tolerance: Flink takes distributed, consistent snapshots of all stages' internal state at regular intervals, and on failure restores every stage to its last consistent snapshot and replays the source stream from the corresponding offset — combining checkpointing (to bound reprocessing) with idempotent/deterministic replay (so replaying the same input from the same offset against the restored state produces the same result) to deliver what it calls exactly-once processing semantics, without requiring a literal distributed transaction spanning every filter in the pipeline.