When a shared upstream pipeline misses its SLA, should downstream pipelines block or run on stale data?

## Why this is a policy question, not a technical one Both behaviours are trivially implementable — the orchestrator can gate on a freshness assertion, or not. The hard part is that blocking and proceeding fail in opposite directions, and which failure you prefer depends on the consumer. A principal-level answer classifies consumers first and only then talks about mechanism. ## What blocking actually buys and costs **Buys:** internal consistency. Everything downstream reflects one coherent state of the world, and nobody publishes a number derived from data that was not there. It also concentrates the incident: one owner, one root cause, one fix. **Costs:** blast radius and recovery shape. A late upstream that gates fifty downstream pipelines converts a single team's problem into a platform-wide stall. Blocked runs accumulate — one per schedule period — and when the upstream finally lands, all of them become runnable at once and contend for the same compute, which is how a two-hour delay turns into a full day of degraded warehouse performance. Blocking also trains people to disable the gate under pressure, which is worse than not having it. ## What proceeding buys and costs **Buys:** availability and a contained failure. A dashboard that shows yesterday's data with a visible date is often perfectly useful; forcing it to show nothing helps nobody. Teams whose input is genuinely tolerant keep working. **Costs:** silent wrongness, and it compounds. Two shapes matter: - **Mixed vintage.** A pipeline joins a fresh fact table to a dimension that failed to refresh. New keys have no match: depending on the join type they either drop rows or land in an "unknown" bucket. The totals still look plausible, the run is green, and the error is discovered weeks later by a business user. - **Propagated staleness.** A consumer of the consumer has no idea any of this happened. Staleness that is explicit at the first hop becomes invisible at the third unless the marker travels with the data. ## The classification I would actually use - **Block** when the output is published outside the company, feeds a regulatory or financial report, drives an automated action (pricing, sending, paying, provisioning), or joins fresh data to the stale dataset in a way that silently drops or misattributes rows. - **Proceed, marked** when the consumer is an internal analytical surface with a human in the loop, the output is fully recomputed on the next successful run, and the staleness is displayed where the human will see it. - **Proceed, degraded** when a partial answer has real value — serve the fresh portion, exclude and label the stale portion, rather than blending them. Notice that the axis is *what the consumer does with the output*, not *how stale the data is*. Six hours is nothing for a monthly trend and catastrophic for an intraday operational feed. ## Make staleness a first-class property of the data The technique that makes "proceed" defensible is carrying the vintage with the data instead of only in the orchestrator's state. Stamp each output with the effective input timestamps it was built from; expose that in the serving layer; and have consuming surfaces show it. A dashboard headed "as of 02:00 yesterday" is an honest artefact. The same dashboard with no stamp is a trap. This also fixes the third-hop problem: the marker propagates because it is a column, not a run state. ```text orders_raw fresh 06:00 customer_dim STALE yesterday 02:00 <- upstream missed SLA -> daily_orders_mart joins both new customers from today have no dim row rows drop or bucket to 'unknown'; totals look normal ``` ## Who gets paged One page, to the upstream owner, with the list of affected downstream datasets attached. Paging every dependant team produces a storm in which nobody owns the fix and everybody loses trust in the alerting. Dependants need *notification*, not a page — ideally delivered into the surface they use, plus an entry on a status page for the platform. ## The organisational half This decision cannot be made per-incident at 03:00 by whoever is on call, because that person does not know the tolerance of fifty consumers. It has to be declared in advance, by the owner of each consuming dataset, as part of that dataset's contract: *maximum acceptable input age, and what to do when it is exceeded*. The platform's job is to make both behaviours one line of configuration, to enforce the declared choice consistently, and to make the resulting staleness visible everywhere the data is used. ## Second-order effects worth naming When a critical upstream regularly misses its SLA, the right answer is usually neither gate nor proceed — it is to reduce the dependency: keep a last-known-good copy of slowly changing inputs so downstream can build against a defined previous vintage rather than an undefined one; split monolithic pipelines so a late partition blocks only what it feeds; and price the incident honestly so the upstream's reliability work gets funded. Blocking policy is a mitigation for a reliability problem, not a substitute for solving it.