How does a scheduler's automatic catchup of missed windows differ from an operator-triggered backfill?

## Two ways the same window gets run again Orchestrators distinguish between windows they *should have run and never did*, and windows an operator *wants run again*. Filling the first set is catchup; requesting the second is a backfill. The runs look identical in the logs — same code, same interval parameters — but who initiates them, how they are bounded, and what they imply about existing data are different. ## Catchup: the scheduler closing its own gaps A scheduled pipeline has a start date and a cadence. The scheduler's job is to ensure a run exists for every window between the start date and now. When you enable a pipeline whose start date is 400 days in the past, there are 400 windows with no run, so it creates 400 runs. Same thing after a pause: unpause a pipeline that was off for a week and the week's windows appear. The consequences are the ones people meet the hard way: - **The stampede.** Hundreds of runs queue instantly. They compete for the same workers, hammer the source system — which may be an operational database that cannot take 200 parallel extracts — and consume real money on a metered warehouse. - **Side effects fire per window.** Every one of those runs executes the whole task graph, including the step that emails a summary or posts to a channel. - **Downstream fan-out.** If downstream pipelines trigger on this one's completion, the stampede propagates. - **Ordering is not guaranteed to be sequential** unless you constrain it, which matters when a window's computation depends on the previous window's output. That is why disabling catchup is such a common default: a newly deployed pipeline should usually start from the next window, with any history loaded through a deliberate, controlled backfill instead. The other common mitigations are setting the start date near the present, and capping how many runs of the pipeline may be active at once. ## Backfill: a bounded, deliberate request A backfill is initiated by a person or an automated job with an explicit range. The typical triggers: - the transformation logic changed and history must be recomputed under the new rules; - a source system was corrected and the old output is now wrong; - a new column or table was added and needs history populated; - a window failed silently and was only noticed later. What distinguishes it operationally is that it is *planned*. You know the range, so you can estimate the cost and duration up front, cap concurrency, route it to isolated compute, order it deliberately, and disable notification steps for the duration. Crucially, a backfill usually targets windows that **already have data**, which is exactly why the task's write must replace rather than append. ## What both require Neither is safe unless the task is interval-scoped and its write replaces the slice it owns. Catchup on an appending task multiplies rows by the number of windows re-run; catchup on a clock-scoped task makes every run resolve to the same output partition and stampede over one target. That is the pairing this whole topic exists to teach: catchup and backfill are only features if idempotence already holds — otherwise they are bulk corruption tools. ## The questions to ask before either happens 1. **How many windows?** Multiply by per-run duration and cost. Three years of hourly windows is over 26,000 runs. 2. **Does window N depend on window N-1?** If a run reads the previous window's output — a running balance, a state table — the runs must be sequential, and parallel catchup will produce garbage. 3. **What side effects fire per run?** Anything leaving the system needs suppressing. 4. **Who consumes the target while this happens?** Rebuilding a published table in place means consumers read a half-rebuilt mart. 5. **Is the source still able to answer for old windows?** Retention, deleted partitions and mutated rows all mean an old window may no longer be reproducible, and replaying it can *overwrite* good historical output with an empty or wrong result — the worst outcome in this whole topic. ## A practical default Deploy pipelines with catchup off and a recent start date, so turning something on is never an incident. Load history explicitly, in controlled chunks, with isolated compute and notifications disabled — and keep that backfill path well-worn and documented, because a team that finds backfilling painful will avoid it, and a team that avoids backfilling ships transformations it is afraid to change.