skip to content

Your nightly SCD Type 2 snapshot load did not run for three days — how do you recover the history?

level: seniorimportance: should knowfreq 40%

answer

  1. what did the source keep, and for how long?
  2. running today's snapshot three times helps nothing
  3. are yesterday's extract files still on disk?
  4. backdating a version re-dates its neighbour
  5. what did the facts join to meanwhile?

basics

~20 s

You can only recover history the source can still replay. With a change log or CDC stream, replay the missed changes in order and insert backdated versions. With only a current-state snapshot, you get one version per changed key — date it from a trustworthy source timestamp, otherwise from the first missed day, and document the gap.

solid answer

~50 s

Start by asking what the source can still tell you, because a snapshot load only ever knew what it saw. If the source keeps a change log, a CDC stream or its own audit history, replay the missed changes in timestamp order and insert one backdated version each — that reconstructs the real history. If the source exposes only current state, the intermediate states are gone: a key that changed twice during the outage yields one version, and the honest choice is to date it from the source's own change timestamp when that is trustworthy, or from the first missed run when it is not, and record the gap in the model's documentation. What you must not do is run the daily job three times against today's snapshot; the first pass writes a version and the next two find nothing, so you get one version dated wrong and a false sense of repair. Then check the facts loaded during the gap, which may point at stale dimension versions.

code

text · 5 lines
text
nightly load runs:   Mon OK   Tue --   Wed --   Thu --   Fri OK
source tier value:   Silver   Gold     Gold     Bronze   Bronze

recoverable from a change log : Silver -> Gold (Tue) -> Bronze (Thu)
recoverable from Friday only  : Silver -> Bronze (one version, start date is a judgment call)

go deeper

for a junior

Understand that a snapshot load only records what it saw when it ran, so a missed night is missing observations — not something a rerun automatically fills in.

for a middle

Be ready to explain why replaying today's snapshot three times produces one version, and what inputs would actually let you reconstruct the missed days.

for a senior

Expect the full incident: triage what the source can replay, apply backdated versions while preserving the window invariants, decide whether the facts loaded during the gap need restating, and add the freshness check that would have caught it.

for a principal

Own the policy: how long sources must retain change history for the warehouse to be recoverable, what an acceptable history gap is, and how gaps get communicated to consumers rather than quietly absorbed.

## First: establish what is actually recoverable A snapshot-based Type 2 load records the source's state at the moments it ran. If it did not run, those states were never observed, and no amount of clever SQL invents them. So the first move is not to write a repair script — it is to find out what the source can still replay: - **A change log, CDC stream, or transaction log with sufficient retention** — full recovery is possible. Replay the missed changes in commit order and insert one version per change, each backdated to its own change time. - **The source keeps its own history or audit table** — usually full recovery too, subject to what that table tracks. - **Only current state** — partial recovery. You will produce at most one new version per changed key, and any intermediate state during the outage is permanently lost. - **Immutable daily extracts already landed in storage** — often overlooked, and often the answer: the files for the missed days may still exist even though the transformation did not run. Process them in date order and the history is exact. Say this out loud in an interview. Candidates who jump straight to "rerun the job" have skipped the only question that determines what is possible. ## Why rerunning the daily job three times does nothing A correctly built load is idempotent (it compares before writing). Point it at today's snapshot three times and the first pass versions the changed keys; the second and third find matching hashes and write nothing. You get exactly one version per changed key, dated at whatever the run supplied — which is the same result as running it once, plus the illusion of having replayed three days. If instead the load is *not* idempotent, three runs give you three identical versions, which is worse. Neither outcome recovers anything. The only version of "just rerun it" that works is replaying three *distinct* inputs — three archived extracts or three CDC windows — with the batch date parameter set to each day in turn, in order. ## Inserting a backdated version is not a plain insert When you do have per-day inputs, inserting a version *between* existing versions is more than an `INSERT`. For each affected key you must: 1. place the new version's start at its real change time; 2. re-end-date the version that previously covered that instant so its window closes where the new one begins; 3. recompute which version is open and carries the current flag — if a later version already exists, the backdated one is not current; 4. re-verify the invariants: no overlapping windows, no gaps, exactly one open row per key. Doing this key by key with ad-hoc updates is where corruption creeps in. If the dimension is small enough, the safer route is often to rebuild it: replay the full ordered change history into a fresh table, validate the invariants, and swap. Rebuilding changes surrogate key assignment, so check what already references those keys before choosing it. ## The fact side of the gap A dimension gap rarely stays contained. Facts loaded during the outage looked up dimension versions that were stale — or, for members created during the gap, found nothing and were assigned a placeholder. After you repair the dimension, those fact rows still point at the wrong surrogate keys and need their lookups redone for the affected window. Whether that restatement is worth doing is a business call about how much the affected reports matter; either way it must be a deliberate decision rather than an oversight. ## Close the loop Two follow-ups separate a repair from an incident response. First, **document the gap** where consumers see it — a period during which a dimension's history is known to be coarser than usual is exactly the kind of caveat an analyst needs before trusting a trend. Second, **fix the detection**, because three days is a long time for a daily job to be silent: add a freshness assertion on the dimension's maximum effective-from date, alert on a missed run rather than on a failed one, and make sure the alert reaches someone. A job that fails loudly for three days is an ops problem; a job that stops silently is a design problem.

  • The source only has current state and a customer changed twice during the outage. What do you record?
    One version. The intermediate state was never observed by anything you can query, so it cannot be recovered — you record the final value and choose its start date honestly, from the source's own change timestamp if trustworthy, otherwise from the first missed run. Then document the coarser interval so consumers do not read the single version as a complete account.
  • When is rebuilding the dimension from scratch better than patching versions in place?
    When you have a full ordered change history and the dimension is small enough to replay cheaply. Rebuilding gives you correct windows and invariants by construction instead of dozens of surgical updates. The blocker is surrogate key assignment: a rebuild renumbers versions, so anything already holding those keys — fact tables, extracts, cached reports — must be remapped or reloaded too.
  • What monitoring would have caught this on day one?
    Freshness assertions rather than failure alerts. Check that the dimension's maximum effective-from is within the expected interval, and alert when a scheduled run does not start at all, not only when one fails. A job that silently stops producing is invisible to failure-based alerting, which is precisely how a one-night gap becomes three days.

saying these in an interview costs you the question

  • Running the daily job three times against today's snapshot
  • Assuming missed intermediate states can be reconstructed from current state
  • Backdating a version without re-end-dating its neighbour
  • Forgetting facts loaded during the gap point at stale versions
  • Leaving the gap undocumented for downstream consumers

context