Why does expiring old snapshots in a lakehouse table break time travel to last month?

## Why expiry has to exist History in a table format is free to *create* and expensive to *keep*. Every update, delete and compaction writes replacement files while the old ones stay alive for as long as some snapshot references them. A table compacted nightly can hold several multiples of its live data in files that only history needs. Expiry is the release valve: drop snapshots older than a retention period, then delete every data file that no surviving snapshot references. That second half is what surprises people. Expiry is not a metadata tidy-up; it **deletes data files**. Once the file is gone, the snapshot that referenced it could not be read even if you somehow kept its metadata. ## What actually breaks After expiry with, say, a seven-day retention: - `TIMESTAMP AS OF` or `VERSION AS OF` a point older than seven days fails — there is no such version to resolve. - Rollback to a pre-horizon version is impossible. **The retention window is the recovery window**, so a bad write discovered on day nine is unrecoverable by rollback. - Reproducing a report or a training set pinned to an old version fails, which is why version ids recorded in an experiment tracker are only as durable as the retention policy behind them. The interview payoff line is the causal one: time travel did not "stop working", a maintenance job did exactly what it was configured to do. ## Diagnosing it When someone reports that time travel broke, work in this order: what version or timestamp were they asking for; what does the table's history show as the oldest surviving version; what is the retention setting; and did a maintenance job run between the last successful attempt and the failure. Almost always the oldest surviving version now sits after the requested point, and the change was a schedule or a config, not the table. ## The two adjacent failures **A long reader losing its files.** A query that resolved a snapshot at planning time reads that snapshot's file list for its whole run. If expiry removes that snapshot and deletes files while the scan is still going, the scan dies with a missing-file error partway through — the reader's isolation guarantee is only as good as the files' continued existence. Keep retention comfortably longer than your longest query, and prefer running expiry when heavy readers are not active. **Orphan cleanup deleting live work.** Orphan-file removal deletes objects under the table location that no snapshot references *and* that are older than an age threshold. In-flight writes look exactly like orphans, because they are unreferenced by design until commit. If the threshold is shorter than the longest write, cleanup can delete a writer's staged files, and the resulting commit references objects that no longer exist. This is the one failure here that produces genuine corruption rather than a failed query, and the mitigation is simple: threshold well above the longest possible write, and never run it with an aggressive value "to save space quickly". ## Policy fixes - **Set retention from the detection window, not from a round number.** How long until a bad write is noticed — a failed data-quality check, a Monday-morning dashboard, a monthly close? Retention must exceed that, plus slack. - **Protect what you must reproduce.** Where the format supports pinning or naming specific snapshots so expiry skips them, use it for audit or model-training points. Where it does not, snapshot the state into a separate table — copy the data, do not rely on history. - **Separate the knobs.** Retention (how far back you can travel), the orphan age threshold (safety margin for in-flight writes), and how often maintenance runs are three independent decisions that are often wrongly set together. - **Watch the cost side too.** Long retention plus frequent compaction is the combination that quietly triples storage, because each rewrite's superseded files stay pinned by every snapshot in the window. ## What expiry does not do It does not compact, it does not fix small files, and it does not remove files a surviving snapshot still needs. It also does not make deleted rows unrecoverable immediately: rows removed for a compliance request remain readable through any unexpired snapshot that still references their file, so your retention period is effectively a floor on how long that data survives.