Why does a Delta Lake time-travel query fail after VACUUM runs?

## Two clocks, not one A Delta table's history has two physical components, and people mentally merge them: 1. **The log.** `_delta_log/00000000000000000040.json` and its neighbours record what each commit added and removed, plus periodic Parquet checkpoints and `_last_checkpoint`. Retention is controlled by `delta.logRetentionDuration`, default **30 days**. Cleanup happens automatically after checkpoints. 2. **The data files.** The Parquet files themselves. Once a commit marks a file `remove`, that file is no longer part of the current table state, but it is still on storage — historical versions still point at it. It becomes eligible for deletion after `delta.deletedFileRetentionDuration`, default **7 days**, and is only actually deleted when someone runs `VACUUM`. Time travel — `SELECT * FROM t VERSION AS OF 40` or `TIMESTAMP AS OF '2026-08-05'` — needs both. The log tells Delta which files made up version 40; the storage layer must still hold them. ## The failure mode With the defaults, a table that is VACUUMed regularly gives you seven days of *usable* time travel and thirty days of *resolvable* versions. A query to a ten-day-old version therefore does not say "that version is gone"; it plans successfully and then fails part-way through the scan with a file-not-found error, usually accompanied by a message noting that files referenced by the transaction log are missing and that this can happen after a manual deletion or an aggressive VACUUM. The same applies to `RESTORE TABLE t TO VERSION AS OF 40`, which needs the same files to be re-addable, and to a Structured Streaming job resuming from an old offset in the table. A second, sharper version of the failure: a query that was **already running** when VACUUM deleted its files. It resolved a file list, then lost the files mid-scan. This is why the retention threshold is not zero by default. ## What VACUUM deletes, exactly `VACUUM tbl [RETAIN n HOURS] [DRY RUN]` deletes, from the table's directory tree: - data files that are **not referenced by the current table version** and whose modification time is older than the retention threshold, and - files under the table directory that the log does not track at all — orphans from failed or abandoned writes. It does **not** touch `_delta_log`; directories beginning with an underscore are skipped, and log cleanup is a separate, automatic mechanism driven by `delta.logRetentionDuration`. `DRY RUN` lists the candidate files without deleting, which is the correct first step on any table you have not vacuumed before. That second bullet is why untracked-file deletion is dangerous with a short window: a long-running writer's not-yet-committed output files look exactly like orphans. ## The retention safety check Delta refuses `RETAIN` values below 168 hours unless you explicitly disable the retention-duration check (`spark.databricks.delta.retentionDurationCheck.enabled = false`). Candidates who casually say "just run `VACUUM RETAIN 0 HOURS` to save storage" are describing the standard way to corrupt a table with concurrent writers, and interviewers notice. The legitimate use of a short retention is a one-off cleanup on a table you know is quiesced. ## Setting the window deliberately Treat the time-travel window as a **contract with consumers**, then set both properties to honour it: ```sql ALTER TABLE analytics.events SET TBLPROPERTIES ( delta.deletedFileRetentionDuration = 'interval 30 days', delta.logRetentionDuration = 'interval 30 days' ); ``` The cost is storage: every rewritten file — and OPTIMIZE and MERGE rewrite a lot — is retained for the whole window. A heavily compacted table with a 30-day window can hold several times its live size. That is the actual tradeoff to reason about: recovery and audit ability against object-storage bill. Also note that changing `deletedFileRetentionDuration` does not resurrect anything already deleted. If a previous VACUUM ran with a 7-day window, the past is gone and only the future window widens. ## Diagnosing it in practice `DESCRIBE HISTORY tbl` shows `VACUUM START` and `VACUUM END` operations with the number of files deleted and the retention actually applied — so you can prove which run removed the files that a failed time-travel query wanted. Pair it with the earliest still-usable version: the oldest commit whose files have not aged past the deleted-file retention. ## The one-line answer Time travel is bounded by file retention, not by log retention, and the two defaults differ. If time travel "stopped working" a week after a cleanup job was introduced, that is not a bug — it is the 7-day default doing exactly what it says.