Why does Iceberg's remove_orphan_files procedure default to a three-day age threshold?

## What an orphan file is An Iceberg table is defined entirely by its metadata: the current `metadata.json`, the snapshots it lists, their manifest lists, and the manifests naming data and delete files. Anything sitting under the table's location that no metadata version references is invisible to the table. That happens whenever a writer uploads files and then fails before committing — a killed Spark job, an OOM executor, a task that was speculatively retried. The files are paid for by the storage bill and read by nothing. `expire_snapshots` cannot clean them up, because it only walks metadata and an orphan is by definition absent from it. `remove_orphan_files` closes that gap by working from the other direction. ## How it decides what to delete ```sql CALL prod.system.remove_orphan_files( table => 'db.events', older_than => TIMESTAMP '2026-08-18 00:00:00', dry_run => true ); ``` It lists every object under the table location, builds the set of all files referenced by table metadata, subtracts one from the other, and deletes what remains — filtered by modification time older than `older_than`. That is a genuinely dangerous operation, because the listing has no way to tell "garbage from a job that died last week" from "data file a job uploaded ninety seconds ago and is about to commit". Both are unreferenced right now. ## Why the window is three days The default `older_than` is three days ago, and it exists solely to stay clear of in-flight writes. The window should exceed the wall-clock duration of your longest write, including retries — a big backfill that runs six hours, a streaming job whose commit stalls, a compaction of a huge table. Teams under storage pressure often shrink it to an hour to reclaim faster; that is precisely how a maintenance job deletes files a concurrent writer is about to reference, producing a table whose manifests point at objects that no longer exist and a `NotFoundException` on the next read. The safest posture is to keep the default and schedule the procedure infrequently — it is a periodic hygiene job, not a per-commit one. ## The other ways it goes wrong **Path form mismatches.** If metadata records `s3://bucket/warehouse/db/events/...` while the listing produces `s3a://bucket/...`, a naive comparison classifies every live file as an orphan. Iceberg guards this with `prefix_mismatch_mode`, whose erroring default aborts rather than delete, plus `equal_schemes` and `equal_authorities` to declare which prefixes are equivalent. Never resolve a mismatch by switching the mode to delete without first confirming the two forms really name the same objects. **A shared or wrong location.** The `location` argument aims the procedure at a specific path. Pointing it above the table, or at a directory two tables share, means the procedure lists files it has no metadata for and deletes another table's data. Its safety model assumes the location belongs to exactly this table. **Registered-but-external files.** Files added with `add_files` from outside the table directory, or metadata living in a separate location, break the assumption that everything under the path is either referenced or garbage. **Cost.** On a table with millions of objects, the full listing is expensive and slow against object storage. That is another reason to run it on a long cadence — weekly or monthly — rather than daily. ## Operating it safely Always run `dry_run => true` first and inspect the returned paths: they should look like orphans (odd task-attempt names, partial partitions), not a coherent slice of the table. Keep the default age window unless you have measured your longest write and deliberately chosen a larger one. Use `max_concurrent_deletes` to bound the load on the object store. And schedule it after compaction and expiry, not before, so that it is not competing with jobs that are actively writing. ## Where it sits in the maintenance sequence The usual ordering is `rewrite_data_files` to fix layout, then `expire_snapshots` to release the files those rewrites replaced, then — much less often — `remove_orphan_files` to sweep what was never committed at all. The first two are metadata-driven and safe by construction; the third is the one that requires care, because it is the only one that deletes a file the table has never known about.