How would you design a maintenance policy for hundreds of Iceberg tables on one platform?

## Why per-table tuning fails at scale Every Iceberg maintenance knob is a table property or a procedure argument, so the naive approach is a bespoke job per table. At a hundred tables that becomes unowned configuration: nobody knows why one table retains 30 days of snapshots and its neighbour retains one, jobs drift out of sync with the tables they maintain, and a table onboarded last month has no maintenance at all. The design goal is that a **new table is maintained correctly by default**. ## Tier the tables, not the jobs Classify by write pattern, because that is what determines every setting: - **Streaming upsert / CDC.** Merge-on-read for deletes and updates, many small files and delete files per hour. Needs frequent compaction, short snapshot retention (commits are numerous), and a smaller target file size if latency matters. - **Hourly or micro-batch append.** Moderate small-file production. Daily compaction usually suffices. - **Daily batch rebuild.** Few large commits. Copy-on-write, little or no compaction, retention chosen for restatement windows. - **Rarely written reference data.** Almost no maintenance; occasional manifest tidy-up at most. Each tier is a named profile with concrete values for `write.delete.mode` / `write.update.mode` / `write.merge.mode`, `write.target-file-size-bytes`, `write.distribution-mode`, `history.expire.max-snapshot-age-ms`, `history.expire.min-snapshots-to-keep`, and `write.metadata.delete-after-commit.enabled` with `write.metadata.previous-versions-max`. ## Make the table carry its own policy Set the profile as table properties at creation. Then one generic maintenance job iterates the catalog and calls `expire_snapshots` with no arguments — the defaults come from the table's own `history.expire.*` values — and `rewrite_data_files` with tier-appropriate options. Configuration lives next to the table, is visible in `SHOW TBLPROPERTIES`, and survives whoever wrote the job. ## Fix the order of operations 1. **`rewrite_data_files`** — fix layout, and on merge-on-read tables materialise delete files away. Enable `partial-progress.enabled` on large tables so a late failure does not discard hours of work, and use `where` to restrict to recently written partitions rather than rewriting history every night. 2. **`expire_snapshots`** — until the snapshots referencing the pre-compaction files are gone, storage holds both copies. Skipping this is why teams report that compaction increased their bill. 3. **`rewrite_manifests`** — occasionally, on tables with many commits, to keep manifest counts and their partition clustering healthy so planning stays fast. 4. **`remove_orphan_files`** — weekly or monthly, keeping the conservative default age window, always dry-run first, never aimed at a shared location. ## Set retention from requirements Snapshot retention buys three things: rollback after a bad write, time travel for reproducibility or audit, and safety for long-running readers. Ask what each table actually needs. A financial table may need month-end states for years — pin those with tags, which survive expiry, rather than raising the general window for every snapshot. A high-frequency streaming table producing a commit a minute cannot retain 30 days without an enormous snapshot log and file count. Retention is a cost decision, and it should be made once per tier with a documented rationale. ## Budget the compute Compaction reads and rewrites real data; a table compacted daily is read and written daily on top of its query load. Justify it where scan cost is high — the tables driving dashboards and frequent joins — and let cold tables sit fragmented. Track outcomes rather than job success: average file size and file count per partition from the `files` metadata table, delete-file counts on merge-on-read tables, snapshot counts, and total table size before and after expiry. Those numbers say whether the policy is working; a green job status does not. ## Governance and blast radius Maintenance runs with write credentials on every table, and `remove_orphan_files` deletes objects the table has never referenced. Give it a dedicated identity, restrict who can change the safety-relevant arguments, and log what each run deleted. Decide in advance what happens when compaction conflicts with a heavy writer — usually schedule around the write window rather than retry into it. Finally, treat the tier profiles as reviewable defaults: when someone needs an exception, the interesting question is whether the tier is wrong for a whole class of tables rather than just for this one.