What happens to existing Iceberg data files when you ALTER TABLE ADD PARTITION FIELD?

## The DDL With the Iceberg Spark SQL extensions the three operations are: ```sql ALTER TABLE prod.db.events ADD PARTITION FIELD region; ALTER TABLE prod.db.events ADD PARTITION FIELD bucket(16, user_id); ALTER TABLE prod.db.events DROP PARTITION FIELD days(event_ts); ALTER TABLE prod.db.events REPLACE PARTITION FIELD days(event_ts) WITH hours(event_ts); ``` Each one completes in the time it takes to write a new table metadata file. None of them reads or writes a single data file. ## What changes in metadata Table metadata holds a list of partition specs, each with a `spec-id`, and a `default-spec-id` naming the one writers must use. Evolving the spec appends a new entry and moves the default; the old entries stay forever, because files written under them still exist. ```json { "default-spec-id": 1, "partition-specs": [ {"spec-id": 0, "fields": [ {"name": "event_ts_day", "transform": "day", "source-id": 3, "field-id": 1000}]}, {"spec-id": 1, "fields": [ {"name": "event_ts_day", "transform": "day", "source-id": 3, "field-id": 1000}, {"name": "region", "transform": "identity", "source-id": 4, "field-id": 1001}]} ] } ``` Two details are worth noticing. The transform names in metadata are singular (`day`, `hour`, `bucket[16]`, `truncate[10]`), while the Spark DDL spells them as functions (`days(...)`, `hours(...)`). And each partition field references its source column by id, not by name, so renaming the source column later does not break the spec. ## What does not change Existing data files are untouched: same paths, same contents, same partition tuple. Existing manifests are untouched too, and every manifest carries the spec id of the entries it holds. Existing snapshots are untouched, so time travel to a snapshot from before the change reads exactly the files and partition values it always did. Queries are untouched, because with hidden partitioning no query ever referenced the partition value in the first place — that is precisely what makes evolution possible. ## How planning copes with several specs A scan may span manifests written under spec 0 and spec 1. Iceberg projects the query's predicates through each spec separately: manifests bound to spec 0 are filtered with spec 0's projection, spec 1's with spec 1's. The practical consequences are worth stating explicitly. - After adding `region`, a query filtering on `region` prunes new files by partition and old files only by their recorded column bounds. Old data is not suddenly organised by region, so it may be read in full. - After `REPLACE PARTITION FIELD days(event_ts) WITH hours(event_ts)`, a one-hour predicate prunes new files to that hour and old files to the containing day, which is the correct and expected outcome. - After `DROP PARTITION FIELD`, files written before the change are still partitioned by the dropped field and still prune on it; files written after are not. ## The v1 wrinkle In v2 tables partition fields carry explicit, unique field ids, so a field can be dropped cleanly. Older v1 tables keep a dropped field in the spec with the `void` transform, which always produces null, so the shape of the partition tuple is preserved. Seeing `void` in a spec is therefore a sign of history, not a mistake. ## When you do want old data reorganised Spec evolution changes the future, not the past. If historical data must be laid out the new way — because the old granularity makes a common query read too much — that is a rewrite of those files under the new spec, done by the table's maintenance procedures rather than by DDL, and it costs full read-and-write IO for the range you rewrite. Deciding whether to rewrite is a cost question: often the honest answer is to evolve the spec, let old partitions age out under the retention policy, and rewrite only the range that is actually hot. ## Failure modes and interview traps The common wrong answer is that Iceberg rewrites data to match the new spec, or that old data becomes unreadable. Neither is true, and the second confusion usually comes from Hive experience where the layout is the contract. A subtler mistake is evolving the spec repeatedly: a table with many specs is legal but harder to reason about, and every scan must consider each one. Another is expecting an immediate query improvement — the win only appears for data written after the change. Finally, remember that adding a partition field increases the number of output files per write, since each commit now produces at least one file per new partition combination; on a small table that can turn one healthy file into dozens of small ones.