How can a table format change a table's partitioning without rewriting existing data?

## Why this was impossible before In a Hive-style table the directory tree *is* the partitioning. `dt=2024-05-01/` means "partitioned by dt" because there is nowhere else for that fact to live. So changing the partitioning means moving every file into a different tree — in practice: create a second table with the new layout, rewrite all history into it (hours to days of compute on a large table), repoint every downstream job and dashboard, and hope nothing was missed. Teams therefore did not change partitioning. They lived with the wrong key, which is exactly why "your partition key turned out to be wrong, now what?" was such a good interview question for so long. ## The change that makes it possible When the table format tracks data files in its own metadata, two facts become storable that a directory tree cannot hold: 1. The table's partitioning is a **declared specification** — an ordered list of (source column, transform) pairs — held in metadata and versioned. 2. Each data file's entry records the **partition values it holds and which version of the specification produced them**. Once those are true, the layout is no longer implied by the path. It is data, and data can be changed with a commit. ## What actually happens on the change Declaring a new layout writes a new specification version and makes it the table's current one. That is the whole operation: no files are read, none are written, none are moved. The commit is as cheap as any other metadata commit and is atomic like any other, so concurrent readers see either the old layout or the new one, never a half-applied state. From that instant: - **Writers** use the current specification. New files are partitioned the new way and tagged with the new version. - **Existing files** are untouched. They remain tagged with the version that produced them, and their recorded partition values remain accurate descriptions of their contents. ## What a reader has to do Because one table now contains files described by two different specifications, scan planning is no longer a single predicate evaluation. The planner groups files by the specification they were written under and prunes each group with the rules that apply to it: ``` spec 0 (month(event_ts)) -> prune by month spec 1 (day(event_ts), bucket(64, user_id)) -> prune by day and bucket ``` Both results are unioned. Correctness is never in question — every file is pruned by criteria that genuinely describe it — but the *quality* of pruning differs by era. ## The consequence people forget **Old data does not gain the new pruning.** If you add `bucket(64, user_id)` today, a query for one user still reads all of the pre-change history at whatever granularity the old specification provided, and only the post-change files narrow to one bucket. The improvement arrives gradually as new data accumulates, and arrives immediately only for queries that already bound themselves to recent time. The symmetric case matters too: if you *coarsen* the layout (hour to day), old finely-partitioned files still prune by hour and are fine; it is the new files that are coarser. ## Evolution is not free of judgment - **Plans get more complex.** Every additional live specification is another planning group. A table that has been re-specified five times carries five sets of pruning rules; two or three eras is normal, a dozen is a smell. - **Downstream assumptions may break.** Jobs that hardcoded a partition column name in a filter, or that enumerate directory paths directly instead of querying the table, will not follow the change. Query the table, never the paths. - **Sorting and clustering are a separate axis.** Changing the partition specification does not reorder rows within files; if pruning inside a partition matters, that is a different lever. ## Format differences worth stating This capability is not uniform. Apache Iceberg supports changing the partition specification in place with exactly the semantics above. Delta Lake does not evolve partition columns of an existing table — they are fixed when the table is created, and the evolvable alternative it offers is liquid clustering, which is a clustering mechanism rather than directory partitioning. Hudi's partitioning is determined by write configuration and is likewise not something you re-declare over existing data. In an interview, say which behaviour you are describing rather than asserting that "table formats support partition evolution" as a universal. ## The follow-up you should expect "So do you go back and rewrite the old data under the new layout?" That is a cost-benefit decision, not a requirement — and the honest default is no, unless the old data is queried often enough to justify re-reading and re-writing all of it.