In Apache Iceberg, what does hidden partitioning change about how a partitioned table is queried?

## The problem it solves In a Hive-style table a partition is a directory named for a column value, such as `/dt=2026-05-01/`. That `dt` is a real column of the table — usually a string someone computed at write time from the timestamp the data actually carries. Two costs follow. First, every reader must know the convention and filter on `dt`; `WHERE event_ts >= TIMESTAMP '2026-05-01 00:00:00'` scans the whole table, because nothing tells the engine that `dt` was derived from `event_ts`. Second, the derived column can drift from its source: a job that computes `dt` in the wrong time zone, or an overwrite that names the wrong partition literal, silently files rows in a partition that contradicts their own data. ## What Iceberg stores instead An Iceberg table carries a **partition spec**: an ordered list of partition fields, each one a source column id, a transform, a partition field id and a generated name. The transforms defined by the format are `identity`, `year`, `month`, `day`, `hour`, `bucket[N]`, `truncate[W]` and `void`. In Spark SQL they are written as functions over a column: ```sql CREATE TABLE prod.db.events ( id bigint, user_id bigint, event_ts timestamp, region string) USING iceberg PARTITIONED BY (days(event_ts), bucket(16, user_id)); ``` Note what is not there: no `dt` column in the schema, and no partition value in any `INSERT`. When a writer produces a data file it computes the transforms over the rows in that file and records the resulting partition tuple — here a day and a bucket number — in the file's manifest entry, next to the file path, row count and per-column lower and upper bounds. The manifest itself records which partition spec id those values belong to. Iceberg does write readable paths such as `data/event_ts_day=2026-05-01/00000-3-abc.parquet`, but that is a convenience for humans. Scan planning never lists directories; it reads manifests and uses the file paths recorded inside them. ## How a query prunes Because the transform is known, Iceberg can convert a predicate on the source column into a predicate on the stored partition value — the format calls this an inclusive projection. `event_ts >= TIMESTAMP '2026-05-01 00:00:00'` projects through the day transform into a lower bound on the day value, and every manifest entry whose partition tuple fails the test is dropped without the data file being opened. The user wrote a predicate on a timestamp and got day-level pruning without knowing the table was partitioned at all. Which predicates project depends on the transform. Order-preserving transforms (`year`, `month`, `day`, `hour`, `truncate`, `identity`) support ranges as well as equality. `bucket[N]` hashes the value, so it projects equality and `IN` only: a `BETWEEN` on a bucketed column matches every bucket. ## Why "hidden" is the right word The partition value is not a column of the table. You cannot select it, and no query needs to mention it. That is the point: the physical layout is a property the table owns, not a convention the users have memorised. Two consequences matter in interviews. - **A class of user error disappears.** There is no partition literal to get wrong, so rows cannot be written into a partition that disagrees with their own values. - **The layout becomes changeable.** Since no query references the partition value, the spec can be evolved later — swapping day granularity for hour, adding a region field — without touching queries. Files already written keep the spec they were written under and are planned with it. ## What it does not give you Hidden partitioning does not make every filter prune. A predicate on a column that is not a partition source can only skip files through the per-file column bounds in the manifests, which works well on sorted or naturally clustered data and poorly otherwise. It does not choose granularity for you either: hourly partitioning of a low-volume table produces the same small-file mess as any over-partitioned layout, and a large bucket count on a small table multiplies file counts. Finally, a predicate the planner cannot see through — a function wrapped around the source column, or a value the engine does not push into the scan — will not project, and planning falls back to bounds-based filtering. ## Interview framing The strongest answer names the mechanism (spec is source column plus transform; the value is stored per file in the manifest; the predicate is projected through the transform at plan time), gives the concrete contrast with a Hive-style derived column, and closes with the operational payoff: no wrong-partition writes, and a layout you are allowed to change your mind about.