What does an Apache Iceberg table store under its metadata/ and data/ directories?

## The two halves of an Iceberg table Apache Iceberg is a **table format**: a specification for a set of files that, taken together, describe a table sitting on top of ordinary data files. Physically an Iceberg table is just a location — `s3://lake/db/events/`, `hdfs://.../events/`, or a local directory — containing two subdirectories by default: `data/` and `metadata/`. `data/` is the **data-file layer**. It contains the actual rows, written in a columnar or row format: Parquet by default, with ORC and Avro also supported. These files are **immutable** — Iceberg never edits a data file in place. Changing rows means writing new files (and, from format version 2 onward, small delete files that mark rows as removed). `metadata/` is the **table layer**. It contains three kinds of file, and knowing which is which is the whole point of this question. ## What lives in metadata/ **1. Metadata files (JSON).** Each commit writes a brand-new JSON metadata file describing the entire table state: `format-version`, `table-uuid`, `location`, the list of `schemas` with `current-schema-id`, the list of `partition-specs` with `default-spec-id`, `sort-orders`, table `properties`, the list of `snapshots`, `current-snapshot-id`, a `snapshot-log` and `metadata-log` of past states, and `refs` (named branches and tags). Naming depends on the catalog: a Hadoop catalog writes `v1.metadata.json`, `v2.metadata.json`, … plus a `version-hint.text`; Hive, Glue, JDBC and REST catalogs write names like `00007-3f9c1a2b-….metadata.json` and keep the pointer to the current one in the catalog itself. **2. Manifest lists (Avro).** One per snapshot, named `snap-<snapshot-id>-<attempt>-<uuid>.avro`. Each row describes one manifest file: its path and length, which partition spec it was written with, how many files it added/kept/deleted, and per-partition-field summaries (`contains_null`, `lower_bound`, `upper_bound`) used to skip whole manifests at planning time. **3. Manifest files (Avro).** Named like `4f2c9a1e-….-m0.avro`. Each row describes one data file (or, in v2+, one delete file): `file_path`, `file_format`, the partition tuple, `record_count`, `file_size_in_bytes`, and column statistics such as `value_counts`, `null_value_counts`, `lower_bounds` and `upper_bounds`. ## Why the split matters In a plain Hive-style table, "what files are in this table?" is answered by **listing directories**. That is slow on object storage, is not atomic, and gives no statistics. In Iceberg it is answered by **reading metadata**: catalog pointer → current metadata file → the current snapshot's manifest list → manifests → the exact set of data-file paths. Two consequences follow directly: - **A file in `data/` that no manifest references is not part of the table.** Copying a Parquet file into the directory by hand does nothing; such a file is an *orphan* and a maintenance job can later delete it. - **Deleting a data file from `data/` does not remove rows** — it breaks the table, because manifests still reference a path that no longer exists, and queries fail with a missing-file error. It also means partition directories are a **convenience, not a mechanism**. Iceberg prunes using partition values stored in manifests, not by parsing directory names, which is why the physical layout can change without breaking queries. ## Where the layout can differ The two-directory layout is the default, not a requirement: - `write.data.path` and `write.metadata.path` can point data and metadata at different prefixes or buckets. - `write.object-storage.enabled` inserts a hash component into data-file paths so that writes spread across object-store partitions instead of hammering one key prefix — the resulting paths look nothing like `ts_day=…/`. - Delete files (v2+) live alongside data files and are tracked by delete manifests, distinguished by a `content` field rather than by directory. ## Reading a listing Given a listing, identify files by shape: `*.metadata.json` is the table state; `snap-*.avro` is a manifest list (one per snapshot); other `*-m*.avro` files are manifests; everything under `data/` is data. If you can name those four things and say which one the catalog points at, you have answered the question. ## Common confusions The manifest list is Avro, not Parquet or JSON. There is one metadata JSON **per commit**, not one per table — old ones remain until a maintenance job cleans them. And `metadata/` holds far more than the schema: schema history, partition-spec history, every retained snapshot, and the pointers that make time travel possible.