Which statistics does an Iceberg manifest store, and how do they prune a scan?

## What a manifest entry records Every row of an Iceberg manifest is a `manifest_entry`: a `status` (ADDED, EXISTING, DELETED), a `snapshot_id`, sequence numbers, and a nested `data_file` struct. That struct is where the statistics live: - `content` — 0 data file, 1 position deletes, 2 equality deletes (v2+) - `file_path`, `file_format`, `partition` (the tuple of transformed partition values) - `record_count`, `file_size_in_bytes` - `column_sizes` — bytes per column, keyed by field ID - `value_counts` — non-null-plus-null value count per column - `null_value_counts`, `nan_value_counts` - `lower_bounds`, `upper_bounds` — the min and max value per column, stored as binary in the column's type encoding - `split_offsets` — where the file can be split for parallel reads - `sort_order_id`, `key_metadata`, and for equality deletes, `equality_ids` Because these are keyed by **field ID**, not by column name, they survive renames: a column renamed from `amt` to `amount` keeps its statistics. ## Two-stage pruning **Stage one — skip manifests.** The manifest list row for each manifest carries a `partitions` array of field summaries: `contains_null`, `contains_nan`, `lower_bound`, `upper_bound` for each partition field. A query filtering `ts >= '2026-05-01'` on a table partitioned by `days(ts)` discards every manifest whose `ts_day` upper bound is earlier — those manifests are never opened. This is what keeps planning cheap on tables with millions of files. **Stage two — skip files.** Inside a surviving manifest, each entry is tested. The partition tuple is checked against the predicate's partition residual, then the per-column bounds are checked directly: `lower_bounds[amount] > 100` fails `amount < 50`; `null_value_counts[x] == record_count` means the file is all-null for `x`, so `x = 5` cannot match. What survives becomes scan tasks. The crucial property: **no data file is opened to decide this.** Formats like Parquet keep their own footer and row-group statistics, and engines do use them for finer-grained skipping once a file is opened — but that requires a read per file. Iceberg's manifests hoist file-level statistics into the table layer so the plan is built from a handful of metadata reads. ## What limits their usefulness Min/max bounds only prune when data is **clustered** by the filtered column. If every file spans the whole value range of `user_id`, no file's bounds exclude any particular user and every file is read. That is why write sort orders and file rewriting matter: they narrow the bounds. Statistics are collected, not guessed — they describe reality, so scattered data yields useless bounds. Collection is also configurable and bounded on purpose. `write.metadata.metrics.default` controls the mode — `none`, `counts`, `truncate(N)` or `full` — with truncated bounds the default for strings so that manifests do not balloon on wide text columns. Metrics are inferred for a limited number of leading columns; beyond that you must opt in per column with `write.metadata.metrics.column.<column>`. A frequent production surprise is a filter column that is deep in a wide schema and therefore has no bounds recorded at all. Truncated bounds stay **correct but inclusive**: a truncated lower bound is rounded down and an upper bound rounded up, so pruning may keep a file it could have skipped, but never skips a file it should have kept. ## Beyond manifests: Puffin statistics Manifest statistics are per-file and exact for min/max, but they say nothing about cardinality. Iceberg therefore supports **Puffin** files — a binary format for statistics and index blobs — referenced from the metadata file's `statistics` list (each entry naming a `snapshot-id`, `statistics-path` and blob metadata). The common blob is a Theta sketch (`apache-datasketches-theta-v1`) giving approximate distinct counts per column, which cost-based optimizers use for join ordering. These help the optimizer choose a plan; they do not prune files. ## Inspecting them Metadata tables expose everything: `db.tbl.files` shows per-file `record_count`, `lower_bounds` and `upper_bounds`; `db.tbl.manifests` shows `partition_summaries` per manifest. When a query reads more files than expected, querying those two tables tells you immediately whether the bounds were absent, useless, or simply not selective. ## Common mistakes Calling these Parquet row-group statistics — they are file-level and stored in the table layer. Assuming every column has bounds — collection is bounded by default. Assuming bounds guarantee pruning — they only help when data is clustered. And assuming truncation causes wrong results — it causes conservative, not incorrect, pruning.