Why does a table format's metadata layer keep its own per-file min/max column statistics?

## Where the statistics live Statistics about data exist at two levels in a lakehouse table, and the distinction is exactly the file-layer/table-layer split. Inside a data file, the file format records information about its own contents so a reader can skip parts of it. Above that, the **table layer** keeps a per-file summary in its metadata: how many rows the file has, how large it is, how many nulls each column holds, and the lower and upper bound of each column's values in that file. The second is a deliberate duplication of information already present in the first, and the duplication is the whole point. ## Why duplicate it Query planning has to decide which files to read before it reads anything. If per-file bounds live only inside the files, the planner has two bad options: open every file's footer — thousands of small, high-latency random reads on object storage, often more expensive than the scan they were meant to avoid — or skip the check and read everything. With the bounds in metadata, planning is a filter over a compact list. A query with `WHERE event_date = '2026-03-01'` compares its predicate to each file's bounds and keeps only files whose range could contain matching rows. On a well-clustered table this drops the file list by orders of magnitude, and the cost of that decision is a handful of metadata reads. ## Two levels of skipping The two levels compose: 1. **Planning-time file pruning** in the table layer removes files entirely — they are never opened, never fetched, never scheduled onto a task. 2. **Read-time skipping** inside each surviving file, using the file format's own internal statistics, avoids decoding chunks of the file that cannot match. Level one saves I/O, request count and scheduling; level two saves decode CPU and bytes transferred. A candidate who conflates them usually cannot explain why planning got slow on a table with a million tiny files: the answer is that level one has to consider every entry in the file list, so metadata volume itself becomes the bottleneck. ## Why bounds sometimes prune nothing Min/max pruning is only as good as the correlation between the predicate column and the physical file layout. Consider a table ingested from a stream, where every micro-batch contains events from every customer. Each file's `customer_id` range then runs from nearly the lowest id to nearly the highest, so a filter on any single customer overlaps every file's range and prunes zero files — while the metadata dutifully reports statistics for all of them. The fix is physical: write or rewrite the data so that values of the filtered column are concentrated in few files, by sorting or clustering during compaction. This is why maintenance jobs typically sort as well as bin-pack. Partitioning does the same thing at a coarser grain, by making the partition value itself part of file placement. ## Practical limits A few caveats worth naming: - **String bounds are usually truncated.** Keeping full values for long text columns would bloat metadata, so bounds are stored as truncated prefixes that still bound the range correctly but prune less precisely. - **Bounds are conservative, never exact.** A file whose range overlaps the predicate may still contain no matching row; pruning avoids false negatives, not false positives. - **Statistics must be produced on write.** Files registered into a table by a path that does not compute statistics carry no useful bounds, and the planner must then assume they might match anything. - **Metadata grows with file count.** Millions of tiny files mean millions of statistics entries to evaluate, which is one more reason the small-files problem hurts even when total data volume is modest. - **Not every column is worth tracking.** Some formats let you limit statistics collection to the columns actually used in filters, trading metadata size against pruning power. ## The interview framing "The file knows about itself; the table knows about all its files." Copying a small summary upward turns per-file knowledge into a planning decision that can be made once, cheaply, for the entire table — and that is the mechanism behind most of the scan-reduction a lakehouse table delivers over a bare directory of files.