Why is listing a directory of Parquet files an unreliable way to define a table's contents?

## The Hive-style assumption The pre-lakehouse arrangement was simple: a metastore entry mapped a table name to a directory, and, for a partitioned table, each partition value to a subdirectory. The table's contents were *defined by* what was in those directories at read time. That works fine on a quiet HDFS cluster with one batch job a night. It breaks in every other situation, and each break is a favourite interview question. ## Failure 1: no atomicity A writer producing 500 output files creates them one at a time. Storage has no notion of "these 500 appear together". Any query that lists the directory in the middle of that job sees an arbitrary prefix of the output — a partial, wrong answer that returns successfully and silently. There is no error to alert anyone. Table formats fix this by writing all the data files first and then making a single atomic metadata commit that moves the table's current version from the old file set to the new one. Until that instant, readers see the old set; after it, the new set. Never a mixture. ## Failure 2: no isolation from rewrites An overwrite or compaction has to both add new files and delete old ones. With a directory as the source of truth, there is an unavoidable window where both are present — every affected row appears twice — or where the old file has been deleted while a running query still holds its path, so the query dies with a file-not-found error. Some engines papered over this with directory rename tricks, but object stores like S3 have no atomic directory rename; a "rename" is a copy of every object followed by deletes, which is neither atomic nor cheap. In a table format the same rewrite is one commit that adds the new files and removes the old ones together, and older versions keep referencing the old files until a retention-based cleanup job removes them, so in-flight readers finish safely. ## Failure 3: no schema authority Each Parquet file carries its own schema. If two jobs write with slightly different schemas — a renamed column, an int that became a long, a nullable field — the directory happily contains both, and the reader has to reconcile them at query time or fail. There is no place that says what the table's schema *is*. The table layer holds one canonical schema and defines how files written under older schemas map onto it, which is what makes safe column add, drop and rename possible without rewriting data. ## Failure 4: listing cost and consistency On object storage, listing is a paginated API call per prefix, returning a limited number of keys per request. A table with a million files needs thousands of round trips before planning can even begin, and a deeply partitioned table multiplies that by the number of prefixes. This is the notorious slow-planning problem of large Hive tables. Historically S3 listings were also eventually consistent, so a just-written file could be absent from a listing entirely; Amazon has since made them strongly consistent, but the cost problem remains. A table format reads its file list from a small number of metadata objects instead, and that list already carries per-file statistics, so many files are eliminated from the plan without being listed or opened at all. ## Failure 5: no history and no way back With a directory, the previous state of the table is simply gone once files are deleted. There is no rollback for a bad overwrite and no way to reproduce yesterday's report. Versioned metadata gives both for free, because old versions reference file sets that are retained until expiry. ## What replaces it The table layer keeps, for each version, an explicit list of data files with their statistics, and a single pointer to the current version. Commits are optimistic: a writer reads the current version, prepares its files, and then swaps the pointer only if nothing else has committed in the meantime; otherwise it retries. That one indirection — a pointer to a file list, rather than a directory scan — is what turns a pile of Parquet into a table. ## The nuance worth stating Directory listing is not merely slower; it is *incorrect* under concurrency. A candidate who only says "listing is slow on S3" has spotted the performance symptom and missed the correctness argument, which is the real reason table formats exist.