Table Format Concepts

Each file costs fixed overhead: a metadata entry to plan, an object-store request to open, and a footer to parse. With thousands of tiny files that per-file overhead dominates, so the engine spends its time on bookkeeping rather than reading rows.

A file format defines how bytes are laid out inside one immutable file. A table format is a metadata layer above a set of such files that records which files, which schema and which version make up the table right now.

Hive-style partitioning splits a table's files into directories named column=value, such as dt=2024-05-01. A query that filters on that column reads only the matching directories, so it touches a fraction of the table.

Every write produces a new immutable snapshot: a record of the complete set of data files that form the table at that instant. Older snapshots are kept, so a time-travel query reads an older file list instead of the current one.

Each writer stages new files, then asks the catalog to move the table's metadata pointer only if it still holds the value that writer read. This conditional swap succeeds for exactly one writer; the loser re-reads, revalidates its change and retries.

Bin-packing compaction groups many small files into bins that add up to a target size, rewrites each bin as one new file, and atomically swaps the table's file list. It preserves row content and order-agnostic semantics; the target is a band, commonly a few hundred megabytes.

A directory listing shows whatever is on storage at that instant, including half-written job output and files a rewrite is about to delete. A table format instead reads an explicit versioned list of files, so every reader sees one consistent set.

Pruning only works on the partition column itself. A predicate on event_ts says nothing about dt, so the engine lists every partition and discards rows afterwards. Filter dt too, or use a format that derives dt from event_ts.

It writes all new data files first, where nothing references them, then writes new table metadata describing the resulting snapshot, and finally swaps a single pointer to that metadata in one atomic operation. Readers see the old state or the new one, never a partial write.

The table records which partition layout each data file was written under. Changing the layout is a metadata commit that applies to new writes only; existing files keep their old partition values, so nothing is rewritten and no downtime is needed.

The REST catalog defines an HTTP protocol where the server owns commit logic, so any client — Python, Rust, Go — can be thin. A Hive Metastore is a Thrift service with a Hive-shaped model that pushes commit and storage logic into each engine's JVM client library.

Write amplification is the ratio of bytes physically written to bytes logically changed. Immutable files force whole-file rewrites, so changing one row in a 512 MB file writes 512 MB. Larger files, frequent small updates, and repeated compaction of the same data all raise it.

So the planner can eliminate whole data files from a query by reading metadata alone. Without those bounds in the table layer, the engine would have to open every file's footer just to learn it contains nothing relevant.

Hash bucketing maps a high-cardinality key onto a fixed number of partitions. Equality lookups on user_id still skip almost everything, while the table keeps a bounded number of partitions holding properly sized files.

Both writers work optimistically against the snapshot they read, then race to swap the current pointer. The swap is conditional, so exactly one wins; the loser re-reads the new snapshot, checks whether the winner's changes conflict with its own, and retries or fails.

Each catalog keeps its own pointer to a current metadata version, so their conditional swaps never see each other. Commits made through one become invisible or get overwritten from the other's view, and a cleanup job run through one deletes files the other still references.

Compaction only removes files from the table's current file list; the objects stay because older table states still reference them and in-flight readers are still reading them. A separate retention-aware cleanup deletes files that no retained state references, after a safety window.

Bin-packing fixes file count; sorting fixes pruning. Pay for a sort-based rewrite only when queries filter on a column whose values are scattered across every file, so per-file statistics overlap and nothing can be skipped. It costs a full shuffle.

Readers resolve a table's contents from its metadata layer, never from a directory listing. Files copied straight into storage were never registered by a commit, so they are orphans: invisible to queries and eligible for deletion by the table's cleanup job.

Cost moves from scanning to planning: the engine must enumerate and filter millions of partition entries before reading anything, files fall far below target size, and skew grows. Coarsen the time grain and hash the customer key into buckets.

Snapshot expiry drops old versions from the table's history and deletes the data files no surviving snapshot still references. Once that runs, any AS OF query or rollback to a point before the retention horizon has neither a snapshot to resolve nor files to read.

Start from which engines must write, since writers need a maintained client for the same catalog and only one catalog may own a table. Then weigh governance, operability and migration cost; readers can be served by federation, but writers cannot be split.

Write a table layer over the existing files instead of copying them: the conversion reads each file's footer for schema and statistics and commits a file list referencing the current paths. You inherit the old file sizes and layout, so plan compaction as a separate follow-up.

Derive retention from how long a bad write takes to detect, then check it against storage cost, the longest running query, and any deletion deadline. Tier it per table rather than setting one platform-wide number, and pin the few versions that must stay reproducible.

An ORC file splits rows into stripes of roughly 64 MB that can be read independently, a footer at the end lists those stripes with the file's schema and column statistics, and row indexes inside each stripe let a reader skip groups of rows that cannot match a filter.

So that access to a table's files follows the grants held in the catalog. The catalog authorizes the operation and returns temporary credentials scoped to that table's storage path, instead of every engine holding broad bucket credentials that bypass table-level permissions.

Snapshots reference files in the same bucket, catalog and account as the live table, so anything that destroys or loses those objects destroys the history with them. Time travel protects against bad writes inside the retention window, not against losing the storage.

Rewrite history only when old data is queried often enough that the mixed layout measurably hurts, and the rewrite fits your compute budget. Otherwise let the new layout govern new writes and let retention retire the old data.