Lakehouse Table Formats

A catalog maps a table name to the location of that table's current metadata, so every engine agrees on which version is live. A storage path only lists files, including leftovers from failed or in-flight writes that no commit includes.

Each file costs fixed overhead: a metadata entry to plan, an object-store request to open, and a footer to parse. With thousands of tiny files that per-file overhead dominates, so the engine spends its time on bookkeeping rather than reading rows.

A file format defines how bytes are laid out inside one immutable file. A table format is a metadata layer above a set of such files that records which files, which schema and which version make up the table right now.

Hive-style partitioning splits a table's files into directories named column=value, such as dt=2024-05-01. A query that filters on that column reads only the matching directories, so it touches a fraction of the table.

Every write produces a new immutable snapshot: a record of the complete set of data files that form the table at that instant. Older snapshots are kept, so a time-travel query reads an older file list instead of the current one.

It removes snapshot entries older than the retention window from table metadata, then deletes the data files, delete files, manifests and manifest lists that no remaining snapshot references. Anything still reachable from a kept snapshot survives, whatever its age.

Instead of rewriting every data file that contains a matching row, the engine writes small delete files that mark rows as removed and commits those. Writes get much faster; reads must merge deletes at scan time until compaction materialises them.

Nothing is rewritten. Iceberg appends a new partition spec to table metadata and points new writes at it; files already written keep the spec id they were written under, and scan planning handles each spec separately.

Iceberg derives partition values from a transform on a real column, so queries filter that column and still prune. Hive-style tables need a separate derived column that every reader must remember to filter on.

The catalog names the current metadata file. That JSON gives the current snapshot, whose manifest-list Avro file names the manifests. Each manifest lists data files with partition values and column bounds, so the reader prunes manifests then files and scans only survivors.

The record key is a row's identity, so Hudi can update or delete that row instead of only appending. The precombine field breaks ties between duplicates of a key, keeping the record with the higher value.

Copy-on-Write merges the update into the base Parquet file and rewrites it as a new file version. Merge-on-Read appends the update as a log block in the same file group, leaving the merge to read time or to a later compaction.

An instant is one entry on the timeline in a Hudi table's .hoodie directory: an action such as commit, deltacommit, compaction or clean, stamped with a monotonic time and a state of requested, inflight or completed.

Almost always compaction is not keeping up, so log files pile up on each file slice and every snapshot query merges more of them. Check the timeline for delta commits since the last completed compaction, then fix scheduling, parallelism or the writer's compaction service.

Copy-on-Write is the default Hudi table type. Updating a record rewrites the whole Parquet base file that holds it as a new file version, so readers just scan the newest base file per file group with no merging.

A Delta table's _delta_log holds the transaction log: numbered JSON commit files listing actions such as add, remove, metaData and protocol, plus periodic Parquet checkpoints and a _last_checkpoint pointer to the newest one.

OPTIMIZE bin-packs many small Parquet files into fewer large ones (roughly 1 GB by default), committing remove and add actions in a new table version. Row content is unchanged, and the replaced files stay on storage until VACUUM deletes them.

A row-level DELETE on a Delta table without deletion vectors rewrites the affected Parquet file: the commit holds a remove tombstone for the old file path, an add for the freshly written file minus those rows, and a commitInfo entry describing the operation.

VACUUM physically deletes data files that the current version no longer references once they are older than the retention threshold (7 days by default), while _delta_log keeps commit history for 30 days. The old version still resolves but its files are gone, so the read fails.

A matched UPDATE or DELETE must have a single, deterministic outcome per target row. If two source rows match the same target row, Delta cannot decide which wins and aborts the merge. The fix is to deduplicate the source to one row per key.