Why does Delta Lake write Parquet checkpoint files into _delta_log?

## The problem checkpoints solve A Delta table's current state is the replay of every action ever committed to `_delta_log`. That is correct but not scalable. A streaming job committing every ten seconds writes 8,640 commit files a day; after a month the log holds a quarter of a million small JSON objects. Replaying them means a quarter of a million round trips to object storage before the first row is read, and object-store LIST calls are paginated and slow. Query planning would dominate query time. ## What a checkpoint is Every `delta.checkpointInterval` commits — 10 by default — the writer that produces that version also writes a checkpoint: a Parquet file named for the version, such as `00000000000000000010.checkpoint.parquet`, sitting in `_delta_log` next to the JSON files. It is not a delta; it is the whole state. Its rows are actions, with a column per action type, and it contains: - every `add` action still live at that version, with `path`, `partitionValues`, `size` and the `stats` used for file skipping; - `remove` tombstones that are still within the tombstone retention window, so a reader of an older version and the cleanup job both still see them; - the current `metaData` (schema, partition columns, properties) and `protocol` (required reader/writer versions and features); - `txn` actions recording the last committed version per streaming application id. Because it is Parquet, it is also columnar and compressed: a reader that only needs paths and partition values does not pay for the statistics columns. ## _last_checkpoint Finding the newest checkpoint by listing `_delta_log` would reintroduce the LIST cost the checkpoint was meant to remove. So Delta maintains `_delta_log/_last_checkpoint`, a small JSON file holding the version of the most recent checkpoint and its size in actions (plus part count for split checkpoints). A reader opens that single object, jumps directly to the named checkpoint, and then reads only the JSON commits with higher version numbers. `_last_checkpoint` is an optimisation, not a source of truth. If it is missing, stale or unreadable, a correct client falls back to listing the log and finding the highest checkpoint itself — slower, but never wrong. Nothing in it can make a reader see incorrect data. ## Very large tables A table with millions of live files produces a checkpoint too large to be one convenient object, so Delta supports splitting a checkpoint into multiple part files, with `_last_checkpoint` recording how many parts to expect. Newer Delta versions add a v2 checkpoint layout, gated by a table feature, where a top-level checkpoint file references sidecar files holding the bulk of the `add` actions — which makes writing a checkpoint on a huge table incremental rather than a full rewrite of the file list every time. ## Checkpoints gate log cleanup Delta keeps commit history for `delta.logRetentionDuration`, 30 days by default. Expired commit files are removed during metadata cleanup, but only when a checkpoint at or after them exists — otherwise deleting a JSON file would break the replay chain and make the table unreadable. This is the direct link between checkpoints and how far back you can time travel: once the commit files for version 40 are cleaned up, `VERSION AS OF 40` fails even though the checkpoint at version 50 is intact, because the exact state at 40 can no longer be reconstructed. ## Operational failure modes The common production symptom is a table whose queries have grown slow to *plan* rather than to scan. Two causes recur. First, checkpointing failing or falling behind — for example a writer that lacks permission to create the checkpoint object, so JSON commits accumulate unbounded and every reader replays thousands of them. Second, an extremely high commit rate: a job committing every few seconds outruns any checkpoint interval in terms of files created, and the honest fix is to commit less often (larger micro-batches) rather than to tune the interval down, since more checkpoints on a large table is itself expensive. The mirror-image mistake is lowering `delta.checkpointInterval` to 1 on a wide table, so every commit rewrites a full state file. On a table with hundreds of thousands of live files that turns a cheap append into an expensive metadata rewrite. ## What checkpoints are not They are not backups, and they are not data — a checkpoint holds file *references*, never rows. They are not required for correctness either: a reader that ignored every checkpoint and replayed all JSON from version 0 would compute exactly the same snapshot, just far more slowly. And they are not a substitute for compacting the data files themselves: a checkpoint makes reading a list of 400,000 tiny files fast, but the query still has to open 400,000 tiny files.