What are the checkpoint files in a Kafka log directory (recovery-point-offset-checkpoint and replication-offset-checkpoint), and what does each track?

## Where these live Every directory listed in `log.dirs` is self-contained: alongside the partition subdirectories (e.g. `my-topic-0/`) sit several small text checkpoint files that record per-partition offsets for *that directory only*. They are rewritten periodically by background tasks and on clean shutdown. ## recovery-point-offset-checkpoint — the durability boundary Kafka does not fsync every message individually (that would be slow); it lets the OS page cache absorb writes and flushes periodically. The **recovery point** for a partition is the offset up to which data has actually been flushed to disk. This file stores `topic partition recoveryPointOffset` for every partition in the directory. - **On a clean shutdown**, Kafka flushes everything and advances each partition's recovery point to the log's end offset. - **On startup**, Kafka must only recover (re-scan/validate) records *after* the recovery point — because everything up to it is known-good on disk. This is exactly what bounds the recovery work discussed under `num.recovery.threads.per.data.dir`. - After an **unclean shutdown**, the recovery point lags the true log end, so Kafka re-validates the tail of each affected log. ## replication-offset-checkpoint — the high watermark The **high watermark (HW)** of a partition is the highest offset that has been replicated to *all* in-sync replicas (the ISR). Consumers can only read up to the HW — messages above it are not yet 'committed' and could be lost if the leader fails. This file stores the HW per partition so it survives a restart. - Without persisting the HW, a restarting broker couldn't immediately know how far reads are safe. - The HW is distinct from the **log end offset (LEO)**, which is the offset of the next record to be appended on this replica. LEO ≥ HW always. ## The other checkpoint files - **`log-start-offset-checkpoint`** — the **log start offset** per partition: the lowest offset still retained, which moves forward as retention deletes old segments or as records are explicitly deleted (DeleteRecords) or compacted. Restores the start offset across restarts. - **`cleaner-offset-checkpoint`** — for compacted topics, tracks how far the **log cleaner** has compacted each partition, so compaction resumes rather than restarting. ## File format Each file is plaintext: ``` 0 <- version 3 <- number of entries my-topic 0 1500 my-topic 1 1490 orders 0 9921 ``` Line 1 = version, line 2 = entry count, then one `topic partition offset` per partition. ## Why this matters operationally - These files are the **durable metadata** that make startup recovery bounded (recovery point) and reads correct after restart (HW). - They are written **per log directory**, so each disk in a JBOD setup carries the checkpoints for the partitions it hosts. Losing a disk loses only that dir's checkpoints, and the affected replicas re-sync from the leader. - Corruption or deletion of these files forces full recovery of the affected partitions (treated like an unclean state). ## Edge cases - A truncated/zero-length checkpoint after a crash mid-write is tolerated: Kafka falls back to full recovery for those partitions. - The recovery point can equal the LEO (fully flushed) or lag it (pending flush). - HW restoration interacts with leader election: a new leader may truncate followers to the HW (or use leader epochs to truncate precisely). ## Bottom line `recovery-point-offset-checkpoint` = how far data is flushed (bounds recovery). `replication-offset-checkpoint` = the high watermark (bounds safe reads). Plus `log-start-offset-checkpoint` and `cleaner-offset-checkpoint` for retention and compaction progress. All per-directory, all small text files, all critical to fast, correct restarts.