How does Kafka use memory-mapped files (mmap) for its offset and time indexes, and how do consumers find a record by offset?

## What needs indexing A partition segment is a big `.log` file of records in offset order. A consumer asks 'give me records starting at offset N'. Scanning the whole file from the start to find N would be O(file size). Kafka needs a fast offset -> file-position lookup. It also needs timestamp -> offset for `offsetsForTimes()` and time-based retention. ## The index files For each segment Kafka keeps: - **`.index`** — maps a **relative offset** (offset minus the segment base offset, stored as 4 bytes) to a **physical byte position** in the `.log` (4 bytes). 8 bytes per entry. - **`.timeindex`** — maps a **timestamp** (8 bytes) to a **relative offset** (4 bytes). - (`.txnindex` for transactions, separately.) ## Sparse, not dense The indexes are **sparse**: Kafka adds an entry only after roughly every `index.interval.bytes` (default 4096) of appended log data, not for every record. This keeps the index small enough to stay resident in memory and mapped cheaply. The cost is that a lookup lands you *near* the target, and you finish with a short sequential scan of the `.log`. ## Memory mapping (mmap) Kafka opens these index files as **`MappedByteBuffer`** (Java's mmap). `mmap` maps the file's bytes into the process's virtual address space; reads/writes become ordinary memory accesses, and the kernel transparently pages data in/out through the **page cache**. Benefits: - No explicit `read()`/`write()` syscall per index access — just pointer arithmetic into the buffer. - The index lives in page cache, so hot indexes are effectively RAM-resident. - Appending an index entry is a memory write; the OS handles persistence. The **active segment's** index is mapped read-write (entries are appended as the log grows); rolled (inactive) segment indexes are typically mapped read-only and trimmed to their exact size. ## The lookup algorithm (offset -> record) 1. **Select segment**: segment files are named by their base offset (e.g. `...0000000368.log`). Kafka finds the segment whose base offset is the largest <= N (a lookup over a sorted skip-list of segments). 2. **Binary search the index**: within that segment's mapped `.index`, binary-search for the largest indexed (relative) offset <= N; read its byte position P. 3. **Sequential scan**: open the `.log` at position P and scan forward record-by-record until reaching offset N (or the first offset >= N). The scan is short because entries are at most `index.interval.bytes` apart. Time lookups (`.timeindex`) work the same way: binary-search by timestamp to get an offset, then resolve the offset via the offset index. ## Edge cases & gotchas - **Crash recovery**: mmap'd dirty index pages are flushed by the OS lazily. After an unclean shutdown, the tail of an index may be missing or corrupt; on startup Kafka **sanity-checks and rebuilds** indexes from the `.log` if needed (recovery can be slow for large logs; `num.recovery.threads.per.data.dir` helps). - **Index size**: `segment.index.bytes` (default 10 MB) caps the index; a segment can roll early if its index fills. - **Why sparse + scan, not a dense map**: a dense per-record index would bloat memory; sparse + sequential scan exploits the fact that the `.log` is already sequential and page-cache-friendly. - mmap on 32-bit address spaces is limited, but Kafka runs on 64-bit so address space is a non-issue.