Mechanically, how does a single compaction pass deduplicate keys, and what is the offset map?

## Goal of a pass A compaction pass turns a region of the log that may contain many records per key into one where each key appears at most once (its newest value), while preserving original offsets and ordering. ## Pass 1 — build the offset map The cleaner scans the **dirty head** and builds an in-memory **offset map** (implementation: `SkimpyOffsetMap`). For each record it records: ``` map[key] = max(offset seen for key) ``` So after pass 1 the map answers 'what is the highest (most recent) offset for this key in the dirty region?'. It is a *skimpy* (compact) hash map storing key-hash → offset to save memory; it stores the key's MD5 hash rather than the raw key. ## Pass 2 — recopy and filter The cleaner then walks the **cleanable log** (the already-clean tail plus the just-mapped head) record by record. A record is **retained** iff: - its key is in the map AND its offset == the mapped (latest) offset, OR - its offset is beyond what pass 1 mapped (newer than the map's coverage), or its key wasn't in the dirty head (then the tail copy is already the latest and is kept). Everything else — older duplicates of a key whose newer version exists — is **dropped**. Retained records are appended to fresh segment files; **offsets are preserved**, so the new log has the same offset values but with holes where records were removed. This is why consumers of compacted topics see non-contiguous offsets. ## Atomic swap New `.clean` segments are built alongside the originals and then swapped in atomically (rename), so readers never see a partially compacted log. The cleaner checkpoint advances to record how far the tail is now clean. ## Memory bounds — why the map size matters The map lives in a fixed buffer: - `log.cleaner.dedupe.buffer.size` — total bytes (default **128 MB**) shared across all cleaner threads. - `log.cleaner.io.buffer.load.factor` — how full the map may get (default **0.9**) before it's considered full. Each entry costs roughly 24 bytes (16-byte MD5 + 8-byte offset). If a single dirty section contains **more unique keys than fit**, the cleaner can only map a prefix of the head; it compacts up to that point and resumes the rest on the next cycle. Symptoms of an undersized buffer are slow compaction progress and growing logs; the fix is a larger dedupe buffer or more threads (which *splits* the buffer, so there is a balance). ## Edge cases - **Tombstones** (null-value records) are kept through compaction for `delete.retention.ms` so consumers learn of deletions, then dropped. - The **active segment** is never part of a pass, so the newest writes are always present in full. - Records keep their **timestamps and headers**; only superseded duplicates are removed. ## Why two passes You cannot decide whether a record is the latest for its key until you have seen the whole region — hence pass 1 maps the maximum offset per key, and pass 2 uses that knowledge to filter.