What does log compaction do to a Kafka-style topic, and how does it differ from simple time/size-based retention?

## Why a partition needs a cleanup policy Every topic partition needs a policy for what happens to old data, because disk isn't infinite. Kafka exposes **two cleanup policies**, and they solve different problems. | The 'delete' policy | The 'compact' policy | |---|---| | Segments (the physical log files a partition is split into) are removed once they exceed a configured age (e.g., 7 days) or the partition exceeds a configured size. | A background thread called the **log cleaner** periodically scans each partition and, for every distinct key, keeps only the record with the highest offset (the most recent write) and removes every earlier record sharing that key. | | It doesn't look at the content of the records at all - once a segment ages out, everything in it is gone, key or no key. | The mechanical result is that a compacted topic converges toward holding exactly one record per key: the latest known value. | ## The 'delete' policy - a history of what happened The 'delete' policy is the default. This is the right policy for genuine event streams: a stream of 'OrderPlaced' events where you care about the full sequence of what happened, and you're fine losing events older than your retention window because downstream consumers have already processed them and the business doesn't need year-old raw events sitting in the log. ## The 'compact' policy - current state The 'compact' policy works differently and solves a different problem: representing current state rather than a history of events. Every record in a topic is a **key-value pair**. - If a producer writes a record with a **null value** for a given key, that acts as a **'tombstone'** - after a configurable delay (`delete.retention.ms`), the compactor removes the key entirely, including the tombstone itself, so deletions eventually get cleaned up too. - It behaves less like an event log and more like a durable, append-only-on-the-wire representation of a key-value table, one you can always fully rebuild by reading the topic from the start. ## Why it exists: backing a state store The reason this exists is that stream-processing frameworks (Kafka Streams' **KTable**, **ksqlDB** tables, **Flink's** changelog streams) need a way to durably back an in-memory state store - things like 'current running total per customer' or 'latest known inventory count per SKU' - so that state survives a process crash or gets rebuilt when a task moves to a different machine. - A raw event-retention topic is the **wrong backing store** for this: if you tried to rebuild state by replaying seven days of raw update events, you'd replay stale intermediate values you no longer care about, and eventually old updates would age out and be lost, silently corrupting the rebuilt state. - A compacted **'changelog' topic** solves both problems: replaying it always yields the current state, and it never loses a key's latest value to a time-based retention cutoff, because compaction is keyed, not age-based. ## The trade-off The trade-off is that compaction costs continuous background I/O and disk churn: - The cleaner thread has to periodically rewrite segments to physically remove superseded records, which is CPU and disk work you don't pay for with simple delete-based retention. - It also changes your **data-modeling contract**: every record must carry a meaningful key, because compaction is entirely key-driven; a topic full of null or duplicate-nonsense keys either compacts uselessly or, worse, silently collapses records you actually wanted to keep distinct. - There's also a subtlety around **timing** - compaction runs asynchronously and isn't instantaneous, so a consumer reading a 'compacted' topic can still transiently see multiple records for the same key if it reads between compaction passes; consumers of compacted topics need to be written expecting 'last write for this key wins' semantics, not 'there is exactly one record per key at all times.' ## Failure modes The most common failure mode in production is choosing the wrong policy for the job: 1. Using plain delete-based retention for what is actually a state changelog, so historical intermediate updates silently age out and a task recovering after a crash rebuilds an incomplete or wrong state store. 2. Or, the reverse, applying compaction to what should be a genuine event history, silently collapsing away events a downstream analytics consumer actually needed (e.g., every price change for an item, not just the latest price). 3. Another common issue is forgetting to send tombstones for deleted entities, so compacted topics accumulate keys forever with no way to signal 'this entity no longer exists.' ## Where it shows up A concrete, widely used example is Kafka Streams itself: every KTable and every stateful operator's local **RocksDB** store is backed by a compacted changelog topic under the hood. When a stream-processing task is reassigned to a new machine (say, after a rebalance or a crash), it rebuilds its local state store by replaying that compacted changelog from the beginning - which is fast and correct precisely because compaction guarantees the replay yields only current values, not the full history of every update ever made.