Retention Policies and Cleanup (Delete)

## What a Kafka topic stores A Kafka **topic** is split into **partitions**; each partition is an append-only **log** persisted on disk. Producers append records to the end (the *head*); consumers read forward by *offset* (a monotonically increasing record position). Because the log only grows, Kafka needs a policy to bound disk usage — that is what `cleanup.policy` controls. ## The two policies - `cleanup.policy=delete` (the default): records are removed once they age out by **time** or **size**. Old data is simply gone. - `cleanup.policy=compact`: Kafka instead keeps the **latest record per key** and removes superseded older values. (Compaction is a sibling topic and not covered here.) You can set both (`delete,compact`) to compact and also cap retention, but the pure-delete case is the focus here. ## How delete decides what to remove Retention is evaluated **per partition** against two limits: - **Time**: `log.retention.ms` (or `.minutes`/`.hours`, with `.ms` winning if several are set). Default is 7 days (`168` hours). A record is eligible once it is older than this. - **Size**: `retention.bytes` — the maximum bytes to keep per partition. Default `-1` (unlimited). When the partition exceeds this, oldest data is removed until it fits. If **both** are set, a segment is removed when **either** limit says it should be — i.e. the more aggressive limit wins. ## Granularity: segments, not records Kafka does not delete individual records. Each partition log is divided into **segment files** (default `segment.bytes=1 GiB`, or rolled by `segment.ms`). A background **retention thread** scans closed (non-active) segments and deletes an entire segment only once **all** records in it are past the limit (based on the segment's largest timestamp for time retention). The **active segment** — the one currently being appended to — is never deleted. This means data can live somewhat past its nominal retention until its segment closes and is fully expired. ## When to use it Use `delete` for event streams, click/telemetry data, application logs, and metrics — anything where old records lose value and you just want a rolling window. Use `compact` when the topic represents the *current state per key* (e.g. a database changelog) that must survive indefinitely. ## Edge cases - Lowering retention does **not** instantly purge data; it takes effect when the retention thread next runs and segments roll/close. - Setting `retention.ms=0` (or a tiny value) effectively makes records eligible almost immediately, but the active segment still survives until it rolls. - Consumer offsets are independent of retention — if data is deleted before a slow consumer reads it, that consumer gets an `OffsetOutOfRange` and must reset.