An ops team needs at least 14 days of message history retained for replay and DR backfill on a high-throughput topic. Which configs do you set and what trade-offs do you weigh?
answer
- retention.ms=1209600000 = 14 days
- retention.bytes must not cut it short (set -1 or large)
- keep cleanup.policy=delete (compact breaks replay)
- segment.ms so old data actually ages out
- disk blowup → tiered storage KIP-405
basics
~20 sSet retention.ms=1209600000 (14 days) and ensure retention.bytes is large enough (or -1) so size doesn't evict early. Keep cleanup.policy=delete. Watch disk: 14 days of high throughput can be huge, so plan capacity and consider tiered storage.
solid answer
~40 sTo guarantee 14 days of replayable history, set the topic's retention.ms=1209600000 and make sure retention.bytes won't cut it short — either -1 (unlimited) or sized above 14 days of data per partition; whichever limit hits first wins. Keep cleanup.policy=delete (compaction would drop superseded keys and break full replay). Account for segment granularity: retention is enforced per segment, so segment.ms/segment.bytes should be modest enough that old data actually ages out and the active segment isn't holding 14 days. The big trade-off is disk: 14 days × high throughput × replication factor can be enormous, so capacity-plan or use tiered/remote storage (KIP-405) to offload old segments to object storage cheaply. Also consider that longer retention enlarges the replay/reprocessing surface and recovery time. Confirm broker defaults (log.retention.ms) don't override intent.
go deeper
Know that retention.ms sets how long data is kept; 14 days is 1209600000 ms.
Coordinate retention.ms, retention.bytes, and cleanup.policy and size the disk.
Bring in segment rolling, tiered storage, and replay/DR-backfill cost trade-offs.
Standardize retention/tiered-storage policy and capacity model across topics and clusters.
## Goal Keep ≥14 days of message history on a busy topic so you can **replay** (re-consume from an old offset/timestamp) and **backfill DR** (re-mirror history into a recovered cluster). ## Primary configs - **`retention.ms=1209600000`** — 14 days in milliseconds. This is the time bound; data older than this becomes eligible for deletion. (Broker-wide default is `log.retention.ms`/`log.retention.hours`; the per-topic override wins.) - **`retention.bytes`** — the size bound **per partition**. The limit reached **first** (time OR size) triggers deletion. If retention.bytes is too small, partitions get trimmed before 14 days. Set it to `-1` (unlimited, rely on time) or to a value comfortably above 14 days of per-partition data. - **`cleanup.policy=delete`** — must stay `delete`. **Compaction would remove superseded records per key**, so a full chronological replay would have holes — wrong for event replay. ## Segment-level reality Retention deletes whole **segments**, and the **active** segment is never deleted. So: - `segment.ms` / `segment.bytes` should roll segments often enough that 14-day-old data actually closes out and ages — but not so often you create millions of tiny files (broker file-handle and metadata overhead). - Because of segment granularity, effective retained history is slightly more than 14 days (the partial active/most-recent-closed segment), which is fine for a *minimum* guarantee. ## The dominant trade-off: disk Approximate storage = `throughput_bytes_per_sec × 1,209,600 s × replication_factor`, summed across partitions. On a high-throughput topic this can be terabytes. Options: - **Capacity-plan** brokers/volumes for the worst case. - **Tiered Storage (KIP-405)** — offload closed segments to cheap object storage (S3/GCS) while keeping recent data local; `remote.storage.enable=true`, with `local.retention.ms`/`local.retention.bytes` bounding the hot tier and `retention.ms` bounding the total. This decouples long retention from local disk cost — ideal for 14-day replay needs. ## Other considerations - Longer retention **enlarges the replay surface** and can lengthen DR backfill/recovery time and reprocessing cost. - Verify monitoring/alerts on disk usage and that broker defaults don't silently shorten retention. - Lowering retention later is destructive on the next cleanup pass — change carefully. ## Quick reference ``` retention.ms=1209600000 # 14 days retention.bytes=-1 # don't evict by size (or size > 14d/partition) cleanup.policy=delete # full chronological history segment.ms=86400000 # roll daily so old data ages out # optional tiered storage: remote.storage.enable=true local.retention.ms=86400000 # keep ~1 day hot, rest in object store ```
- Why not use cleanup.policy=compact to save space here?Compaction keeps only the latest record per key, deleting superseded events. A full chronological replay would then have gaps, so it breaks event replay. Use delete for time-based history.
- How do you keep 14 days without exploding local disk?Enable Tiered Storage (KIP-405): keep recent data local (local.retention.ms/bytes) and offload older closed segments to object storage, while retention.ms still bounds total history. This decouples long retention from local disk cost.
saying these in an interview costs you the question
- Setting retention.ms but leaving a small retention.bytes that evicts early.
- Using compact for a topic that needs full chronological replay.
- Ignoring replication factor and partition count when sizing disk.
- Forgetting that segment rolling affects when data actually ages out.