skip to content

Explain the difference between local.retention.ms/bytes and retention.ms/bytes on a tiered-storage topic, and how to size them.

level: middleimportance: must knowfreq 60%

answer

  1. total = both tiers; local = disk only
  2. default −2 = inherit total; −1 = infinite total
  3. local <= total (hard constraint)
  4. local sizes hot disk; total sizes remote cost
  5. upload-behind blocks local delete

basics

~20 s

retention.ms/bytes is total retention across local + remote storage. local.retention.ms/bytes controls how long/how much data stays on local broker disk before it can be deleted (after upload). Local must be <= total, and local sizes your disk.

solid answer

~50 s

On a tiered topic there are two retention horizons. `retention.ms` / `retention.bytes` define the total time/size a record is kept across both tiers — when exceeded, segments are deleted from remote storage. `local.retention.ms` / `local.retention.bytes` define how long/how much data is retained on the broker's local disk; once a segment is older/over this and has been successfully uploaded, it's eligible for local deletion. So local retention sizes your hot/local disk footprint, while total retention drives remote (cheap) storage cost. Defaults: local.retention.* of -2 means 'inherit the corresponding total retention' (effectively no early local deletion beyond total). You typically set local.retention small (hours/a few GB) so most data lives remotely. Constraint: local.retention must not exceed total retention. Size local to cover your consumers' lag and replay needs plus replication catch-up, since reads inside the local window avoid remote-fetch latency.

go deeper

for a junior

Knows local.retention sizes disk; retention.ms/bytes is the overall lifetime.

for a middle

Explains the upload-then-delete dependency and the local<=total constraint with the -2 default.

for a senior

Sizes local retention against consumer lag, replica catch-up, and ingest rate; reasons about upload-lag risk.

for a principal

Builds a capacity model tying local.retention to disk SKU and total retention to object-store cost across the fleet.

**The two horizons.** A tiered topic has a *total* retention and a *local* retention. - **Total retention** — `retention.ms` (time) and `retention.bytes` (size, per partition). This is the lifetime of data across *both* local disk and remote store. When a segment ages past `retention.ms` or the partition exceeds `retention.bytes`, the data is deleted from wherever it lives, including remote storage. This is the knob that bounds your remote (object-store) cost. - **Local retention** — `local.retention.ms` and `local.retention.bytes`. These govern only the *local broker disk*. A rolled segment becomes eligible for local deletion once (a) it has been **successfully uploaded** to remote storage, and (b) it exceeds `local.retention.ms`/`local.retention.bytes`. This is the knob that bounds your local SSD/disk footprint. **Default sentinel value −2.** `local.retention.ms` and `local.retention.bytes` default to **−2**, meaning 'use the value of the corresponding total retention' (`retention.ms`/`retention.bytes`). With the default, local retention effectively equals total retention, so nothing is deleted locally earlier than total — you must explicitly lower local retention to get a small hot tier. (Note `−1` means infinite for the *total* retention configs.) **Hard constraint.** `local.retention.ms <= retention.ms` and `local.retention.bytes <= retention.bytes`. You cannot keep data locally longer than it's kept overall. **Sizing local retention — what to cover:** 1. **Consumer lag / replay window.** Reads within the local window are served from local disk (fast, no remote fetch). Size local retention to cover normal consumer lag and common reprocessing so you don't pay remote-fetch latency/cost on the hot path. 2. **Replica catch-up.** Followers replicate from the local log; a re-bootstrapping or lagging follower needs the data locally. Too-small local retention can force more remote involvement / re-replication friction. 3. **Disk budget.** local_disk_per_partition ≈ ingest_rate × local.retention.ms (or directly local.retention.bytes), × replication factor across the broker's partitions. **Sizing total retention — what to cover:** business replay/compliance window. Because remote storage is cheap and large, total retention can be days/weeks/months without growing local disk. **Edge cases:** - If uploads fall behind (slow RSM/backend), segments are *not* deleted locally even past local.retention until upload succeeds — so local disk can temporarily exceed the nominal local budget. Monitor upload lag. - `retention.bytes` is per-partition; multiply by partition count and replication factor for cluster sizing. - Setting local.retention very small minimizes disk but increases remote-fetch frequency for any consumer that lags beyond it.

  • What does local.retention.ms = -2 mean, and how does it differ from -1 on retention.ms?
    local.retention.* = −2 means inherit the corresponding total retention (so no earlier local deletion). retention.ms = −1 means infinite/unlimited total retention. They are different sentinels for different configs.
  • If your RSM/backend slows down and uploads lag, what happens to local disk usage relative to local.retention.bytes?
    Segments can't be deleted locally until they're uploaded, so local disk can exceed the local.retention budget. You must monitor remote-copy lag and may need to throttle ingest or scale the backend.

saying these in an interview costs you the question

  • Saying local.retention is the total lifetime of the data — that's retention.ms.
  • Forgetting the constraint local.retention <= total retention.
  • Assuming a segment is deleted locally on schedule even if its upload hasn't completed.
  • Confusing -2 (inherit total, for local.*) with -1 (infinite, for total retention).

context

resolving links…