skip to content

How do you set and inspect retention on a single topic without changing the whole broker, and how do broker-level vs topic-level configs interact?

level: middleimportance: should knowfreq 48%

answer

  1. topic override beats broker default
  2. kafka-configs.sh --alter --add-config
  3. --delete-config reverts to default
  4. log. prefix on broker keys only
  5. --describe --all shows source

basics

~10 s

Use kafka-configs.sh (or kafka-topics.sh --config) to set per-topic overrides like retention.ms and retention.bytes. A topic-level config always overrides the broker default (log.retention.*). Inspect with --describe.

solid answer

~40 s

Per-topic retention is set with topic-level config keys (retention.ms, retention.bytes) which override the broker defaults (log.retention.ms/.minutes/.hours, log.retention.bytes). Set them at create time with kafka-topics.sh --create --config retention.ms=..., or alter live with kafka-configs.sh --alter --entity-type topics --entity-name <topic> --add-config retention.ms=86400000. Remove an override with --delete-config retention.ms to fall back to the broker default. Inspect effective and overridden values with kafka-configs.sh --describe --entity-type topics --entity-name <topic> (and --all to see defaults too), or via the AdminClient. Precedence: an explicit topic override wins; if absent, the broker static/dynamic default applies. Note the per-topic keys are unit-suffixed (retention.ms, segment.ms) while broker keys carry the log. prefix (log.retention.ms, log.segment.bytes) — a common naming gotcha.

go deeper

for a junior

Know retention can be set per topic and inspected with --describe.

for a middle

Use kafka-configs.sh to add/delete overrides and explain topic-vs-broker precedence and the log. prefix asymmetry.

for a senior

Explain dynamic config propagation, the describe sources, and that altering doesn't instantly free disk.

for a principal

Standardize topic-config governance (defaults, overrides, automation via AdminClient) across teams and environments.

## Two config scopes Kafka retention can be set at two levels: - **Broker level (defaults)**: `log.retention.ms` / `.minutes` / `.hours`, `log.retention.bytes`, `log.segment.bytes`, `log.segment.ms`. These apply to every topic that doesn't override them. - **Topic level (overrides)**: `retention.ms`, `retention.bytes`, `segment.bytes`, `segment.ms`, `cleanup.policy`. These apply to one topic only. Note the **naming asymmetry**: broker keys use the `log.` prefix; the per-topic equivalents drop it. This trips people up constantly. ## Precedence An explicit **topic-level override always wins** over the broker default. If a topic has no override for a key, the broker's effective default (static config file or dynamically set) applies. ## Setting at create time ``` kafka-topics.sh --bootstrap-server b:9092 --create --topic events \ --partitions 6 --replication-factor 3 \ --config retention.ms=86400000 --config retention.bytes=10737418240 ``` ## Altering a live topic Use `kafka-configs.sh` (the modern way; `kafka-topics.sh --alter --config` is deprecated for configs): ``` kafka-configs.sh --bootstrap-server b:9092 --alter \ --entity-type topics --entity-name events \ --add-config retention.ms=3600000 ``` The change is dynamic — no broker restart — and takes effect on the next retention check; it does not instantly purge existing data. ## Removing an override (revert to broker default) ``` kafka-configs.sh --bootstrap-server b:9092 --alter \ --entity-type topics --entity-name events \ --delete-config retention.ms ``` ## Inspecting ``` kafka-configs.sh --bootstrap-server b:9092 --describe \ --entity-type topics --entity-name events ``` Add `--all` to also list inherited defaults and their source (DEFAULT_CONFIG, STATIC_BROKER_CONFIG, DYNAMIC_TOPIC_CONFIG). Programmatically, `AdminClient.describeConfigs` / `incrementalAlterConfigs` do the same. ## Gotchas - Changing retention does not reclaim disk immediately — the retention thread acts on the next pass and only on closed segments. - To force quick deletion for cleanup, you can temporarily set a tiny `retention.ms`, wait for the checker, then restore — but the active segment still survives until it rolls. - `retention.bytes` is per partition; multiply by partition count. - Setting `cleanup.policy` is also a per-topic override here (`delete` vs `compact`).

  • After lowering retention.ms on a live topic, why isn't disk freed immediately?
    The retention thread runs periodically (log.retention.check.interval.ms) and only deletes closed segments fully past the limit; the active segment must also roll first.
  • What's the difference between log.retention.ms and retention.ms?
    log.retention.ms is the broker-wide default; retention.ms is the per-topic override (no log. prefix) that takes precedence for that topic.

saying these in an interview costs you the question

  • Saying you must restart brokers to change retention (it's a dynamic config).
  • Using log.retention.ms as a per-topic key (wrong scope/prefix).
  • Expecting --alter to purge data instantly.
  • Forgetting --delete-config to revert to the broker default.

context

resolving links…