skip to content

Explain per-topic config overrides versus broker defaults: how does precedence work for something like retention.ms?

level: middleimportance: must knowfreq 65%

answer

  1. broker log.retention.ms vs topic retention.ms
  2. override wins, else inherit
  3. delete override to revert (not set-to-default)
  4. DYNAMIC_TOPIC_CONFIG vs DEFAULT_CONFIG
  5. kafka-configs.sh --describe shows source

basics

~10 s

Brokers have cluster-wide defaults (e.g. log.retention.ms). A topic can override the equivalent topic-level config (retention.ms). If a topic-level override exists it wins; otherwise the broker default applies.

solid answer

~40 s

Many settings exist as both a broker-level default and a topic-level override, often with different names — e.g. broker log.retention.ms vs topic retention.ms, broker log.segment.bytes vs topic segment.bytes. Precedence: a per-topic override always takes priority over the broker default for that topic. If no topic override is set, the topic inherits the broker default at the time of evaluation. You set topic overrides at creation (--config retention.ms=...) or later via kafka-configs.sh --alter --entity-type topics --add-config, or AdminClient.incrementalAlterConfigs. To revert a topic to the broker default, you DELETE the override (not set it to the default value), so the topic tracks the broker default again. kafka-configs.sh --describe shows which configs are DYNAMIC_TOPIC_CONFIG (overridden) vs DEFAULT_CONFIG (inherited).

go deeper

for a junior

Know topic overrides beat broker defaults and where retention is set.

for a middle

Explain the broker/topic name mapping, how to set overrides, and the delete-to-revert subtlety.

for a senior

Use --describe source labels to audit drift and reason about why pinning vs inheriting matters operationally.

for a principal

Define config governance: which settings are centrally managed broker defaults vs per-topic exceptions, and how to detect override drift.

## Two levels of configuration Kafka configs that govern log behavior usually exist at **two levels**: - **Broker default** — cluster/broker-wide, e.g. `log.retention.ms`, `log.segment.bytes`, `log.cleanup.policy`. These set the baseline for every topic on the broker. - **Topic override** — per-topic, e.g. `retention.ms`, `segment.bytes`, `cleanup.policy`. These names usually drop the `log.` prefix. ## Precedence rule For a given topic, **a per-topic override wins over the broker default**. If the topic has no override for that config, it inherits the broker default. There is no merging — it's a simple 'override if present, else inherit'. Example: broker has `log.retention.ms=604800000` (7 days). Topic `audit` sets `retention.ms=2592000000` (30 days). Records in `audit` are kept 30 days; every other topic without an override keeps 7 days. ## Setting an override At creation: ``` kafka-topics.sh --create --topic audit --partitions 3 --replication-factor 3 \ --config retention.ms=2592000000 ``` After creation: ``` kafka-configs.sh --bootstrap-server localhost:9092 --alter \ --entity-type topics --entity-name audit \ --add-config retention.ms=2592000000 ``` Programmatically use `AdminClient.incrementalAlterConfigs` with an `AlterConfigOp` of type SET. ## Reverting to the broker default Crucial subtlety: to make a topic follow the broker default again, you must **DELETE** the override, not set it to the current default value. Setting it explicitly pins it; deleting it lets the topic track the broker default if that default later changes. ``` kafka-configs.sh --alter --entity-type topics --entity-name audit \ --delete-config retention.ms ``` ## Inspecting source `kafka-configs.sh --describe --entity-type topics --entity-name audit` shows each config with its **source**: - `DYNAMIC_TOPIC_CONFIG` — an explicit per-topic override. - `DEFAULT_CONFIG` / `STATIC_BROKER_CONFIG` — inherited from the broker. This lets you see exactly which values are overridden vs inherited. ## Edge cases - A topic override set to the same value as the broker default is still recorded as an override (it won't track future broker default changes). - Some broker configs have no topic-level equivalent and cannot be overridden per topic.

  • How do you make an overridden topic config go back to following the broker default?
    Delete the override with --delete-config; setting it to the default value would pin it and stop it tracking future broker default changes.
  • Why do the broker and topic config names differ (log.retention.ms vs retention.ms)?
    Broker-level configs use the log.* prefix as cluster defaults; the topic-level override uses the unprefixed name. They map to the same behavior but live at different scopes.

saying these in an interview costs you the question

  • Saying you revert a topic to default by setting it to the default value (you must delete the override)
  • Claiming broker defaults override topic settings
  • Confusing log.retention.ms (broker) with retention.ms (topic) as the same config name
  • Thinking changing the broker default retroactively changes topics that have explicit overrides

context

resolving links…