In a KRaft cluster, how do feature flags and metadata.version replace inter.broker.protocol.version, and how do you upgrade metadata.version during a rolling upgrade?

seniorshould knowfreq 45%

answer

KRaft: no ZK, no inter.broker.protocol.version
metadata.version = the feature flag (KIP-584/778)
roll binaries first, flag stays put
kafka-features.sh upgrade --metadata <V>
controller validates all nodes support target

basics

~20 s

KRaft has no ZooKeeper, so there is no inter.broker.protocol.version to set in server.properties. Instead the cluster has a feature flag named metadata.version that gates new behavior. You roll new binaries first (metadata.version stays put), then explicitly raise it with kafka-features.sh upgrade once all nodes are on the new version.

solid answer

~50 s

Under KRaft (KIP-500), cluster metadata lives in an internal Raft log managed by controllers, not ZooKeeper, so inter.broker.protocol.version is gone. Its replacement is the metadata.version feature flag — one of Kafka's named feature flags (KIP-584/KIP-778) — which encodes the metadata-record schema/behavior level the whole cluster agrees on. The upgrade is still two-phase in spirit: (1) rolling-restart all controllers and brokers onto the new binaries; metadata.version does not change automatically, so the cluster keeps behaving at the old level and stays downgradable. (2) Once every node runs the new binaries, you raise the flag explicitly with kafka-features.sh upgrade --metadata <VERSION> (or --feature metadata.version=...). The controller validates that all nodes support the target level before committing it to the metadata log. You can inspect current/available levels with kafka-features.sh describe. Some metadata.version downgrades are unsafe/unsupported, so check before raising.

Under KRaft (KIP-500), cluster metadata lives in an internal Raft log managed by controllers, not ZooKeeper, so `inter.broker.protocol.version` is gone. Its replacement is the **`metadata.version`** feature flag — one of Kafka's named *feature flags* (KIP-584/KIP-778) — which encodes the metadata-record schema/behavior level the whole cluster agrees on. The upgrade is still two-phase in spirit: (1) rolling-restart all controllers and brokers onto the new binaries; `metadata.version` does not change automatically, so the cluster keeps behaving at the old level and stays downgradable. (2) Once every node runs the new binaries, you raise the flag explicitly with `kafka-features.sh upgrade --metadata <VERSION>` (or `--feature metadata.version=...`). The controller validates that all nodes support the target level before committing it to the metadata log. You can inspect current/available levels with `kafka-features.sh describe`. Some metadata.version downgrades are unsafe/unsupported, so check before raising.

go deeper

for a junior

Know that KRaft uses metadata.version (a feature flag) instead of inter.broker.protocol.version.

for a middle

Know the two steps: roll binaries, then kafka-features.sh upgrade --metadata; and that the flag doesn't auto-advance.

for a senior

Explain controller validation, the downgrade-safety distinction, and the mapping from IBP to a finalized cluster-wide feature flag.

for a principal

Define KRaft upgrade/rollback policy, treat metadata.version bumps as commit points, and distinguish them from ZK-to-KRaft migration.

Background: KRaft vs. ZooKeeper. Classic Kafka stored cluster metadata (topics, partitions, ISR, ACLs, configs) in ZooKeeper, an external coordination service. KRaft (KIP-500) removes ZooKeeper: a quorum of controller nodes maintains the metadata in an internal Raft replicated log (@metadata topic). Brokers replay that log to learn cluster state. This changes how upgrades express 'what behavior level is the cluster on.'

Feature flags (KIP-584, and KIP-778 for KRaft). Kafka has a generic mechanism of named, integer-versioned feature flags. Each flag has a min and max supported version per node and a single finalized level agreed cluster-wide and stored in the metadata log. New cluster-wide behaviors are gated behind a flag level so the cluster only enables them once every node can handle them.

metadata.version is the key flag. The most important feature flag is metadata.version. It encodes the version/schema of the metadata records the controllers write and the behaviors that depend on them — it is the direct KRaft successor to inter.broker.protocol.version. While a cluster runs at a given metadata.version, controllers only write metadata records that nodes at that level understand, guaranteeing mixed-version safety.

Why there's no inter.broker.protocol.version in server.properties anymore. In KRaft you do not (and generally cannot meaningfully) set inter.broker.protocol.version to drive the upgrade; the behavior level is the finalized feature flag in the metadata log, not a static config on each broker. Setting it has no effect on the modern KRaft upgrade path. (metadata.version is bootstrapped at cluster format time via kafka-storage.sh format --release-version and thereafter changed only through the features tooling.)

The rolling upgrade procedure.

Roll the binaries. Rolling-restart every controller and every broker onto the new Kafka version, one node at a time, waiting for health (no under-replicated partitions, controllers caught up) between nodes. metadata.version does not auto-advance — the cluster keeps operating at the old level and remains downgradable to the old binaries.
Finalize the new level. After all nodes are confirmed on the new binaries, raise the flag explicitly:

kafka-features.sh --bootstrap-server host:9092 upgrade --metadata 3.7

(or --feature metadata.version=<N>). The active controller validates that every node advertises support for the requested level; if any node is too old, the upgrade is rejected, preventing you from enabling behavior a node can't handle. Once committed to the metadata log, the new behaviors turn on cluster-wide. 3. Verify with kafka-features.sh describe, which lists each feature's finalized level and the supported range.

Downgrades. Some metadata.version transitions are unsafe to downgrade because newer metadata records can't be represented at the older level. The tooling distinguishes safe vs. unsafe downgrades; kafka-features.sh downgrade may refuse or require --unsafe. Treat raising metadata.version as a commit point, just like the message-format bump in the ZooKeeper world.

Mapping to the old mental model.

inter.broker.protocol.version (static per-broker config) → metadata.version (cluster-wide finalized feature flag).
'phase 2 rolling restart that raises IBP' → 'run kafka-features.sh upgrade once', no restart needed to flip the flag.
The two-phase discipline (deploy new code, then enable new behavior, keeping the mixed window safe and reversible) is identical.

Gotchas. (1) Don't run kafka-features.sh upgrade until all nodes are upgraded — though the controller's validation will reject it anyway. (2) ZK-to-KRaft migration is a distinct, more involved procedure, not a plain rolling upgrade. (3) Other feature flags exist (e.g., for specific subsystems); metadata.version is the umbrella one for general upgrades.

**Background: KRaft vs. ZooKeeper.** Classic Kafka stored cluster metadata (topics, partitions, ISR, ACLs, configs) in **ZooKeeper**, an external coordination service. **KRaft** (KIP-500) removes ZooKeeper: a quorum of **controller** nodes maintains the metadata in an internal **Raft** replicated log (`@metadata` topic). Brokers replay that log to learn cluster state. This changes how upgrades express 'what behavior level is the cluster on.' **Feature flags (KIP-584, and KIP-778 for KRaft).** Kafka has a generic mechanism of named, integer-versioned **feature flags**. Each flag has a min and max supported version per node and a single *finalized* level agreed cluster-wide and stored in the metadata log. New cluster-wide behaviors are gated behind a flag level so the cluster only enables them once every node can handle them. **`metadata.version` is the key flag.** The most important feature flag is **`metadata.version`**. It encodes the version/schema of the metadata records the controllers write and the behaviors that depend on them — it is the direct KRaft successor to `inter.broker.protocol.version`. While a cluster runs at a given `metadata.version`, controllers only write metadata records that nodes at that level understand, guaranteeing mixed-version safety. **Why there's no inter.broker.protocol.version in server.properties anymore.** In KRaft you do **not** (and generally cannot meaningfully) set `inter.broker.protocol.version` to drive the upgrade; the behavior level is the *finalized feature flag in the metadata log*, not a static config on each broker. Setting it has no effect on the modern KRaft upgrade path. (`metadata.version` is bootstrapped at cluster format time via `kafka-storage.sh format --release-version` and thereafter changed only through the features tooling.) **The rolling upgrade procedure.** 1. **Roll the binaries.** Rolling-restart every controller and every broker onto the new Kafka version, one node at a time, waiting for health (no under-replicated partitions, controllers caught up) between nodes. **`metadata.version` does not auto-advance** — the cluster keeps operating at the old level and remains downgradable to the old binaries. 2. **Finalize the new level.** After all nodes are confirmed on the new binaries, raise the flag explicitly: ``` kafka-features.sh --bootstrap-server host:9092 upgrade --metadata 3.7 ``` (or `--feature metadata.version=<N>`). The active controller **validates that every node advertises support** for the requested level; if any node is too old, the upgrade is rejected, preventing you from enabling behavior a node can't handle. Once committed to the metadata log, the new behaviors turn on cluster-wide. 3. **Verify** with `kafka-features.sh describe`, which lists each feature's finalized level and the supported range. **Downgrades.** Some `metadata.version` transitions are **unsafe** to downgrade because newer metadata records can't be represented at the older level. The tooling distinguishes safe vs. unsafe downgrades; `kafka-features.sh downgrade` may refuse or require `--unsafe`. Treat raising `metadata.version` as a commit point, just like the message-format bump in the ZooKeeper world. **Mapping to the old mental model.** - `inter.broker.protocol.version` (static per-broker config) → `metadata.version` (cluster-wide finalized feature flag). - 'phase 2 rolling restart that raises IBP' → 'run kafka-features.sh upgrade once', no restart needed to flip the flag. - The two-phase discipline (deploy new code, *then* enable new behavior, keeping the mixed window safe and reversible) is identical. **Gotchas.** (1) Don't run `kafka-features.sh upgrade` until *all* nodes are upgraded — though the controller's validation will reject it anyway. (2) ZK-to-KRaft migration is a distinct, more involved procedure, not a plain rolling upgrade. (3) Other feature flags exist (e.g., for specific subsystems); `metadata.version` is the umbrella one for general upgrades.

What happens if you run kafka-features.sh upgrade --metadata to a level one broker doesn't yet support?
The active controller validates support across all nodes and rejects the upgrade; the finalized level is not changed, so the cluster keeps operating at the old metadata.version. This guard prevents enabling behavior a node can't handle.
Why might a metadata.version downgrade be refused or require an --unsafe flag?
Newer metadata records written at the higher level may have no representation at the lower level, so downgrading would lose or corrupt metadata. The tooling marks such transitions unsafe and refuses (or demands explicit --unsafe acknowledgment).

saying these in an interview costs you the question

Setting inter.broker.protocol.version in server.properties to drive a KRaft upgrade (it has no effect).
Thinking metadata.version advances automatically when you upgrade binaries.
Believing all metadata.version changes are freely reversible.
Confusing a routine KRaft rolling upgrade with a ZooKeeper-to-KRaft migration.

In a KRaft cluster, how do feature flags and metadata.version replace inter.broker.protocol.version, and how do you upgrade metadata.version during a rolling upgrade?

§ answer

basics

solid answer

§ go deeper

▾level notes4 levels

▾deep dive~3 min

▾common follow-ups2 questions

▾red flags4 flags

§ context

▾related questions4 questions

▾related topics3 topics

answer

go deeper

context