skip to content

How do you provision the KRaft controller quorum and enable migration mode for a KIP-866 migration?

level: middleimportance: must knowfreq 55%

answer

  1. zookeeper.metadata.migration.enable=true on BOTH
  2. format with EXISTING cluster-id (from /cluster/id)
  3. controllers need zookeeper.connect too
  4. metadata.version >= 3.4-IV0
  5. controller.quorum.voters shared list

basics

~10 s

Start a new KRaft controller quorum with process.roles=controller and zookeeper.metadata.migration.enable=true, pointing it at the existing ZooKeeper. It must reuse the cluster's existing cluster.id and connect to the same ZK.

solid answer

~40 s

You stand up a brand-new dedicated KRaft controller quorum (e.g., 3 nodes, process.roles=controller, a controller.quorum.voters list, and controller listeners). On those controllers you set zookeeper.metadata.migration.enable=true and supply the existing zookeeper.connect so the controller can read ZK and join the cluster. Critically, the KRaft controllers must be formatted with the cluster's existing cluster ID — you pass it to kafka-storage.sh format --cluster-id <existing-id> (read from ZooKeeper's /cluster/id or your brokers). The metadata.version feature must be 3.4-IV0 or higher because that is the lowest version supporting migration. Once the quorum is up in migration mode, it reads ZK metadata and waits. The brokers, still in ZK mode, are then reconfigured with zookeeper.metadata.migration.enable=true plus the controller.quorum.voters/controller.listener.names so they register with the new KRaft controllers and the migration can begin.

go deeper

for a junior

Know the master switch is zookeeper.metadata.migration.enable=true.

for a middle

Be able to list controller config, the cluster-id reuse, and the broker reconfig.

for a senior

Explain the version gate (3.4-IV0) and why both sides need the flag and shared voters list.

for a principal

Reason about security listener parity, fleet rollout order, and pre-flight validation of cluster.id/metadata.version.

## The two participants A ZK-to-KRaft migration involves two sets of nodes: 1. **The existing brokers** — currently running in ZooKeeper mode. 2. **A new KRaft controller quorum** — separate processes you provision specifically for the migration. (In ZK mode the controller was just one of the brokers; in KRaft the controller is a distinct role.) ## Step 1 — Format the KRaft controllers with the EXISTING cluster ID KRaft nodes store a formatted metadata log directory. You format it with `kafka-storage.sh format`. The single most important detail: you must pass the cluster's **existing** cluster ID, not generate a new one (`kafka-storage.sh random-uuid`). The existing ID lives in ZooKeeper under `/cluster/id`. Using a fresh UUID would make the KRaft quorum a *different* logical cluster and the brokers would refuse it. ```bash bin/kafka-storage.sh format \ --cluster-id <EXISTING_CLUSTER_ID> \ --config controller.properties ``` ## Step 2 — Controller config for migration mode The controller `*.properties` must contain at least: - `process.roles=controller` - `node.id=<unique>` and `controller.quorum.voters=<id>@host:port,...` - `controller.listener.names=CONTROLLER` and a matching `listeners` - **`zookeeper.metadata.migration.enable=true`** — the master switch that puts the controller in migration mode. - **`zookeeper.connect=<same ZK as the brokers>`** — so the controller can read the existing metadata and coordinate. Without the ZK connection string the migration controller cannot import existing state. ## Step 3 — Metadata/version gate The KRaft `metadata.version` feature flag must be at least **3.4-IV0**, the first metadata version that understands migration records. On the broker side the equivalent gate is `inter.broker.protocol.version` being high enough (>= 3.4). If you try to migrate below this, the controller refuses. ## Step 4 — Reconfigure brokers to talk to the new controllers The brokers stay in ZK mode for now but must be told about the migration and the new controllers. You add to each broker: - `zookeeper.metadata.migration.enable=true` - `controller.quorum.voters=<same list as the controllers>` - `controller.listener.names=CONTROLLER` and a security/listener mapping for it and roll the brokers. They register with the KRaft quorum. At that point the KRaft controller can become active and the dual-write phase starts. ## Edge cases - **Mismatched cluster.id** is the classic failure — double-check it. - The migration controllers should be *new* nodes; do not try to convert an existing broker in place into the controller mid-migration. - Security: the CONTROLLER listener typically needs the same auth (e.g., mTLS/SASL) you use cluster-wide, or controllers and brokers cannot connect.

  • Why must you reuse the existing cluster ID when formatting the KRaft controllers?
    Because the brokers belong to a cluster identified by that ID. A new random UUID makes the controllers a different logical cluster and the brokers reject it; the ID lives in ZooKeeper at /cluster/id.
  • Do the KRaft migration controllers need a zookeeper.connect setting?
    Yes. During migration the controller reads existing metadata from ZooKeeper and dual-writes to it, so it must be configured with the same zookeeper.connect as the brokers.

saying these in an interview costs you the question

  • Generating a new random cluster ID for the KRaft controllers instead of reusing the existing one.
  • Forgetting zookeeper.connect on the migration controllers — they cannot import ZK state without it.
  • Setting the migration flag only on controllers and not on the brokers (both sides need it).
  • Trying to migrate below metadata.version 3.4-IV0.

context

resolving links…