ZooKeeper-to-KRaft Migration

## Background: what ZooKeeper did for Kafka For most of Kafka's history, the cluster relied on **Apache ZooKeeper** — a separate distributed coordination service — to store all *cluster metadata*: the list of topics, their partitions and replica assignments, in-sync replica (ISR) sets, controller election, broker registration, ACLs, and dynamic configs. One broker was elected **controller** and was responsible for reading/writing this metadata in ZooKeeper and propagating it to other brokers. This design had drawbacks: ZooKeeper is a second system to operate and secure, metadata propagation was slow at scale (the controller had to fetch and broadcast full state), and very large clusters hit scaling limits on partition counts and failover time. ## KRaft: Kafka's replacement for ZooKeeper **KRaft** (Kafka Raft) moves metadata management *inside* Kafka. A dedicated **controller quorum** (typically 3 or 5 nodes) runs the **Raft consensus protocol** and stores metadata as an event log in an internal topic, `__cluster_metadata`. Brokers consume this log to learn the cluster state. There is no external ZooKeeper. KRaft was introduced in KIP-500, became production-ready in Kafka 3.3, ZooKeeper mode was deprecated in **3.5**, and ZooKeeper support was **removed in Kafka 4.0**. So migration is mandatory for anyone who wants to upgrade past 3.x. ## What KIP-866 adds KIP-866 is the **dual-write migration** mechanism. The core problem: you cannot simply stop ZooKeeper and start KRaft, because the metadata lives in ZooKeeper and the brokers are running. KIP-866 lets you migrate *online* (no full downtime) by: 1. **Provisioning a KRaft controller quorum** configured in migration mode. 2. The KRaft controller **copies existing metadata out of ZooKeeper** into the KRaft log. 3. Entering a **dual-write phase**: the KRaft controller becomes the active controller but writes every metadata change to *both* KRaft *and* ZooKeeper. This keeps ZK in sync so you can still roll back. 4. **Rolling the brokers** one at a time from ZooKeeper mode into KRaft mode. 5. **Finalizing** the migration, after which ZooKeeper is no longer written to and can be decommissioned. ## Why online matters Kafka often backs critical pipelines; a multi-hour outage to migrate is unacceptable. The dual-write design means producers and consumers keep running throughout, and operators retain a **rollback window** while ZooKeeper is still receiving writes. ## Edge cases / caveats - All brokers and controllers must run a Kafka version that supports migration (3.4+ for the feature, with 3.5/3.6 strongly recommended for stability). - The cluster's **metadata version** (`inter.broker.protocol.version` in ZK mode; the `metadata.version` feature in KRaft) must be high enough. - Migration is a one-way *intent*: once finalized, you cannot return to ZooKeeper.