Why was ZooKeeper deprecated and removed from Kafka (KIP-500)? What concrete problems did the ZK dependency cause?

## Background For over a decade Kafka required an external **ZooKeeper** ensemble to hold cluster metadata and run controller election. **KIP-500** ('Replace ZooKeeper with a Self-Managed Metadata Quorum') proposed eliminating that dependency. (The *replacement design* — KRaft, the metadata quorum — is covered by a sibling topic; here we focus on *why ZK had to go*.) ## The concrete pain points ### 1. Two systems to operate ZooKeeper is its own distributed system with its own deployment, configuration, JVM tuning, monitoring, upgrade cadence, and backup story. Operators had to be experts in *both* Kafka and ZK. ZK was also a **separate failure domain**: if the ZK ensemble lost quorum, Kafka couldn't elect leaders or change metadata even though the brokers themselves were fine. More moving parts = more ways to fail and more on-call burden. ### 2. Metadata scalability bottleneck Metadata changes (topic creates, leader changes, ISR updates) were serialized through the controller and ZK. Two costs dominated: - **Controller failover**: a new controller reloaded the *entire* metadata set from ZK to rebuild memory — an O(number of partitions) read storm. At hundreds of thousands of partitions this took a long time, during which leadership changes stalled. - **Broker startup / metadata propagation**: pushing full metadata via ZK and `UpdateMetadata`/`LeaderAndIsr` RPCs didn't scale incrementally. This effectively capped the number of partitions a cluster could safely run. ### 3. Consistency and code complexity The controller kept an in-memory cache of ZK state and synchronized the two. Keeping an external store and the in-memory model consistent created subtle bugs, propagation lag, and a lot of glue code. There was no single, ordered log of metadata changes — just point-in-time znode state plus watches. ### 4. Security and configuration duplication ZK had its own authentication (e.g. SASL) and ACL model, distinct from Kafka's. Securing a cluster meant securing two systems with two models, and misconfigured ZK ACLs were a real risk (anyone with ZK write access could corrupt cluster metadata). ## What replaced it (briefly) KRaft turns metadata itself into an internal, ordered, **Raft-replicated log** managed by dedicated controller nodes. Because the active controller is just the leader of that log, it already holds metadata in memory — failover becomes ~the time to elect a Raft leader instead of an O(partitions) reload. One system, one security model, incremental metadata propagation, far higher partition ceilings. ## Timeline / status - **KIP-500**: the umbrella proposal to remove ZK. - **Kafka 2.8**: early-access KRaft (not production). - **Kafka 3.3**: KRaft marked production-ready. - **Kafka 3.5**: ZooKeeper mode deprecated. - **Kafka 4.0**: ZooKeeper mode removed by default; clusters run KRaft. ## Edge cases / gotchas - 'Removing ZK' isn't *only* about speed — operational simplicity and a single security/config surface were equally motivating. - Migration from ZK to KRaft was non-trivial; KIP-866 provided a dual-write migration path so existing clusters could move without downtime. - ZK wasn't 'bad' — it was a sound choice early on; Kafka simply outgrew the two-system architecture as scale and operational expectations rose.