Before KRaft, what did Kafka use ZooKeeper for, and what kinds of cluster metadata lived in it?
answer
- ZK = metadata, not message data
- broker registry / topics / ISR / controller / ACLs
- ephemeral znodes for liveness
- controller watches ZK
- separate cluster, second failure domain
basics
~20 sZooKeeper was an external coordination service that stored Kafka's cluster metadata: which brokers are alive, the list of topics and partitions, partition leaders and ISR, controller election, and ACLs/configs. Kafka depended on it to run.
solid answer
~40 sZooKeeper (ZK) was a separate distributed coordination service Kafka required for cluster-wide metadata and consensus. It stored the broker registry (which brokers are alive), topic and partition definitions, partition leadership and the in-sync replica (ISR) set, dynamic configs, ACLs, and delegation tokens. It also handled controller election: the broker that won became the controller and managed leader assignment and failover. Brokers registered ephemeral znodes for liveness, and the controller watched ZK for membership changes. ZK was the durable source of truth for metadata, while the actual record data lived in Kafka log segments on the brokers. KIP-500 later replaced ZK with KRaft because the two-system architecture added operational complexity and a metadata-scaling bottleneck.
go deeper
Know that ZK was an external service holding cluster metadata, and that records were never stored in it.
Be able to enumerate concrete categories: broker registry, topic/partition assignment, leader+ISR, controller, configs/ACLs.
Explain the control-plane vs data-plane split and how ZK liveness/watches drove leader failover.
Frame ZK as a second failure domain and metadata bottleneck that motivated KIP-500, and reason about operational tradeoffs of a two-system design.
## What ZooKeeper is Apache ZooKeeper is a standalone distributed coordination service. It exposes a small filesystem-like tree of nodes called **znodes**, each holding a little data, and it provides strong consistency (linearizable writes) via its own consensus protocol (ZAB). Many distributed systems used it as a 'source of truth' for small but critical shared state. ## Why Kafka needed it A Kafka cluster is many independent broker processes. They must agree on facts that no single broker owns: *which brokers are alive, what topics exist, who leads each partition, which replicas are caught up*. Kafka (pre-2.8, fully removed by default in 4.0) delegated this agreement to ZooKeeper. Kafka stored **metadata** in ZK; the actual message data stayed in Kafka's own log segment files on the brokers' disks. ZK never held your records — only the control-plane state. ## What lived in ZooKeeper - **Broker registry** — `/brokers/ids/<id>`: an ephemeral znode per live broker with its host/port. Disappears when the broker's ZK session ends, giving liveness. - **Topics & partitions** — `/brokers/topics/<topic>`: replica assignment per partition. - **Partition state** — leader and the **ISR** (in-sync replica set) per partition, used during leader election/failover. - **Controller** — `/controller`: an ephemeral znode marking which broker is the elected controller. - **Configs / ACLs / quotas / delegation tokens** — under `/config`, `/kafka-acl`, etc. - **Consumer offsets (very old clients)** — older consumers stored offsets in ZK before the `__consumer_offsets` topic became the default. ## How it was used at runtime The **controller** broker watched ZK znodes (via **watches** — one-shot change notifications). When a broker died, its ephemeral znode vanished, ZK fired a watch, and the controller recomputed leaders for affected partitions and pushed updates to brokers. ## Edge cases / gotchas - ZK was a *separate cluster* to run, secure, and tune — a second failure domain. - Metadata changes funneled through ZK, which became a scaling bottleneck at high partition counts (slow controller failover, slow startup). - A ZK outage froze metadata changes (no new leaders elected) even if brokers were healthy. KIP-500 set the direction to remove ZK; KRaft (the replacement) folds metadata + consensus into Kafka itself.
- Did Kafka store the actual messages/records in ZooKeeper?No. Records live in Kafka's own log segment files on broker disks. ZooKeeper only held control-plane metadata (broker registry, partition leadership/ISR, configs, ACLs, controller election).
- Where did consumer offsets live — ZK or Kafka?Very old consumers committed offsets to ZK, but modern Kafka stores them in the internal compacted topic __consumer_offsets. Offsets-in-ZK was deprecated well before KIP-500.
saying these in an interview costs you the question
- Saying Kafka stored message data in ZooKeeper (it never did — only metadata).
- Claiming ZK was just for the controller; it held broad metadata (topics, ISR, ACLs, configs).
- Confusing ZK with KRaft — ZK is the legacy external system, KRaft is the replacement.