skip to content

What were the key ZooKeeper znode paths Kafka used (e.g. /brokers, /controller, /admin), and how could you inspect them?

level: middleimportance: nice to knowfreq 35%

answer

  1. /brokers/ids, /brokers/topics
  2. /controller + /controller_epoch
  3. /admin/{delete_topics,reassign_partitions}
  4. /config, /kafka-acl
  5. zookeeper-shell.sh; KRaft = kafka-metadata-shell.sh

basics

~10 s

Kafka kept its metadata under well-known ZK paths: /brokers/ids (live brokers), /brokers/topics (topic/partition assignment), /controller (current controller), /controller_epoch, /admin (admin operations like deletes/reassignments), /config, and /kafka-acl. You inspected them with zookeeper-shell.sh.

solid answer

~30 s

Kafka organized its metadata under a set of well-known znode paths. `/brokers/ids/<id>` (ephemeral) listed live brokers and their endpoints; `/brokers/topics/<topic>` held partition-to-replica assignments and, under `partitions/<n>/state`, the leader+ISR. `/controller` (ephemeral) named the active controller and `/controller_epoch` held the fencing epoch. `/admin` carried transient admin operations such as `/admin/delete_topics` and `/admin/reassign_partitions`. `/config` stored entity (topic/broker/user/client) config overrides, and `/kafka-acl` plus `/kafka-acl-changes` stored ACLs. Older consumers also used `/consumers/<group>/offsets`. You could inspect these with `zookeeper-shell.sh <zkhost>:2181` and run `ls /brokers/ids`, `get /controller`, etc. Note in KRaft none of these exist — metadata lives in Kafka's internal `__cluster_metadata` log and is read via `kafka-metadata-shell.sh`/`kafka-metadata-quorum.sh`.

go deeper

for a junior

Recognize the main paths: /brokers, /controller, /admin, /config.

for a middle

Map each path to its meaning (registry, leader+ISR, controller, admin queues) and know zookeeper-shell.sh.

for a senior

Discuss chroot namespacing, /admin as transient work queues, and the KRaft equivalents (kafka-metadata-shell).

for a principal

Explain the internal-contract nature of the layout and the runbook-migration implications when moving ZK->KRaft.

## The znode layout ZooKeeper exposes a tree; Kafka rooted its state at conventional paths (optionally under a chroot like `/kafka` if configured in `zookeeper.connect`): - **`/brokers/ids/<id>`** — *ephemeral*. One per live broker, JSON with `host`, `port`, listener endpoints, rack. Disappearance = broker down. - **`/brokers/topics/<topic>`** — *persistent*. JSON map of partition -> replica broker list (the assignment). - **`/brokers/topics/<topic>/partitions/<n>/state`** — leader, leader epoch, and **ISR** for that partition. - **`/brokers/seqid`** — helper for broker id auto-generation. - **`/controller`** — *ephemeral*. Holds the broker id of the active controller. Basis of controller election. - **`/controller_epoch`** — *persistent*. Monotonic fencing token bumped each new controller. - **`/admin/delete_topics/<topic>`** — queued topic deletions the controller processes. - **`/admin/reassign_partitions`** — pending partition reassignment plan. - **`/admin/preferred_replica_election`** — request to rebalance to preferred leaders. - **`/config/{topics,brokers,clients,users}/<entity>`** — dynamic config overrides. - **`/kafka-acl/...`** and **`/kafka-acl-changes`** — authorizer ACLs and a change-notification channel. - **`/isr_change_notification`** — brokers signal ISR shrink/expand to the controller. - **`/consumers/<group>/...`** — legacy (old high-level consumer) offsets/ownership; superseded by the `__consumer_offsets` topic. ## Inspecting them Kafka shipped `bin/zookeeper-shell.sh`, a thin wrapper over the ZK CLI: ``` bin/zookeeper-shell.sh localhost:2181 ls /brokers/ids get /brokers/ids/0 get /controller get /controller_epoch ls /admin/delete_topics ``` This was useful for debugging stuck deletes/reassignments or confirming the controller. **Caveat**: directly *writing* to these znodes could corrupt cluster state — they're an internal contract, not a public API. ## After KRaft Under KRaft these paths simply don't exist. Metadata is an internal topic, `__cluster_metadata`, replicated by the controller quorum. You inspect it with `kafka-metadata-shell.sh` (browse the metadata log like a filesystem) and `kafka-metadata-quorum.sh describe` (quorum status). This is a frequent gotcha when porting old runbooks: the `zookeeper-shell.sh` recipes no longer apply. ## Edge cases / gotchas - A **chroot** in `zookeeper.connect` (e.g. `host:2181/kafka`) namespaces all the above under `/kafka`, letting multiple Kafka clusters share one ZK ensemble. - `/admin/*` nodes are transient work queues; their presence usually means an operation is in progress (or stuck). - Manually editing znodes is unsupported and dangerous; use Kafka admin tools instead.

  • What lives under /admin and why is it usually transient?
    /admin holds queued control operations the controller consumes: /admin/delete_topics, /admin/reassign_partitions, /admin/preferred_replica_election. They appear while an operation is pending and are removed once processed, so a lingering node often signals a stuck operation.
  • How do you inspect cluster metadata under KRaft, where these znodes don't exist?
    Use kafka-metadata-shell.sh to browse the internal __cluster_metadata log like a filesystem, and kafka-metadata-quorum.sh describe to view the controller quorum/leader and lag.

saying these in an interview costs you the question

  • Saying you should edit znodes directly to fix issues — that risks corrupting cluster state; use admin tools.
  • Assuming zookeeper-shell.sh recipes work on KRaft clusters — those znodes don't exist there.
  • Confusing /controller (ephemeral, current controller) with /controller_epoch (persistent fencing counter).

context

resolving links…