skip to content

What is a Kafka broker, and what role does broker.id play in a cluster?

level: juniorimportance: must knowfreq 75%

answer

  1. one server = one broker
  2. broker.id = unique integer
  3. duplicate id -> startup fails
  4. stable id reclaims partitions
  5. KRaft uses node.id

basics

~10 s

A broker is a single Kafka server that stores topic partition data and serves produce/fetch requests. broker.id is its unique numeric identifier within the cluster; no two brokers may share the same id.

solid answer

~40 s

A Kafka broker is one server process in a Kafka cluster. It persists messages to disk as partition log segments, hosts leader and follower replicas of partitions, and handles client produce and fetch requests over the network. A cluster is just several brokers cooperating. Each broker has a unique broker.id (an integer) that identifies it in cluster metadata; replica assignments, leadership, and ISR membership all reference brokers by this id. If broker.id is unset, modern Kafka can auto-generate one (broker.id.generation.enable), but in production you normally pin it so a restarting broker reclaims its identity and its data. Duplicate ids cause startup failure. In KRaft mode the equivalent is node.id, which must be unique across both broker and controller roles.

go deeper

for a junior

Know a broker is one Kafka server and broker.id is its unique number.

for a middle

Explain why id stability matters for reclaiming partitions and the auto-generation behavior.

for a senior

Discuss KRaft node.id, reserved.broker.max.id, and the re-replication consequences of id changes.

for a principal

Reason about identity/storage coupling in capacity planning, node replacement runbooks, and KRaft migration.

## What a broker is Apache Kafka is a distributed log: producers append messages, consumers read them, and the data is stored durably on disk. A **broker** is a single Kafka **server** (one JVM process, usually one machine/container). A **cluster** is a set of brokers that coordinate to store and serve data together. Each broker is responsible for a subset of the data. Topics are split into **partitions** (ordered append-only logs), and each partition is stored as a sequence of **log segment** files on a broker's disk. A partition is **replicated** across several brokers for fault tolerance: one broker holds the **leader** replica (handles reads/writes) and others hold **follower** replicas (copy the leader). ## broker.id Every broker needs a stable identity so the cluster can say things like "partition foo-0's leader is broker 3, followers are brokers 1 and 5." That identity is **broker.id**, a non-negative integer configured in `server.properties`. Key facts: - **Uniqueness is mandatory.** Two brokers with the same id cannot both join; the second fails to start. - **Stability matters.** Replica assignments persist this id. If a broker restarts with the *same* id and the *same* log directories, it reclaims its partitions and resumes as a replica. If the id changes, the cluster sees a brand-new empty broker and the old replicas are considered offline. - **Auto-generation.** With `broker.id` left at the default `-1` and `broker.id.generation.enable=true`, the broker generates an id (starting at `reserved.broker.max.id + 1`). Convenient for ephemeral dev clusters, risky for stateful production nodes. - **KRaft note.** In KRaft mode (ZooKeeper-free), the unifying identifier is **node.id**; for a node that is purely a broker it serves the same purpose as broker.id, but node.id must be unique across the *combined* broker+controller id space. ## Edge cases - Reusing a dead broker's id on fresh storage makes the cluster try to use it as a replica with no data, triggering re-replication. - `reserved.broker.max.id` (default 1000) bounds manually-assigned ids when auto-generation is on; manual ids above it conflict with the generator's range.

  • What happens if you start two brokers with the same broker.id?
    The second broker fails to register/start because broker.id must be unique across the cluster; the cluster rejects the duplicate.
  • Why pin broker.id in production instead of auto-generating it?
    A pinned, stable id lets a restarting broker reclaim its existing partition replicas and data directories instead of being treated as a new, empty node that triggers re-replication.

saying these in an interview costs you the question

  • Saying broker.id can be reused freely across brokers
  • Confusing broker.id with topic/partition ids
  • Claiming a broker stores whole topics rather than partition replicas
  • Saying every broker holds a full copy of all data

context

resolving links…