Explain the process.roles, node.id, and controller.quorum.voters configuration properties in KRaft. What does each control?

## The three identity configs When you configure a KRaft node you must answer three questions: *what does this node do?*, *who is it?*, and *who are the controllers?* These map to three properties. ### 1. `process.roles` This declares the node's **role(s)**. Valid values: - **`broker`** — the node stores topic data and serves producers/consumers, but does **not** vote in the metadata quorum. It is an observer of `__cluster_metadata`. - **`controller`** — the node participates in the metadata Raft quorum (it is a **voter**); it can become the active controller. A pure controller does **not** serve client data. - **`broker,controller`** — **combined mode**: one process plays both roles. If `process.roles` is set, the node runs in KRaft mode. (In the old days an unset `process.roles` meant ZooKeeper mode; since Kafka 4.0 ZooKeeper mode is gone.) **Combined vs dedicated**: combined mode is great for laptops, demos, and tiny clusters because you run fewer processes. For production it is discouraged — a controller competing for CPU/heap with broker data traffic makes failover and metadata latency less predictable, and isolation/security boundaries are cleaner with **dedicated controllers** (commonly 3 controller-only nodes plus N broker-only nodes). ### 2. `node.id` A **unique integer** identifying this node within the cluster. It unifies what used to be `broker.id`. The same ID is used to refer to the node in `controller.quorum.voters`. Two nodes must never share a `node.id`. For controllers, this ID is the node's identity in the Raft group. ### 3. `controller.quorum.voters` A **static list** of the controller voters in the form: ``` controller.quorum.voters=1@host1:9093,2@host2:9093,3@host3:9093 ``` Each entry is `{node.id}@{host}:{controller-listener-port}`. Every node in the cluster — brokers and controllers alike — needs this so it can locate the controllers. The controllers use it to form their Raft quorum and elect a leader. Because it is static, the membership is fixed at the listed set; you cannot casually add/remove voters. ### Related listener config Controllers expose a dedicated listener (e.g. `CONTROLLER`) named in `controller.listener.names`, and brokers connect to it. The port in the voter list is that controller listener's port (commonly 9093). ## Cluster bootstrapping Before first start you generate a cluster ID with `kafka-storage.sh random-uuid` and format each node's storage with `kafka-storage.sh format --cluster-id <id> --config server.properties`. This writes the `meta.properties` containing the `node.id` and cluster ID and initializes `__cluster_metadata`. ## Dynamic quorums (newer Kafka) The static `controller.quorum.voters` is rigid. **KIP-853 (dynamic KRaft quorums)** lets you **add and remove voters at runtime**. With it you configure **`controller.quorum.bootstrap.servers`** (a discovery endpoint list) instead of a fixed voter set, and use `kafka-metadata-quorum.sh` to add/remove controllers. This is important for operations like replacing a failed controller without a full reconfiguration. ## Common pitfalls / edge cases - Reusing a `node.id` across two nodes corrupts cluster identity. - Listing the wrong port (broker listener instead of controller listener) in `controller.quorum.voters` breaks quorum formation. - Mixing combined and dedicated controllers inconsistently, or changing a node's role after format, can require re-formatting. - The number of voters should be **odd** (typically 3 or 5) so a clear majority exists.