In a dynamic KRaft cluster, what is the difference between an observer and a voter, and how does a node move between these roles?
answer
- voter votes + counts toward majority
- observer replicates only, no vote
- brokers are always observers
- join as observer, catch up, then promote
- AddRaftVoter promotes, RemoveRaftVoter demotes
basics
~20 sA voter is a controller in the quorum's voter set that votes in elections and counts toward majority commits. An observer replicates the metadata log but doesn't vote or count. A node joins as an observer, catches up, then add-controller promotes it to voter.
solid answer
~40 sIn KRaft, brokers and not-yet-added controllers act as observers: they fetch and replicate the __cluster_metadata log to stay current, but they do not vote in leader elections and do not count toward the majority needed to commit. Voters are the controllers in the active voter set; they vote, can be elected leader, and their acknowledgments are what commit metadata. With dynamic quorums, a controller formatted with --no-initial-controllers starts as an observer, replicates the log to catch up, and is then promoted to voter by AddRaftVoter (kafka-metadata-quorum add-controller), which commits a VotersRecord. RemoveRaftVoter demotes it back out of the voter set. Keeping the add as a two-step observe-then-promote process means the node is fully caught up before it starts affecting availability, since a far-behind voter would slow commits.
go deeper
Know that voters vote and observers only replicate; a node joins as observer then gets promoted.
Explain commit-majority weighting and the AddRaftVoter/RemoveRaftVoter role transitions.
Discuss why catch-up before promotion protects availability and how brokers differ from controllers.
Reason about quorum sizing, fault tolerance changes during role transitions, and monitoring lag before promotion.
## Two roles in the KRaft log KRaft replicates a single metadata log, `__cluster_metadata`. Every participant **fetches** from the leader to stay current, but participants fall into two roles: - **Voter** — a controller that is in the cluster's **voter set**. Voters: (1) cast votes in leader elections, (2) can *become* the leader, and (3) their fetch acknowledgments **count toward the majority** required to commit a record. The number of voters determines fault tolerance (majority = `floor(N/2)+1`). - **Observer** — a participant that **replicates the log but does not vote and does not count toward commit majorities**. Brokers are always observers (they need metadata but never lead the quorum). A controller that has not yet been added to the voter set is also an observer. ## How a node changes role (dynamic quorums) 1. **Join as observer:** format with `kafka-storage format --no-initial-controllers`. The node boots, reads `controller.quorum.bootstrap.servers`, and begins fetching the metadata log as an observer. It has a `directory.id` but is not in the voter set. 2. **Catch up:** it replicates until its log end offset is close to the leader's. You can monitor via `kafka-metadata-quorum describe --status` (look at the observer's lag / last fetch). 3. **Promote to voter:** `kafka-metadata-quorum add-controller` issues **AddRaftVoter**; the leader commits a `VotersRecord` adding the node. From then on it votes and counts toward majority. 4. **Demote / leave:** `remove-controller` issues **RemoveRaftVoter**, committing a `VotersRecord` that drops it back to a non-voter (then you stop it). ## Why observe before promote A voter that is far behind still counts toward the **majority needed to commit**. If you promoted an empty/lagging node immediately, the leader might need its ack to reach majority while it is still catching up, **slowing or stalling commits** and reducing effective availability. Catching up as an observer first avoids this. ## Edge cases - **Brokers vs controllers:** brokers are observers by design and are never promoted to voters; only controller-role nodes join the voter set. - **Leader is always a voter:** an observer can never be elected leader. - **Static mode:** the voter set is fixed in config; observers (brokers) still exist but you cannot promote a controller at runtime. ## Summary Observer = replicate-only, no vote, no commit weight; voter = full quorum member. Dynamic reconfiguration is precisely the controlled, log-committed promotion (AddRaftVoter) and demotion (RemoveRaftVoter) of controllers between these roles.
- Are brokers voters in a KRaft quorum?No. Brokers are always observers — they replicate the metadata log but never vote, count toward commit majority, or become controller leader.
- What goes wrong if you promote a lagging observer to voter immediately?As a voter it counts toward the commit majority while still behind, so the leader may need its ack to reach majority, slowing or stalling commits until it catches up.
saying these in an interview costs you the question
- Saying observers vote but their vote 'counts less' — observers do not vote at all.
- Claiming brokers can be promoted to voters.
- Thinking an observer can be elected controller leader.
- Believing promotion is instant regardless of catch-up state.