Walk through how you add a new controller voter to a running dynamic KRaft cluster and how you remove one.

## Goal Grow or shrink the controller quorum without downtime and without editing config on every node. Dynamic reconfiguration (KIP-853) does this by writing membership changes into the metadata log itself. ## Adding a controller — step by step 1. **Provision** the host and pick a unique `node.id`. 2. **Format storage as an observer:** ``` kafka-storage format --cluster-id <id> \ --config server.properties \ --no-initial-controllers ``` `--no-initial-controllers` means the node does *not* assume it is part of the initial voter set; it boots as a **non-voting observer**. Formatting also generates its `directory.id` into meta.properties. 3. **Configure** `controller.quorum.bootstrap.servers` to point at existing controllers, set `process.roles=controller` (or `broker,controller`), and start the node. 4. **Catch up:** as an observer it replicates `__cluster_metadata` from the leader until its log nearly matches. You can watch lag with `kafka-metadata-quorum describe --status` (LastFetchTimestamp / log end offset). 5. **Promote to voter:** ``` kafka-metadata-quorum --bootstrap-controller host:9093 \ add-controller ``` This sends **AddRaftVoter** with the node's id, directory.id, and endpoints. The leader appends a **VotersRecord** that includes the new voter. Once that record is **committed** (replicated to a majority of the *new* configuration), the node is a full voter. ## Removing a controller 1. Run: ``` kafka-metadata-quorum --bootstrap-controller host:9093 \ remove-controller --controller-id <id> \ --controller-directory-id <dirId> ``` This sends **RemoveRaftVoter**. The leader commits a VotersRecord dropping that voter. 2. After the change commits, **shut down** the node. (Shutting it down first, while still a voter, shrinks your live majority and is the wrong order.) ## One change at a time Membership changes are applied **single-server at a time** (add one or remove one). Each change is a **joint-consensus** transition that requires agreement from both the old and new voter sets before committing, which guarantees no two disjoint majorities can form. You cannot batch a +2/-2 swap as one atomic step; do them sequentially. ## Availability math Keep a majority alive throughout. Example: going 3→5, after adding the 4th voter a majority is 3 of 4; after the 5th it is 3 of 5. When removing, remove before stopping so you never drop below majority of the *current* voter set. ## Common pitfalls - Forgetting `--no-initial-controllers` (the node tries to bootstrap its own quorum). - Promoting before the observer has caught up (a far-behind voter weakens availability until it catches up). - Stopping a node before remove-controller commits. - Passing the wrong directory.id to remove-controller (it must match the voter set entry).