Explain how --initial-controllers and the bootstrap.checkpoint mechanism work for first-boot quorum bootstrap in newer Kafka versions, and how that differs from controller.quorum.voters.

## The problem with static quorums In the original KRaft design every node lists the controllers explicitly: ``` controller.quorum.voters=1@host1:9093,2@host2:9093,3@host3:9093 ``` This is a **static** membership: the set of voters is baked into config on every node. It works, but it's rigid — you cannot cleanly **add or remove a controller** at runtime, because all nodes must agree on the voter set and there is no protocol record of membership changes. Replacing a dead controller meant a careful, error-prone reconfiguration. ## KIP-853: dynamic quorums **KIP-853** introduces **dynamic controller membership**, generally available around Kafka **3.9 / 4.0**. Two pieces change: 1. **Discovery vs. membership are separated.** Instead of `controller.quorum.voters`, dynamic clusters use: ``` controller.quorum.bootstrap.servers=host1:9093,host2:9093,host3:9093 ``` This is only a list of **endpoints to contact** to find the quorum — it is *not* the authoritative voter set. The real voter set lives **inside the metadata log** as records and can change over time. 2. **The genesis voter set is recorded at format time** via a snapshot rather than config. ## directory.id With dynamic quorums, each storage directory gets a **`directory.id`** (a UUID written into `meta.properties` by format). A controller voter is identified by the pair **(node id, directory id)**. Tracking storage identity — not just node id — lets the quorum distinguish a controller that was rebuilt on fresh disks from the original, which is essential for safe add/remove. ## --standalone and --initial-controllers At format time you declare the seed quorum: - **`--standalone`** (`-s`): format a single controller as a one-node initial quorum (you grow it later with add-controller). - **`--initial-controllers`** (`-I`): provide the full initial voter set explicitly, e.g. ``` bin/kafka-storage.sh format -t $CLUSTER_ID -c controller.properties \ --initial-controllers "0@host0:9093:<dir-uuid-0>,1@host1:9093:<dir-uuid-1>,2@host2:9093:<dir-uuid-2>" ``` Each entry is `id@host:port:directory.id`. You format every initial controller with the **same** `--initial-controllers` string so they all agree on the genesis set. - **`--no-initial-controllers`**: format a node that will *join* an existing dynamic quorum later (it does not seed membership). ## bootstrap.checkpoint For a dynamic quorum, the formatter writes a **`bootstrap.checkpoint`** file into the metadata log directory. This is a **Raft snapshot at the special offset `-1`** (the "bootstrap" snapshot, sometimes shown as `00000000000000000000-0000000000.checkpoint`-style genesis) that encodes the **initial voter set** chosen by `--standalone`/`--initial-controllers`, plus a bootstrap metadata version. On **first boot**, the controllers read `bootstrap.checkpoint` to establish the genesis quorum and the initial **metadata.version** feature level. After that, all membership changes are appended to the metadata log and replicated by Raft; the checkpoint is no longer the source of truth for membership. ## Runtime membership changes Once bootstrapped dynamically you manage controllers with: ``` bin/kafka-metadata-quorum.sh --command-config ... add-controller bin/kafka-metadata-quorum.sh ... remove-controller --controller-id <id> --controller-directory-id <dir> ``` These append membership records to the metadata log — no rolling config edit across the fleet. ## Static vs dynamic — how to tell - **Static:** `controller.quorum.voters` set; format does **not** need `--initial-controllers`; no `bootstrap.checkpoint` voter set; membership frozen. - **Dynamic:** `controller.quorum.bootstrap.servers` set; format uses `--standalone`/`--initial-controllers`; `bootstrap.checkpoint` written; membership mutable via kafka-metadata-quorum. You should not mix both styles for the same cluster. ## Edge cases - Formatting initial controllers with mismatched `--initial-controllers` strings produces inconsistent genesis quorums that won't converge. - The directory.id must match what you list; if you wipe a controller's disk it gets a new directory.id and must be re-added as a new voter. - Migrating an existing static cluster to dynamic membership is a deliberate, version-gated operation, not automatic.