How does metadata propagate from the active controller to brokers in KRaft, and how does this differ from the ZooKeeper-era controller model?

## The KRaft propagation path (pull, log-based) 1. The **active controller** appends each metadata change as a **record** to the single-partition `__cluster_metadata` Raft log and commits it once a majority of voters acknowledge. 2. **Brokers are observers** (non-voting). Each broker runs a fetch loop that **pulls** committed records from the active controller, exactly as a follower fetches from a partition leader. 3. The broker **applies records in offset order** to its in-memory **metadata image** (the "MetadataDelta/MetadataImage" structures) — learning, for example, that it is now leader for partition X or that topic Y was created. 4. The broker tracks the **metadata offset** it has applied and reports it back (via heartbeats), so the controller knows how far behind each broker is (its **metadata lag**). A broker that falls too far behind can be **fenced** until it catches up. 5. On startup or reconnection, a broker loads the **latest snapshot** and then replays only the **tail** after that offset — bounded, fast catch-up. Key property: **incremental deltas**. Each change is one small record. Brokers never need a full metadata dump for routine changes; they just consume the next records. ## Broker registration and heartbeats Brokers **register** with the controller (a record in the log) and send periodic **heartbeats** (the `BrokerHeartbeat` RPC). The controller uses these to track liveness and metadata catch-up, replacing ZooKeeper ephemeral znodes for membership. A broker that stops heartbeating is **fenced** (excluded from leadership) and eventually unregistered. ## The ZooKeeper-era model (push, RPC-based) — the contrast - Membership/metadata lived in **ZooKeeper**. One broker was the **controller**, elected via a ZooKeeper ephemeral node. - The controller **watched** ZooKeeper for changes and **pushed** updates to brokers using **`LeaderAndIsr`** RPCs (telling a broker which partitions it leads/follows) and **`UpdateMetadata`** RPCs (cluster-wide topic/partition/leader maps). - On **controller failover**, the new controller had to **read all metadata from ZooKeeper** and **resend full state** to every broker — an O(metadata-size) operation taking many seconds in large clusters, during which the cluster was effectively unmanaged. - This fan-out-of-full-state design **limited partition counts** (practically the low hundreds of thousands) and made recovery slow. ## Why KRaft is better - **Near-instant failover**: every controller voter already has the full metadata log in memory; the new active controller is immediately up to date — no ZooKeeper reload. - **Incremental propagation**: brokers consume deltas, not full snapshots, so per-change cost is tiny and **millions of partitions** become feasible. - **Single ordered source of truth**: an offset-ordered log instead of fan-out RPCs that could arrive out of order or get lost. - **Observable lag**: per-broker metadata offset makes "is everyone caught up?" directly measurable. ## Edge cases - A broker with stale metadata (high lag) may briefly act on old leadership info; the offset tracking + fencing bound this window. - Snapshots are still needed: a brand-new or very-stale broker gets a snapshot rather than replaying from offset 0. - The data-plane RPCs that clients use (Produce/Fetch) are unchanged; only the *metadata* propagation mechanism changed. - `LeaderAndIsr`/`UpdateMetadata` RPCs are removed in KRaft — a frequent gotcha for people debugging based on old runbooks.