How do metadata snapshots and log replay enable fast controller failover in KRaft compared with the ZooKeeper-based architecture?

## The ZooKeeper baseline In the legacy architecture, ZooKeeper stored cluster metadata and Kafka elected one broker as *the controller*. That controller cached metadata in memory. On controller failure, a new controller was elected and had to *load the full metadata state from ZooKeeper* — every topic, partition, ISR, config — over many ZooKeeper round-trips. For large clusters (hundreds of thousands of partitions) this **cold load** was a known bottleneck, making failover slow (tens of seconds to minutes) and limiting cluster size. Metadata propagation to brokers was also a fan-out of RPCs, not a log. ## The KRaft model Metadata is a replicated Raft log (`__cluster_metadata`) owned by a dedicated controller quorum (typically 3 or 5 controllers). - One controller is the Raft leader (**active controller**); the rest are followers (**hot standbys**). - All of them continuously replicate and *replay* the log into their own in-memory metadata state. - Brokers are observers that also replay the log to build their local metadata cache. ## Why snapshots matter for failover A standby controller bootstraps from the latest snapshot (jumping to offset X) and then keeps applying the tail. So at any moment a standby is fully materialized up to nearly the log end. When it must take over, there is no separate 'load state' phase — its state machine is already warm. Snapshots also bound how long a *restarted* controller takes to become caught-up: load snapshot + replay small tail, rather than replay from offset 0. ## The failover sequence - (1) Active controller fails / loses leadership. - (2) Raft runs a leader election among in-sync followers (the one with the most up-to-date log wins, by Raft rules). - (3) The new leader resolves any uncommitted tail (entries above the previous high-water mark may be truncated per Raft). - (4) Because it already has the materialized state, it immediately begins serving as active controller — handling broker heartbeats, leader elections for partitions, etc. The whole thing is seconds, not minutes. ## Broker side Brokers don't depend on a single controller pushing them state; they **pull the log**. So a controller failover does not require re-pushing metadata to every broker — brokers just keep tailing the new leader's log. This decoupling is part of why KRaft scales to many more partitions. ## Edge cases - A standby that has lagged badly (e.g. was down) may need a snapshot install before it is a viable candidate. - The quorum must keep enough in-sync members for an election to succeed (majority). - The replay determinism guarantees all in-sync controllers agree on state, so any of them is a correct successor. ## Net effect KRaft converts failover from a cold, external-store full-load into a **warm-standby Raft handoff**, which is the architectural reason KRaft advertises faster controlled and uncontrolled failover and higher partition scalability than ZooKeeper-based Kafka.