How does KRaft replicate the metadata log, and why is it described as pull-based rather than push-based Raft?

## Push vs pull, defined - **Push-based replication (classic Raft)**: the **leader initiates** by sending **AppendEntries** RPCs containing new log entries to each follower, and uses them as heartbeats too. - **Pull-based replication (KRaft)**: the **follower initiates** by sending a **Fetch** request asking 'give me records starting at offset X'; the leader replies with records. Nothing is replicated until a follower asks. ## Why KRaft chose pull Kafka already has a mature, high-throughput **replica fetcher** model: partition followers fetch from the partition leader. By making the metadata quorum also pull-based, KRaft reuses the same log segment format, the same fetch path, the same request **purgatory** (delayed-request handling), and the same offset/epoch bookkeeping. Less new code, more reuse of proven components. ## The core RPCs - **Vote**: sent by candidates during elections (covered separately). - **BeginQuorumEpoch**: the **leader** sends this to voters to announce 'I am leader for epoch E.' Note this is leader-initiated — it is how a freshly elected leader makes voters start fetching from it, since voters otherwise wouldn't know whom to Fetch from. - **Fetch**: the workhorse. Each follower/observer sends Fetch with `fetchOffset` and `lastFetchedEpoch`. The leader validates the epoch (divergence detection), returns records, and notes the follower's progress. - **EndQuorumEpoch**: sent by a leader that is gracefully resigning (e.g., controlled shutdown) so a new election starts promptly instead of waiting for timeouts. - **FetchSnapshot**: used when a follower is too far behind and the leader has compacted/snapshotted the metadata log; the follower fetches the snapshot, then resumes Fetch. ## How commit works under pull Because followers report their `fetchOffset` in each Fetch, the leader continuously learns each voter's replicated position. When a given offset has been fetched by a **majority of voters** (including the leader's own copy), that offset is **committed**, and the leader advances the **high watermark (HWM)**. The HWM is then propagated to followers in subsequent Fetch responses. ## Divergence / truncation The follower includes `lastFetchedEpoch` in Fetch. If the leader sees the follower's log diverges (the follower has records from an epoch the leader's history doesn't match at that offset), it tells the follower the **diverging offset/epoch**, and the follower **truncates** its log back to the last common point before continuing. This is how an old leader's uncommitted tail is cleaned up. ## Operational consequences - **Liveness detection**: a leader detects an unresponsive follower by the absence of Fetch requests; a follower detects a dead leader by Fetch failures/timeouts, which triggers a new election. - **Snapshots over infinite logs**: the metadata log is compacted into periodic snapshots so new/lagging nodes don't replay all history — they FetchSnapshot then tail with Fetch. - **Observers scale reads**: brokers as observers can fetch without affecting quorum, so adding brokers does not change commit latency. ## Pitfalls to avoid - Saying KRaft uses AppendEntries — it does not; that is classic Raft. - Thinking BeginQuorumEpoch carries the data — it only announces leadership; data flows via Fetch. - Assuming the leader pushes heartbeats — in steady state, followers drive the interaction by polling with Fetch.