What happens when a KRaft node has fallen so far behind that the metadata log records it needs have already been truncated?

## Setup — the truncation horizon After a durable snapshot at offset X, KRaft may delete log segments fully covered by X. The earliest offset the leader still has in its log is therefore somewhere >= the start of retained segments. Call the boundary the ***log-start offset* / snapshot horizon**. A node whose next-needed offset is below this horizon cannot be served record-by-record because those records no longer exist. ## The problem A follower catches up by sending `Fetch` requests asking for records starting at its current offset. If that offset is below the leader's log-start offset, the leader has nothing to send — the records are gone. Without a remedy, the follower could never catch up. ## The mechanism — snapshot transfer (FetchSnapshot) Raft solves this with InstallSnapshot; KRaft's analogue is the `FetchSnapshot` RPC. The flow: - (1) The leader's Fetch response signals the follower that the requested offset is no longer available and points it to the latest snapshot (by snapshot id = end offset + epoch). - (2) The follower issues `FetchSnapshot` requests to download that snapshot, possibly in multiple chunks for a large image. - (3) Once fully received and validated, the follower *installs* it: it discards its (stale) state and adopts the snapshot's materialized state as of offset X. - (4) The follower then resumes ordinary `Fetch` from offset X+1, replaying the tail normally. From here it is back on the standard replay path. ## Why this is correct The snapshot is, by construction, the result of replaying 0..X. Installing it puts the follower in the exact state it would have had by replaying every record up to X — so subsequent tail replay yields identical state to a full replay. Determinism preserved. ## Why the leader can truncate aggressively Because snapshot install always exists as a fallback, the leader does not need to retain the entire log to accommodate the slowest member. It retains some recent log for normal incremental catch-up and falls back to snapshot transfer for far-behind members. This is the design choice that keeps the log bounded. ## Trade-offs and edge cases - (1) A snapshot install transfers the *full* materialized state, which for a large cluster (hundreds of thousands of partitions) can be sizable and bandwidth/IO heavy — much more than a small record delta, so it is the exceptional path, not the norm. - (2) Tuning retention/snapshot thresholds affects how often nodes fall over the horizon: very aggressive truncation plus brief outages can force more snapshot installs. - (3) The follower must validate snapshot integrity (offset/epoch) before adopting it. - (4) During install the follower is not yet a usable replica/observer; for a controller it cannot be an election candidate until caught up. - (5) This is exactly the path a long-offline broker or controller takes on rejoin. ## Operational signal Seeing FetchSnapshot activity / snapshot loading on rejoin indicates a node fell behind the retained log, often after a lengthy outage or under very high metadata churn with tight snapshot thresholds.