skip to content

How does KRaft decide a metadata record is committed, and how does the high watermark advance?

level: seniorimportance: must knowfreq 50%

answer

  1. committed = on a majority of voters
  2. HWM = highest committed offset
  3. leader computes HWM from reported fetch offsets
  4. only commit in current epoch (Figure-8 fix)
  5. HWM <= LEO

basics

~20 s

A record is committed once it has been replicated (fetched) by a majority of the voters. The high watermark is the highest offset known to be on a majority; the leader advances it as Fetch offsets arrive, and only records at or below the HWM are visible/applied.

solid answer

~50 s

Each Fetch request reports a voter's `fetchOffset`, telling the leader how much that voter has replicated. The leader computes the **high watermark (HWM)** as the highest offset that a **majority of voters** (including itself) have stored. A record at or below the HWM is **committed** — durable against the loss of a minority of voters — and only then is it applied to the metadata state machine and exposed to observers. KRaft adds a safety rule from Raft: a leader only advances the HWM for records in its **own current epoch**; it does not directly commit a leftover record from a prior epoch by majority count alone (the *Raft commitment restriction*), avoiding the classic Figure-8 anomaly. Once an in-epoch record commits, earlier records commit transitively. The HWM is piggybacked back to followers in Fetch responses so they learn what is safe to apply. Uncommitted tail records from a deposed leader are truncated when a follower detects epoch divergence.

go deeper

for a junior

Know that a record is committed when a majority of voters have it, and the HWM marks the committed boundary.

for a middle

Explain that the leader uses reported fetch offsets to advance the HWM and that uncommitted tails get truncated.

for a senior

Articulate the majority-overlap durability argument and the current-epoch commit restriction (Figure-8).

for a principal

Reason about quorum sizing trade-offs, HWM vs LEO, snapshot interactions, and how truncation preserves linearizability.

## Definitions - **Offset**: the position of a record in the log (0,1,2,...). - **Committed**: a record is committed when it is guaranteed durable and will never be lost or reordered even across leader failures — concretely, when it lives on a **majority of voters**. - **High watermark (HWM)**: the highest offset that is committed. Records above the HWM are not yet safe and must not be exposed or applied. - **State machine**: the in-memory metadata image built by replaying committed records. ## How the leader computes the HWM Every voter, on each **Fetch**, reports the offset it has reached (`fetchOffset` = the next offset it wants = one past what it has). The leader keeps, for each voter, the highest offset that voter has acknowledged. To find the committed point, the leader takes the offset such that a **majority** of voters (counting its own log) are at or beyond it. That offset minus one is the new HWM. Example with 3 voters (majority = 2): leader at offset 100, follower A at 100, follower B at 95. The leader plus A are at 100, that's a majority, so HWM = 99 (offset 99 is committed). B will catch up later. ## Why a majority With N voters, the cluster tolerates the loss of `floor((N-1)/2)` voters. Any future leader needs votes from a majority, which must overlap with the majority that holds each committed record — so a committed record survives into every future leader's log. This is the same overlap argument that makes elections safe. ## The current-epoch commit restriction (subtle but important) Raft (and KRaft) forbid a new leader from committing a record left over from a **previous epoch** purely because it now sits on a majority. The reason is the famous **Raft Figure 8** scenario: a record could appear majority-replicated, then be overwritten after a leadership change, violating durability. The fix: a leader only advances the HWM when a record **from its own epoch** reaches a majority. Doing so transitively commits all earlier records (including the old-epoch ones) safely. In practice, a new KRaft leader appends a control/leader-change record in its new epoch, and committing that record carries the prior tail across the line. ## Propagating and applying - The leader includes the current HWM in **Fetch responses**, so followers and observers learn how far they may safely apply records to their metadata image. - Brokers (observers) apply records only up to the HWM they have been told about, ensuring they never act on uncommitted metadata. ## Truncation of uncommitted tails If an old leader had appended records that never reached a majority and a new leader was elected without them, followers that did receive that tail detect the divergence via `lastFetchedEpoch` in their next Fetch, are told the **diverging offset**, and **truncate** those uncommitted records. No committed record is ever truncated. ## Edge cases - **Even number of voters**: a majority of 4 is still 3, so 4 voters tolerate only 1 failure — same as 3 but with higher quorum cost; odd counts (3,5) are preferred. - **Lagging minority**: the HWM advances based on the fastest majority, so one slow voter does not block commits. - **HWM vs LEO**: the **log end offset (LEO)** is the last appended offset (may be uncommitted); the HWM is the last committed offset. HWM <= LEO always.

  • Why can't a new KRaft leader commit a record from a previous epoch just because it now sits on a majority?
    Because of the Raft Figure-8 anomaly: such a record could later be overwritten after another leadership change, breaking durability. A leader only advances the HWM for a record in its own epoch, which transitively and safely commits the earlier tail.
  • What is the difference between the log end offset (LEO) and the high watermark (HWM)?
    LEO is the offset just past the last appended record (which may be uncommitted/uncovered by a majority); HWM is the highest committed offset. HWM is always <= LEO.
  • Does one slow follower delay commits in a 5-voter quorum?
    No. Commit requires only a majority (3 of 5), so the leader advances the HWM once the fastest 3 (including itself) have the record; the slow follower catches up later.

saying these in an interview costs you the question

  • Saying a record is committed once any single follower has it
  • Claiming all voters must have a record before it commits (it's a majority, not all)
  • Ignoring the current-epoch commit restriction / Figure-8 problem
  • Confusing HWM with LEO
  • Saying observers count toward the commit majority

context

resolving links…