skip to content

What is the difference between the log-end-offset (LEO) and the high watermark (HW) on a Kafka partition leader, and why does it matter to consumers?

level: juniorimportance: must knowfreq 70%

answer

  1. LEO = next offset to write
  2. HW = min(ISR LEOs) = committed boundary
  3. consumers read below HW only
  4. HW hides un-replicated tail
  5. acks=all waits for HW

basics

~20 s

LEO is the offset just past the last message written to the log. HW is the highest offset that has been replicated to all in-sync replicas. Consumers can only read up to (below) the HW, so they never see un-replicated records.

solid answer

~40 s

Every replica tracks its log-end-offset (LEO): the offset of the next record to be appended, i.e. one past the last byte in its local log. The leader also tracks the high watermark (HW): the minimum LEO across all replicas currently in the ISR (in-sync replica set). The HW is the boundary of 'committed' data — records below it exist on every in-sync replica. Consumers are only allowed to fetch messages up to HW-1; anything between HW and LEO is written but not yet fully replicated and is invisible to consumers. This guarantees the read-your-committed-data property: a consumer never sees a record that could later be lost if the leader fails before that record is replicated.

go deeper

for a junior

Recall the two definitions and that consumers read below the HW.

for a middle

Explain HW = min(ISR LEOs) and how Fetch requests propagate LEO/HW.

for a senior

Connect HW to acks/min.insync.replicas durability guarantees and ISR eviction.

for a principal

Reason about HW lag, the read-committed guarantee, and how the boundary interacts with failover design.

## The two offsets A Kafka partition is an append-only log. Each replica (one leader, several followers) keeps its own copy of that log and tracks two key positions: - **LEO (Log-End-Offset):** the offset that will be assigned to the *next* record appended. If a replica has stored records at offsets 0..99, its LEO is 100. Every replica has its own LEO; the leader's LEO advances first when a producer writes, and followers catch up by fetching. - **HW (High Watermark):** the highest offset that is *committed*, meaning it has been replicated to **all replicas currently in the ISR** (In-Sync Replica set). Concretely, the leader computes `HW = min(LEO of all ISR members)`. ## Why consumers stop at the HW Consumers may only read records with offset **< HW**. Records in the range `[HW, LEO)` are physically present on the leader but have not yet been confirmed on every in-sync replica. If the leader crashed at that moment, a new leader elected from the ISR might not have those records, so they could vanish. By hiding them, Kafka guarantees that **once a consumer sees a record, that record is durable** (survives a leader failure within the ISR). ## How it advances 1. Producer sends a batch to the leader. Leader appends it → leader LEO jumps. 2. Followers send Fetch requests. Each Fetch from a follower also reports that follower's current LEO (its fetch offset). 3. The leader updates its view of each follower's LEO and recomputes `HW = min(ISR LEOs)`. 4. The new HW is sent back to followers piggybacked on the next Fetch response, so followers learn the committed boundary slightly after the leader. ## Edge cases / nuances - With **acks=all** + `min.insync.replicas`, a producer is acknowledged only when the record's offset is below the HW (committed on the required number of in-sync replicas). With **acks=1**, it is acknowledged on leader append, before the HW advances — that record can be lost if the leader fails first. - The HW can never exceed any ISR member's LEO, but it can lag the leader's LEO arbitrarily if followers are slow (they get evicted from the ISR via `replica.lag.time.max.ms` so the HW can keep moving). - Followers also track their own HW (learned from the leader); this matters during truncation and leader election.

  • If a producer uses acks=1, can a consumer ever observe a record that later gets lost?
    No. The consumer still only reads below the HW. acks=1 means the producer is acked at leader append (before HW advances), so the producer can lose data, but the consumer-visible boundary is still the HW — it never exposes the un-replicated tail.
  • How does the leader learn each follower's LEO?
    Followers report their current fetch offset (their LEO) inside every Fetch request they send to the leader. The leader uses those to recompute HW = min(ISR LEOs).

saying these in an interview costs you the question

  • Saying the HW is the last offset written to the leader (that's the LEO).
  • Claiming consumers can read up to the LEO.
  • Confusing HW with the committed *consumer* offset (__consumer_offsets) — different concept.
  • Saying HW = max of replica LEOs (it is the min across the ISR).

context

resolving links…