When a consumer reads from a follower (KIP-392), what is the upper bound on the data it can read, and what consistency/lag implications does that create?
answer
- bounded by follower's own high-watermark
- HW propagates one round-trip behind leader
- freshness drops, correctness preserved
- uncommitted never served
- out-of-ISR follower → fall back / re-home
basics
~20 sA follower can only serve records up to its own high-watermark — the committed offset it has replicated and acknowledged. If the follower lags the leader, the consumer reads slightly older data, and end-to-end latency can grow, though it never reads uncommitted records.
solid answer
~50 sThe high-watermark (HW) is the offset up to which all in-sync replicas have the data; it's the committed boundary consumers are allowed to see. When fetching from a follower, the consumer is bounded by *that follower's* HW, which can trail the leader's HW because HW propagation to a follower takes an extra replication round-trip. Consequences: (1) end-to-end latency rises by roughly the follower's replication/HW-propagation lag; (2) a consumer can momentarily observe an offset that 'isn't there yet' on the chosen follower and must wait or re-home. Correctness is preserved — you never read uncommitted/unreplicated data — but freshness drops. If a follower falls out of the ISR, it's no longer eligible and the consumer falls back to another replica or the leader. This is why latency-sensitive workloads sometimes keep reading the leader despite the egress cost.
go deeper
Just know follower reads can be a bit behind the leader and may add a little latency.
Define high-watermark and state that the follower's HW (not the leader's) bounds the read.
Explain HW propagation lag, ISR eligibility, re-homing on disruption, and the latency-vs-cost trade-off.
Advise which workloads should use FFF vs stay on the leader, and tie follower lag monitoring to consumer freshness SLAs.
## Key terms - **Log end offset (LEO):** the next offset to be written on a given replica — how far that replica's local log goes. - **High-watermark (HW):** the offset up to which the record is considered **committed**, meaning every in-sync replica has it. Consumers are only allowed to read *below* the HW; records between HW and LEO exist physically but aren't yet visible to consumers because they're not guaranteed durable. - **ISR (in-sync replicas):** the set of replicas caught up enough with the leader to count toward commitment. ## How HW moves The leader advances its HW once all ISR followers have fetched up to a given offset. The leader then *tells* followers the new HW value on their next fetch response. So a follower learns the new HW one replication round-trip *after* the leader advanced it. This means **a follower's view of the HW lags the leader's HW** by at least one round-trip, often more under load. ## Why this bounds follower reads When a consumer fetches from follower F, F will only serve records up to **F's own HW**. Even if the leader has committed offset 1000, if F's HW is currently 980, the consumer sees up to 980 from F. So: - **Freshness / latency:** end-to-end latency (produce → consumer sees it) increases by F's HW-propagation + replication lag. For a tight SLA this matters. - **Apparent stalls:** if the consumer's committed/target offset is ahead of F's HW momentarily, it waits; KIP-392 also handles cases where the consumer must fall back (e.g., if the requested offset isn't yet on the follower, the broker can redirect it). ## Correctness is still safe You never read **uncommitted** data from a follower, because HW *is* the committed boundary and the follower won't serve past its HW. So FFF doesn't weaken the consumer's delivery guarantees — it only trades a little freshness for locality. ## ISR / eligibility edge cases - Only **in-sync** replicas are selectable. A lagging follower that drops out of the ISR stops being chosen; the consumer re-homes (to another same-rack ISR replica, else the leader). - On leadership change or ISR shrink, the **preferred read replica** the consumer cached can become stale; the consumer refreshes metadata and re-selects, so there's a brief window of cross-AZ reads after a disruption. - A follower's HW can also be temporarily reset/truncated during certain recovery scenarios; the consumer handling accounts for offset-out-of-range by re-checking with the leader. ## Practical guidance - Use FFF for **cost-sensitive, latency-tolerant** consumers (analytics, batch, lag-tolerant streams). - Keep **latency-critical** consumers on the leader, or accept the extra lag knowingly. - Monitor follower replication lag — large lag directly becomes consumer freshness lag under FFF.
- Can fetch-from-follower ever return uncommitted (un-replicated) data?No. A follower serves only up to its high-watermark, which is the committed boundary. The trade is freshness/latency, never a weaker durability/visibility guarantee.
- What happens to a consumer's chosen follower if that follower drops out of the ISR?It becomes ineligible; the consumer refreshes metadata and re-selects another in-rack ISR replica or falls back to the leader, briefly incurring cross-AZ reads.
saying these in an interview costs you the question
- Claiming follower reads can expose uncommitted data — they can't (HW-bounded).
- Saying FFF has zero latency impact — it adds the follower's replication lag.
- Confusing LEO with HW; consumers read up to HW, not LEO.