What is the ISR, and how does Kafka decide whether a follower is in-sync or should be removed from it?

middlemust knowfreq 58%

answer

ISR = leader + caught-up followers
Time-based: replica.lag.time.max.ms (30s)
Covers both stalled and slow followers
Replaced replica.lag.max.messages
Only ISR can be elected / counts toward HW

basics

~20 s

The ISR (in-sync replica set) is the leader plus all followers caught up with it. A follower is dropped from the ISR if it hasn't fetched up to the leader's log end within replica.lag.time.max.ms, and re-added once it catches back up.

solid answer

~50 s

The ISR is the set of replicas — the leader plus its sufficiently caught-up followers — that are considered fully replicated for a partition. Only ISR members are eligible to be promoted to leader (under clean election) and only ISR LEOs count toward the high watermark. Kafka decides membership by time, not message count: a follower stays in the ISR as long as it has caught up to the leader's log-end offset at some point within the last replica.lag.time.max.ms (default 30 s). The leader tracks, per follower, the last time it was fully caught up. If a follower neither catches up to the current LEO nor sends a fetch within that window, the leader shrinks the ISR by removing it (recorded in metadata via the controller/KRaft). When the lagging follower fetches back up to the leader's LEO, it is re-added. This time-based rule (since KIP-1xx era, replacing the old replica.lag.max.messages) handles both slow and stalled followers uniformly.

The **ISR** is the set of replicas — the leader plus its sufficiently caught-up followers — that are considered fully replicated for a partition. Only ISR members are eligible to be promoted to leader (under clean election) and only ISR LEOs count toward the high watermark. Kafka decides membership by *time*, not message count: a follower stays in the ISR as long as it has caught up to the leader's log-end offset at some point within the last **`replica.lag.time.max.ms`** (default 30 s). The leader tracks, per follower, the last time it was fully caught up. If a follower neither catches up to the current LEO nor sends a fetch within that window, the leader shrinks the ISR by removing it (recorded in metadata via the controller/KRaft). When the lagging follower fetches back up to the leader's LEO, it is re-added. This time-based rule (since KIP-1xx era, replacing the old `replica.lag.max.messages`) handles both slow and stalled followers uniformly.

go deeper

for a junior

Know the ISR is the set of replicas that are caught up with the leader.

for a middle

Explain the time-based replica.lag.time.max.ms rule and why it replaced the message-count rule.

for a senior

Connect ISR shrink/expand to HW advancement, min.insync.replicas, and unclean election trade-offs.

for a principal

Reason about ISR dynamics under failure, the availability/durability spectrum, and fleet-level monitoring (UnderReplicatedPartitions).

Definition. The ISR (in-sync replica set) is the subset of a partition's replicas that are currently keeping up with the leader. It always includes the leader itself and any followers deemed in-sync. It is dynamic — replicas join and leave as their replication health changes.

Why it matters.

Commit / high watermark: Only ISR members' LEOs are used to compute the high watermark (HW = min ISR LEO). A record is committed when all ISR members have it.
Leader election: Under clean leader election, only an ISR member can be promoted when the leader fails — guaranteeing the new leader already holds all committed data.
Durability gate: min.insync.replicas plus acks=all require the ISR to be at least a given size for writes to be accepted.

The membership rule (time-based). A follower is in-sync if it has fully caught up to the leader's log-end offset within the last replica.lag.time.max.ms (default 30000 ms). The leader maintains, per follower, the timestamp of the last moment that follower's fetch reached the leader's current LEO. Two failure modes are both covered:

A follower that stops fetching entirely (crashed, network partition) won't update its caught-up timestamp -> evicted after the window.
A follower that fetches but can't keep pace (slow disk, overloaded) keeps falling behind the moving LEO and never reaches it within the window -> evicted.

This unified time metric replaced the older, brittle replica.lag.max.messages count, which mis-fired during traffic bursts (a sudden spike could push a healthy follower past a message-count threshold even though it was fine).

Shrink and expand.

Shrink: When the leader detects a follower has exceeded the lag window, it removes it from the ISR and propagates the change (via the controller in ZK mode, or the KRaft metadata log). HW may then advance based on the remaining, smaller ISR — so an evicted slow follower no longer holds back commits.
Expand: When that follower fetches back up to the leader's LEO, the leader adds it to the ISR again.

Interplay with durability. If the ISR shrinks to just the leader and min.insync.replicas=2, acks=all produces are rejected (NotEnoughReplicas) — Kafka refuses to commit data that would live on too few brokers. If unclean.leader.election.enable=true and the ISR becomes empty, a non-ISR replica may be elected leader, risking data loss for availability.

Observability. A persistently small ISR (UnderReplicatedPartitions / kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions > 0) signals replication problems — slow brokers, network, or undersized fetchers.

**Definition.** The **ISR (in-sync replica set)** is the subset of a partition's replicas that are currently keeping up with the leader. It always includes the leader itself and any followers deemed in-sync. It is dynamic — replicas join and leave as their replication health changes. **Why it matters.** 1. **Commit / high watermark:** Only ISR members' LEOs are used to compute the high watermark (HW = min ISR LEO). A record is committed when all ISR members have it. 2. **Leader election:** Under clean leader election, only an ISR member can be promoted when the leader fails — guaranteeing the new leader already holds all committed data. 3. **Durability gate:** `min.insync.replicas` plus `acks=all` require the ISR to be at least a given size for writes to be accepted. **The membership rule (time-based).** A follower is **in-sync** if it has fully caught up to the leader's log-end offset within the last `replica.lag.time.max.ms` (default 30000 ms). The leader maintains, per follower, the timestamp of the last moment that follower's fetch reached the leader's current LEO. Two failure modes are both covered: - A follower that **stops fetching entirely** (crashed, network partition) won't update its caught-up timestamp -> evicted after the window. - A follower that **fetches but can't keep pace** (slow disk, overloaded) keeps falling behind the moving LEO and never reaches it within the window -> evicted. This unified time metric replaced the older, brittle `replica.lag.max.messages` count, which mis-fired during traffic bursts (a sudden spike could push a healthy follower past a message-count threshold even though it was fine). **Shrink and expand.** - **Shrink:** When the leader detects a follower has exceeded the lag window, it removes it from the ISR and propagates the change (via the controller in ZK mode, or the KRaft metadata log). HW may then advance based on the remaining, smaller ISR — so an evicted slow follower no longer holds back commits. - **Expand:** When that follower fetches back up to the leader's LEO, the leader adds it to the ISR again. **Interplay with durability.** If the ISR shrinks to just the leader and `min.insync.replicas=2`, `acks=all` produces are rejected (NotEnoughReplicas) — Kafka refuses to commit data that would live on too few brokers. If `unclean.leader.election.enable=true` and the ISR becomes empty, a non-ISR replica may be elected leader, risking data loss for availability. **Observability.** A persistently small ISR (UnderReplicatedPartitions / `kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions` > 0) signals replication problems — slow brokers, network, or undersized fetchers.

Why did Kafka move from replica.lag.max.messages to a time-based ISR rule?
A message-count threshold falsely evicted healthy followers during traffic bursts: a sudden spike of writes could push the leader far ahead in message count momentarily even though the follower was replicating fine. A time-based rule (caught up within replica.lag.time.max.ms) reacts to sustained lag, not transient bursts, and uniformly catches both stalled and slow followers.
How does shrinking the ISR affect the high watermark and durability?
The HW is the min LEO over the current ISR, so removing a slow follower lets the HW advance based on the faster remaining replicas (improving availability/latency). But it also means committed data now lives on fewer replicas; if the ISR drops below min.insync.replicas, acks=all writes are rejected to protect durability.
What metric would alert you that ISRs are unhealthy?
UnderReplicatedPartitions (kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions) being greater than zero indicates partitions whose ISR is smaller than the replication factor — a sign of slow/failing followers, network issues, or insufficient fetcher capacity.

saying these in an interview costs you the question

Saying ISR membership is based on a message-count lag threshold (deprecated replica.lag.max.messages)
Claiming any replica, not just ISR members, can be cleanly elected leader
Forgetting that the leader is itself part of the ISR
Thinking ISR is static rather than dynamically shrinking/expanding

What is the ISR, and how does Kafka decide whether a follower is in-sync or should be removed from it?

§ answer

basics

solid answer

§ go deeper

▾level notes4 levels

▾deep dive~2 min

▾common follow-ups3 questions

▾red flags4 flags

§ context

▾related questions6 questions

▾related topics3 topics

answer

go deeper

context