Walk through what happens, step by step, when a follower falls out of the ISR and later rejoins. Who decides, and how is the change propagated?

## Who owns ISR membership: the leader The **partition leader** is the authority on which followers are in-sync. It is the only replica that sees every follower's fetch progress, so it is the natural decision-maker. The **controller** (cluster coordinator) is the authority on *persisting and broadcasting* the ISR, not on computing it. ## Step-by-step: shrink (follower falls out) 1. **Tracking.** For each follower, the leader records the last time that follower's fetch reached the leader's **log end offset (LEO)** — the caught-up timestamp. 2. **Periodic check.** The leader runs a scheduled task (default every `replica.lag.time.max.ms / 2`) checking each follower's timestamp against `now`. 3. **Decision.** A follower whose gap exceeds `replica.lag.time.max.ms` (default 30s) is marked out of sync. 4. **Propose the new ISR.** The leader sends an **`AlterPartition`** request to the **controller** (this is the KRaft path; in pre-KRaft/ZooKeeper deployments the leader wrote the new ISR to the partition's state znode in ZooKeeper). 5. **Controller commits.** The controller validates the request against the current **leader epoch** (a monotonically increasing fencing number that rejects stale proposals from a deposed leader), commits the new ISR to the cluster metadata log, and propagates updated `LeaderAndIsr`/metadata to brokers. 6. **High watermark recomputed.** The leader now computes the high watermark over the *smaller* ISR. Consequences: acks=all requests commit once the remaining (fewer) ISR members ack — potentially faster but less redundant; if ISR size drops below `min.insync.replicas`, acks=all producers receive `NotEnoughReplicasException` / `NotEnoughReplicasAfterAppendException`. ## Step-by-step: expand (follower rejoins) 1. The ejected follower keeps (or resumes) fetching from the leader. 2. It replicates the backlog until its fetch finally requests the leader's **current** LEO — i.e., it is fully caught up. 3. The leader proposes an **ISR expansion** via the same `AlterPartition` path; the controller commits it and bumps metadata. 4. The high watermark and durability guarantees return to full redundancy. Important: a rejoining follower must catch up to the *moving* LEO, not a frozen snapshot — under heavy write load this can take a while, and the follower stays out until it converges. ## Fencing with leader epoch Every ISR change is tagged with the **leader epoch**. If a stale leader (e.g., one that was network-partitioned and superseded) tries to alter the ISR, the controller rejects the request because its epoch is behind. This prevents split-brain ISR corruption. ## Observability Key JMX metrics: - **`IsrShrinksPerSec`** / **`IsrExpandsPerSec`** — rate of ISR membership changes; sustained nonzero values signal flapping followers. - **`UnderReplicatedPartitions`** — partitions where ISR size < replication factor; should be 0 in steady state. - **`UnderMinIsrPartitionCount`** — partitions whose ISR is below min.insync.replicas (writes failing). ## Edge cases - **Flapping:** a follower hovering near the threshold can repeatedly shrink/expand, churning metadata. Tune `replica.lag.time.max.ms` up or fix the underlying slowness (disk, network, GC). - **Leader failure during shrink:** if the leader dies mid-decision, the controller elects a new leader from the *committed* ISR, never from a follower that was never in-sync (assuming unclean election is disabled).