Replication and Durability

Kafka promotes one of the in-sync follower replicas to be the new leader. Clients get an error, refresh metadata, find the new leader's broker, and resume producing/consuming there. No manual action is needed.

After a successful write the data is in the broker's RAM (the OS page cache), and with acks=all also in the RAM of other replicas. It isn't necessarily on the physical disk yet — the operating system writes it to disk a bit later.

LEO is the offset just past the last message written to the log. HW is the highest offset that has been replicated to all in-sync replicas. Consumers can only read up to (below) the HW, so they never see un-replicated records.

The ISR is the subset of a partition's replicas (leader plus followers) that are fully caught up with the leader's log. Assigned replicas (AR) are all replicas; ISR is the healthy, up-to-date ones eligible to serve and be elected leader.

Each partition has one leader replica that handles all produce and consume requests. The other replicas are followers; they only copy the leader's data and stand by to take over if the leader fails.

min.insync.replicas sets the minimum number of in-sync replicas that must acknowledge a write before the broker accepts it. It only takes effect when the producer uses acks=all; otherwise the broker ignores it.

The preferred leader is the first broker listed in a partition's replica assignment (the assigned-replica list). Kafka tries to make it the leader so leadership is spread evenly across brokers and no single broker is overloaded.

broker.rack is a per-broker string tagging which rack or availability zone a broker is in. Kafka uses these tags to spread a partition's replicas across different racks, so one rack failing doesn't take down all copies.

Unclean leader election lets Kafka promote a replica that is NOT fully up to date (out of the in-sync set) to leader when no in-sync replica is available. The unclean.leader.election.enable flag turns this on or off. It trades possible data loss for availability.

If min.insync.replicas equals RF, every replica must be in sync to accept writes, so losing or lagging just one broker stops all writes. Setting it to RF-1 (e.g. 2 with RF=3) keeps you durable while tolerating one failure.

acks=all makes the producer wait until all in-sync replicas have the record. min.insync.replicas sets how many replicas must be in sync for that write to be allowed. With replication factor 3, set min.insync.replicas=2 and acks=all so you can lose one broker without losing data and without halting writes.

Kafka acknowledges writes once enough replicas have the data in memory, not once it's flushed to disk. Replication across machines is faster and safer than a per-message fsync, which would be slow and still vulnerable to a single disk failure.

A follower stays in the ISR if it fetched up to the leader's latest offset within replica.lag.time.max.ms (default 30s). Miss that window and the leader removes it; catch back up and it rejoins. The old count-based check (lag in messages) misjudged bursty traffic, so it was replaced by time.

The ISR (in-sync replica set) is the leader plus all followers caught up with it. A follower is dropped from the ISR if it hasn't fetched up to the leader's log end within replica.lag.time.max.ms, and re-added once it catches back up.

Each follower broker runs ReplicaFetcherThreads. A thread sends Fetch requests to the leader asking for records starting at the follower's current log-end offset, appends what it gets to its local log, and advances its offset — repeating continuously.

The leader rejects new acks=all writes with NotEnoughReplicasException, so producers fail and retry. The partition becomes read-still-available but write-blocked until enough followers rejoin the ISR.

auto.leader.rebalance.enable (default true) lets the controller automatically run preferred leader election. The check interval sets how often it inspects imbalance, and the per-broker percentage is the imbalance threshold that triggers a rebalance for a broker.

It decides whether a partition can elect a leader from a replica that was NOT in the in-sync set when all in-sync replicas are down. Enabled = stay available but possibly lose data; disabled (default) = stay consistent but the partition goes offline until an in-sync replica returns.

On rejoining, a follower truncated everything above its high watermark, assuming records above the HW were uncommitted. But because the follower's HW lags, this could throw away records that were actually committed, or let two replicas keep different records at the same offset — causing data loss or log divergence.

Each leader is assigned a monotonically increasing leader epoch, and the log records which epoch produced each offset range (the leader-epoch cache). On becoming a follower, the broker asks the leader, via OffsetsForLeaderEpoch, for the end offset of its last known epoch, and truncates exactly there — the true point where the logs diverge — instead of guessing with the HW.

A record appended on the leader is only 'committed' once all in-sync followers have replicated it. The high watermark marks the highest committed offset; consumers can only read up to it, so un-replicated tail records stay invisible.

RF=3/min.isr=2 lets you survive one broker failure while still accepting writes and keeping two durable copies. RF=3/min.isr=3 blocks writes on any single failure; RF=2/min.isr=2 blocks writes on any single failure and has no spare copy.

KIP-392 lets a consumer read from a follower replica in its own rack/AZ instead of always from the leader. You set replica.selector.class on the broker and client.rack on the consumer. It cuts cross-AZ network cost and latency.

Each partition keeps 2 of its 3 replicas because they're in different AZs. Leaders that were in the dead AZ fail over to surviving replicas, ISR drops to 2 which still meets min.insync.replicas=2, so acks=all producers and consumers keep working with no data loss.

The lagging replica becomes leader at its own (shorter) log end. Records the old leader had committed but this replica never fetched are gone. When the old replica returns, it must truncate its longer log down to the new leader's offset to re-join, throwing away those records.

The producer's send to the old leader fails with a retriable error like NOT_LEADER_OR_FOLLOWER. The client marks its metadata stale, asks any broker for fresh metadata, learns the new leader, and retries automatically within delivery.timeout.ms.

Followers fetch from the leader. Each Fetch response carries the leader's current HW. The follower applies it after appending the fetched data. Because the HW arrives in the next response, a follower's HW always trails the leader's by one fetch round-trip.

min.insync.replicas is a broker default that can be overridden per topic. The full contract needs three parts: replication factor at topic creation, min.insync.replicas as a topic config, and acks=all on the producer.

Run kafka-leader-election.sh with --election-type PREFERRED to move leadership back to preferred replicas. PREFERRED only elects in-sync replicas (safe). UNCLEAN can elect an out-of-sync replica to restore availability, risking data loss.