Replication and Durability

Kafka assigns each partition's replicas by cycling through racks so no two replicas share a rack until it runs out of racks. With replication factor 3 and 3 racks, each replica lands in a different rack; with fewer racks than RF, some replicas double up.

Enable it for high-volume, loss-tolerant streams like metrics or clickstream logs where staying available matters more than a few lost records. Never enable it for financial transactions, payments, or audit logs where losing a committed record is unacceptable — keep those fail-closed.

Broker configs like default.replication.factor and min.insync.replicas set what new auto-created topics get. Each topic can override these at creation or via config alteration. Use safe broker defaults, then override per topic when a workload needs different durability or spans data centers.

The leader epoch is a number that increases by one each time a partition gets a new leader. It lets brokers and clients tell stale leadership apart from current, so followers truncate correctly and old leaders can't corrupt the log.

Unclean leader election lets an out-of-sync replica become leader when no in-sync replica survives. It restores availability but can lose acknowledged data. It's off by default; enable it only when uptime matters more than durability.

Because Kafka acknowledges writes when replicas have the data in memory (not on disk), acknowledged records can be lost only if every in-sync replica loses power at nearly the same time before any of them flushes to disk. That's the correlated power-loss window.

flush.messages and flush.ms tell a broker to fsync its log to disk after a certain number of messages or after a time interval. They're usually left at defaults because Kafka relies on replication for durability, so forcing extra fsyncs just hurts throughput.

Under-replicated partitions with all brokers up means followers can't keep up, not that brokers are down. Check follower I/O (disk, network, GC), look at IsrShrinks/ExpandsPerSec for flapping, inspect which brokers are dropping out, and either fix the bottleneck or tune replica.lag.time.max.ms / replica fetcher threads.

The leader detects a follower exceeding replica.lag.time.max.ms, shrinks the ISR, and persists the new ISR via the controller (AlterPartition in KRaft, ZooKeeper in older versions). When the follower catches back up to the leader's LEO, the leader expands the ISR and re-propagates it.

replica.fetch.max.bytes caps bytes per partition per fetch; replica.fetch.wait.max.ms is the max long-poll wait when no data is ready; num.replica.fetchers sets how many fetcher threads a follower runs per source broker to parallelize replication.

With min.insync.replicas equal to RF, taking any one broker down for maintenance drops the ISR from 3 to 2, below the floor of 3, so all acks=all writes fail. Lowering min.insync.replicas to 2 fixes it while keeping dual-copy durability.

Preferred leader election only changes which existing replica leads. kafka-reassign-partitions.sh actually moves or changes replicas across brokers — needed for adding/removing brokers, fixing skewed data placement, or changing replication factor. It copies data, so it is heavyweight.

acks=all + min.insync.replicas controls write-time durability (rejecting writes when too few replicas are in sync); unclean.leader.election.enable controls failover behavior when all ISR are gone. Together false + acks=all + min.insync.replicas>=2 gives a CP topic that fails closed; unclean=true makes it fail open with possible loss.

Use RF=3, min.insync.replicas=2, acks=all, unclean.leader.election.enable=false on the broker/topic, plus enable.idempotence=true on the producer (with retries and proper delivery.timeout.ms). acks=all/MISR=2 stops loss on the broker side but doesn't prevent producer retries from creating duplicates or unclean election from truncating data.

Strongest durability: acks=all, min.insync.replicas=2 (RF=3), and unclean.leader.election.enable=false. Then a write is acked only after the HW advances past it (committed on >=2 in-sync replicas), and leader-epoch truncation prevents divergence on failover. The trade-off is higher latency and reduced availability when replicas fall behind.

When the ISR shrinks, fewer replicas hold committed data. min.insync.replicas sets the floor: below it, acks=all writes are rejected (availability lost to protect durability). unclean.leader.election lets an out-of-sync replica become leader if ISR is empty — restoring availability but risking data loss.

Map broker.rack to AZs, use RF=3 across 3 AZs with min.insync.replicas=2 and unclean.leader.election=false for durability. Enable KIP-392 fetch-from-follower (RackAwareReplicaSelector + client.rack) to cut cross-AZ read costs. The main tradeoffs are cost vs freshness and balanced AZ sizing.

First weigh data loss vs downtime: if waiting for an ISR replica to return is acceptable, wait. If availability must be restored now and loss is tolerable, force it — temporarily set unclean.leader.election.enable=true (per-topic), or run kafka-leader-election.sh with --election-type UNCLEAN, then revert. Document the loss.

When the old leader comes back as a follower, it may hold uncommitted records past the high watermark that the new leader never had. Using leader-epoch lookups it finds the divergence point, truncates the extra records, then re-fetches from the new leader to catch up and rejoin the ISR.

Often disable auto.leader.rebalance.enable and trigger preferred election deliberately after each restart batch, scoped via JSON and staggered, so leadership churn is controlled. Watch for client metadata storms, imbalance from out-of-sync preferred replicas, and unclean-election durability risks.