As a principal engineer, how do acks, min.insync.replicas, the high watermark, and leader epochs together determine whether an acknowledged write can ever be lost? What configuration gives the strongest durability and what are the trade-offs?

## The durability chain Whether an *acknowledged* write can be lost depends on four interacting controls: ### 1. acks (producer) - `acks=0`: fire and forget — lost on any failure. - `acks=1`: acked when the **leader** appends (before the HW advances). If the leader dies before a follower replicates that record, it is lost even though the producer got a success. - `acks=all` (a.k.a. `-1`): the producer is acked only when the record's offset has been **committed**, i.e., is below the HW — replicated to **all replicas currently in the ISR**. This is the prerequisite for no-loss. ### 2. min.insync.replicas (topic/broker) `acks=all` alone is weak if the ISR has shrunk to just the leader (then 'all ISR members' = 1). `min.insync.replicas=N` makes a Produce with acks=all **fail with NotEnoughReplicas** unless at least N replicas are in sync. With **RF=3 and min.insync.replicas=2**, every committed record lives on >=2 brokers, so any single broker loss is survivable. Setting it equal to RF maximizes durability but means *any* replica outage stops writes. ### 3. High watermark The HW is the mechanism that *defines* 'committed' for acks=all: a record is acknowledged precisely when the HW passes its offset. It also guarantees consumers never read past it, so no consumer ever observes a record that could later be lost. ### 4. Leader epochs (KIP-101/279) Even with acks=all and min.insync.replicas=2, the **pre-KIP-101** HW-based truncation could silently lose or diverge committed records during failover. Leader-epoch-based truncation closes that hole by truncating to the actual lineage divergence point. This is the part many candidates forget: the producer/ack settings guarantee a record is *committed*, but **epoch-based recovery is what keeps it committed across leader changes.** ### 5. unclean.leader.election.enable The last gap: if all in-sync replicas are down and `unclean.leader.election.enable=true`, Kafka may elect an **out-of-sync** replica (one missing committed records) as leader to restore availability — losing committed data. Set it to **false** to forbid this (the partition stays offline until an in-sync replica returns). This is the CP-vs-AP knob. ## The strongest-durability recipe - Replication factor 3 - `acks=all` - `min.insync.replicas=2` - `unclean.leader.election.enable=false` - `enable.idempotence=true` (avoids duplicates from retries; default true in modern clients) and bounded/inf retries With this, a committed (acked) record survives any single-broker failure, is never re-ordered or duplicated, and is never dropped or diverged on failover. ## Trade-offs - **Latency:** acks=all waits for replication round-trips; p99 rises, especially across racks/AZs. - **Availability:** if the ISR falls below min.insync.replicas (e.g., two of three brokers slow/down), the partition rejects writes (NotEnoughReplicas). You traded availability for consistency. - **Throughput:** stronger settings + idempotence cap in-flight tuning (max.in.flight.requests.per.connection<=5 with idempotence). - **Operational:** min.insync.replicas=RF is fragile — a single rolling restart can stall producers; min.insync.replicas=2 with RF=3 is the common balance. ## Common pitfalls - Setting acks=all but leaving min.insync.replicas=1 → 'all ISR' can be just the leader → false sense of safety. - Leaving unclean.leader.election.enable=true on a critical topic. - Assuming consumer-visible (below HW) implies producer-durable under acks=1 — it does not; acks=1 records can be lost while still never being consumer-visible before commit.