You must guarantee no acknowledged message is ever lost or silently duplicated end-to-end. Which producer, topic, and broker settings do you combine, and why is acks=all/MISR=2 alone insufficient?

**Goal.** 'No acknowledged message lost, none silently duplicated' is a *pipeline* property; each component has a failure mode that the others don't cover. **Layer 1 — broker/topic durability (prevents loss of accepted writes).** - `acks=all`: producer waits for the full ISR before considering the write done. - `min.insync.replicas=2` (with RF=3): a write is only accepted while >=2 replicas are in sync, so a committed record is on at least two brokers and survives one broker death. - `unclean.leader.election.enable=false`: when all ISR members for a partition are down, the partition goes offline rather than electing a behind replica. This closes the truncation hole — otherwise a stale leader could discard records that acks=all had already acknowledged. **This is why acks=all + MISR=2 alone is insufficient: with unclean=true, the worst case still loses committed data.** **Layer 2 — producer idempotence (prevents silent duplicates).** Durability *requires* the producer to retry on transient failures (NotEnoughReplicas, leader change, timeout). A naive retry can write the same record twice if the first attempt actually succeeded but the ack was lost. `enable.idempotence=true` solves this: the broker assigns the producer a **Producer ID (PID)** and tracks a monotonic **sequence number** per (PID, partition). Duplicate or out-of-order batches are rejected by the broker, giving exactly-once *delivery to a partition* and preserving ordering even with retries. Idempotence also implicitly sets acks=all, retries>0, and max.in.flight.requests.per.connection<=5 — so enabling it is the second half of 'no loss, no dup.' It is the default in clients since Kafka 3.0. **Layer 3 — config consistency.** Keep `retries` effectively unbounded (idempotence sets Integer.MAX_VALUE) and tune `delivery.timeout.ms` to bound total retry time; if it expires during an ISR outage the send fails (surfaced to the app) rather than being silently dropped — the app must handle that error to remain truly lossless. **Layer 4 — exactly-once across stages (optional, for read-process-write).** Idempotence covers a single producer to single partitions. For atomic writes spanning multiple partitions/topics, or to tie consuming + producing into one atomic unit (e.g. Kafka Streams EOS), use **transactions**: set a `transactional.id`, wrap sends in beginTransaction/commitTransaction, commit consumer offsets within the transaction, and have downstream consumers use `isolation.level=read_committed` so they never see aborted records. **Putting it together (typical lossless producer):** - Broker/topic: RF=3, min.insync.replicas=2, unclean.leader.election.enable=false. - Producer: acks=all, enable.idempotence=true, retries=MAX, delivery.timeout.ms sized for tolerable retry window, max.in.flight<=5. - Consumer (for EOS): read_committed; for read-process-write, transactions. **Edge cases.** (1) Idempotence is per producer *session*; a producer restart gets a new PID, so cross-restart dedup needs transactions or app-level keys. (2) If the app ignores a failed send after delivery.timeout.ms, you reintroduce loss — error handling is part of the guarantee. (3) Disk-level durability (`flush`) is usually left to replication rather than fsync-per-message; relying on replication is the standard model, so losing power to multiple brokers simultaneously is the residual risk RF mitigates. (4) MISR=2 with RF=3 means a second concurrent failure stops writes (availability), which is the deliberate price of the durability guarantee.