Design an end-to-end no-data-loss producer/topic configuration. Beyond acks=all, what settings are required and what failure modes remain?

## Goal 'No data loss' means: once the producer's `send()` reports success, the record is never lost despite broker failures. Achieving it requires aligning producer, topic, and broker settings — `acks=all` alone is insufficient. ## Producer settings - **`acks=all`**: wait until the record is committed across the ISR (see committed-vs-leader question). Mandatory. - **`enable.idempotence=true`** (default since 3.0): the producer gets a producer ID and tags each record with a monotonic sequence number per partition. Brokers dedup retries and detect gaps, giving **exactly-once-into-the-log** and ordering even with retries. It *requires* `acks=all`, `retries>0`, and `max.in.flight.requests.per.connection ≤ 5`; the client enforces these. - **`retries`** (effectively ∞ by default) and **`delivery.timeout.ms`** (default 120000): bound total delivery time. Set high enough that a rolling restart or brief ISR shrink (which yields retriable NotEnoughReplicas) doesn't fail the send. - **Application discipline**: never ignore the `Future`/callback. With async `send()`, a swallowed exception is silent loss. Block on critical sends or handle the callback and propagate failures. ## Topic settings - **`replication.factor=3`**: three copies per partition. - **`min.insync.replicas=2`**: under acks=all, require 2 in-sync copies. Tolerates 1 broker down while still accepting writes; every committed record is on ≥2 brokers. ## Broker settings - **`unclean.leader.election.enable=false`** (default false in modern Kafka): forbids electing a replica that is **not** in the ISR as leader. An unclean election would promote an out-of-date replica whose log is behind the HW, **truncating committed records** — direct data loss. Keeping this false means that if no ISR replica is available the partition goes offline rather than losing data (consistency over availability). ## Rack/AZ awareness - **`broker.rack`** + rack-aware assignment spreads the 3 replicas across racks/availability zones, so a single-AZ outage doesn't take all replicas of a partition. ## Remaining failure modes (honest limits) 1. **All ISR replicas lost simultaneously** (e.g. correlated power/AZ failure of the brokers holding a partition) — the committed record is gone. Mitigate with cross-AZ/rack placement and, for the highest tier, geo-replication (MirrorMaker 2 / Cluster Linking) — but cross-cluster replication is async and has its own RPO. 2. **Producer-side crash before send completes** — records buffered in the client (not yet acknowledged) are lost. Mitigate with synchronous critical sends or an outbox pattern. 3. **Consumer-side gaps** — true end-to-end no-loss requires the consumer to **commit offsets only after processing succeeds** (at-least-once) or use transactions/exactly-once. Auto-commit before processing can drop records logically even if Kafka never lost them. 4. **Disk corruption / fsync timing** — Kafka relies on the OS page cache and flushes lazily; a simultaneous power loss of multiple brokers before flush can lose recently-committed data. Replication across machines is the primary defense, not per-broker fsync. ## The complete recipe (one line) Producer: `acks=all, enable.idempotence=true, delivery.timeout.ms` tuned + handle callbacks. Topic: `RF=3, min.insync.replicas=2`. Broker: `unclean.leader.election.enable=false`, rack-aware. Consumer: commit after process.