As an architect, how do you choose among at-most-once, at-least-once (with idempotent consumers), and exactly-once for a given pipeline, and what are the trade-offs?

## The decision is economic, not religious The right guarantee is the cheapest one that meets correctness needs. Frame it as two costs: - **Cost of loss**: what breaks if a record is silently dropped? - **Cost of a duplicate**: what breaks if a record is processed twice? ## At-most-once — when loss is cheaper than a duplicate Pick when duplicates are actively harmful or pointless and occasional gaps are fine, and you want the lowest latency / highest throughput. Examples: best-effort metrics, sampled telemetry, ephemeral signals where a stale resend is worse than a missing point. Achieved with commit-before-process and/or `acks=0`/no retries. Trade-off: you *will* lose data under failure — only acceptable when that data is non-critical. ## At-least-once + idempotent consumers — the pragmatic default Most real pipelines land here. You guarantee no loss (process-then-commit, `acks=all`, retries) and make duplicates harmless by **idempotent processing**: - **Upsert by business key** (insert-or-update) so reprocessing converges to the same state. - **Dedup table** keyed by `(topic, partition, offset)` or a producer-supplied idempotency key. - **Idempotent downstream APIs** (PUT-like semantics, idempotency-Key headers). Why it's often best: simpler operationally than EOS, lower latency, and — critically — it **works across external systems** (databases, HTTP services) that Kafka transactions cannot enroll. The dedup cost is paid at the sink, where you have full control. ## Exactly-once — when both loss and duplicates are unacceptable AND it's Kafka-to-Kafka Use EOS (`processing.guarantee=exactly_once_v2` in Streams, or the transactional producer API) for stateful Kafka-internal transforms where correctness is non-negotiable: running balances, exactly-once aggregations, dedup-sensitive joins. Costs to budget: - **Latency**: read_committed consumers buffer until commit markers; transaction commit adds round-trips. Tune `commit.interval.ms` (in Streams) to trade latency vs throughput. - **Throughput**: transaction coordination and markers add overhead vs plain produce. - **Operational complexity**: `transactional.id` management, fencing, coordinator load, hanging-transaction edge cases (`transaction.timeout.ms`). - **Scope limit**: covers only Kafka topics + offsets. Any external side effect needs its own idempotency / outbox — EOS does **not** magically make a Postgres write exactly-once. ## The classic anti-pattern Teams reflexively ask for exactly-once. Often the real requirement is 'the final state is correct,' which **at-least-once + idempotent sink** satisfies more cheaply and with broader reach. Reserve EOS for cases where you genuinely cannot make the sink idempotent and both loss and duplicates are intolerable, and the data path is inside Kafka. ## Quick decision guide | Need | Choose | |---|---| | Loss tolerable, dupes harmful, min latency | at-most-once | | No loss; dupes manageable via idempotent sink; external systems involved | at-least-once + idempotency (default) | | No loss AND no dupes; Kafka→Kafka transform; correctness critical | exactly-once (transactions/Streams) | ## Cross-cutting reminders - Durability config (`acks=all`, `min.insync.replicas>=2`, RF>=3) underpins *all* no-loss options — without it even at-least-once can lose data on broker failure. - Ordering, partitioning by key, and consumer rebalance behavior interact with whichever guarantee you pick.