At a system-design level, how do stream-processing frameworks achieve 'exactly-once' processing guarantees for stateful aggregations, tying together offset commits, state updates, and output writes - and what breaks that guarantee?

## The default is at-least-once Stream processing frameworks default to **at-least-once** semantics: on a crash, some records get reprocessed, and if the processing has any side effect (incrementing a counter, appending to a downstream topic), that side effect can happen more than once for the same input record. For many use cases this is tolerable, or made tolerable by making the downstream effect idempotent (e.g., upserting by key instead of incrementing). But some workloads - financial aggregation, billing, exactly-metered usage counting - genuinely need each input event to affect the result exactly once, no more, no less, even across crashes and rebalances. ## The three operations that must become atomic The engineering answer is to make the three operations that constitute 'processing one record' atomic with each other: - **(1)** advancing the consumer offset past the input record; - **(2)** updating the operator's internal state (say, an aggregation's running total, written to its changelog); - **(3)** producing any output record downstream. Left independent, any one of these three can succeed while another fails on a crash, and that partial completion is exactly what breaks exactly-once. ## Two mechanisms: Kafka Streams and Flink | Kafka Streams | Flink | |---|---| | A single Kafka transaction: the changelog write, the output-topic write, and the offset commit, via an idempotent, transactional producer. | Coordinated distributed checkpoints combined with a two-phase-commit sink protocol for exactly-once output to systems that support it (including Kafka). | **Kafka Streams** solves this using Kafka's transactional producer/consumer protocol: the framework wraps the changelog write, the output-topic write, and the offset commit into a single Kafka transaction using an idempotent, transactional producer identified by a stable transactional ID per task. Kafka's transaction coordinator ensures all writes in that transaction become visible atomically - either every one of them is committed, or (if the task crashes before completing) the entire transaction is aborted and none of the writes ever become visible to a consumer configured to read only committed data. On restart, the task resumes from the last offset that was part of a successfully committed transaction, reprocessing the input it never actually finished, exactly as if the failed attempt had never started - no double-counting, no gaps. **Flink** achieves a similar guarantee through a different mechanism: coordinated distributed checkpoints (via a barrier-based protocol) that capture a globally consistent snapshot of both operator state and source-offset position across the whole pipeline, combined with a two-phase-commit sink protocol for exactly-once output to systems that support it (including Kafka). On failure, the entire job rolls back to the last completed checkpoint and reprocesses from there, with the two-phase-commit sink ensuring partially-written output from a failed attempt never becomes visible. ## The trade-off The trade-off for all of this is real: transactional writes and coordinated checkpoints add meaningful throughput and latency overhead compared to the plain at-least-once path. - Kafka transactions add commit-protocol round trips and increase end-to-end latency (commonly tuned via a commit-interval setting, trading off latency against transaction overhead). - Flink's checkpointing similarly adds periodic synchronization pauses and I/O for durable snapshots. You're deliberately paying throughput and latency to buy correctness, and for a large share of streaming workloads - dashboards, approximate metrics, anything already tolerant of occasional duplicates - that cost isn't justified, so at-least-once with idempotent sinks remains the pragmatic default. ## Where the guarantee ends The scope limit that principals need to internalize is where this guarantee actually ends: it holds strictly within the **transactional boundary** the framework controls - Kafka topics in, Kafka topics (or a two-phase-commit-capable sink) out. The instant a stateful operator's processing has a side effect the framework doesn't control transactionally, that side effect reverts to at-least-once (or worse, at-most-once if it happens before the commit) regardless of how the internal Kafka-to-Kafka path is configured. Such side effects: - calling an external HTTP API; - writing to a non-transactional database; - sending an email. A team that configures exactly-once semantics on their Kafka Streams topology and then calls a third-party payment API as a side effect inside a processor has not actually achieved exactly-once payment calls; they've achieved exactly-once for the Kafka-internal bookkeeping and left the payment call exposed to duplication on retry, unless that call is independently made idempotent (e.g., via an idempotency key). ## Two further failure modes Two further failure modes matter operationally. 1. **First, 'zombie fencing'.** If a task is presumed dead (e.g., after a rebalance timeout) but is actually still alive and slow, it must be fenced off from completing its in-flight transaction so it can't commit stale output after a new instance has already taken over its partitions and started producing - Kafka's transactional protocol handles this via **producer epochs** that invalidate an older, stale transactional producer. 2. **Second, deliberate replay.** Exactly-once guarantees don't survive a manual, out-of-band offset reset for reprocessing - deliberately rewinding offsets to reprocess history is, by definition, intentionally reprocessing, and any exactly-once bookkeeping only guarantees the original run was exactly-once, not that a manual replay produces disjoint output from the first pass; operators doing a deliberate replay for reprocessing need a separate strategy (like writing to a new output topic/table version) to avoid duplicating already-committed results. ## Where it shows up A concrete example: a billing system aggregating metered API usage per customer per billing period uses Kafka Streams with exactly-once semantics enabled end-to-end, reading usage events and writing running totals to an output topic that a billing service consumes - guaranteeing no customer is ever double-billed or under-billed due to a crash or rebalance during aggregation, at the cost of somewhat higher latency between an event occurring and its usage total updating, which the team accepted as the correct trade-off given the financial stakes.