When building an exactly-once read-process-write pipeline on Kafka, why is isolation.level=read_committed required on the consuming side, and what breaks if you forget it?

## What exactly-once on Kafka actually is Kafka's exactly-once semantics (EOS) for stream processing is a **chain** of two cooperating mechanisms: 1. **Transactional producers** (a `transactional.id` + the idempotent producer) that write output records and the consumer's input offsets **atomically**. The standard read-process-write loop is: poll input, process, `producer.sendOffsetsToTransaction(offsets, consumerGroupMetadata)` to record how far the input was consumed *inside the same transaction* as the outputs, then `commitTransaction()`. Commit makes the outputs and the offset advance visible together; abort makes neither visible. 2. **read_committed consumers** downstream, which by LSO gating + aborted-transaction filtering only ever observe committed outputs. These two halves are both necessary. The producer side gives *atomic write of outputs + offsets*; the consumer side gives *no observation of non-committed data*. EOS is the composition. ## Why read_committed is mandatory on the consuming side Suppose stage A transactionally produces to topic T but a particular transaction **aborts** (crash, validation failure, explicit rollback). The records were physically written to T before the abort. If stage B reads T with the default **read_uncommitted**, it will: - read those aborted records, - process them, - and produce derived effects (writes, side effects, its own outputs). Now a unit of work that A rolled back has nonetheless propagated downstream. The atomicity A paid for is undone one hop later. You observe **duplicates or phantom records** — precisely the failure EOS is supposed to eliminate. The same happens for *uncommitted-but-still-open* transactions: B might process records that are later aborted. So read_committed is not an optimization; it is a **correctness requirement** for the consuming end of every EOS hop. ## What you must set, end to end - Producer: `enable.idempotence=true` (implied with a transactional.id), a unique-and-stable `transactional.id`, `acks=all`. Use `beginTransaction`/`sendOffsetsToTransaction`/`commitTransaction`. - Consumer: **`isolation.level=read_committed`**, and `enable.auto.commit=false` (offsets are committed through the producer's transaction, not by the consumer auto-commit). - Broker: `transaction.state.log.replication.factor` and `transaction.state.log.min.isr` sized for durability (commonly 3/2 in production). ## Kafka Streams does this for you With **`processing.guarantee=exactly_once_v2`**, Kafka Streams automatically configures transactional producers, disables consumer auto-commit, and sets `isolation.level=read_committed` on its internal consumers. Hand-rolled consumer/producer pipelines get none of this automatically — forgetting read_committed is a classic silent EOS bug because everything *appears* to work until a transaction aborts under failure. ## The subtle failure mode The bug is silent in the happy path: with no aborts, read_uncommitted and read_committed deliver the same records, tests pass, demos work. It only manifests under the exact failure conditions EOS exists to handle — producer crashes, rollbacks, rebalances during a transaction — making it a dangerous omission to catch only in production. ## Costs you accept In exchange you accept LSO-gating latency (records visible at commit, not produce) and head-of-line blocking from long transactions — so EOS pipelines favor short, frequent transactions and tuned `transaction.timeout.ms`.