What does idempotence guarantee versus what it does NOT, and when do you need transactions instead?
answer
- idempotence = per-partition, per-session
- no cross-partition / cross-session atomicity
- transactions = transactional.id + initTransactions
- epoch fencing kills zombies (ProducerFencedException)
- read_committed + __transaction_state coordinator
basics
~10 sIdempotence guarantees exactly-once writes to a single partition within one producer session. It does NOT give cross-partition atomicity, cross-session deduplication, or atomic consume-process-produce. Those require transactions (transactional.id, initTransactions, begin/commit).
solid answer
~50 sIdempotence is the narrow guarantee: for one producer session, retried batches are deduplicated per partition, so each record lands exactly once on its partition. Its boundaries: (1) per-partition only — a single send() to multiple partitions isn't atomic; (2) per-session only — a producer crash/restart gets a fresh PID, so duplicates across restarts aren't prevented; (3) no read-process-write atomicity. When you need to atomically write to multiple partitions/topics, or atomically commit consumer offsets together with output records (the consume-transform-produce pattern in Kafka Streams or stream processors), you need transactions: set a stable transactional.id, call initTransactions(), then beginTransaction()/commitTransaction()/abortTransaction(). Transactions build on idempotence (they require it) and add a transaction coordinator, producer epoch fencing of zombie producers, and the read_committed isolation level on consumers so they only see committed records. Idempotence alone = exactly-once-per-partition; transactions = exactly-once end-to-end (EOS).
go deeper
Know idempotence stops retry duplicates on one partition.
State the three things idempotence does not cover.
Draw the idempotence-vs-transactions boundary and name the transactional API and read_committed.
Explain epoch fencing, the transaction coordinator/__transaction_state, and choose the right guarantee for a given pipeline (e.g., Kafka Streams exactly_once_v2).
## Two layers of exactly-once Kafka's exactly-once story has two layers, and conflating them is the most common interview mistake. ### Layer 1 — Idempotent producer (enable.idempotence=true) Guarantees: a retried batch is written **exactly once to its partition**, for the **lifetime of one producer instance (session)**. Does NOT guarantee: - **Cross-partition atomicity.** If `send()` fans out to 3 partitions and the app crashes after 1 succeeds, the other 2 may not be written — there's no all-or-nothing. - **Cross-session dedup.** A crash/restart yields a **new PID**, so records re-sent by application logic after restart are new records, not recognized duplicates. - **Consume-process-produce atomicity.** Reading from input, processing, and writing output + committing the input offset are independent; a crash between them causes reprocessing/duplicates downstream. ### Layer 2 — Transactions (Exactly-Once Semantics, EOS) When you need atomicity across partitions/topics or atomic offset commits, use **transactions**: ```java props.put("transactional.id", "orders-processor-1"); // stable, identity-bound KafkaProducer<K,V> p = new KafkaProducer<>(props); p.initTransactions(); // fences previous producer with same id try { p.beginTransaction(); p.send(rec1); p.send(rec2); // to any partitions/topics p.sendOffsetsToTransaction(offsets, groupMeta); // atomically commit input offsets p.commitTransaction(); } catch (KafkaException e) { p.abortTransaction(); } ``` Transactions add: - **transactional.id** — a stable identity that survives restarts; the **transaction coordinator** persists state in the `__transaction_state` topic. - **Producer epoch fencing** — on `initTransactions()` the epoch is bumped, so a **zombie** (old, hung) instance with the same `transactional.id` is **fenced** (its writes are rejected with ProducerFencedException). This is the cross-session protection idempotence alone lacks. - **Atomic multi-partition commit** — all records and offsets in the transaction become visible together (or not at all) via control records (commit/abort markers). - **read_committed consumers** — set `isolation.level=read_committed` so consumers skip aborted records and don't read past an open transaction (the Last Stable Offset). ## Decision rule - Single producer just wants no duplicate retries → **idempotence** (default, free, low overhead). - Need all-or-nothing across multiple partitions/topics, or exactly-once stream processing (Kafka Streams `processing.guarantee=exactly_once_v2`) → **transactions**. ## Why idempotence is a prerequisite Transactions **require** idempotence (the transactional producer turns it on implicitly), because the same PID + sequence machinery is what makes the per-partition writes inside a transaction deduplicated; transactions then layer atomicity and fencing on top.
- How do transactions prevent a zombie (restarted-but-old) producer from writing?On initTransactions() the coordinator bumps the producer epoch for that transactional.id. The old instance's epoch is now stale, so its writes are fenced with ProducerFencedException.
- What consumer setting completes the exactly-once picture?isolation.level=read_committed, so consumers only read committed transactional records and stop at the Last Stable Offset, skipping aborted ones.
- Does a transactional producer still need enable.idempotence?It's implied/required — transactions are built on the idempotent PID+sequence machinery, so enabling a transactional.id turns idempotence on automatically.
saying these in an interview costs you the question
- Saying idempotence gives end-to-end exactly-once — it's only per-partition, per-session.
- Claiming idempotence survives producer restarts.
- Confusing idempotence with transactions (transactional.id, commit/abort).
- Forgetting read_committed on the consumer side when claiming EOS.
- Saying transactions don't require idempotence.