skip to content

Which producer configs does enable.idempotence require, and what happens if you set a conflicting value?

level: middleimportance: should knowfreq 58%

answer

  1. acks=all, retries>0, in-flight<=5
  2. unset -> auto-filled defaults
  3. conflict -> ConfigException at construct
  4. delivery.timeout.ms is the real bound
  5. ordering preserved even with 5 in-flight

basics

~10 s

Idempotence requires acks=all, retries>0, and max.in.flight.requests.per.connection<=5. If you explicitly set a conflicting value (e.g., acks=1), the producer fails fast with a ConfigException.

solid answer

~40 s

When enable.idempotence=true, the producer needs three things to make dedup work: acks=all (so an acknowledged write is durable on all in-sync replicas and never silently lost), retries>0 (otherwise there are no retries to deduplicate), and max.in.flight.requests.per.connection<=5 (the broker only caches the last five batches' sequence metadata for dedup). If you don't set these, the producer auto-fills compatible defaults (acks=all, retries=Integer.MAX_VALUE, in-flight=5). But if you explicitly set a conflicting value such as acks=0 or acks=1, retries=0, or in-flight=6+, the producer throws a ConfigException at construction time rather than silently disabling the guarantee. This fail-fast behavior protects you from thinking you have exactly-once-per-partition when you don't.

go deeper

for a junior

Remember acks=all and bounded in-flight are needed.

for a middle

List all three required configs and that conflicts throw ConfigException.

for a senior

Explain auto-fill vs fail-fast and that delivery.timeout.ms is the practical retry bound.

for a principal

Discuss the historical silent-downgrade behavior and ordering guarantees with in-flight up to 5.

## The required trio `enable.idempotence=true` only works if three other settings are compatible: | Config | Required value | Why | |---|---|---| | `acks` | `all` (a.k.a. `-1`) | Idempotence dedups retries, but if the write itself can be lost (acks=0/1), there's nothing durable to dedup against. `all` means every in-sync replica acknowledged. | | `retries` | `> 0` | The whole point is to make **retries** safe; with `retries=0` there are no retries, so the producer would fail on the first transient error instead of retrying idempotently. | | `max.in.flight.requests.per.connection` | `<= 5` | The broker caches sequence metadata for only the last 5 batches per producer-partition. More outstanding requests could let a retried duplicate escape the dedup window, and could also break ordering. | ## Auto-fill vs. fail-fast Two distinct behaviors: - **You leave them unset** → the producer **auto-configures** compatible defaults: `acks=all`, `retries=Integer.MAX_VALUE` (bounded in practice by `delivery.timeout.ms`, default 120000 ms), `max.in.flight.requests.per.connection=5`. - **You explicitly set a conflicting value** → the producer **throws `ConfigException`** when you construct it. Examples that fail: `acks=1` (or `0`), `retries=0`, `max.in.flight.requests.per.connection=6`. Note a historical subtlety: in older Kafka (pre-3.0) idempotence was **off** by default, and if you turned it on while leaving a conflicting `acks` it would sometimes silently downgrade. Modern clients **fail fast** so you can't accidentally lose the guarantee. ## delivery.timeout.ms matters Because `retries` is effectively unlimited, the real bound on how long the producer keeps retrying a record is `delivery.timeout.ms` (default 2 minutes), which caps the total time from `send()` to success/failure including all retries and `request.timeout.ms`. Tuning idempotent producers is usually about `delivery.timeout.ms`, not `retries`. ## Ordering bonus With idempotence on, even with up to 5 in-flight requests, the producer **preserves ordering** on retries — because sequence numbers let the broker reject out-of-order batches, the client re-sends in the correct order. Without idempotence, in-flight > 1 plus retries can reorder records.

  • If retries defaults to Integer.MAX_VALUE, what actually bounds retrying?
    delivery.timeout.ms (default 120000 ms) caps total time from send() to terminal success/failure, including all retries and request timeouts.
  • Can idempotence preserve ordering with multiple in-flight requests?
    Yes. With idempotence on, up to 5 in-flight requests still keep per-partition ordering because sequence numbers let the broker reject out-of-order batches and the client re-sends correctly.

saying these in an interview costs you the question

  • Saying acks=1 is fine with idempotence — it throws ConfigException.
  • Claiming you must set retries to a specific finite number — it defaults to Integer.MAX_VALUE and is bounded by delivery.timeout.ms.
  • Saying the producer silently disables idempotence on conflict — modern clients fail fast.
  • Stating in-flight must be 1 — it can be up to 5 and still keep ordering with idempotence.

context

resolving links…