skip to content

How can max.in.flight.requests.per.connection cause records to be reordered within a partition, and how does idempotence fix it?

level: seniorimportance: must knowfreq 65%

answer

  1. in-flight 5 + retry = reorder risk
  2. Old fix: max.in.flight=1
  3. Idempotence: PID + per-partition seq nums
  4. enable.idempotence default since 3.0
  5. idempotence requires max.in.flight<=5, acks=all

basics

~10 s

With multiple in-flight batches and retries, a retried earlier batch can land after a later one, reordering within a partition. Enabling the idempotent producer preserves order even with up to 5 in-flight requests.

solid answer

~40 s

max.in.flight.requests.per.connection controls how many unacknowledged produce requests a producer sends to a single broker connection concurrently (default 5). Without idempotence, if batch 1 fails and is retried while batch 2 already succeeded, the retried batch 1 is appended after batch 2 — reordering records within the same partition even though they share a key. Historically the fix was max.in.flight=1 (at a throughput cost). The modern fix is the idempotent producer (enable.idempotence=true, default since Kafka 3.0): each producer gets a PID and per-partition monotonic sequence numbers, so the broker rejects out-of-order or duplicate sequences and the client re-sequences retries. Idempotence preserves ordering AND exactly-once-per-partition de-dup with max.in.flight up to 5. If you set max.in.flight>5 with idempotence, the producer config validation fails.

go deeper

for a junior

Aware that retries can reorder and that there's a producer setting involved.

for a middle

Explains the retry-overtakes-batch scenario and the max.in.flight=1 fix.

for a senior

Explains PID + sequence numbers, the <=5 constraint, and config coupling (acks=all).

for a principal

Reasons about exactly-once boundaries, transactions for cross-session guarantees, and throughput-vs-ordering policy across the platform.

## The setting `max.in.flight.requests.per.connection` is the maximum number of produce requests the client will send to *one broker connection* without yet receiving acknowledgements. Default is **5**. Higher in-flight = better pipelining/throughput; lower = stronger ordering. ## How reordering happens (non-idempotent) Producers batch records per partition and send batches as requests. Suppose for one partition the producer sends: - Request A (records 1-10) - Request B (records 11-20) concurrently (in-flight = 2). If **Request A fails** (e.g. transient `NotEnoughReplicas`) but **Request B succeeds**, then on retry, Request A is re-sent and appended *after* B. The partition log now contains 11-20 then 1-10 — **reordered within a single partition**, defeating the per-key ordering you relied on. With `retries>0` and `max.in.flight>1`, this is a real hazard. ## Old fix Set `max.in.flight.requests.per.connection=1`. Only one unacked request at a time, so a retry can never be overtaken. Cost: no pipelining, lower throughput. ## Modern fix: the idempotent producer Set `enable.idempotence=true` (the **default since Kafka 3.0**). Mechanism: - The producer is assigned a **Producer ID (PID)** by the broker. - Each record batch carries the PID and a **monotonically increasing sequence number per partition**. - The broker tracks the last successfully written sequence per (PID, partition). It only accepts the *next expected* sequence. An out-of-order batch is rejected with `OutOfOrderSequenceException`; a duplicate (already-seen sequence) is silently dropped (de-dup). - The client buffers and **re-sequences retries** so they're reapplied in order. Result: even with `max.in.flight` up to **5**, ordering within a partition is preserved AND duplicates from retries are eliminated. This is *exactly-once-per-partition* semantics at the produce level. ## Config constraints With `enable.idempotence=true`: - `max.in.flight.requests.per.connection` must be **<= 5** (else config validation throws `ConfigException`). - `acks` must be `all`. - `retries` must be > 0. The modern client sets these automatically when idempotence is on. ## Edge cases - Setting `acks=1` or `acks=0` disables the idempotence guarantee. - A producer session expiry / PID reset (e.g. after `delivery.timeout.ms` exhaustion) can still surface `OutOfOrderSequenceException`, which is fatal and requires a new producer. - Idempotence is *per producer session per partition* — it does not de-dup across producer restarts unless you use transactions with a stable `transactional.id`.

  • With enable.idempotence=true, what is the maximum allowed max.in.flight.requests.per.connection and why?
    5. The broker only buffers a bounded window of sequence numbers per partition to re-order/de-dup; beyond 5 it can't guarantee correct re-sequencing, so config validation rejects values >5.
  • What other producer configs does enabling idempotence force?
    acks=all and retries>0; the client sets these implicitly. Overriding acks to 0/1 makes the config invalid.
  • Does idempotence guarantee ordering across producer restarts?
    No. Idempotence is per producer session (PID). Cross-session de-dup/ordering requires transactions with a stable transactional.id.

saying these in an interview costs you the question

  • Saying max.in.flight only affects throughput, not ordering
  • Claiming you must set max.in.flight=1 even with idempotence enabled
  • Thinking idempotence allows unlimited in-flight requests
  • Believing idempotence de-dups across producer restarts without transactions
  • Forgetting that acks=all is required for the idempotence guarantee

context

resolving links…