skip to content

Compare automatic offset commits (enable.auto.commit) with manual commits. What are the trade-offs and failure modes?

level: middleimportance: must knowfreq 85%

answer

  1. auto-commit fires inside poll(), not a timer
  2. default interval 5000ms
  3. commits returned-by-poll, not processed
  4. commitSync = blocking+retry+throws; commitAsync = fast+no retry
  5. commitAsync in loop, commitSync in finally

basics

~20 s

With enable.auto.commit=true, the consumer commits the current position periodically (auto.commit.interval.ms, default 5000ms) during poll(). It's simple but can lose or duplicate records on crash. Manual commits (commitSync/commitAsync after processing) give you control over timing and delivery semantics.

solid answer

~40 s

enable.auto.commit=true (the default) makes the consumer commit its current fetch position automatically during poll() once auto.commit.interval.ms (default 5000) has elapsed. It commits what has been *returned by poll*, not necessarily what you've finished processing. So if you poll a batch, start processing, and crash mid-batch after an auto-commit fired, those records are skipped — data loss (at-most-once leaning). Conversely you can reprocess up to the last interval — duplicates. Manual commits (set enable.auto.commit=false) let you commit *after* processing completes: commitSync (blocking, retried, throws on failure) for correctness at shutdown/critical points, or commitAsync (non-blocking, no retry) for throughput in the hot loop. Manual commits enable reliable at-least-once. The common pattern: commitAsync per batch, commitSync in a finally block on close.

go deeper

for a junior

Know auto-commit is the simple default (every ~5s) and manual commits give more control.

for a middle

Explain that auto-commit fires inside poll, commits what poll returned, and the resulting loss/duplicate windows; contrast commitSync vs commitAsync.

for a senior

Map each mode to delivery semantics and articulate the commitAsync-in-loop + commitSync-in-finally pattern and why async doesn't retry.

for a principal

Discuss when exactly-once via transactions is required vs idempotent sinks, and design commit cadence against rebalance and throughput targets.

## The two modes ### Automatic (enable.auto.commit=true — the default) The consumer commits offsets for you. Mechanically: **commits happen inside `poll()`**, not on a background timer. On each `poll()`, the client checks whether `auto.commit.interval.ms` (default **5000ms**) has elapsed since the last auto-commit; if so, it commits the **current position** (the offsets of records already *returned by previous polls*). Key subtlety: it commits what poll has **handed to you**, regardless of whether your code finished processing them. Sequence of danger: 1. `poll()` returns records 50–99, advancing position to 100. 2. You process 50–70. 3. Next `poll()` fires an auto-commit of 100 (interval elapsed). 4. Crash. On restart you resume at 100 → records 71–99 are **lost** (never processed but marked done). The inverse — duplicates — happens when you process records, then crash *before* the next poll's auto-commit, so the work since the last commit is redone. So auto-commit is fundamentally **best-effort** and gives neither clean at-least-once nor exactly-once. ### Manual (enable.auto.commit=false) You call commits explicitly **after** processing: - **`commitSync()`** — blocks until the broker acknowledges, **retries** automatically on retriable errors, and **throws** on failure so you can react. Use at shutdown, before rebalance, or at critical checkpoints. - **`commitAsync()`** — fires and returns immediately with an optional callback. **No retries** (a later commit would override a stale one anyway). Higher throughput, but a lost async commit can mean reprocessing. ## The canonical safe pattern ``` try { while (running) { var records = consumer.poll(d); process(records); consumer.commitAsync(); // fast path per loop } } finally { try { consumer.commitSync(); } // durable final commit finally { consumer.close(); } } ``` commitAsync keeps the loop fast; the final commitSync guarantees the last batch is durably recorded before shutdown. ## Delivery semantics mapping - Auto-commit: ambiguous; can drop or duplicate. - Manual commit *after* processing: **at-least-once** (process → commit; crash before commit → reprocess, never lose). - Manual commit *before* processing: at-most-once (rarely wanted). - Exactly-once: needs transactions (sendOffsetsToTransaction) or an idempotent/transactional sink, not bare commits. ## Why auto-commit still exists For stateless, idempotent, or loss-tolerant pipelines it is simple and adequate. The trade is convenience vs control.

  • Does auto-commit run on a background thread?
    No. The auto-commit check and the commit itself happen synchronously inside poll(). If you stop calling poll(), no auto-commits occur. This is why long processing between polls can also affect commit timing.
  • Why does commitAsync not retry but commitSync does?
    A failed async commit shouldn't be retried because a newer commit (higher offset) may have already succeeded; retrying the older one could move the committed offset backwards. commitSync blocks in order, so retrying is safe and it surfaces errors to the caller.

saying these in an interview costs you the question

  • Saying auto-commit happens on a background timer thread (it happens during poll)
  • Claiming auto-commit gives exactly-once or guaranteed no-loss
  • Recommending commitSync in the tight loop for throughput-sensitive apps (use commitAsync there)
  • Saying commitAsync retries on failure

context

resolving links…