When a Kafka consumer processes a batch of 10 records and commits only the offset of the last record after the whole batch succeeds, what happens on a crash after record 6 finishes, and why is committing after every single record not simply 'safer'?

## What commit ordering actually decides Offset commit ordering refers to two related decisions: 1. **At what granularity you commit** — after every single record, after every batch/poll, or on a timer. 2. **In what order commits actually land** relative to the processing they represent, especially when commits are issued asynchronously. In Kafka, a commit doesn't acknowledge an individual message the way a queue ack does; it writes a single cursor value per partition — 'the next offset this consumer group should read from here' — to an internal offsets topic. Because the guarantee is expressed as one monotonically-advancing number per partition rather than one flag per message, committing offset `110` implicitly tells the broker 'everything up through 109 on this partition is done,' even if what actually happened was 'I processed records 100-109 as one batch and I'm only checkpointing once, at the end.' That single design choice is why granularity matters so much: it directly sets the size of the replay window on crash. ## Why a cursor instead of per-message acks This design exists because per-message acknowledgment, as RabbitMQ or JMS do, requires the broker to track delivery state for every in-flight message individually, which doesn't scale to the sustained high-throughput, ordered-log workloads Kafka targets. A single per-partition cursor is cheap to store and cheap to advance, but it forces an all-or-nothing checkpoint over whatever batch of records preceded it. Consumers therefore have to explicitly choose how often to move that cursor, trading throughput against the size of the 'at risk' window. ## The granularity choices - **After every single record.** Committing after every single record minimizes the replay window — at most one record gets reprocessed after a crash — but every commit is itself a network round trip to the offsets topic, so throughput drops sharply, often by an order of magnitude, under `commitSync()`. - **Once per polled batch.** Committing once per polled batch amortizes that cost and is the common production pattern, but it means a crash after processing record 6 of a 10-record batch causes all 10 to be replayed, not just the unfinished 4, because the cursor hasn't moved past record 0 of that batch yet. - **Async steady-state batches with a final `commitSync()`.** Using `commitAsync()` for steady-state batches with a final `commitSync()` on shutdown gets most of the throughput of infrequent commits with a smaller staleness window, but introduces its own hazard: async commits can complete out of order, so a consumer must never let an older, in-flight async commit overwrite a newer one, which client libraries handle by tracking a monotonically increasing sequence and discarding stale commit callbacks. ## Failure modes 1. **Duplicate storms after a restart.** The most common production issue is teams choosing large batch/commit intervals purely for throughput and then being surprised by 'duplicate storms' after any restart or rolling deploy, because every restart replays the entire uncommitted tail of the last batch, not just the record that was mid-flight. 2. **Committing for a reassigned partition.** A second failure mode is committing an offset for a partition the consumer no longer owns after a rebalance: if a `commitAsync()` call issued just before a rebalance completes after the partition has been reassigned, it can commit a stale offset that causes the new owner to skip records it hasn't processed yet, or briefly conflicts with the new owner's own commits — this is why offset commits should be tied to the consumer-group generation and why manual-commit code typically calls a final synchronous commit inside the rebalance-listener's `onPartitionsRevoked` callback before giving up a partition. 3. **Porting the ack model between broker types.** A third, subtler failure is out-of-order manual commits when porting code between broker types: in RabbitMQ, acking message 8 before message 5 is safe per-message, but mistaking RabbitMQ's per-message ack model for Kafka's single-cursor model, or vice versa, can silently change the delivery guarantee. ## A worked pattern A concrete production pattern: an order-processing consumer polls 200 records per batch, writes each order to a database inside the loop, and calls `commitSync()` only once after the whole batch of 200 finishes. Throughput is good because there's one round trip per 200 records instead of 200. But a crash after order #150 replays all 200 orders on restart, so the database write must be an idempotent upsert keyed on order ID regardless of batch size — the batch-size choice only changes how many redundant upserts happen, not whether they can happen at all. Shrinking the batch size or adding a mid-batch async commit every 20 records reduces the blast radius of a crash but never removes the need for the write itself to tolerate replay, which is exactly why commit granularity is a throughput/staleness knob, not a substitute for making the write itself safe to repeat.