Concretely, how does a Kafka-style consumer track 'what have I already processed' using offsets, and how is that different from how an SQS or RabbitMQ consumer acknowledges a message?

## The log side: a bookmark you own In a Kafka-style log, a partition is literally an **append-only array**, and an offset is just an integer index into it. - A consumer group's committed position is stored centrally (Kafka keeps it in an internal topic, `__consumer_offsets`), and reading via `poll()` never removes or modifies anything. - Only explicitly committing the offset (auto-commit on an interval, or manual sync/async commit) advances the consumer's bookmark. - Crucially, multiple consumer groups reading the same partition each hold a completely independent offset, so one group's progress has zero effect on another's. ## The queue side: state the broker keeps for you In a queue, the broker instead maintains real **per-message state**: available, in-flight, or deleted. When a consumer receives a message, it becomes invisible (SQS's visibility timeout) or unacked (RabbitMQ), and the client must send back an explicit acknowledgment referencing that specific message before the broker purges it. If no ack arrives within the window, the broker assumes failure and makes the message available again for redelivery, possibly to an entirely different consumer instance. ## Why the split exists This split exists because the two models optimize for different guarantees. - **The queue model** optimizes for 'this piece of work must be done by exactly one worker, and once done, it's off the books' - minimal broker-side state, self-cleaning. - **The log model** optimizes for 'this fact must be available to many independent readers, at their own pace' - since there's no shared notion of 'done,' each reader must independently bookmark its own position. ## The trade-offs The trade-offs cut both ways. Offset commits are cheap and batchable - you can commit once per thousand messages or once per few seconds - and replay is as simple as resetting the offset backward. But the ordering of commit versus processing matters: 1. **Committing before finishing work** risks silent message loss if the consumer crashes mid-batch (at-most-once). 2. **Committing after finishing work** risks reprocessing the same messages on restart (at-least-once, requiring idempotent handling). True exactly-once needs either idempotent side effects keyed by offset/event-ID, or a transactional API that ties the offset commit and the output write into one atomic operation. Acks give simpler single-message semantics with automatic redelivery and dead-lettering built in, but that per-message state doesn't scale to letting ten independent services each read the same message without physically duplicating it into ten queues. ## Production failure modes Production failure modes differ accordingly. - **Logs** suffer from consumer-group lag creeping unnoticed until someone checks a lag metric, and from 'rebalance storms' - when a consumer joins or leaves a group, partitions get reassigned, and the new owner resumes from the last committed offset, which can be behind where the old owner actually got to, causing a brief burst of duplicate processing. - **Queues** suffer from in-flight message limits (SQS caps how many unacked messages can be outstanding per queue at once, and once that cap is hit no new messages can be received until some are acked or expire) and from poison-pill loops where a message keeps failing and keeps getting redelivered until a `maxReceiveCount` setting finally routes it to a dead-letter queue. ## Two workers, two outcomes A concrete contrast: - **The queue side** - a checkout worker on SQS receives an order message with a 30-second visibility timeout; if the database write actually takes 45 seconds, the message reappears and a second worker may pick it up concurrently, causing double processing unless the write is idempotent. - **On the log side**, an analytics consumer on Kafka commits offsets every 5 seconds in batches of thousands of events; if it crashes mid-batch, it resumes from the last committed offset on restart and reprocesses the last few seconds of events - which is safe here because the downstream aggregation upserts by event ID rather than blindly incrementing counters.