A consumer pulls a message off a queue and crashes before sending an acknowledgement back to the broker. What happens to that message, and what delivery guarantee does this behavior typically produce?

## What an acknowledgement decides Acknowledgement is the mechanism a message broker uses to know when it's safe to consider a message done and remove or advance past it — and it's the single biggest lever determining what delivery guarantee your system actually gets. ## The ack cycle **Mechanism, step by step.** 1. When a consumer receives a message, most brokers don't delete it immediately. Instead they mark it as **in-flight** or invisible and start a timer — RabbitMQ calls this an unacked delivery tied to the consumer's channel, SQS calls it a visibility timeout, Kafka (which works slightly differently, via consumer offsets rather than per-message acks) tracks a committed offset per partition. 2. The consumer processes the message and, only after finishing successfully, sends an **acknowledgement** (RabbitMQ: `basic.ack`; SQS: `DeleteMessage` call; Kafka: commit the offset past that record). 3. Only at that point does the broker consider the message consumed and either delete it (queue model) or advance the read cursor (log model, where the message physically stays but is no longer re-read by that consumer group on restart). 4. If the consumer never sends that ack — because it crashed, the process was killed, the network dropped, or it explicitly rejected the message — the broker's timer expires and it treats the message as failed: it becomes visible/available again and gets redelivered, either back to the same consumer or to a different one in a competing-consumers setup. ## Why the broker waits **Why this design exists:** it's the only way to survive consumer failure without losing work. - If the broker deleted a message the instant it handed it out (sometimes called **auto-ack** or fire-and-forget), any consumer crash between receipt and completion would silently lose that message forever — unacceptable for anything business-critical like payments or order processing. - Requiring an explicit ack after successful completion means the broker, not the consumer, is the source of truth for whether the work got done, and a crashed consumer simply results in the work being retried by someone else. ## Duplicates are the cost **Trade-off — duplicates are the cost.** The mechanism that prevents message loss inherently creates the possibility of duplicate delivery. Consider the exact crash timing: a consumer finishes the actual business work (charges a credit card, writes a database row) but crashes in the tiny window before the ack packet reaches the broker. From the broker's point of view, no ack arrived, so it redelivers — and now the business work runs a second time. This is why virtually every at-least-once system is described as giving at-least-once delivery, not exactly-once, and why this needs to be paired with **idempotent processing**: the consumer's handler should be written so that applying the same message twice, matched by a message ID, idempotency key, or natural business key, has the same effect as applying it once — for example, setting the account balance to X rather than adding $10 to the balance, or checking whether a given message ID has already been processed before doing the side-effecting work. ## Failure modes in production - **Acks before the work finishes.** The most common bug in production is a consumer that acks the message before finishing the work (to be safe against long processing times, or because of a badly placed try/catch), which silently converts the system to at-most-once — if the process then crashes mid-work, the message is gone and the work never completes, with no error raised anywhere. - **Forgetting a redelivery limit.** The opposite failure — forgetting to configure a sane redelivery/retry limit — lets a poison message (one that always fails to process, due to a malformed payload for example) get redelivered forever, endlessly cycling through consumers, consuming CPU and log volume without ever making progress. - **Visibility-timeout mis-tuning.** A third failure mode: setting it too short for the actual processing time means the broker gives up and redelivers a message that's still being legitimately processed, causing two consumers to work on it concurrently and possibly double-write; setting it far too long means a genuinely crashed consumer's message sits invisible and unprocessed for an unnecessarily long time before anyone else picks it up. ## An idempotency key absorbing a redelivery A concrete example: an e-commerce order-processing worker consumes `ChargeCard` messages from an SQS queue. The worker successfully calls the payment gateway and gets a success response, but the EC2 instance is terminated by a spot-instance reclaim before the `DeleteMessage` call completes. SQS's visibility timeout expires, the message reappears, and a second worker instance picks it up and calls the payment gateway again. Because the team stamped every `ChargeCard` message with a unique idempotency key passed straight through to the payment gateway's API, the gateway recognizes the duplicate key and returns the original charge's result instead of charging the customer twice — the at-least-once redelivery is absorbed safely by that idempotency design rather than causing a double charge.