A message queue delivers the same order-created message to a consumer twice because the consumer crashed after processing it but before acknowledging it. What must the consumer do so processing the message twice doesn't cause a duplicate charge or duplicate shipment?

## Why the same message arrives twice **At-least-once delivery** is the dominant guarantee in production messaging systems: brokers like Kafka, SQS, RabbitMQ, and Pub/Sub choose to redeliver a message rather than risk losing it, because the alternative, **at-most-once**, silently drops messages on any failure between fetch and acknowledgment. The price of that safety choice is duplicates: a consumer can process a message, start doing side effects, then crash or lose its network connection before it acknowledges (acks) the message, and the broker, having no proof the message was handled, redelivers it. The same order-created message might now arrive at the consumer twice, spaced by anywhere from milliseconds to hours later after a redeploy. ## How an idempotent consumer works **Mechanism:** an idempotent consumer is one whose observable effect from processing a message N times (N >= 1) is identical to processing it once. Concretely, the consumer keeps a record of which messages, or which business operations, it has already applied: - Before applying a new message's effect, it checks that record. - If the message's identity already appears there, it **skips the side effect** — or safely reapplies a value that lands on the same result, such as setting a field to an absolute value rather than incrementing it — and simply acks. The **identity** used for that check matters. It can be: - the message's own unique ID assigned by the producer or broker, or - a business-level key such as an order ID plus an operation type, deliberately chosen so retries of the same producer-side action collapse to the same key. ## The reliability model underneath **Why it exists:** this is purely a consequence of choosing at-least-once delivery as the reliability model. Distributed systems cannot cheaply guarantee exactly-once delivery across a network boundary, so the practical compromise used by essentially every message broker is deliver at least once, and let the consumer make redelivery harmless. 'Exactly-once processing' as marketed by some systems is really 'exactly-once effect,' built by combining at-least-once delivery with an idempotent consumer; the achievement lives in the application logic, not magic in the wire protocol. ## What it costs **Trade-offs:** making a consumer idempotent costs something on both sides. - **On the correctness side**, if you get it wrong and a duplicate slips through, the failure is silent and customer-visible (double charge, duplicate email, doubled inventory decrement), which is often far more expensive to unwind than the extra engineering to prevent it. - **On the engineering side**, idempotency adds a persistent lookup or a schema constraint to every message handled, which costs latency (an extra read or an upsert with a uniqueness check) and storage (a growing table of processed identifiers). Teams sometimes accept a narrower guarantee, duplicates are rare and reconciled nightly, when the operation is cheap to reverse and the volume of true duplicates is low, trading strict correctness for simplicity. ## Where it breaks **Failure modes in production:** 1. **The most common failure is choosing the wrong deduplication key**, for example deduplicating on a broker-generated message ID when the producer itself retried and got a new ID for what is logically the same event, so the duplicate check never fires. 2. **Another is a check-then-act race:** two threads or two consumer instances read 'not yet processed' at nearly the same instant, both proceed, and both apply the side effect before either writes the 'processed' marker; this needs a database-level unique constraint or conditional write, not just an application-level if-check. 3. **A third is unbounded growth of the dedup store:** if processed-message records are never expired, the lookup table grows forever and eventually degrades lookup latency or blows storage budgets. ## The same pattern at an API boundary **A concrete real-world scenario:** Stripe's payment API is idempotent by design: clients pass an `Idempotency-Key` header with every charge request, and Stripe stores the key with the first response for a defined window. If the same key is replayed because the client's HTTP call timed out and it retried, Stripe returns the original charge's result instead of creating a second charge. That is the idempotent-consumer pattern applied at an HTTP API boundary rather than a message queue, and it illustrates the general shape: - identify the logical operation with a stable key chosen by the caller, - remember what happened the first time, - and short-circuit repeats.