Why do Competing Consumers setups typically guarantee at-least-once message delivery rather than exactly-once, and what must consumer code do to stay correct under that guarantee?

## The guarantee and where it comes from Delivery semantics describe the guarantee a messaging system makes about how many times a given message will be delivered to a consumer, and Competing Consumers systems almost universally land on **at-least-once** rather than exactly-once. The mechanism is the acknowledgment protocol described by the visibility-timeout pattern: a consumer receives a message, does its work, and then explicitly tells the broker "done, delete this." The broker cannot distinguish between two situations that look identical from its side: 1. The consumer crashed before finishing the work, so the message genuinely needs to be retried. 2. The consumer finished the work perfectly but crashed (or the network dropped the response) before the acknowledgment reached the broker. In case 2, the work already happened, but the broker has no way to know that — from its point of view, it just didn't hear back in time. The only safe default is to treat both cases the same way and redeliver, because silently treating case 1 as case 2 (i.e., never retrying) would lose messages, which is almost always the worse failure. The cost of that choice is that case 2 becomes a duplicate: the same message really was processed successfully once already, and it gets processed again. ## Why exactly-once rarely survives **Exactly-once delivery** — meaning each message is guaranteed to produce its effect exactly one time, no more, no less — is a much stronger guarantee that requires the broker and the consumer's side effects to be coordinated as a single atomic unit, typically via a shared transactional store or a two-phase-commit-like protocol. A few specialized systems approximate this in narrow contexts (Kafka's transactional producer/consumer APIs offer "exactly-once semantics" when both the read and the write stay inside the Kafka ecosystem), but the moment a consumer's side effect reaches outside that boundary — writing to an external database, calling a third-party API, sending an email — there is no longer a shared transaction to make atomic, and the guarantee reduces back to at-least-once from the application's perspective. This is why the practical, portable rule for Competing Consumers is: assume at-least-once, and design accordingly, rather than chase a stronger guarantee that rarely survives contact with real infrastructure. ## Idempotency, the constraint the consumer absorbs The trade-off, then, isn't really a choice — it's a constraint the consumer must absorb. This is where **idempotency** comes in: a consumer operation is idempotent if running it N times has the same observable effect as running it once. - **Make the operation naturally idempotent.** The cleanest way to get this is to design the operation itself to be naturally idempotent — an "update the account balance to $500" message is idempotent by construction (it's a SET), whereas "add $50 to the account balance" is not (running it twice adds $100). - **Deduplication keyed on something stable.** When the operation can't be made naturally idempotent — e.g., "charge the customer's card" really is an action, not a state assignment — the standard technique is deduplication keyed on something stable: the message's unique ID, or a business-level idempotency key the producer attaches (many payment APIs formalize this as an explicit idempotency key the caller supplies). - **Record the keys already processed.** The consumer records processed keys, typically via a database unique constraint or a fast key-value store with a TTL, and checks that record before doing the side effect — if the key's already been seen, it short-circuits and returns the prior result instead of repeating the action. ## What the failure mode looks like The failure mode when this discipline is skipped shows up concretely and often expensively. - A **payment-processing consumer** that isn't idempotent, running behind a Competing Consumers queue, will occasionally double-charge a customer: the charge succeeds, the acknowledgment is lost to a network blip or a mid-flight crash, the message is redelivered to (possibly) a different instance, and it charges the card again because nothing recorded that the first charge already happened. - An **inventory-decrement consumer** that isn't idempotent will oversell stock under load, because a redelivered "decrement by 1" message decrements again. These bugs are notoriously hard to catch in testing because they only manifest under the exact conditions Competing Consumers is built to handle — crashes, redeliveries, timing races — which are rare and load-dependent, so they surface in production, at scale, and often correlate with deploys (which kill consumer instances mid-message) or transient network issues, not with any obvious code change. ## Where it shows up A concrete, well-known real-world pattern: AWS's own SQS documentation explicitly states standard queues provide at-least-once delivery (with occasional out-of-order delivery too) and recommends designing consumers to be idempotent; SQS FIFO queues add deduplication (via a short dedup window keyed on message content or an explicit deduplication ID) specifically to narrow — not eliminate — the practical rate of duplicates, while still describing the overall contract as effectively-once rather than a hard guarantee, precisely because the same crash-before-ack ambiguity exists at the FIFO layer too.