Why does an idempotent receiver need to detect duplicate messages explicitly, rather than relying on the messaging channel to guarantee exactly-once delivery?

## Three delivery guarantees Delivery guarantees for messaging systems come in three flavors: | Guarantee | What it promises | |---|---| | **at-most-once** | send and forget, may lose the message | | **at-least-once** | retry until acknowledged, may deliver more than once | | **exactly-once** | deliver precisely one time, no losses, no duplicates | Exactly-once sounds like the obvious thing everyone wants, but it is provably very hard to achieve end-to-end across independent processes connected by an unreliable network, because the fundamental ambiguity can't be eliminated: when a sender doesn't receive an acknowledgment, it cannot distinguish between 'the message never arrived' and 'the message arrived, was processed, and only the ack was lost on the way back.' The only safe default in that ambiguous case is to retry, and retrying is what turns at-least-once into the practical, dominant guarantee real messaging infrastructure provides. The **idempotent receiver** pattern exists to make that retry safe: instead of relying on the channel to prevent duplicates (which it fundamentally cannot do reliably), the receiver itself is built to produce the same result whether a given message arrives once or five times. ## Two ways to make a retry safe Mechanically, there are two complementary techniques. 1. The first is **deduplication by identity**: every message carries a unique identifier (often the same ID used for correlation, or a dedicated message ID), and the receiver keeps a durable record — a dedupe table, typically with the message ID as a unique key — of IDs it has already fully processed. When a message arrives, the receiver checks that record first; if the ID is already present, it skips reprocessing (or returns the cached result of the first processing) rather than re-executing the business logic. Crucially, the check-and-record step must happen atomically with the business-logic side effect — usually inside the same database transaction that applies the change — otherwise a crash between 'do the work' and 'record that I did it' reopens the exact duplicate-processing window the pattern exists to close. 2. The second technique is designing the operation itself to be **naturally idempotent** regardless of duplicate detection: an operation like 'set account balance to $500' is idempotent by construction (applying it twice yields the same end state as applying it once), whereas 'add $100 to account balance' is not (applying it twice adds $200). Where possible, favoring naturally idempotent operations (absolute sets, upserts keyed on a natural identifier, PUT-style semantics) reduces reliance on the dedupe table as the sole safety net. ## Why commands feel this first This pattern exists because commands, in particular, tend to have real-world, often **irreversible side effects** — charging a card, shipping a package, sending an email — and at-least-once delivery means every command consumer must assume duplicates are a normal, expected occurrence, not a rare edge case. Skipping this is one of the most common and expensive mistakes in message-driven systems: it looks like it works in every test and demo (where retries rarely trigger), and then fails in production exactly when the network is already unreliable — which is precisely when retries fire most. ## The trade-off The trade-off is **storage and complexity versus safety**. - Maintaining a dedupe table costs a persistent record per processed message (or at least a bounded, time-windowed subset of recent ones, since keeping every ID forever is usually unnecessary and unbounded), plus the discipline of wrapping the check atomically with every side effect, in every consumer, for the life of the system. - Skipping it is cheaper up front but means every retry-driven duplicate becomes a real bug: a customer charged twice, an email sent three times, inventory decremented for the same order twice. - Some systems try to sidestep the storage cost by making every operation naturally idempotent instead, but that's not always possible — sending an email is inherently not idempotent no matter how it's implemented, since the side effect (a message landing in someone's inbox) can't be 'overwritten' by a repeat. ## Failure modes Failure modes cluster around the atomicity gap and around scope. - **The atomicity gap:** a receiver records 'processed' before or after — but not atomically with — the actual side effect, so a crash in between leaves either a duplicate charge (side effect happened, record didn't, so a retry redoes it) or a permanently skipped operation (record happened, side effect crashed, so the retry is wrongly treated as already-done and silently dropped) — both are silent data-correctness bugs that don't throw an exception. - **Scope failures** happen when deduplication keys on the wrong thing — for instance, deduplicating on payload content instead of a stable message ID, which breaks the moment two legitimately different messages happen to have identical content, or deduplicating per-consumer-instance instead of per logical consumer role, which fails the moment you scale to multiple instances behind a competing-consumer queue and each instance has its own, un-shared dedupe table. ## Putting it together A concrete scenario: `PaymentService` receives a `ChargeCard` command with message ID `msg-9911`. It begins a database transaction, checks a `processed_messages` table for `msg-9911`, finds nothing, charges the card via the payment gateway, and inserts `msg-9911` into `processed_messages`, all before committing. The gateway call succeeds but the network drops before `PaymentService` can acknowledge the message back to the queue, so the queue's at-least-once guarantee redelivers the same `ChargeCard` command a minute later. This time the dedupe check finds `msg-9911` already present, skips the gateway call entirely, and the customer is charged exactly once despite the message arriving twice.