A client's call to submit a `ChargeCard` command times out on the network, so the client's retry logic resends the exact same command. How do you design the command-handling path so the retry doesn't charge the card twice, and where does that check actually have to live?

## The property and the mechanism Idempotency in command handling is the property that processing the same command more than once produces the same effect as processing it exactly once. The mechanism for enforcing it on a naturally non-idempotent operation like charging a card starts with the client attaching a unique identifier to the command before it's ever sent — either the command's own natural ID, if the client generates one, or an explicit idempotency key field. 1. **When the handler receives the command**, before performing the actual side effect, it looks up that key in a dedup store: a table or index that records which command IDs have already been processed and, ideally, what the outcome was. 2. **If the key is found**, the handler skips the real work entirely and returns the previously recorded result, so the caller gets the same answer it would have gotten the first time, without the card being charged again. 3. **If the key is not found**, the handler performs the charge and, in the same database transaction, writes a row recording that this key has now been processed, along with enough information to answer a future duplicate. ## Why the guarantee moves to the application layer This mechanism exists because network delivery is fundamentally uncertain: when a client's request times out, there is no way to distinguish 'the server never received it' from 'the server received it, processed it, and the response was lost on the way back.' A system that tried to guarantee exactly-once delivery over the network would be fighting an impossible problem in the general case — you cannot make an unreliable network reliable by trying harder at the transport layer alone. The practical, well-established solution is to **accept at-least-once delivery**, meaning duplicates will happen, and instead make duplicate processing harmless by tracking what has already been done. This shifts the guarantee from the network layer, where it can't be made, to the application layer, where it can. ## The trade-off The trade-off is extra state and extra work on every command: - **a dedup table** to maintain, - **an index** to keep fast, and - **a decision about retention** — keeping every idempotency key forever is wasteful, but purging them too aggressively reopens the double-processing window for a client that retries after a long delay. There's also a design cost in deciding what counts as 'the same command': if a client accidentally reuses an idempotency key for a genuinely different charge amount, naive implementations might silently return the wrong cached result, so many systems also store enough of the original request to detect and reject a key reused with different content, rather than assuming any repeat of the key means an identical retry. ## Failure modes 1. **The most damaging production failure mode** is implementing the dedup check and the actual write as two separate, non-atomic steps: first a SELECT to see if the key exists, then, if not found, a separate step to perform the charge and insert the key. Under concurrency — two duplicate requests arriving close together, which is exactly the scenario retries create — both can pass the SELECT before either has inserted its key, and both proceed to charge the card. The fix is to make the check and the write atomic with each other, typically by relying on a unique constraint on the idempotency key column and letting the second insert fail with a constraint violation, which the handler catches and treats as 'already processed,' or by doing the check-and-insert inside one transaction with appropriate isolation. 2. **A second failure mode is confusing naturally idempotent operations with ones that need enforcement**: an operation like `SetShippingAddress`, which just overwrites a field, produces the same end state no matter how many times it's applied and needs no dedup key at all, while an operation like `AddItem` or `ChargeCard`, whose effect compounds with each application, absolutely does. | Naturally idempotent | Needs enforcement | |---|---| | `SetShippingAddress` just overwrites a field | `AddItem` or `ChargeCard` compounds with each application | | produces the same end state | needs an explicit idempotency key | ## A real-world pattern A concrete real-world pattern: payment APIs such as Stripe's charge endpoint require callers to supply an `Idempotency-Key` header on write operations; the server stores that key alongside the result of the first successful request for a bounded retention window, and any subsequent request with the same key returns the original result verbatim rather than creating a second charge — which is precisely the client-times-out-and-retries scenario this question describes, solved at the API boundary rather than left to the client to avoid retrying altogether.