Your POST endpoint honours the Idempotency-Key header, but the operation it performs is a call to an external payment provider you do not control. How do you keep the stored idempotency record and the external side effect consistent across crashes?

## The core impossibility An idempotency-key implementation whose effect is local is easy: commit the business row and the completed dedupe record in one database transaction, and every crash leaves you in a consistent state. The hard version is when the effect lives in someone else's system — a payment provider, a shipping carrier, an email sender. You cannot enlist their API in your transaction, and they will not participate in two-phase commit. So there is always a window where your local record and their state can disagree. The design goal shifts from "never disagree" to **"always be able to converge"**. ## Intent-first ordering The ordering that survives crashes is: 1. **Commit intent.** Write and commit the `(scope, key)` record as `in_progress`, with a request fingerprint and a lease expiry, *before* any outbound call. This is durable evidence that an attempt is under way, which is what recovery will key off. 2. **Perform the effect**, passing your key downstream. 3. **Commit the outcome.** Write the provider's result into the record and mark `completed`. Compare the alternatives and why they fail: - *Call first, record after.* A crash after the charge leaves no local trace at all. The retry sees an unknown key and charges again — unless the provider deduplicates, which is exactly why step 2's key pass-through carries so much weight. - *Mark completed before confirming.* A crash or a provider failure after the mark means every replay confidently reports success for a payment that never happened. This is the worst failure mode because it is invisible: no error, no alert, just a customer who was never charged and an order that shipped. ## Key pass-through: making recovery converge The single most important technique is to send **your** idempotency key as the provider's `Idempotency-Key`. It has the effect of extending your dedupe boundary into their system: any number of retries or takeovers of the same logical operation collapse to one charge at their end, because both systems now agree on the operation's identity. Derive their key deterministically from yours (the key itself, or a hash of key + operation name if you make several distinct downstream calls in one request — each distinct call needs a stable, distinct key). Never generate a fresh downstream key per attempt; that is the same bug as regenerating your own key per retry, just one layer down. ## Recovery from an abandoned in-progress record When a later retry finds an in-progress record with an expired lease, it may take over. Take over **atomically** — a conditional update that re-stamps the lease so only one taker wins — and set the lease longer than the maximum plausible execution time, because taking over an operation that is still running is how you reintroduce the duplicate. Then prefer **query over re-execute**: if the provider exposes lookup by idempotency key or by your client reference, read their state and reconcile into your record. Re-calling with the same key is the fallback and is safe if their dedupe window covers your recovery window — note that their TTL is now part of your correctness argument, so it belongs in your design docs, not just theirs. Some operations simply cannot be resolved automatically (provider has no lookup, key window expired). For those, the record should transition to a state that a reconciliation job or a human queue picks up. Designing that terminal state explicitly, rather than leaving rows stuck in-progress forever, is what separates a system that degrades from one that silently loses money. ## Storage placement follows from all this The record must be **durable and transactional with your local state**. That argues for the primary relational database rather than a cache: an eviction, a failover that loses recent writes, or a non-replicated node means a lost key, and a lost key for a payment means a second real charge. Caches are fine when the effect is cheap to repeat; they are not fine here. ## Events and downstream fan-out If completing the operation also publishes an event (`PaymentCaptured`), do not publish inside the request path — a crash between commit and publish loses the event, and publishing before commit can announce work that rolls back. Write the event to an **outbox table in the same transaction as the completed record**, and let a relay publish it at-least-once. Consumers then need their own idempotency, keyed on the event id — the same pattern recursing one level out. ## What a principal-level answer emphasises - The guarantee you can actually offer is **at-most-once effect with at-least-once attempt and a convergence path**, not distributed atomicity. - The provider's idempotency semantics and retention window become part of *your* correctness argument. - Every state must be recoverable: no row may sit in-progress forever, and there must be a defined path — automatic reconciliation or an operational queue — for the cases automation cannot close. - Observability is part of the design: alert on in-progress records older than the lease, on takeover rates, and on reconciliation mismatches, because these are the signals that the invisible failure mode is happening.