skip to content

Concurrency and Idempotency

15 roadmaps13 questions

Making writes safe when clients race and retry: optimistic concurrency with preconditions, idempotency keys for non-idempotent operations, and retry-tolerant endpoint design. Interviewers focus here because it is where REST design meets distributed-systems correctness.

part ofAPI stylesoverview, primer and where to startread it →
on this pageshow

explore

questions

13

What is the Idempotency-Key HTTP header pattern used by payment and other write APIs, who generates the key, and which requests should carry one?

level: juniorimportance: must knowfreq 58%

answer

  1. Client generates UUID per intent, reuses on every retry
  2. Header, not body — gateways and SDKs can see it
  3. First call executes and stores; repeats replay stored response
  4. Only for POST / non-idempotent PATCH
  5. Not needed for GET, PUT, DELETE

basics

~20 s

The client generates a unique key (usually a UUID) per logical operation and sends it as the Idempotency-Key request header. The server records the key with the result; if the same key arrives again it returns the stored response instead of re-executing. It makes retrying a POST safe.

solid answer

~50 s

POST is not idempotent, so when a response is lost the client cannot tell whether the charge happened. The `Idempotency-Key` header fixes that at the application level. **The client** generates the key — one per logical operation, typically a UUIDv4 — and **reuses the same key on every retry** of that operation. A new logical operation gets a new key. Generating it client-side is essential: only the client knows that attempt 2 is a retry of attempt 1. **The server** looks the key up. First time: execute, store the outcome against the key, respond. Subsequent times: return the **stored response** without re-executing, so the caller sees the original 201 and the original resource id. Carry a key on state-creating, non-idempotent requests: `POST /payments`, `POST /orders`, `POST /transfers`. It is unnecessary on GET (safe) and on PUT/DELETE (already idempotent by method). Send it as a request header, not in the body, so gateways and middleware can act on it.

code

http · 22 lines
http
POST /v1/payments HTTP/1.1
Host: api.example.com
Idempotency-Key: 8f14e45f-ceea-467a-9f5a-3c3f2f0d1b77
Content-Type: application/json

{"amount":2000,"currency":"usd","source":"card_1"}

HTTP/1.1 201 Created
Location: /v1/payments/pay_9K2

--- response lost; client retries with the SAME key ---

POST /v1/payments HTTP/1.1
Host: api.example.com
Idempotency-Key: 8f14e45f-ceea-467a-9f5a-3c3f2f0d1b77
Content-Type: application/json

{"amount":2000,"currency":"usd","source":"card_1"}

HTTP/1.1 201 Created
Idempotent-Replayed: true
Location: /v1/payments/pay_9K2

go deeper

for a junior

Explain the ambiguity a lost POST response creates, that the client generates a UUID and reuses it on retries, and that the server replays the stored response.

for a middle

Add where the pattern applies (POST and non-idempotent PATCH, not GET/PUT/DELETE), why it is a header, and that the server needs a durable dedupe store.

for a senior

Frame it as an application-level contract layered over HTTP because the protocol offers nothing here, and flag the hard parts you'd address next: scope, TTL, concurrent retries, payload mismatch.

for a principal

Discuss it as a platform contract — SDKs that attach keys automatically, uniform semantics across services, storage and retention cost, and the boundary between this and optimistic concurrency control.

## The problem it solves HTTP gives PUT and DELETE idempotency, so a client library may safely re-send them when a response is lost. POST has no such guarantee, and POST is what you use for "charge this card" or "place this order". When the connection drops with no response, the client faces an ambiguity it cannot resolve: the request may never have arrived, or it may have been fully processed with only the response lost. Retrying risks a double charge; not retrying risks a payment the customer thinks failed but which actually went through. HTTP itself offers no mechanism to resolve this. The `Idempotency-Key` header is the industry's application-level answer — popularised by Stripe and other payment APIs, and now the subject of an IETF draft (`Idempotency-Key` header field). It gives the server a way to recognize "this is the same logical operation I already handled". ## How it works, end to end 1. The client decides to perform an operation (charge $20). Before sending anything, it **generates a unique key** — a UUIDv4 is the standard choice — and persists it alongside its intent if the operation matters enough to survive a client restart. 2. It sends `POST /payments` with `Idempotency-Key: 8f14e45f-…` and the request body. 3. The server looks up the key within its scope. Not found → it executes the operation, stores the key together with the resulting status, headers and body, and responds normally (say 201 Created). 4. The response is lost. The client retries **the identical request with the identical key**. 5. The server finds the key, does **not** execute again, and returns the stored response: same 201, same payment id. The client ends up with the original outcome. Exactly one payment exists. ## Who generates the key, and why it must be the client Only the client knows that its second attempt is a retry of the first, rather than a genuinely new operation. If the server generated keys, every request would get a fresh one and nothing would ever deduplicate. So the key is client-supplied and its lifetime is tied to the client's *intent*, not to the request. This has a practical consequence people get wrong: the key must be generated **before** the first attempt and reused across all retries of that intent, including retries after a process restart. If your retry loop generates a new UUID each pass, you have built an elaborate way of doing nothing. Conversely, a client that reuses one key for two genuinely different operations will silently get the first operation's response back for the second — so keys must never be derived from something non-unique like a user id or a timestamp with second resolution. A good key is: unique per operation, opaque, unguessable enough that another tenant can't collide with it, and long-lived enough to survive the client's retry window. UUIDv4 satisfies all of that. Some clients derive it deterministically from the operation's own natural identity (order id + attempt intent), which is fine as long as the derivation is genuinely unique per intended effect. ## Where it belongs and where it doesn't **Use it on non-idempotent state-changing requests**: `POST /payments`, `POST /orders`, `POST /transfers`, and non-idempotent `PATCH` operations such as balance adjustments. **Don't bother on**: GET and HEAD (safe — nothing to deduplicate); PUT and DELETE (already idempotent by method, since re-sending the same replacement or removal converges). Adding keys everywhere just adds storage and confusion. **Send it as a header, not a body field.** A header is visible to gateways, proxies, logging and routing layers, it is method- and content-type-agnostic, and it keeps deduplication independent of your payload schema. It also means an SDK can attach it automatically to every mutating request without understanding the body. ## What the pattern does not do It is not distributed transactions, and it is not a general concurrency-control mechanism — it doesn't stop two *different* clients writing conflicting updates (that is what ETag preconditions are for). It only collapses repeats of the *same* logical operation. It is also not free: the server must maintain a durable dedupe store, decide the key's scope, decide how long keys live, handle two retries arriving concurrently, and decide what to do when the same key arrives with a different payload. Those are the follow-up questions that separate a candidate who has read about the header from one who has implemented it. ## The one-line version The client names the operation with a unique key; the server promises that a key is executed at most once and that later arrivals get the original answer back. That turns an unsafe POST retry into a safe one.

  • Should a client generate a new key on each retry attempt?
    No — that defeats the entire mechanism. The key identifies the logical operation, not the HTTP attempt, so all retries of one intent must carry the identical key. A new key is generated only when the client genuinely wants a second, separate operation.
  • Do you need an idempotency key on PUT and DELETE?
    Normally not, because those methods are already idempotent by definition — re-sending the same replacement or removal converges on the same state. A key can still help if you want to replay the exact original response body or if the handler has non-convergent side effects, but the common case does not need it.
  • Why is the key sent as a header rather than a field in the JSON body?
    A header is independent of the payload's content type and schema, so a gateway, proxy or SDK can read and act on it without parsing the body. It also lets a client library attach the key automatically to every mutating request, and keeps deduplication metadata out of your domain model.

Like a ticket number at a counter: you keep the same ticket while you wait, and showing it again gets you the same order, not a second one.

saying these in an interview costs you the question

  • Saying the server generates the idempotency key.
  • Generating a fresh key on each retry attempt.
  • Putting the key in the request body instead of a header.
  • Requiring keys on GET requests, which are already safe.
  • Believing the header alone provides idempotency with no server-side dedupe store.

context

resolving links…

open as a page

You are implementing support for the Idempotency-Key request header on a POST endpoint. What does the server store, and what is the request-handling flow for a first request versus a replay?

level: middleimportance: must knowfreq 55%

basics

~20 s

Store a row keyed by (scope, key) holding a fingerprint of the request, a state (in-progress / completed), and the saved status, headers and body. First request: insert the row, execute, save the response. Repeat: find the completed row and return the stored response without re-executing.

open as a page

Your HTTP client times out waiting for a response to a POST request. What do you actually know about whether the server applied it, and how should the client behave?

level: middleimportance: must knowfreq 56%

basics

~20 s

Almost nothing: a timeout means you lost the answer, not that the work did not happen. The request may have been fully applied. Retrying a non-idempotent POST can duplicate the effect, so either make the operation retry-safe first, or reconcile by querying before retrying.

open as a page

A client's write to your API is rejected with HTTP 412 because the version token it sent was stale. Walk through what the client should do next, and what the API has to provide for that step to be possible.

level: middleimportance: should knowfreq 40%

basics

~20 s

Re-read the resource to get current state and a fresh validator, reconcile the user's intended change against what actually changed, then resend with the new validator. Never blindly resend the same body with a refreshed token, and cap the retry loop.

open as a page

A client reuses the same Idempotency-Key HTTP header value but sends a different request body than the first time. How should the API respond, and how does the server detect this?

level: middleimportance: should knowfreq 42%

basics

~20 s

Reject it without executing anything. The server stores a fingerprint (hash of method, path and body) with each key and compares on every arrival; a mismatch means the client has a bug, so return an error — commonly 422 Unprocessable Content — rather than replaying or executing.

open as a page

How would you design a resource-creation call so that a client retrying after a network failure cannot create two records — by having the client choose the resource's URL and using PUT instead of POST to a collection?

level: middleimportance: should knowfreq 32%

basics

~20 s

Let the client generate the identifier (a UUID) and PUT to /orders/{id}. The URL names one resource, so a replay writes the same record instead of creating a second. Guard the first write with If-None-Match: * so an existing resource is not silently overwritten.

open as a page

Your API currently accepts unconditional PUT requests, and you want every write to a sensitive resource to carry a precondition, rejecting the ones that do not with HTTP 428 Precondition Required. How would you roll that out across existing clients?

level: seniorimportance: should knowfreq 26%

basics

~20 s

Decide which resources genuinely need it, ship the ETag on reads first, then run a warn phase where unconditional writes are accepted but counted per client. Once that count hits zero for the resources you are enforcing, flip them to 428 behind a flag with a documented, actionable error.

open as a page

You are adding ETag-based preconditions to a resource with many independently edited fields and embedded sub-objects. How do you decide what the ETag value should be computed over, and what breaks if you choose badly?

level: seniorimportance: should knowfreq 34%

basics

~20 s

Compute the validator over exactly the state the endpoint protects. Whole-resource validators are simple but cause false conflicts when clients edit unrelated fields; finer granularity means exposing those parts as their own resources with their own ETags, not narrowing the token on the same URL.

open as a page

Two retries carrying the same Idempotency-Key HTTP header arrive at your API at the same moment, before the first has finished. What must the server guarantee, and how do you implement it?

level: seniorimportance: should knowfreq 45%

basics

~20 s

Exactly one may execute. Claim the key with an INSERT under a unique constraint before doing any work — the database picks the winner. The loser sees an in-progress record and returns 409 Conflict (or waits briefly), never executing. In-progress rows need a lease so a crash doesn't block the key forever.

open as a page

When implementing the Idempotency-Key HTTP header, what namespace should a key be scoped to, how long should stored keys be retained, and what happens to a client that retries after that retention window expires?

level: seniorimportance: should knowfreq 38%

basics

~20 s

Scope keys per authenticated principal (API key, account or tenant) — never globally, or one tenant's key can collide with another's and replay their response. Retain typically 24 hours. After expiry the key is unknown, so a retry re-executes and duplicates the effect; publish the window and require clients to stop retrying before it.

open as a page

Design an API for an operation that takes several minutes to finish, using HTTP 202 Accepted plus a separate status resource. How do client retries fit into that design?

level: seniorimportance: should knowfreq 38%

basics

~20 s

The submit call validates, persists the job, and returns 202 with a Location header pointing at a status resource. The client polls that URL for state and follows a link to the result on completion. Retrying the submit must resolve to the same job; polling a status URL is always a safe GET.

open as a page

A service starts degrading and every client begins retrying, which keeps it down. How would you set retry policy across clients — backoff, jitter, attempt caps, budgets — and what should the API communicate through the HTTP Retry-After response header?

level: principalimportance: should knowfreq 42%

basics

~20 s

Retries multiply load exactly when capacity is gone. Use exponential backoff with jitter, a small attempt cap, and a retry budget capping retries as a fraction of successful traffic. Retry at one layer only, honour Retry-After on 429 and 503, and shed load rather than queueing it.

open as a page

Your POST endpoint honours the Idempotency-Key header, but the operation it performs is a call to an external payment provider you do not control. How do you keep the stored idempotency record and the external side effect consistent across crashes?

level: principalimportance: nice to knowfreq 30%

basics

~20 s

You cannot make a local commit and a remote call atomic, so you record intent first, then call, then record the outcome — and you pass your key through as the provider's idempotency key. A crash mid-flight is then recoverable: re-calling the provider with the same key deduplicates at their end.

open as a page