skip to content

What do proxies, message queues and SDK retry policies assume about HTTP method idempotency, and how does that break an API that exposes state-changing operations as POST?

level: seniorimportance: must knowfreq 50%

answer

  1. infrastructure trusts the method declaration - it cannot read your logic
  2. proxy/mesh retries + SDK backoff default to GET/PUT/DELETE
  3. queues are at-least-once: every consumer is a retried request
  4. duplicates emerge only under timeouts/deploys - invisible in tests
  5. dedupe key committed in the same transaction; retention > retry window

basics

~20 s

They assume the method declaration is true: GET/PUT/DELETE are replayable, POST is not. Queues add at-least-once delivery, so any POST-shaped consumer duplicates. The result is duplicate writes appearing under load or partial failure, with no single component at fault.

solid answer

~50 s

Every layer acts on the method's declared properties. - **Proxies, load balancers and service meshes** retry idempotent methods on connection failure or timeout, often silently, and may replay a request against a second upstream after the first appears unhealthy. - **HTTP client libraries and cloud SDKs** ship retry-with-backoff defaults scoped to GET/PUT/DELETE, sometimes extended to any 5xx or 429. - **Message queues** guarantee at-least-once delivery: a consumer that acknowledges after processing will reprocess anything whose ack is lost. Effectively every queue-driven handler is a retried request. Against a POST-shaped write, the failure mode is duplicate records that only appear during timeouts, deploys or upstream slowness - so they are invisible in tests and arrive as an incident. The practical hardening: never expose a state-changing GET; keep gateway retries off POST unless the route is documented as deduplicating; make queue consumers idempotent on a business key rather than trusting exactly-once; and where duplication is expensive, add a server-side deduplication key committed in the same transaction as the effect.

code

http · 14 lines
http
POST /payments HTTP/1.1
Idempotency-Key: 8f3d1b2a-77c4-4a91-9a2e-0d5c1f6b3e88
Content-Type: application/json

{"amount":4200,"currency":"EUR"}

HTTP/1.1 201 Created
Location: /payments/pay_7731

--- replay of the identical request ---

HTTP/1.1 200 OK
Idempotent-Replayed: true
Location: /payments/pay_7731

go deeper

for a junior

Say that proxies, clients and queues retry automatically, so a POST that creates something can run twice; keep it to the mechanism.

for a middle

Name the specific layers and their defaults, and explain that at-least-once queue delivery makes every consumer effectively a retried request.

for a senior

Diagnose it as an emergent, load-correlated failure with no single faulty component, and give the hardening order including transactional deduplication and observability.

for a principal

Treat idempotency as a system-wide invariant every intermediary depends on, and decide where deduplication lives, what it costs in storage and latency, and how retention is chosen against retry windows.

## The shared assumption Generic HTTP infrastructure cannot inspect your business logic. Its only signal about whether a request may be replayed is the **method**, and it treats the declaration as truth. That assumption is what makes caches, retrying proxies and automatic client retries possible at all - and it is what turns a lie about method semantics into a distributed correctness bug. ## Who replays what **Caches and prefetchers.** Anything declared GET may be stored and served without reaching your server, and may be fetched with no user intent by crawlers, link scanners, prefetching browsers and preview generators in chat clients. A state-changing GET therefore fires unprompted. This is not theoretical - link-prefetching agents following GET-based administrative links has destroyed real data. **Proxies, load balancers and service meshes.** These retry idempotent methods on connection errors, timeouts and configured status codes, and can hedge a slow request onto a second upstream. Envoy-style retry policies and Kubernetes ingress controllers all expose this, and the retry is invisible to your application - the second upstream sees an ordinary first request. **Client libraries and SDKs.** Cloud SDKs and HTTP clients ship retry-with-backoff by default. The scoping is usually method-based, but plenty of configurations retry any 5xx or 429 regardless of method, and application developers enable "retry all" without auditing routes. **Message queues.** Every mainstream broker offers at-least-once delivery. If a consumer processes a message and then fails before acknowledging, the message is redelivered and processed again. So any HTTP write driven by a queue consumer is, structurally, a retried request - the same duplicate-execution hazard as a POST retry, with the added twist that redelivery can happen minutes later, defeating short-lived deduplication windows. ## Why it breaks POST-shaped APIs specifically A POST that creates or charges has no replay protection of its own: two identical POSTs are two creations, by design. Layer the above on top and duplicates appear whenever the network is imperfect - which is exactly when nobody is watching: - A p99 latency spike trips a gateway timeout, the gateway retries, the original request completes anyway - two records. - A rolling deploy drops in-flight connections; the client library retries - two records. - A queue consumer crashes after the downstream POST but before acking - two records. The defining property is that no single component is buggy. Each behaved as configured and as documented. The duplicates emerge from the composition, which is why they survive code review and integration tests and surface as an incident with double charges or double emails. ## Hardening, in order of leverage 1. **Never expose a state-changing GET.** This is non-negotiable; caches and prefetchers will exercise it without a user. 2. **Audit retry configuration per route, not globally.** Default gateway and client retries to GET, PUT and DELETE. Turn on POST retries only for routes documented as deduplicating. 3. **Prefer state-describing operations.** Where the client can choose the identifier, `PUT /resources/{client-uuid}` makes retry safety a property of the method rather than of configuration - infrastructure gets it right for free. 4. **Make queue consumers idempotent on a business key.** Do not rely on exactly-once semantics; treat redelivery as certain and deduplicate on an order id, event id or hash. Retention must cover the broker's redelivery window, which can be long. 5. **Add a server-side deduplication key for expensive POSTs.** A caller-supplied unique key, stored **in the same transaction as the effect**, lets the server return the first outcome on a replay. Scope keys per caller, define what happens when a replay arrives with a different body (reject with 422), return 409 while the original is still in flight, and set retention beyond the longest realistic retry window. 6. **Make duplicates observable.** Alert on deduplication hits and on same-key repeats. A rising deduplication rate is an early signal that something upstream started retrying. ## The framing that reads as senior Say that idempotency is not a property of one endpoint but a **system invariant**: every intermediary between client and database assumes it, so violating it produces failures that are emergent, load-correlated and unattributable to any single component.

  • Why is it not enough to check 'does a row with this key already exist' before inserting?
    Because the check and the insert are two steps, and concurrent retries can both pass the check before either commits, producing two rows. The deduplication record must be written under a uniqueness constraint in the same transaction as the effect, so the database serialises the race and the loser detects the conflict.
  • A queue broker advertises exactly-once delivery. Can you drop consumer-side deduplication?
    No. Such guarantees hold only inside the broker's own transactional boundary and do not extend to side effects your consumer performs against an external system, such as an HTTP POST or an email send. Once processing crosses that boundary the delivery can be repeated from the consumer's perspective, so business-key deduplication in the consumer stays necessary.

saying these in an interview costs you the question

  • Assuming duplicates are a client bug rather than an emergent property of retry layers
  • Trusting a queue's exactly-once claim to cover external side effects
  • Enabling gateway retries on all methods to improve availability
  • Checking for an existing record before inserting instead of relying on a uniqueness constraint in one transaction
  • Setting deduplication key retention shorter than the broker or client retry window

context

resolving links…