In an async request-reply system, a client's initial POST times out on the network before the client receives the 202 response, so the client doesn't know whether the server actually accepted the job. What should the API design do to prevent this from silently creating duplicate jobs, and what other failure modes does a production implementation of this pattern need to guard against?

## The shape of the problem Production async request-reply systems fail in a small, well-understood set of ways, and each one maps to a specific, well-established mitigation: 1. duplicate submission 2. the worker crashing mid-processing 3. the poll storm 4. orphaned job records 5. duplicate webhook delivery ## First — the duplicate submission The first failure mode is the one in the question: a client submits the initial `POST`, the network drops the response before the client sees the 202, and the client — reasonably, since it cannot distinguish 'server never got it' from 'server got it but the ack was lost' — retries the exact same submission. Without any safeguard this creates a second, duplicate job doing the same work twice, which is wasteful at best and dangerous at worst if the operation has real-world side effects like charging a card or sending an email. The standard fix is a **client-generated idempotency key**, often a UUID, attached to the submission request. The server stores this key alongside the job it creates, and if it later receives another `POST` carrying the same key, it recognizes the retry and returns the existing job's status or location rather than starting new work. This differs from deduplicating by request-body content, because two requests with identical-looking payloads might legitimately be separate operations submitted on purpose; the idempotency key represents a specific attempt, letting the server say 'same key equals same attempt' without having to infer intent from the payload. ## Second — the worker crashing mid-processing A second failure mode is the worker crashing mid-processing after the job's status has already been flipped to 'running,' leaving the status stuck there forever with no update. - **Visibility timeout.** Queue-backed architectures address this with a visibility timeout: when a worker pulls a job off the queue, the message becomes invisible to other workers for a configured period so it isn't processed twice concurrently, and if the worker crashes before finishing and deleting the message, the message reappears once that timeout expires so another worker can retry it. - **Heartbeat or overall timeout.** This must be paired with the status store itself reflecting stalled jobs, typically via a heartbeat the worker writes periodically, or an overall timeout after which a job stuck in 'running' too long is flipped to 'failed' so a polling client isn't left waiting indefinitely for something that will never finish. ## Third — the poll storm A third failure mode is the poll storm: many clients polling a status endpoint aggressively with no backoff, especially when a batch UI action starts many jobs at once and all their clients begin polling in near lock-step. This is mitigated with a `Retry-After` header the server tunes to suggest sensible intervals, combined with: - enforced **exponential backoff and jitter** on the client side - potentially **rate limiting** on the status endpoint itself as a backstop ## Fourth — orphaned job records A fourth is orphaned job records: without a TTL or scheduled cleanup, job and result records accumulate indefinitely even though no client will ever query most of them again, which is purely an operational and cost problem rather than a correctness one, but it compounds over time in a busy system. ## Fifth — duplicate webhook delivery A fifth failure mode is specific to systems using webhook callbacks rather than polling: at-least-once delivery means the same completion notification can be delivered more than once, so the client must dedupe by job id or delivery id and treat duplicate delivery as normal rather than as an error. ## What to return for a job the server has lost Finally, consider what a status endpoint should return if the server genuinely has no record of a job at all, for instance after a database failover lost recent writes — it should return an explicit 'unknown' or `404` response rather than a generic error or an indefinite hang, so the client can clearly decide whether to resubmit rather than being stuck unable to tell 'never existed' from 'transient failure, keep polling.' A well-known real-world instance of the idempotency-key safeguard is Stripe's API, which accepts an `Idempotency-Key` header on `POST` requests specifically so that network retries of payment-creation calls never double-charge a customer.