Once the response is sent, a failure in the deferred work has no caller to return to — how do you design for that?

## Why a late failure is silent While a handler is running, failure has somewhere to go. It becomes a status code, the caller sees it, a client may retry, a user may complain, and the endpoint's error rate moves. Every one of those signals comes from the fact that somebody is waiting. Move the work behind the response and all of them vanish at once. There is no connection to write an error to, no client that will retry, no user who knows to complain, and the accepting endpoint keeps reporting success — it did succeed, at the only thing it promised. Failure after the response is therefore silent **by construction**, and the whole design problem is to put deliberate channels where the caller used to be. ## The response defines what a late failure means | What the response said | What a later failure means | What the design owes the user | |---|---|---| | "Done" — the action is complete | The response was untrue and the user acted on it | Do not respond this way unless the effect is already durable | | "Accepted" — we will do this | An expected, recoverable case | A way to see the outcome, or a follow-up message | | Nothing about the work at all | Purely internal; the user is unaffected | Operator visibility only | This table is the first design decision, not a detail. Much of the pain of deferred work comes from a response that claimed completion for something that had merely been scheduled. ## Three replacements for the caller 1. **The system tells itself.** Retry transient faults with backoff and a cap, then move the item to a dead-letter place with its failure cause attached. Because retries replay work, the effect must be idempotent — a natural key, a conditional write, or a state check — or recovery becomes duplication. 2. **The system tells an operator.** Counters for items accepted, completed and failed; the size of the dead-letter place; and the **age of the oldest unprocessed item**. That last one is the signal most teams are missing. 3. **The system tells the user.** A status they can check, a notification when the work completes or fails permanently, or an inbox message. If the user can only find out by noticing the absence of something, the design is incomplete. ## What to alert on - **Backlog age**, not backlog size. Size depends on throughput; age directly measures how stale the oldest obligation is. - **Dead-letter arrivals**, because each one is an obligation that nothing else will fulfil. - **Failure rate per stage**, not just a global rate, so a single failing downstream is distinguishable from a broad fault. - **Completion rate against acceptance rate.** If acceptances exceed completions for long, the gap is real work quietly going missing. The classic gap is alerting only on errors. A consumer that has crashed, deadlocked, or is stuck retrying one poison item emits **no errors at all**; the error rate is flat, every dashboard looks healthy, and work simply stops. Only age-based signals catch that. ## Carry the thread of identity forward The request that accepted the work has a correlation identifier. Copy it into the durable record and into every log line and metric the deferred work emits. Without it, an incident that starts as "this customer's order never completed" cannot be traced past the accepting request, because nothing downstream carries anything that connects to it. With it, one identifier follows the obligation from acceptance to completion or to the dead-letter place — which is what makes reconciliation possible at all. ## Reconciliation closes the loop Even with retries and alerts, some items end up in a state nobody watched. A periodic pass that looks for records accepted long ago and still unfinished is cheap to run and catches everything the live path missed: items lost before they reached a consumer, items whose consumer died mid-flight, items stuck in a state transition. Treat that sweep as part of the design rather than as an incident tool — it is the only component that answers the question "is anything we accepted still undone?" without being told where to look. ## The summary Deferring work moves failure out of the request's blast radius and into a place with no natural reporter. Design the reporter: retry then dead-letter for the machine, age and counters for the operator, a status or a message for the user, a correlation identifier tying them together, and a reconciliation sweep that assumes all of it can still miss something.