How would you decide which post-response work may stay in the serving process and which needs a durable handoff?

## The axis that actually decides it The question is usually posed as "is this too slow to do in the request?", which is the wrong axis for the wrong reason. Duration decides whether work belongs *before* the response; it says nothing about what happens if the work never runs. The deciding question is: > If this item vanished silently during a routine deploy, who would notice, and what would it take to repair? Answers range from "nobody, and nothing" to "the customer, and a manual reconciliation against a payment provider". That range, not a stopwatch, is what separates the options. | Axis | Pushes toward in-process | Pushes toward a durable handoff | |---|---|---| | Cost of silent loss | Negligible, regenerable | A person must repair it, or a user was told it happened | | What the response claimed | Nothing, or an internal effect | The response said the action happened | | Volume and burstiness | Small, smooth, bounded | Spiky; needs a buffer between arrival and processing | | Capacity coupling | Cheap work that will not disturb serving | Heavy work that would compete with request handling | | Retry semantics | Nothing to retry, next request repeats it | Needs backoff, caps and a dead-letter place | | Operational cost | Nothing extra to run | A consumer, its alerts, and a second failure domain | ## Three tiers, not two 1. **Best-effort, in-process.** Metrics, cache warming, an optional trace export. Requirements: a bounded pool with a rejection policy, a timeout per task, counters for submitted, completed, rejected and failed, and an explicit written note that losing these is acceptable. 2. **Durable record, simple runner.** The obligation is written in the same durable store the request already touches, before the response goes out. A poller or sweeper claims unfinished records and performs them, with attempts and last error kept as columns of the record. This gives recoverability without introducing anything new to operate — often the right answer for a service that already has a database and no message infrastructure. 3. **Durable handoff to a dedicated system.** Warranted when the work needs fan-out to several consumers, ordering guarantees, isolation so heavy processing cannot disturb serving, or independent scaling of the workers. It is the most capable option and the most expensive: a second failure domain, poison-message handling, backlog alerting, and one more thing to upgrade. The middle tier is the one teams skip. They oscillate between "just spawn a task" and "stand up a broker", when a record with a state column plus a periodic sweep provides the durability that actually mattered. ## What a handoff costs, honestly - **A second failure domain.** The broker's availability becomes your endpoint's availability if you enqueue synchronously inside the request. - **Duplicate delivery.** Most systems deliver at least once, so every effect must be idempotent regardless of tier. - **Two places where the truth lives.** A record written in one store and a message sent to another can diverge; either enqueue from the record, or accept reconciliation. - **New operational surface.** Backlog alerts, dead-letter review, consumer deploys, and an on-call story for the consumer as well as the service. These are worth paying when the loss cost justifies them, and pure overhead when it does not — which is why the decision is made per obligation, not once per service. ## Capacity is the constraint people forget Work kept in the serving process shares that process's threads, memory and dependency connections with request handling. A burst of deferred tasks raises request latency in a way that looks exactly like a serving problem, and autoscaling on request metrics will happily add instances that each start doing more background work. Any in-process tier therefore needs a hard bound, so the failure mode under load is countable rejection rather than an unpredictable slowdown of the whole service. ## Making it a rule other teams can apply Technology rules age badly; promise rules do not. A durable version is: **if a response tells the user something happened, the effect must be durable before that response is written.** It needs no knowledge of which handoff mechanism is currently favoured, it is checkable in review by reading the response body, and it puts the argument where it belongs — on what the endpoint claims. Pair it with one supported, documented path for tier 2 so that the easy choice is also the correct one, and a periodic reconciliation sweep so that unfinished obligations surface even when a tier was chosen wrongly.