A handler responds 200 and then runs the real work on a thread it spawned itself — what breaks in production?

## Why it looks correct Answering the caller quickly and doing the slow part afterwards is the right instinct. The response is not waiting on an email, a thumbnail or a downstream write, so the client sees low latency and the endpoint looks healthy in every dashboard. In a demo and in most tests, the work also happens: the process stays up for the whole scenario, the task runs, the effect appears. This pattern usually survives months of production traffic before anyone notices what it costs. ## What is actually missing 1. **Durability.** The only evidence the work is owed is an object in memory. When the process is replaced — and in a normally operated system it is replaced constantly, for deploys, scaling and host maintenance — everything unfinished disappears. There is no list of what was lost, so nobody can reconcile it afterwards. 2. **Backpressure.** Spawning per request lets arrival rate set concurrency. A traffic spike creates tasks as fast as requests arrive, and those tasks contend with request handling for threads, memory and dependency connections. The failure looks like a serving problem, so scaling on request metrics adds capacity for the wrong cause. 3. **Retry and dead-lettering.** A dependency that is briefly unavailable turns a recoverable fault into permanent loss. There is nothing to hold the item while the dependency recovers and nowhere to park an item that keeps failing. 4. **Visibility.** The caller is gone, so a failure produces no error response, no client retry and no support ticket. Unless counters were added deliberately, the loss rate is unknown — and "unknown" is usually reported as zero. ## In-process task versus recorded handoff | Property | Self-spawned in-process task | Obligation recorded durably first | |---|---|---| | Survives a restart | No — lost with the process | Yes — the record outlives the process | | Knows what was lost | Nothing to reconcile against | The unfinished records are the list | | Behaviour under a burst | Unbounded growth, competes with serving | Backlog grows where it can be measured | | Transient dependency failure | Permanent loss | Retried, then dead-lettered | | Failure visibility | Only what you counted | Backlog age, failure counts, dead-letter count | | Cost to operate | Nothing extra to run | A consumer or sweeper, and its alerts | ## If it genuinely must stay in-process Sometimes a durable handoff is not worth it — the work is regenerable, best-effort, or trivially repeated by the next request. Then make the losses bounded and known: - Submit to a **fixed-size pool with a rejection policy**, never an unbounded spawn, so a burst produces countable rejections instead of resource exhaustion. - Give every task a **timeout**, so one stuck call cannot hold a worker forever. - Make the effect **idempotent**, so a client retry or a later repair pass cannot double it. - Export counters for **submitted, completed, rejected and failed**, so "we lose a few" is a measured number and not a hope. - Write down, in the code and in the runbook, **which work may be lost**, so the next person does not quietly add something that may not. ## The smallest change that fixes it properly Record the intent as part of the write the request was already making: a row, a document, an entry with a state field, committed before the response goes out. Once the obligation is durable, who performs it becomes an implementation choice — a consumer, a periodic sweep for unfinished records, or even the same process, which may now crash without losing anything. That is the real insight: **durability comes from the record, not from the runner.** A background thread with a durable record behind it is recoverable; a durable-looking queue client called from a thread that nothing recorded is not. ## How it shows up in an incident The usual report is not an outage. It is "some customers did not get their confirmation last Tuesday", and the counts never quite reconcile. Correlating the gaps with deploy timestamps is the diagnosis, because the loss window is exactly the moment the process was replaced. By then there is no record of the missing items, so the recovery is a manual reconstruction from whatever other trace the request left — which is the strongest argument for making the obligation durable at the moment it is accepted.