Blocking, Async & Streaming

Thread-per-request frameworks give each request a pooled worker thread for the whole exchange. Event-loop frameworks run handlers as short turns on a few shared loop threads. Coroutine frameworks suspend the handler at await points and release the carrier thread meanwhile.

Streaming keeps memory flat and bytes moving. The handler writes pieces the framework sends as they are produced, so a large response costs a small buffer per request instead of its full size, and the client sees data sooner.

A request timeout bounds how long the framework waits for a response, not how long the handler runs. When it fires, the framework abandons the result and writes an error response; the handler usually keeps executing.

A failed awaitable is delivered to the framework's error-mapping stage, the same one that handles a thrown error, and becomes an error status. An awaitable that never settles produces no response at all: the exchange stays open holding its resources.

The handler worker count is the ceiling: one request occupies one worker for its whole exchange, waiting included. Beyond it requests sit in the server's connection or dispatch queue, so clients see queue time, then connection refusals or timeouts, while handler timings still look healthy.

Headers travel ahead of the body, so the first flush freezes the status code, every header field, cookies and the body's framing choice. Later changes to them are lost; only more body bytes can follow.

Nothing forcibly stops running code, so an abandoned handler keeps its worker until its current call returns. It stops early only if the framework signals cancellation and the handler reaches a point where it can observe it.

An after-response hook runs framework-scheduled code once a request's response has been written, for cleanup, metrics and emitting events. It stays in the serving process and unsupervised: it cannot change the response, and its failures have no caller left to tell.

Nothing durable records that the work is owed, so a deploy, crash or scale-in silently loses whatever is in flight or queued in memory. There is also no backpressure, no retry, no dead-letter, and no caller left to report a failure to.

Request-scoped objects — request and response, body streams, per-request context, a borrowed connection — are torn down or returned to a pool when the response ends. Deferred code holding a reference finds them closed, cleared, or already serving another request.

Submit the blocking call to a bounded pool of your own, compose the response mapping onto the handle that submission returns, and return that handle. Never wait for the result inside the handler: that reintroduces the block.

Writing is not sending. The framework buffer, a content encoder, the socket, an intermediary and the client's parser can each hold bytes, so incremental writes arrive as one batch unless every layer is flushed.

Two ways: a write to the closed connection fails, or the transport reports the close and the framework fires a disconnect event or cancellation signal. A handler that buffers notices nothing until its final write.

A wrapper that does its after-work right after calling the next stage runs that work when the handler returns its handle, not when the response exists. Timings, cleanup and error catching all land in the wrong place.

Thread-keyed storage assumes one thread serves one request end to end. Under loop or coroutine execution a thread interleaves many requests and a request touches several threads, so context goes missing after a resumption, or belongs to somebody else.

A stalled loop stops every duty it owns: reading and writing its sockets, completing handshakes, firing its own timers, often accepting connections. Sockets stay with the loop that accepted them, so the damage covers that loop's share of clients.

A framework does one of three things: the write blocks and pins whatever runs the handler, unsent bytes queue in memory and grow per slow connection, or the write signals not-ready so the handler must pause producing.

Timed-out handlers are still running. Confirm by comparing in-flight count with response rate and sampling where workers are parked, then bound the slow call itself, propagate cancellation into it, and cap concurrency to that dependency rather than enlarging the pool.

Replace the missing caller with three channels: automatic recovery (bounded retry, then a dead-letter place), operator visibility (failure counters, backlog age, a correlation identifier), and a user-facing status or follow-up. And never claim in the response what the deferred work has not yet done.

Convert endpoints that hold many concurrent requests whose time is mostly spent waiting on remote calls that can genuinely be awaited. Leave compute-bound endpoints, and those fronting a hard downstream limit, alone: converting them only relocates the queue.

Choose from the workload and the ecosystem, not from fashion: connection count and waiting time argue for loop or coroutine models, processor-bound work does not, and blocking client libraries can cancel the benefit entirely. Weigh it as a stack-wide, hard-to-reverse commitment.

Measure what one open stream costs on the real execution model, find the limit that binds first, cap admission below it, bound every stream's lifetime so the fleet stays drainable, and bound per-stream buffers.

Decide by the cost of losing an item, not by how long it takes. Best-effort, regenerable work can stay in the process behind bounds; anything the user was told happened, or that a person would have to repair, must be recorded durably before you respond.