A client reads a streamed response far slower than the handler produces it — what does the framework do with the unsent data?

A streaming handler produces at whatever rate its source allows. The consumer reads at whatever rate its network, its CPU and its own patience allow. When the second is slower than the first, the difference has to go somewhere, and there are essentially three places it can go: into the producer (it waits), into memory (the bytes pile up), or into the bin (something is discarded). In practice a framework's write API lands in one of these three shapes, and knowing which one you are on decides how the service fails under load. ## Why the difference cannot just vanish The socket's send buffer is finite. Once it is full the transport will not accept more bytes until the consumer acknowledges what it already has, which it does at its own pace. From the handler's point of view the write call has nowhere to put the data. What happens next depends entirely on the shape of the write API the framework hands the handler. ## The three outcomes | Shape of the write API | What happens on a slow reader | The resource that runs out | |---|---|---| | Blocking write | The call parks until space frees up | The thread or worker, held for the slowest client's duration | | Non-blocking write with an unbounded queue | Unsent bytes accumulate in process memory | Memory, and the whole instance with it | | Write that signals "not ready" | The producer is expected to pause and resume | Nothing, if the handler honours the signal | 1. **It blocks.** The write parks until the transport accepts more. Nothing is lost and memory stays flat, but whatever executes the handler is pinned for as long as the slow consumer takes. In a model that dedicates a worker per in-flight request this converts one slow reader into one unavailable worker, and enough of them into a service that accepts no new requests at all. It is also why a single slow reader can hold a pooled database cursor open for minutes. 2. **It queues.** The write returns immediately and the framework holds the unsent bytes. Memory now grows at the difference between produce rate and drain rate, per slow connection. This is the most dangerous shape, because it looks perfect in testing — fast readers barely queue anything — and it fails as an out-of-memory event under exactly the conditions that matter: mobile clients, congested links, a consumer doing expensive work per record. 3. **It pushes back.** The write returns a signal, a completion handle or a "wait for drain" callback, and the contract is that the handler stops producing until the transport is ready again. This is the only shape that bounds memory and the producer without discarding anything — but only if the handler actually honours it. A handler that ignores the signal and keeps writing has silently opted into shape 2. ## Discarding, and when it is correct For a stream of live updates, buffering is often the wrong answer anyway: a consumer minutes behind does not want the backlog, it wants the current state. Those streams are built on a **bounded** per-connection queue plus a discard policy: - **Drop oldest** — keep the newest state, right for telemetry, prices, positions and progress updates. - **Drop newest** — keep an intact prefix, right when early records are the meaningful ones. - **Collapse** — replace the queued value for a key with the latest one, so a slow consumer gets a compacted view rather than a backlog. - **Disconnect the consumer** — the honest option when the data must not be silently incomplete: close the stream and let the client come back for a fresh, consistent read. The important half of that decision is the contract. A consumer that does not know discards are possible will treat the stream as complete and be quietly wrong, so the discard policy has to be part of the documented interface, not an implementation detail. ## Operating this in production - **Measure per-connection buffered bytes**, not just total memory. The distribution is what tells you a few consumers are dragging, long before the aggregate looks alarming. - **Bound every queue.** An unbounded buffer is a deferred out-of-memory event with a slow fuse; a bounded one with an explicit policy fails in a way you chose. - **Set a write or idle-progress timeout**, so a consumer that has effectively stopped reading does not hold resources indefinitely. - **Cap concurrent streams per instance**, because each one holds a connection, a buffer and, depending on the execution model, something more expensive. - **Watch for the load-test blind spot.** Fast local consumers never trigger any of this; you have to test with a deliberately throttled reader to see which of the three shapes you are actually running. ## The takeaway Ask one question of any streaming code: *when the consumer is slower than the producer, what gives?* If the answer is "the thread waits", you have a concurrency ceiling. If it is "memory grows", you have an incident waiting for a bad network. If it is "the producer pauses", check that the handler really honours the signal — and if the data is live, ask whether the right answer is to discard rather than to hold anything at all.