Chunked & Streamed Bodies

A response body can be produced in one of two shapes, and the choice decides how much memory the service holds, how soon the client can start working, and what the framework is still able to do about the response later. ## The two shapes In the **buffered** shape the handler builds the whole body first — a string, a byte array, a serialized object — and hands that finished value to the framework. The framework knows the exact size before anything leaves the process: it can set a length header, compress the whole payload, and write it in one pass. The handler has already returned by the time the first byte reaches the network. In the **streamed** shape the framework hands the handler a sink — a writer, an output stream, a generator it pulls from, or a callback it invokes with each piece — and the handler produces the body in parts. Bytes leave the process while the handler is still running, and the handler is done only when it closes the sink. ## Where the memory goes The decisive number is rarely the size of one response. It is **size x concurrency**, because every in-flight request holds its own copy: - A 200 MB export built in memory costs about 200 MB of live data per concurrent download; twenty of them is roughly 4 GB, on top of everything else the process is doing. - Serialization usually costs more than the final size, because an intermediate representation and the encoded bytes coexist for a moment. - The streamed version holds only a small write buffer (typically kilobytes) plus whatever single row, record or chunk is being produced right now — a per-request cost that does not grow with the body. - The failure is non-linear. Memory looks fine at low concurrency and the service falls over the moment concurrency crosses a line, which is why this shows up as an incident rather than as a slow regression. Large, short-lived buffers are also the worst possible shape for a garbage-collected runtime: they are allocated in one piece, survive long enough to be promoted, and are then discarded, which is exactly the pattern that produces long pauses. ## What the client experiences Streaming does not make the network faster and does not reduce the bytes transferred. What it moves is **when** the first useful byte arrives. A buffered build of a large body looks to the client like a hang — nothing arrives until the whole thing is ready, which can exceed a client or intermediary read timeout even though the server is working correctly. A streamed body starts arriving almost immediately, so a consumer can parse, render or write to disk progressively, and the connection shows steady activity instead of silence. ## What streaming costs | Concern | Buffered body | Streamed body | |---|---|---| | Peak memory per request | Proportional to the body | A small write buffer | | Time to first byte | After the whole body is built | Almost immediately | | Length known up front | Yes, the framework can set it | No, unless you know it another way | | Resources held during transfer | Released when the handler returns | Held until the last byte is written | | Reacting to a mid-body failure | Full freedom, nothing has been sent | Constrained, the response is already committed | Three of those deserve emphasis. First, without a known length the framework must use a framing that does not need one, and the client cannot show a percentage-of-total progress bar. Second, a cursor, connection or file handle the handler reads from stays open for the whole transfer, so the slowest client now decides how long a scarce resource is pinned. Third, the response commits as soon as the first bytes are flushed, which changes what can still be done if the producer fails halfway. ## When buffering is the better choice Streaming is not a free upgrade. Prefer a buffered body when: 1. The body is small — for typical record-sized responses the buffer is irrelevant and the buffered path is simpler and easier to fail out of. 2. The exact length or a digest of the whole body is required before sending. 3. Failing cleanly matters more than starting early, because a buffered response can still be replaced by an error response. 4. The result is cacheable or reusable, in which case something has to hold the whole value anyway. ## A working rule Stream when the body is unbounded or large relative to the per-request memory budget, or when the first part is useful before the last part exists. Buffer otherwise. And measure the real thing: peak memory per in-flight request and time to first byte are the two numbers that tell you which shape you actually have, regardless of which one the code appears to describe.