A streaming handler writes rows as it produces them, yet the client receives them all at once at the end — why?

Writing is not sending. A handler's write call hands bytes to the layer below it, and every layer between the handler and the consumer's parser is allowed to hold onto those bytes until it has a reason to pass them on. When all of them hold, the stream still *works* — it is simply delivered as one batch at the end, which defeats the entire point of streaming. ## The layers that can hold your bytes 1. **The framework's response buffer.** Most frameworks accumulate small writes and pass them down when the buffer fills, because one syscall per row would be ruinous. A handler writing short rows may fill that buffer only after thousands of them, or never. 2. **A content encoder.** Compression works on blocks and needs input before it can emit anything. An encoder placed in front of the output has its own buffer, and flushing the writer without flushing the encoder simply moves the data from one buffer to another. 3. **The socket send buffer and the transport's own batching.** The operating system may coalesce small writes to avoid sending many tiny segments, which adds a delay that is invisible from application code. 4. **An intermediary.** A reverse proxy, gateway, load balancer or content-delivery network may buffer the whole upstream response before forwarding any of it — often the default, because it frees the upstream connection sooner. This is one of the most common causes in production, and **no application-level flush can override it**. 5. **The consuming client.** A client that reads complete lines, complete records or a fully parsed document surfaces nothing to the caller until its own parser is satisfied, even when the bytes arrived on time. ## What a flush does, and what it does not A **flush** is an explicit instruction to push whatever is buffered at that layer down to the next one. It only affects the layer you call it on, so on a path with an encoder and a writer, both must be flushed. And it is always a request: a flush can push bytes into the socket, but it cannot make an intermediary forward them, and it cannot make a client's parser emit an incomplete record. Flushing is not free, either. Every flush costs at least one write down the stack and typically one network segment, so flushing per byte or per tiny record trades throughput and CPU for latency. The usual compromise is to flush on a **meaningful unit** — a complete record, a batch of rows, or a time interval — rather than on every write. | Symptom | Likely layer | What to change | |---|---|---| | Nothing until the very end, any size | An intermediary buffering the whole response | Disable response buffering on that hop for this route | | Output arrives in large, even blocks | Framework or encoder buffer | Flush at record boundaries, flush the encoder too | | Bytes arrive on time but the app sees nothing | The consuming client's parser | Read incrementally, parse per record | | Fine locally, batched in production | Only production has the intermediary | Compare a direct call with a call through the edge | ## How to diagnose it in the right order The path has several buffers and guessing wastes a day, so bisect it instead: 1. Read the endpoint **directly from the server process**, with a consumer that prints bytes as they arrive with timestamps. If the timing is correct here, the handler and framework are fine and the problem is in front of them. 2. Repeat **through each hop** — the reverse proxy, then the edge — and the hop where the timing collapses is the buffering one. 3. If even the direct read batches, remove the encoder and re-test to separate the compressor from the writer, then check whether the handler flushes at all. 4. Only then look at the client library, by comparing what a raw byte-level reader sees against what the high-level parser surfaces. ## Design consequences - **Choose a flush unit deliberately.** Per record is right for a low-rate event stream; per batch or per time window is right for a high-volume export where per-record flushing would dominate the cost. - **Treat the whole path as part of the feature.** A streaming endpoint that is never exercised through the same hops as production is not tested. Buffering at the edge is configuration, and configuration is where the behaviour actually lives. - **Be careful with compression on low-rate streams.** An encoder needs input to emit a block, so it can add latency precisely where latency is the feature; either flush it per record, accepting worse compression, or leave the route uncompressed. - **Do not confuse a hold with a hang.** A stream held in a buffer and a stalled producer look identical from the outside. Timestamped byte-level reads are what distinguish them, which is why the diagnosis starts there. The one-line version worth remembering: a write reaches the next buffer, a flush pushes it one layer further, and delivery to the consumer requires **every** layer on the path to cooperate.