How would you set response buffering and commit policy across services that return both small payloads and very large exports?

## Frame the decision correctly The question is not "buffer or stream". It is **where the commit point should sit by default, who may move it, and what they owe when they do**. Commit is the moment the response stops being revisable, so moving it earlier buys latency and memory and sells away error reporting. That is a platform-wide tradeoff, and letting each handler make it implicitly - by producing a payload that happens to exceed a buffer nobody chose - is the failure mode to design out. ## The default: commit late For the overwhelming majority of responses, hold the whole body and commit at the end. This gives you: - a **replaceable response**: a failure at any point can still become a proper error status with a proper body; - a **declared length**, which clients can use for progress and completeness checks; - **one code path** for status, headers and cookies, with no size-dependent behaviour; - **simple testing**, because a small fixture exercises the same path as production. Size the buffer above the high percentile of normal payloads, so the overflow trigger is not hit by accident. Then price it: buffered bytes are per in-flight request, so the worst case is roughly buffer size times peak concurrency, and that number has to live alongside the memory budget, the concurrency limit and the payload caps. A buffer chosen without that multiplication is a capacity incident waiting for a traffic peak. ## The exception: commit early, by contract Some responses genuinely cannot be buffered - long exports, results assembled from a slow source, output whose first bytes matter for perceived latency. Make this an explicit, reviewable choice rather than an emergent one, and attach obligations to it: 1. **A completeness signal.** After commit, truncation is the only failure channel left, so the payload needs an end marker, a trailing status field or a declared item count that the client verifies. Without it, a partial answer is indistinguishable from a whole one. 2. **Abort on failure.** A streamed response that fails should end the connection rather than finish cleanly, so careful clients see an incomplete transfer. 3. **Bounded memory per response.** Streaming exists to cap memory; if the producer materialises everything first anyway, you have paid both costs. 4. **Separate instrumentation.** Post-commit failures are recorded as successes, so they need their own counter and alert, plus client-side tracking of short reads and parse failures. 5. **Client expectations documented.** Consumers must know the response may be delimited rather than length-declared, and how to detect a partial one. ## Where the policy is enforced | Layer | What it should own | |---|---| | Shared framework configuration | Default buffer size, concurrency and payload caps, error-stage behaviour on a committed response | | Service template or platform library | The sanctioned streaming helper that carries the completeness convention | | Review and architecture guidance | When an endpoint may stream at all, and what its consumers were told | | Observability defaults | Post-commit failure metric, truncated-response and short-read signals | | Edge and gateway configuration | Whether intermediaries re-buffer, and any response-size or timeout ceilings | The last row is easy to forget. A proxy in front of the service may buffer a response the application flushed early, converting a stream back into a whole message, or may impose its own size and time limits. Streaming is an end-to-end property; it is verified on the real path, not in a unit test. ## Tradeoffs to state out loud - **Memory versus recoverability.** Bigger buffers mean more responses stay replaceable, and more bytes held under load. These pull in opposite directions and the resolution depends on payload distribution and concurrency, not on principle. - **Latency versus honesty.** Early first bytes look better and make failures unreportable. For an interactive surface the trade is often worth it; for a machine consumer that retries on status, it is usually not. - **Uniformity versus fit.** One global buffer size is easy to reason about and wrong for outliers; per-endpoint tuning fits better and multiplies the configurations people must understand. Prefer one default plus a small number of named profiles. - **Caps as a safety net.** A maximum response size that refuses rather than streams is sometimes the right answer, especially for endpoints that were never designed to produce bulk output - pagination or an asynchronous export job may serve the user better than a giant response. ## How to roll it out Measure the current payload distribution and the existing overflow rate first, so the buffer default is evidence-based. Land the metric before the change, so you can see post-commit failures disappear. Convert the loudest offenders to the sanctioned streaming path, and only then tighten the default. The goal is that **no team ever discovers the commit point from a production incident** - they either stay on the buffered default or opt in knowingly.