skip to content

In Cloudflare Workers, how do you proxy or transform a multi-gigabyte CSV download without buffering it, and why not call text()?

level: middleimportance: must knowfreq 50%

answer

  1. bodies are streams, read once
  2. return the stream, not the string
  3. a readable and writable pair
  4. pipe without awaiting
  5. fixed length versus chunked

basics

~20 s

Return the origin's body stream, or pipe it through a TransformStream, so bytes flow to the client as they arrive. text() reads the whole body into the isolate's 128 MB of memory and consumes the one-use body stream.

solid answer

~40 s

A Worker should treat the body as a stream. To pass a large CSV through unchanged, return `new Response(upstream.body, upstream)`: the runtime pumps bytes from origin to client without your code touching them, and an origin's compressed encoding can pass straight through. To change or inspect it on the way, pipe it through a `TransformStream` with `upstream.body.pipeThrough(ts)` or an unawaited `pipeTo()`; the runtime keeps the pipe running after the handler returns, so no `ctx.waitUntil()` is needed. Calling `text()` or `arrayBuffer()` instead loads the whole file into the isolate, whose 128 MB limit a multi-gigabyte export cannot fit, and it consumes the body: a second read throws `Body has already been used`. Streamed bodies go out chunked unless you write through a `FixedLengthStream`.

code

javascript · 21 lines
javascript
export default {
  async fetch(request, env, ctx) {
    const upstream = await fetch("https://legacy.example.com/exports/orders.csv");
    if (!upstream.ok || upstream.body === null) return upstream;

    let bytes = 0;
    const counter = new TransformStream({
      transform(chunk, controller) {
        bytes += chunk.byteLength;
        controller.enqueue(chunk);
      },
      flush() {
        console.log(`orders.csv streamed ${bytes} bytes`);
      },
    });

    const response = new Response(upstream.body.pipeThrough(counter), upstream);
    response.headers.set("Content-Disposition", 'attachment; filename="orders.csv"');
    return response;
  },
};

go deeper

for a junior

Recall that bodies are one-use streams, that text() reads everything into memory, and that returning upstream.body forwards a download without buffering it.

for a middle

Explain the TransformStream pair, why pipeTo() stays unawaited, why chunk boundaries ignore CSV rows, and when FixedLengthStream gives a Content-Length.

for a senior

Diagnose a proxy that fails on large exports: find the buffering read, swap it for a stream, and check that every tee() branch has a reader.

for a principal

Set a team rule that proxy Workers never buffer bodies of unknown size, and review new code for text(), json() and clone() on paths that carry user downloads.

## Bodies are one-use streams In Cloudflare Workers the body of a `Request` or `Response` is a `ReadableStream`. Methods such as `text()`, `json()` and `arrayBuffer()` read that stream **to the end** and return the result, which has two consequences: - **Memory.** The whole payload is materialised inside the isolate, and an isolate has 128 MB of memory, shared with everything else it is doing. A multi-gigabyte CSV export cannot fit; the runtime can fail with errors such as `Memory limit would be exceeded before EOF`. - **Consumption.** A body can be read once. A second `text()` throws `TypeError: Body has already been used. It can only be used once. Use tee() first if you need to read it twice.` and `bodyUsed` turns `true`. Streaming avoids both: the Worker starts sending the response status and headers at once and forwards bytes as the origin produces them. ## Pass-through: the cheapest case If the Worker only forwards the download, return the origin's stream: ```javascript const upstream = await fetch(originUrl); return new Response(upstream.body, upstream); ``` The documentation calls this already optimal, since none of your code runs per chunk, and as long as the body is not read and `Content-Encoding` is kept, compressed data from the origin passes through without being decompressed and recompressed. Wrapping it in a new `Response` is only needed to edit headers such as `Content-Disposition`. ## Transforming on the way: TransformStream A `TransformStream` is a pair: a `writable` side that you feed and a `readable` side that you return as the body. With the standard constructor (on by default for compatibility dates since 2022-11-30) you pass a `transform(chunk, controller)` callback that can inspect, change or drop each chunk, and an optional `flush()` that runs at the end. 1. `fetch()` the origin. 2. Create the `TransformStream` with the per-chunk logic. 3. Connect them: `upstream.body.pipeThrough(ts)` returns the readable side directly, or call `upstream.body.pipeTo(ts.writable)` **without `await`** and return `ts.readable`. 4. Return `new Response(readable, upstream)`. Awaiting `pipeTo()` before returning stalls the handler: nobody reads the readable side until the response is returned, so the pipe cannot finish. Not awaiting it is safe because the runtime keeps the invocation alive while the response body is still streaming, so this does not need `ctx.waitUntil()`. Chunks are byte arrays of arbitrary size and their boundaries do not follow the data: a CSV row can be split across two chunks. A transform that works per row must decode with `TextDecoder` in streaming mode and carry the partial last line into the next call. ## Reading the same body twice | Need | Tool | Cost | |---|---|---| | Forward unchanged | return `upstream.body` | none | | Change bytes in flight | `TransformStream` | one callback per chunk | | Two consumers, such as client plus hash | `body.tee()` or `response.clone()` | may buffer for the slower branch | | Hash while streaming | `tee()` into `crypto.DigestStream` | one extra branch | `tee()` and `clone()` split one stream into two. The runtime applies backpressure according to the branch with the most unread data, which limits pile-up when the source honours backpressure, but a branch that is read late or never may still force bytes to be held, so give every branch a reader. ## Content-Length and other details - A streamed body is sent with **chunked** transfer encoding. If the client needs a `Content-Length` (for a progress bar) and the size is known, write through a `FixedLengthStream(length)`, which errors if the byte count differs. - A `Content-Length` set by hand on the response is ignored; the runtime computes it. - `IdentityTransformStream` is a byte-only pass-through whose readable side supports bring-your-own-buffer reads. - Streams can only be used inside a handler's request context, never at module scope. ## Diagnosing a proxy that fails on big exports A typical report is that small exports download fine while large ones fail, arrive late, or never start. Work through it in order: 1. Search the code path for `text()`, `json()`, `arrayBuffer()` and `formData()` on the upstream body; each one buffers. 2. Look for `clone()` or `tee()` where one branch is never read, such as a clone kept for logging. 3. Check that a `pipeTo()` before the `return` is not awaited. 4. Confirm the response body is the stream (`upstream.body` or a transform's readable side), not a string built from it. Once the body streams, time to first byte no longer grows with file size and memory use stays flat.

  • Why is upstream.body.pipeTo(writable) left unawaited in the documentation's example?
    The readable side is only consumed once the handler returns the `Response`. Awaiting `pipeTo()` first waits for a reader that does not exist yet, so the handler stalls. Left unawaited, the pipe runs while the response streams, and the runtime keeps the invocation alive until the body finishes or the client disconnects, without `ctx.waitUntil()`. `pipeThrough()` avoids the floating promise altogether.
  • The client needs a Content-Length for a streamed download whose size the origin reported. What do you do?
    Write the bytes through `new FixedLengthStream(length)` and return its readable side. The runtime then sends `Content-Length` instead of chunked encoding, and the stream errors if more or fewer bytes than declared pass through. Setting the header by hand does nothing because the runtime ignores it.
  • You need both to stream the CSV to the client and to compute its SHA-256. How?
    Split the body with `upstream.body.tee()`. Return one branch in the response and pipe the other into `new crypto.DigestStream("SHA-256")`, then await its `digest` promise. Both branches are read concurrently, and backpressure follows the slower reader instead of the runtime holding the whole file.

Streaming a body is a hose run straight from the tap to the garden: water arrives as it flows and nothing has to hold all of it. Calling text() is filling a bucket first, which only works if the bucket is bigger than everything the tap will deliver, and the water poured into it is gone from the hose.

  • Calling await res.text() is fine for any size because Workers bodies are lazy.
  • A response body can be read twice if the first read was text().
  • pipeTo() must be awaited before returning, or the stream is cut off.
  • An unawaited pipe needs ctx.waitUntil() to survive after the handler returns.
  • A transform callback receives one CSV row per chunk.