In Workers AI, how do you stream a text-generation model's draft reply to the browser, and what does stream: true change?
answer
- time to first word
- a flag in the inputs, not the options
- pass the stream straight into a Response
- text/event-stream
- max_tokens still caps the draft
basics
~20 sSet stream: true in the inputs of env.AI.run(); it then resolves to a ReadableStream of server-sent events, which the Worker returns as a Response with content-type text/event-stream so the browser shows the draft as it is generated.
solid answer
~40 s`stream: true` is a model input, so it goes in the second argument of `env.AI.run()`. The call then resolves to a `ReadableStream` of server-sent events that delivers text as it is generated, instead of a finished `{ response }` object. The Worker hands that stream to `new Response(stream, { headers: { "content-type": "text/event-stream" } })` without buffering it, and the agent's browser renders text as events arrive. Streaming changes when text arrives, not how much is generated: the same tokens are billed, and `max_tokens`, which defaults to 256 on many catalogue models such as `@cf/meta/llama-3.3-70b-instruct-fp8-fast`, still caps the draft. Once the first byte is sent, the status is fixed at 200, so a failure mid-stream has to be signalled inside the stream or noticed by the client.
code
javascript · 21 linesexport default {
async fetch(request, env) {
const { ticket } = await request.json();
let stream;
try {
stream = await env.AI.run("@cf/meta/llama-3.3-70b-instruct-fp8-fast", {
messages: [
{ role: "system", content: "Draft a polite, specific reply to this support ticket." },
{ role: "user", content: ticket },
],
max_tokens: 800,
stream: true,
});
} catch (err) {
return Response.json({ error: "draft unavailable" }, { status: 503 });
}
return new Response(stream, {
headers: { "content-type": "text/event-stream" },
});
},
};go deeper
Remember that stream: true is an input, that the result is a ReadableStream, and that it is returned with content-type text/event-stream.
Explain what changes and what does not: time to first token improves, while tokens billed and the max_tokens cap stay the same.
Show the error split: failures before the first byte become statuses, failures after it must be handled by the client, and truncated drafts usually trace to max_tokens.
Decide which endpoints stream at all: streaming suits a person watching, while background summaries are simpler to store and retry unstreamed.
## Why stream a draft reply A support agent waiting for a 600-word draft reply would otherwise stare at a spinner for the whole generation time. **Streaming** sends each piece of text as the model produces it, so the first words appear quickly and the agent can start reading. The total generation time is unchanged; only the **time to first token** improves. ## Turning it on In Workers AI, `stream` is a **model input**: it sits in the second argument of `env.AI.run()`, beside `messages` and `max_tokens`. Model schemas describe it as a boolean, default `false`, that streams the response back incrementally using server-sent events (SSE). - Without it, the promise resolves once generation is complete, to `{ response, usage }`. - With it, the promise resolves to a `ReadableStream` whose bytes are SSE events, each carrying the next fragment of text as the model produces it. - The third argument (`gateway`, `extraHeaders`, `rejectIfBusy` and the rest) is for call options. Its type has no `stream` field, so a flag placed there does not make a streaming call. ## Forwarding the stream to the browser The Worker does not read the stream itself. It passes it as the body of a new `Response`: ```javascript return new Response(stream, { headers: { "content-type": "text/event-stream" }, }); ``` Bytes flow from the model through the Worker to the client as they arrive, and the Worker never holds the whole draft in memory. On the client, `EventSource` can consume a GET endpoint; a POST that carries the ticket text is read with `fetch()` and the body's reader, parsing the `data:` lines as they arrive. Two details keep the browser side honest: - **Chunk boundaries are arbitrary.** A network read can end in the middle of an event, so the client buffers text until it sees the blank line that ends an SSE event before parsing it. - **The agent may leave.** If the agent closes the tab, the client disconnects; the draft is then lost unless the Worker also kept a copy, which is why storing the finished text is a separate decision from streaming it. Nothing between the model and the browser should buffer the body. A proxy or a framework layer that collects the whole response before sending it turns a streaming endpoint back into a slow one, so test the endpoint end to end and watch the first words arrive. ## What streaming does not change | Concern | Without `stream` | With `stream: true` | |---|---|---| | Neurons billed | input and output tokens | the same tokens | | Length cap | `max_tokens` | `max_tokens` | | What the call resolves to | the finished `{ response, usage }` | a stream that fills as tokens are generated | | HTTP status to the browser | chosen after the result is known | fixed at 200 once streaming begins | Three consequences follow: - **Length.** Many catalogue models, such as `@cf/meta/llama-3.3-70b-instruct-fp8-fast`, `@cf/openai/gpt-oss-120b` and `@cf/mistralai/mistral-small-3.1-24b-instruct`, default `max_tokens` to 256. A draft that stops mid-sentence is usually that default, not a streaming fault, so set `max_tokens` for the length you want. - **Errors.** A failure before streaming starts (an unknown model, an exhausted free allocation) makes the `env.AI.run()` promise reject, and the Worker can still return a proper error status. A failure after the first byte cannot change the status, so the client must handle a stream that ends early. - **Keeping the text.** To store the finished draft as well as stream it, split the stream (for example with `tee()`) and consume the second branch after the response is returned, as the Workers handler material describes for background work. ## When not to stream Streaming earns its keep when a person is watching. A ticket summary written into a database by a background job gains nothing from it; a plain call that resolves to `{ response }` is simpler to store and retry. The bge embedding models have no `stream` input at all. ## A checklist for the draft-reply endpoint 1. Put `stream: true` and an explicit `max_tokens` in the inputs. 2. Return the stream as the `Response` body with `content-type: text/event-stream`. 3. Catch the error that `env.AI.run()` throws before streaming starts and map it to a status. 4. Make the client tolerate a stream that ends early, and offer a retry.
- A streamed Workers AI draft always stops after a paragraph or two; what is the likely cause?The model's `max_tokens` default. Many catalogue text models, including `@cf/meta/llama-3.3-70b-instruct-fp8-fast` and `@cf/openai/gpt-oss-120b`, default it to 256 tokens, and streaming does not lift that cap. Set `max_tokens` in the inputs to the length the draft needs, within the model's context window.
- Does streaming a Workers AI response cost fewer neurons than waiting for the whole reply?No. Neurons are charged on the tokens processed and generated, which are the same either way. Streaming improves how soon the first words appear, not how much compute the reply takes. To spend less, shorten the prompt, lower `max_tokens` or choose a cheaper model.
- stream: true goes in the third options argument of env.AI.run().
- A streamed reply is billed for fewer neurons than a buffered one.
- The Worker must read the whole stream before returning a Response.
- A cut-off streamed draft means the stream broke, not that max_tokens was reached.
- An error midway through a stream can still turn the response into a 500.