How do you stream a language model's tokens to a browser as Server-Sent Events from a FastAPI 0.143 path operation?
answer
- a response class from fastapi.sse
- yield, not return
- event, id, retry and comment
- data is always JSON-encoded
- raw_data for a done marker
basics
~20 sMake the path operation a generator and set response_class=EventSourceResponse from fastapi.sse (0.135+). Each yielded value becomes one event with JSON in its data field; yield ServerSentEvent to set event, id, retry, comment or unencoded raw_data.
solid answer
~40 sI write the endpoint as an `async def` generator that awaits the model and `yield`s each token, and set `response_class=EventSourceResponse`, imported from `fastapi.sse` (added in 0.135). A plain yielded value is JSON-encoded into the event's data, validated first if the return type is `AsyncIterable[Item]`. To name the event or give it an id, I yield `ServerSentEvent(data=..., event="token", id=str(n))`; `data` is always JSON, so a bare string arrives quoted, and `raw_data` sends text as-is, like a `[DONE]` marker. FastAPI sets `Cache-Control: no-cache` and `X-Accel-Buffering: no`, and sends a ping comment after 15 idle seconds. It works with POST too, so the prompt can travel in a JSON body.
code
python · 27 linesfrom collections.abc import AsyncIterable
import anyio
from fastapi import FastAPI
from fastapi.sse import EventSourceResponse, ServerSentEvent
from pydantic import BaseModel
app = FastAPI()
class ChatRequest(BaseModel):
prompt: str
async def generate_tokens(prompt: str) -> AsyncIterable[str]:
for token in ["Hello", ",", " world"]:
await anyio.sleep(0.05) # stands in for awaiting the model
yield token
@app.post("/chat/stream", response_class=EventSourceResponse)
async def chat_stream(body: ChatRequest) -> AsyncIterable[ServerSentEvent]:
n = 0
async for token in generate_tokens(body.prompt):
n += 1
yield ServerSentEvent(data={"token": token}, event="token", id=str(n))
yield ServerSentEvent(raw_data="[DONE]", event="done")go deeper
Remember the two parts: a yielding path operation and response_class=EventSourceResponse from fastapi.sse.
Explain the ServerSentEvent fields, why data is always JSON-encoded, when raw_data is the right choice, and what the return annotation validates.
Show you know what FastAPI does for long-lived streams: the 15-second ping, the no-cache and no-buffering headers, def generators in the thread pool, and cleanup in finally.
Decide how the team's streaming contract signals completion and errors in-band, and whether endpoints take prompts by POST body rather than query strings.
## The two pieces: a generator and EventSourceResponse **Server-Sent Events (SSE)** is a way for a server to push a sequence of small text events over one long HTTP response with the `text/event-stream` media type. Browsers understand it natively, which makes it a common transport for chat interfaces that show a model's answer token by token. FastAPI added first-class SSE support in **0.135**. It needs two things on a path operation: 1. The function **yields** instead of returning, so FastAPI can send each item as soon as it exists. 2. The decorator sets `response_class=EventSourceResponse`, imported from `fastapi.sse`. `EventSourceResponse` is a subclass of Starlette's `StreamingResponse` whose media type is `text/event-stream`. The class is mostly a marker: FastAPI's routing layer sees it on a generator endpoint and encodes each yielded item as an event. ## Plain items versus ServerSentEvent There are two ways to feed the stream: - **Yield plain data** (a Pydantic model, dict, list, string, number). Each item is JSON-encoded and placed in the event's data field. With a return type such as `AsyncIterable[Token]`, FastAPI validates each item against `Token` and documents it in OpenAPI. - **Yield `ServerSentEvent` objects** when you need the other event fields. FastAPI skips item-type validation for these, so you can mix event kinds in one stream. | `ServerSentEvent` field | Type | What it is for | |---|---|---| | `data` | any JSON-serializable value | the payload, **always** JSON-encoded | | `raw_data` | `str` | a payload sent as-is, without JSON encoding | | `event` | `str` | an event name a browser listener can subscribe to | | `id` | `str` | an event id; a reconnecting browser sends the last one back | | `retry` | `int` (milliseconds, `>= 0`) | a reconnection delay hint | | `comment` | `str` | a comment line that clients ignore | FastAPI validates these fields when you build the object: `event` and `id` must be single lines, `id` must not contain a null character, `retry` must be non-negative, and `data` and `raw_data` cannot both be set. A token containing a newline is therefore safe in `data`, but not in `event`. When the generator yields plain models under an annotation such as `AsyncIterable[Token]`, the generated OpenAPI document lists the response as `text/event-stream` with an event `itemSchema` whose `data` field carries `Token`'s schema as JSON content, so client generators and readers of `/docs` can see the payload shape. With `ServerSentEvent` items, the schema shows only the generic event fields. The stream ends when the generator returns; FastAPI sends no closing event of its own, which is why a done marker is a convention you add. ## data versus raw_data Because `data` is always JSON-encoded, `ServerSentEvent(data="Hello")` sends the string with its quotes, and the browser must `JSON.parse` it. That is usually what you want: tokens with spaces, quotes or newlines survive intact. Use `raw_data` only for text you control completely, such as a `[DONE]` sentinel that tells the client the answer is finished, or pre-formatted log lines. The exact text format of events, reconnection with the last event id, and proxy buffering are covered with the Server-Sent Events protocol. ## What FastAPI adds for you - `Cache-Control: no-cache` on the response, so the stream is not cached. - `X-Accel-Buffering: no`, which tells some reverse proxies not to buffer the response. - A keep-alive comment (`: ping`) whenever the generator has been silent for **15 seconds**, so idle intermediaries do not drop a slow stream. - Any HTTP method: SSE over `POST` works, so a long prompt and chat history can be sent as a validated JSON body rather than a query string. - `def` generators are advanced in the thread pool, so a blocking model client does not freeze the event loop. ## A token-streaming endpoint, step by step 1. Validate the request body with a Pydantic model, as for any POST. 2. Await the model's async token iterator inside the generator. 3. Yield `ServerSentEvent(data={"token": token}, event="token", id=str(n))` for each token. 4. Yield a final event, for example `ServerSentEvent(raw_data="[DONE]", event="done")`. 5. Put any cleanup (closing the model stream, recording usage) in `try`/`finally`, because a closed tab ends the generator early. A browser's built-in `EventSource` can only issue GET requests, so a POST stream is read with `fetch` and a streaming body reader, or with a client library that parses event streams.
- Why does ServerSentEvent(data="Hello") arrive with quotation marks, and when would you avoid that?FastAPI always JSON-encodes `data`, so a string becomes a JSON string literal and the client parses it with `JSON.parse`. That keeps spaces and newlines intact. When the client expects bare text, such as a `[DONE]` sentinel or a preformatted log line, send it with `raw_data` instead; the two fields cannot be set together.
- Does a return type of AsyncIterable[ServerSentEvent] validate the payloads inside each event?No. FastAPI treats `ServerSentEvent` as a transport wrapper, so it skips item-type validation and leaves it out of the item schema. Its own field checks still run when the object is built. To validate payloads, yield plain model instances with an `AsyncIterable[Model]` annotation, or build the payload as a model before wrapping it.
- What does FastAPI send if the model takes longer than 15 seconds to produce the next token?It inserts a keep-alive comment, `: ping`, on the stream after 15 seconds without an event. Browser clients ignore comments, but the bytes keep idle proxies and load balancers from treating the connection as dead.
- SSE in FastAPI needs a third-party package; the framework has no native support.
- ServerSentEvent's data field sends a plain string without quotes.
- EventSourceResponse only works on GET path operations.
- You must set Cache-Control and buffering headers yourself on every SSE endpoint.
- Returning a list of events from the function streams them one by one.
Assumes FastAPI 0.143; fastapi.sse.EventSourceResponse and ServerSentEvent arrived in 0.135, and validation of the event and id fields in 0.136.2.