skip to content

JSON Lines & SSE

A path operation that yields streams JSON Lines or, with EventSourceResponse, Server-Sent Events. Interviewers ask how a model's tokens are streamed and what a client disconnect does.

on this pageshow

explore

questions

5

How do you stream a language model's tokens to a browser as Server-Sent Events from a FastAPI 0.143 path operation?

level: middleimportance: must knowfreq 45%

answer

  1. a response class from fastapi.sse
  2. yield, not return
  3. event, id, retry and comment
  4. data is always JSON-encoded
  5. raw_data for a done marker

basics

~20 s

Make the path operation a generator and set response_class=EventSourceResponse from fastapi.sse (0.135+). Each yielded value becomes one event with JSON in its data field; yield ServerSentEvent to set event, id, retry, comment or unencoded raw_data.

solid answer

~40 s

I write the endpoint as an `async def` generator that awaits the model and `yield`s each token, and set `response_class=EventSourceResponse`, imported from `fastapi.sse` (added in 0.135). A plain yielded value is JSON-encoded into the event's data, validated first if the return type is `AsyncIterable[Item]`. To name the event or give it an id, I yield `ServerSentEvent(data=..., event="token", id=str(n))`; `data` is always JSON, so a bare string arrives quoted, and `raw_data` sends text as-is, like a `[DONE]` marker. FastAPI sets `Cache-Control: no-cache` and `X-Accel-Buffering: no`, and sends a ping comment after 15 idle seconds. It works with POST too, so the prompt can travel in a JSON body.

code

python · 27 lines
python
from collections.abc import AsyncIterable

import anyio
from fastapi import FastAPI
from fastapi.sse import EventSourceResponse, ServerSentEvent
from pydantic import BaseModel

app = FastAPI()


class ChatRequest(BaseModel):
    prompt: str


async def generate_tokens(prompt: str) -> AsyncIterable[str]:
    for token in ["Hello", ",", " world"]:
        await anyio.sleep(0.05)  # stands in for awaiting the model
        yield token


@app.post("/chat/stream", response_class=EventSourceResponse)
async def chat_stream(body: ChatRequest) -> AsyncIterable[ServerSentEvent]:
    n = 0
    async for token in generate_tokens(body.prompt):
        n += 1
        yield ServerSentEvent(data={"token": token}, event="token", id=str(n))
    yield ServerSentEvent(raw_data="[DONE]", event="done")

go deeper

for a junior

Remember the two parts: a yielding path operation and response_class=EventSourceResponse from fastapi.sse.

for a middle

Explain the ServerSentEvent fields, why data is always JSON-encoded, when raw_data is the right choice, and what the return annotation validates.

for a senior

Show you know what FastAPI does for long-lived streams: the 15-second ping, the no-cache and no-buffering headers, def generators in the thread pool, and cleanup in finally.

for a principal

Decide how the team's streaming contract signals completion and errors in-band, and whether endpoints take prompts by POST body rather than query strings.

## The two pieces: a generator and EventSourceResponse **Server-Sent Events (SSE)** is a way for a server to push a sequence of small text events over one long HTTP response with the `text/event-stream` media type. Browsers understand it natively, which makes it a common transport for chat interfaces that show a model's answer token by token. FastAPI added first-class SSE support in **0.135**. It needs two things on a path operation: 1. The function **yields** instead of returning, so FastAPI can send each item as soon as it exists. 2. The decorator sets `response_class=EventSourceResponse`, imported from `fastapi.sse`. `EventSourceResponse` is a subclass of Starlette's `StreamingResponse` whose media type is `text/event-stream`. The class is mostly a marker: FastAPI's routing layer sees it on a generator endpoint and encodes each yielded item as an event. ## Plain items versus ServerSentEvent There are two ways to feed the stream: - **Yield plain data** (a Pydantic model, dict, list, string, number). Each item is JSON-encoded and placed in the event's data field. With a return type such as `AsyncIterable[Token]`, FastAPI validates each item against `Token` and documents it in OpenAPI. - **Yield `ServerSentEvent` objects** when you need the other event fields. FastAPI skips item-type validation for these, so you can mix event kinds in one stream. | `ServerSentEvent` field | Type | What it is for | |---|---|---| | `data` | any JSON-serializable value | the payload, **always** JSON-encoded | | `raw_data` | `str` | a payload sent as-is, without JSON encoding | | `event` | `str` | an event name a browser listener can subscribe to | | `id` | `str` | an event id; a reconnecting browser sends the last one back | | `retry` | `int` (milliseconds, `>= 0`) | a reconnection delay hint | | `comment` | `str` | a comment line that clients ignore | FastAPI validates these fields when you build the object: `event` and `id` must be single lines, `id` must not contain a null character, `retry` must be non-negative, and `data` and `raw_data` cannot both be set. A token containing a newline is therefore safe in `data`, but not in `event`. When the generator yields plain models under an annotation such as `AsyncIterable[Token]`, the generated OpenAPI document lists the response as `text/event-stream` with an event `itemSchema` whose `data` field carries `Token`'s schema as JSON content, so client generators and readers of `/docs` can see the payload shape. With `ServerSentEvent` items, the schema shows only the generic event fields. The stream ends when the generator returns; FastAPI sends no closing event of its own, which is why a done marker is a convention you add. ## data versus raw_data Because `data` is always JSON-encoded, `ServerSentEvent(data="Hello")` sends the string with its quotes, and the browser must `JSON.parse` it. That is usually what you want: tokens with spaces, quotes or newlines survive intact. Use `raw_data` only for text you control completely, such as a `[DONE]` sentinel that tells the client the answer is finished, or pre-formatted log lines. The exact text format of events, reconnection with the last event id, and proxy buffering are covered with the Server-Sent Events protocol. ## What FastAPI adds for you - `Cache-Control: no-cache` on the response, so the stream is not cached. - `X-Accel-Buffering: no`, which tells some reverse proxies not to buffer the response. - A keep-alive comment (`: ping`) whenever the generator has been silent for **15 seconds**, so idle intermediaries do not drop a slow stream. - Any HTTP method: SSE over `POST` works, so a long prompt and chat history can be sent as a validated JSON body rather than a query string. - `def` generators are advanced in the thread pool, so a blocking model client does not freeze the event loop. ## A token-streaming endpoint, step by step 1. Validate the request body with a Pydantic model, as for any POST. 2. Await the model's async token iterator inside the generator. 3. Yield `ServerSentEvent(data={"token": token}, event="token", id=str(n))` for each token. 4. Yield a final event, for example `ServerSentEvent(raw_data="[DONE]", event="done")`. 5. Put any cleanup (closing the model stream, recording usage) in `try`/`finally`, because a closed tab ends the generator early. A browser's built-in `EventSource` can only issue GET requests, so a POST stream is read with `fetch` and a streaming body reader, or with a client library that parses event streams.

  • Why does ServerSentEvent(data="Hello") arrive with quotation marks, and when would you avoid that?
    FastAPI always JSON-encodes `data`, so a string becomes a JSON string literal and the client parses it with `JSON.parse`. That keeps spaces and newlines intact. When the client expects bare text, such as a `[DONE]` sentinel or a preformatted log line, send it with `raw_data` instead; the two fields cannot be set together.
  • Does a return type of AsyncIterable[ServerSentEvent] validate the payloads inside each event?
    No. FastAPI treats `ServerSentEvent` as a transport wrapper, so it skips item-type validation and leaves it out of the item schema. Its own field checks still run when the object is built. To validate payloads, yield plain model instances with an `AsyncIterable[Model]` annotation, or build the payload as a model before wrapping it.
  • What does FastAPI send if the model takes longer than 15 seconds to produce the next token?
    It inserts a keep-alive comment, `: ping`, on the stream after 15 seconds without an event. Browser clients ignore comments, but the bytes keep idle proxies and load balancers from treating the connection as dead.

  • SSE in FastAPI needs a third-party package; the framework has no native support.
  • ServerSentEvent's data field sends a plain string without quotes.
  • EventSourceResponse only works on GET path operations.
  • You must set Cache-Control and buffering headers yourself on every SSE endpoint.
  • Returning a list of events from the function streams them one by one.

Assumes FastAPI 0.143; fastapi.sse.EventSourceResponse and ServerSentEvent arrived in 0.135, and validation of the event and id fields in 0.136.2.

open as a page

In FastAPI 0.143, what does a path operation send when it uses yield instead of return and sets no response_class?

level: juniorimportance: should knowfreq 35%

basics

~20 s

Since FastAPI 0.134 it streams JSON Lines: each yielded item is serialized to one JSON line and sent as application/jsonl while the function keeps producing. An AsyncIterable[Item] or Iterable[Item] return type validates and filters every item.

open as a page

In FastAPI 0.143, how does a yielding path operation's output differ under the default response class, EventSourceResponse and StreamingResponse?

level: middleimportance: should knowfreq 28%

basics

~20 s

With no response_class FastAPI sends JSON Lines; with EventSourceResponse it sends Server-Sent Events with JSON data; with StreamingResponse it passes each chunk through untouched, so only the first two validate items against the return type.

open as a page

When a user closes the browser tab mid-answer, what happens to a FastAPI 0.143 path operation still yielding a language model's tokens?

level: seniorimportance: should knowfreq 40%

basics

~20 s

Under Uvicorn, Starlette's streaming response notices the disconnect and cancels the stream: an async def generator gets CancelledError at its current await and its finally runs. A def generator's in-flight step finishes in its thread first.

open as a page

Why does a FastAPI 0.143 log-tail path operation that raises HTTPException(404) before its first yield send clients a 200 and a broken body?

level: seniorimportance: should knowfreq 30%

basics

~20 s

A yielding path operation's body, even the code before its first yield, runs only when the stream is iterated, after the 200 status and headers are sent. Raise the 404 from a dependency, which FastAPI solves before the response starts.

open as a page