How do you stream LLM tokens from inside a LangGraph node to the client?

## The mechanism When you stream a compiled graph with `stream_mode="messages"`, LangGraph attaches a streaming-capable callback to the model calls made inside nodes. Each token the provider returns becomes an `AIMessageChunk`, and LangGraph pairs it with a metadata dict before yielding `(chunk, metadata)`. The useful metadata keys are the LangGraph ones — `langgraph_node` (the node whose body made the call), `langgraph_step` (which superstep), plus the tags and metadata you attached to the model or the call config. Everything you need for routing tokens to the right place in a UI is in that dict; the chunk itself only knows it is text. ## Why .invoke() inside a node still streams This surprises people. A node written as `return {"messages": [llm.invoke(state["messages"])]}` still produces tokens on the stream. The chat-model base class checks, at call time, whether a streaming-capable handler is attached, and if so it uses the provider's streaming API internally while still returning one complete message to your node. So the node's own contract is unchanged — it gets a whole `AIMessage` — while the tokens leak out through the callback channel as they arrive. Two things switch this off. A model constructed with streaming disabled (`disable_streaming=True`) will not emit chunks. And a provider or model configuration that does not support streaming at all — some structured-output or tool-forcing paths — returns in one shot, so there is nothing incremental to emit. ## Filtering: the part people get wrong A real agent graph has more model calls than the user should see: a routing classifier, a query rewriter, a summariser that compacts history, a grader in a self-correction loop. All of them stream. If you pipe the raw message stream to the browser, the user watches the classifier deliberate and then sees the answer typed over the top of it. Two filters are idiomatic: - **By node**: skip unless `metadata["langgraph_node"] == "answer"`. Simple, but couples the transport layer to node names. - **By tag**: bind a tag to the model instance you consider user-facing (`llm.with_config(tags=["user_facing"])`) and filter on the tags present in the metadata. Survives node renames and works when one node hosts several model calls. ## Chunks that are not text Not every chunk carries prose. During a tool-calling turn the model streams partial tool-call arguments; those arrive as `AIMessageChunk`s whose `content` is empty and whose `tool_call_chunks` hold fragments of a JSON argument string. Rendering them naively gives the user a burst of nothing, or worse, half-formed JSON. Guard with `if chunk.content:` before writing to the wire, and accumulate tool-call chunks only if you actually intend to show the arguments forming. ## Combining with step-level modes Token output alone is a thin UI: the user sees text appear but not that a search ran. The common production shape is `stream_mode=["updates", "messages"]`, which yields `(mode, payload)` tuples so one connection carries both. Map step updates onto status chips ("Searching…", "Reading 3 sources…") and message chunks onto the answer body. ## Async, subgraphs and the callback-propagation trap Use `astream()` in a server. On Python 3.10, async callback propagation is not automatic through context variables: if a node's async body calls a model without threading the `RunnableConfig` it was given down into that call, the callback never attaches and the tokens vanish silently — no error, just an empty token stream. Accepting `config: RunnableConfig` in the node signature and passing it through to the model call fixes it, and is harmless on newer Python. Tokens from models called inside a nested graph do surface in the parent's message stream. If you also want the nested graph's step updates, pass `subgraphs=True`, which prefixes chunks with a namespace tuple identifying the nested graph. ## What to say about failure If the run raises mid-stream, the client has already received a partial answer. Decide the contract: emit a terminal error event and let the UI mark the message as incomplete, or buffer server-side until a step boundary before flushing. Silently ending the stream is the worst option, because the client cannot distinguish it from a short answer.