skip to content

Inside a FastAPI async def endpoint, how do run_in_threadpool, anyio.to_thread.run_sync and iterate_in_threadpool differ for running blocking code?

level: middleimportance: should knowfreq 30%

answer

  1. one call versus an iterator
  2. keyword arguments and partial
  3. a limiter argument
  4. every next() is a thread hop

basics

~10 s

run_in_threadpool from fastapi.concurrency wraps anyio.to_thread.run_sync and also forwards keyword arguments; run_sync takes positional arguments plus limiter and abandon_on_cancel; iterate_in_threadpool turns a blocking iterator into an async one, running each next() on a worker thread.

solid answer

~40 s

All three put blocking work on anyio worker threads and, by default, borrow from the same 40-token limiter FastAPI uses for `def` endpoints. `run_in_threadpool(func, *args, **kwargs)`, importable from `fastapi.concurrency`, is Starlette's thin wrapper: it binds the arguments with `functools.partial` and awaits `anyio.to_thread.run_sync`. Calling `anyio.to_thread.run_sync(func, *args)` directly accepts positional arguments only, but exposes `limiter=`, to cap one kind of call on its own `CapacityLimiter`, and `abandon_on_cancel=`. `iterate_in_threadpool(iterator)` is an async generator for `async for`: each `next()` runs as its own thread call, which suits a blocking cursor or file reader but costs a hand-off per item. By contrast `asyncio.to_thread` uses asyncio's own executor, which anyio's limiter does not count.

code

python · 33 lines
python
import functools
import time
from collections.abc import Iterator

import anyio.to_thread
from fastapi import FastAPI
from fastapi.concurrency import iterate_in_threadpool, run_in_threadpool

app = FastAPI()


def render_report(report_id: int, *, fmt: str) -> str:
    time.sleep(0.1)  # blocking work from a synchronous library
    return f"report {report_id} as {fmt}"


def blocking_rows() -> Iterator[dict]:
    for n in range(3):
        time.sleep(0.05)  # each next() blocks, like a synchronous cursor
        yield {"row": n}


@app.get("/reports/{report_id}")
async def report(report_id: int):
    # run_in_threadpool forwards keyword arguments
    text = await run_in_threadpool(render_report, report_id, fmt="pdf")
    # anyio.to_thread.run_sync takes positional arguments only
    same = await anyio.to_thread.run_sync(
        functools.partial(render_report, report_id, fmt="pdf")
    )
    # each next() call on the iterator runs on a worker thread
    rows = [row async for row in iterate_in_threadpool(blocking_rows())]
    return {"text": text, "same": text == same, "rows": rows}

go deeper

for a junior

Know that run_in_threadpool from fastapi.concurrency lets an async def endpoint await a blocking call without freezing the event loop.

for a middle

Explain the differences: keyword arguments through run_in_threadpool, positional-only run_sync with limiter and abandon_on_cancel, and iterate_in_threadpool running each next() on a thread.

for a senior

Give a slow call its own CapacityLimiter, choose the cancellation behaviour on purpose, and batch small iterator items instead of paying a thread hop per item.

for a principal

Standardise on the anyio-based helpers so the service has one thread budget to size and monitor, and make bespoke limiters a reviewed exception.

## Why you need them FastAPI runs a plain `def` endpoint on a worker thread for you, but inside an **`async def`** endpoint every line runs on the event loop. When such an endpoint has to call blocking code, a synchronous library, a file read or a blocking cursor, you move that call to a thread yourself and `await` the result. FastAPI gives you, or re-exports, three helpers for it, all built on **anyio**, the concurrency library Starlette and FastAPI run on. | Helper | Import | Arguments | Use it for | |---|---|---|---| | `run_in_threadpool` | `from fastapi.concurrency import run_in_threadpool` | `(func, *args, **kwargs)` | one blocking call, the same way FastAPI runs `def` code | | `anyio.to_thread.run_sync` | `import anyio.to_thread` | `(func, *args, abandon_on_cancel=..., limiter=...)` | one blocking call that needs its own limiter or cancellation behaviour | | `iterate_in_threadpool` | `from fastapi.concurrency import iterate_in_threadpool` | `(iterator)` | consuming a blocking iterator with `async for` | ## run_in_threadpool `run_in_threadpool` is Starlette's helper, and `fastapi.concurrency` re-exports it. Its whole body binds the arguments with `functools.partial` and awaits `anyio.to_thread.run_sync`. This is exactly what FastAPI itself calls for `def` endpoints and dependencies, so: - it **forwards keyword arguments**: `await run_in_threadpool(render_report, 7, fmt="pdf")` works; - it uses anyio's **default thread limiter**, the 40 tokens per event loop shared with every `def` endpoint and dependency; - it has no options, so it suits the ordinary case. ## anyio.to_thread.run_sync Calling anyio directly gives two knobs, at the price of a narrower signature: 1. **Positional arguments only.** `run_sync(render_report, 7, fmt="pdf")` raises a `TypeError`, because `run_sync` has no `fmt` parameter and does not forward keywords. Wrap the call: `run_sync(functools.partial(render_report, 7, fmt="pdf"))`. 2. **`limiter=`** takes an `anyio.CapacityLimiter`. Giving one slow kind of call its own limiter, say five tokens for report rendering, stops a burst of it from taking every default token that the `def` endpoints need. 3. **`abandon_on_cancel=`**, `False` by default. A running thread cannot be interrupted. With the default, a cancelled awaiting task stays shielded until the thread finishes, so cleanup sees a finished call; with `True` the task moves on immediately and the thread keeps running unattended, its result discarded. ## iterate_in_threadpool Some blocking APIs hand you an **iterator**: a cursor that fetches rows lazily, a file read line by line, a paginated synchronous client. Iterating it in an `async def` endpoint would block the loop on every `next()`. `iterate_in_threadpool(iterator)` returns an **async iterator** you consume with `async for`: - it calls `iter()` on what you pass, then runs each `next()` as a separate `anyio.to_thread.run_sync` call; - `StopIteration` cannot cross the thread boundary cleanly, so Starlette converts it into a private exception and ends the loop for you; - each item costs a thread hand-off and a limiter token, so for many tiny items it can be cheaper to fetch a whole batch with one `run_in_threadpool` call. Returning a streamed response built from such an iterator is the streaming material; here the point is consuming one inside your own code. ## Which pool, and asyncio's own helpers All three helpers draw on anyio's **default thread limiter** unless you pass `limiter=` to `run_sync`. `asyncio.to_thread` and `loop.run_in_executor` look similar but use **asyncio's own default executor**, a separate pool that anyio's limiter neither counts nor caps. Mixing both in one app means two thread budgets to reason about, which is why FastAPI code usually sticks to the anyio-based helpers. In one in-process check with the default limiter lowered to one token, three 0.2 s `run_in_threadpool` calls took about 0.6 s, while three `asyncio.to_thread` calls took about 0.2 s, because they never touched the limiter. ## Why not just declare the endpoint def Often that is the simplest answer: if the whole endpoint is blocking work, plain `def` lets FastAPI run it through `run_in_threadpool` for you. The helpers earn their place when the endpoint is **mostly async**, say it awaits an async client and then calls one synchronous library, or when one call needs a separate limiter. Turning such an endpoint into `def` would lose its `await` calls, and splitting it into a `def` dependency only works when the blocking part can run before the endpoint body. ## Choosing quickly - one blocking call, nothing special: `run_in_threadpool`; - one blocking call that must not starve other work, or needs its cancellation behaviour chosen: `anyio.to_thread.run_sync` with `limiter=` or `abandon_on_cancel=`; - a blocking iterator: `iterate_in_threadpool`, or one batched `run_in_threadpool` call when items are small.

  • When is iterate_in_threadpool the wrong choice in a FastAPI endpoint?
    When the iterator yields many small items quickly. Every `next()` becomes its own thread call, with a hand-off and a limiter token each, so the overhead can outweigh the work. Fetching a batch with one `run_in_threadpool(lambda: list(rows))` call, or a few large chunks, does far fewer hops.
  • Do run_in_threadpool calls in FastAPI compete with def endpoints for threads?
    Yes. `run_in_threadpool` calls `anyio.to_thread.run_sync` with no limiter, so it borrows from the same default limiter, 40 tokens per event loop, that `def` endpoints and `def` dependencies use. Pass your own `CapacityLimiter` to `run_sync` to give a slow call a separate budget.

  • anyio.to_thread.run_sync forwards keyword arguments to the function
  • iterate_in_threadpool runs the whole iterator in one worker thread call
  • asyncio.to_thread shares the same 40-token limit as def endpoints
  • Cancelling the awaiting task stops the worker thread immediately
  • run_in_threadpool uses a separate pool from the one def endpoints use