A FastAPI service whose def endpoints call a slow blocking database driver stalls near 40 concurrent requests while CPU stays low; what caps it, and how do you raise the cap?
answer
- def code borrows a token
- anyio's default capacity limiter
- forty tokens per event loop
- set total_tokens in the lifespan
basics
~20 sEvery def endpoint and dependency borrows a token from anyio's default thread limiter, 40 per event loop, and later calls wait for one. Raise it in the lifespan with anyio.to_thread.current_default_thread_limiter().total_tokens, sized against the database pool and memory.
solid answer
~40 sFastAPI runs `def` endpoints and dependencies through `run_in_threadpool`, which calls `anyio.to_thread.run_sync`, and each call borrows a token from anyio's **default capacity limiter**: 40 tokens, created per event loop. With every token held by a slow database call, the 41st `def` call waits for a free token, while `async def` endpoints keep answering. You raise the cap by setting `anyio.to_thread.current_default_thread_limiter().total_tokens` inside the `lifespan`, because the limiter lives on the running loop and calling it at import time raises `NoEventLoopError`. The same tokens also serve `run_in_threadpool` calls you make, `iterate_in_threadpool`, sync background tasks, `FileResponse` reads and spooled `UploadFile` I/O. More tokens help only if the database pool can serve that many connections and memory can hold the threads; otherwise the queue just moves into the driver.
code
python · 25 linesfrom contextlib import asynccontextmanager
import anyio.to_thread
from fastapi import FastAPI
@asynccontextmanager
async def lifespan(app: FastAPI):
# must run on the event loop: at import time this raises NoEventLoopError
limiter = anyio.to_thread.current_default_thread_limiter()
limiter.total_tokens = 80 # anyio's default is 40 per event loop
yield
app = FastAPI(lifespan=lifespan)
@app.get("/internal/threads")
async def thread_pool_stats():
stats = anyio.to_thread.current_default_thread_limiter().statistics()
return {
"total": stats.total_tokens,
"borrowed": stats.borrowed_tokens,
"waiting": stats.tasks_waiting,
}go deeper
Remember that def endpoints run on a limited pool of worker threads, 40 by default, and that extra requests wait for a free one.
Explain the capacity limiter: each def endpoint, def dependency and run_in_threadpool call borrows a token per call, from a limiter that lives on each event loop.
Diagnose the plateau from low CPU, tasks_waiting above zero and fast async endpoints, then raise total_tokens in the lifespan only as far as the database pool and memory allow.
Set the service's thread budget against connection pools and replica memory, and decide which endpoints get their own limiter or an async client instead.
## Where the 40 comes from FastAPI runs every plain `def` **path operation** and every `def` **dependency** through `run_in_threadpool`, Starlette's helper re-exported by `fastapi.concurrency`. That helper calls `anyio.to_thread.run_sync`, and anyio guards every such call with a **capacity limiter**: a counter of tokens that a call must borrow before it gets a worker thread and gives back when it finishes. Unless a call passes its own `limiter`, anyio uses the **default thread limiter**. On the asyncio backend anyio creates it lazily as `CapacityLimiter(40)` and stores it per event loop. Starlette's documentation spells it out: the default pool is 40 tokens, so at most 40 threads run at the same time, and the limit is shared with FastAPI's sync dependencies. ## What the stall looks like Picture `def` endpoints whose database driver takes 300 ms per call under load: - the first 40 concurrent calls each borrow a token and run on worker threads; - the 41st call **waits for a token**, on the event loop, without blocking it; - `async def` endpoints keep answering at normal speed, because they borrow no token; - CPU stays low, since every thread is waiting on the database. Throughput is therefore capped near **tokens divided by seconds per call**. Two details make it worse than it looks: 1. tokens are borrowed **per call, not per request**: a request with a `def` dependency and a `def` endpoint borrows a token for each in turn (and once more to validate a `def` endpoint's return value against a response model), so slow dependencies eat into the same budget; 2. the pool is **shared** by more than your endpoints. | Also borrows from the default limiter | Why | |---|---| | `await run_in_threadpool(...)` and `anyio.to_thread.run_sync(...)` you write | same helper, no `limiter` passed | | `iterate_in_threadpool(...)` | one `run_sync` call for every `next()` | | sync tasks in `BackgroundTasks` | Starlette runs them through `run_in_threadpool` | | `FileResponse` | file reads go through anyio's thread-backed file API | | `UploadFile` reads and writes | once the spooled file has rolled over to disk | `asyncio.to_thread` and `loop.run_in_executor` are different: they use asyncio's own default executor, which anyio's limiter does not count. ## Raising the cap safely The limiter belongs to the **running event loop**, so set it from code that runs on that loop. The `lifespan` is the natural place: ```python @asynccontextmanager async def lifespan(app: FastAPI): limiter = anyio.to_thread.current_default_thread_limiter() limiter.total_tokens = 80 yield ``` Calling `current_default_thread_limiter()` at module import time raises `NoEventLoopError`, because no loop is running yet. Because the limiter is per event loop, each worker process has its own, so the effective pool is the per-process value times the number of processes. Before raising it, size against what the threads wait on: - **database connections**: if the connection pool holds fewer connections than there are tokens, extra threads queue inside the driver instead of the limiter, and you have added threads for nothing; - **memory**: each worker thread carries its own stack, and Starlette's docs warn that more threads can cost performance and memory; - **the GIL**: more threads help waiting work, not CPU-bound work. ## Measuring before and after Read the limiter's state rather than guessing: `current_default_thread_limiter().statistics()` returns `borrowed_tokens`, `total_tokens` and `tasks_waiting`. A `tasks_waiting` above zero during the stall confirms the diagnosis; `borrowed_tokens` pinned at the total while CPU is idle says the threads are waiting on I/O. In one in-process run with the limiter set to 2, four requests that each ran a 0.3 s `def` dependency and a 0.3 s `def` endpoint took about 1.2 s (2.4 thread-seconds over 2 tokens), while an `async def` endpoint called at the same moment answered in about a millisecond. ## When raising it is the wrong fix - If the slow work is one blocking client, an awaitable client removes the thread and the token altogether. - If one endpoint should not starve the rest, give its offloaded calls their own `anyio.CapacityLimiter` through the `limiter` argument of `anyio.to_thread.run_sync`, so a burst there cannot take every default token. - If the threads are busy **computing** rather than waiting, more tokens only add contenders for the GIL; that work needs another process, not a bigger thread pool. - If the database connection pool is the real ceiling, tokens beyond its size only move the queue from the limiter into the driver; the two budgets have to be sized together.
- Why can't the FastAPI thread limit be raised at module import time?anyio stores the default limiter per event loop, and `anyio.to_thread.current_default_thread_limiter()` needs a running loop to find it; at import there is none, so it raises `NoEventLoopError`. Setting `total_tokens` inside the `lifespan` changes the limiter of the loop that will serve requests.
- How do you stop one slow FastAPI endpoint from using all 40 default tokens?Keep it `async def` and offload its blocking call with `anyio.to_thread.run_sync(func, limiter=report_limiter)`, where `report_limiter` is an `anyio.CapacityLimiter` with a small `total_tokens`. Its calls then queue on their own limiter and leave the default pool to the other `def` endpoints and dependencies.
- FastAPI starts a new thread for every def request with no upper limit
- The thread limit is a Uvicorn setting passed on the command line
- async def endpoints also queue when the 40 tokens are used up
- Raising total_tokens always raises throughput, whatever the database pool size
- asyncio.to_thread draws from the same 40 tokens as def endpoints