skip to content

A FastAPI service whose def endpoints call a slow blocking database driver stalls near 40 concurrent requests while CPU stays low; what caps it, and how do you raise the cap?

level: seniorimportance: should knowfreq 45%

answer

  1. def code borrows a token
  2. anyio's default capacity limiter
  3. forty tokens per event loop
  4. set total_tokens in the lifespan

basics

~20 s

Every def endpoint and dependency borrows a token from anyio's default thread limiter, 40 per event loop, and later calls wait for one. Raise it in the lifespan with anyio.to_thread.current_default_thread_limiter().total_tokens, sized against the database pool and memory.

solid answer

~40 s

FastAPI runs `def` endpoints and dependencies through `run_in_threadpool`, which calls `anyio.to_thread.run_sync`, and each call borrows a token from anyio's **default capacity limiter**: 40 tokens, created per event loop. With every token held by a slow database call, the 41st `def` call waits for a free token, while `async def` endpoints keep answering. You raise the cap by setting `anyio.to_thread.current_default_thread_limiter().total_tokens` inside the `lifespan`, because the limiter lives on the running loop and calling it at import time raises `NoEventLoopError`. The same tokens also serve `run_in_threadpool` calls you make, `iterate_in_threadpool`, sync background tasks, `FileResponse` reads and spooled `UploadFile` I/O. More tokens help only if the database pool can serve that many connections and memory can hold the threads; otherwise the queue just moves into the driver.

code

python · 25 lines
python
from contextlib import asynccontextmanager

import anyio.to_thread
from fastapi import FastAPI


@asynccontextmanager
async def lifespan(app: FastAPI):
    # must run on the event loop: at import time this raises NoEventLoopError
    limiter = anyio.to_thread.current_default_thread_limiter()
    limiter.total_tokens = 80  # anyio's default is 40 per event loop
    yield


app = FastAPI(lifespan=lifespan)


@app.get("/internal/threads")
async def thread_pool_stats():
    stats = anyio.to_thread.current_default_thread_limiter().statistics()
    return {
        "total": stats.total_tokens,
        "borrowed": stats.borrowed_tokens,
        "waiting": stats.tasks_waiting,
    }

go deeper

for a junior

Remember that def endpoints run on a limited pool of worker threads, 40 by default, and that extra requests wait for a free one.

for a middle

Explain the capacity limiter: each def endpoint, def dependency and run_in_threadpool call borrows a token per call, from a limiter that lives on each event loop.

for a senior

Diagnose the plateau from low CPU, tasks_waiting above zero and fast async endpoints, then raise total_tokens in the lifespan only as far as the database pool and memory allow.

for a principal

Set the service's thread budget against connection pools and replica memory, and decide which endpoints get their own limiter or an async client instead.

## Where the 40 comes from FastAPI runs every plain `def` **path operation** and every `def` **dependency** through `run_in_threadpool`, Starlette's helper re-exported by `fastapi.concurrency`. That helper calls `anyio.to_thread.run_sync`, and anyio guards every such call with a **capacity limiter**: a counter of tokens that a call must borrow before it gets a worker thread and gives back when it finishes. Unless a call passes its own `limiter`, anyio uses the **default thread limiter**. On the asyncio backend anyio creates it lazily as `CapacityLimiter(40)` and stores it per event loop. Starlette's documentation spells it out: the default pool is 40 tokens, so at most 40 threads run at the same time, and the limit is shared with FastAPI's sync dependencies. ## What the stall looks like Picture `def` endpoints whose database driver takes 300 ms per call under load: - the first 40 concurrent calls each borrow a token and run on worker threads; - the 41st call **waits for a token**, on the event loop, without blocking it; - `async def` endpoints keep answering at normal speed, because they borrow no token; - CPU stays low, since every thread is waiting on the database. Throughput is therefore capped near **tokens divided by seconds per call**. Two details make it worse than it looks: 1. tokens are borrowed **per call, not per request**: a request with a `def` dependency and a `def` endpoint borrows a token for each in turn (and once more to validate a `def` endpoint's return value against a response model), so slow dependencies eat into the same budget; 2. the pool is **shared** by more than your endpoints. | Also borrows from the default limiter | Why | |---|---| | `await run_in_threadpool(...)` and `anyio.to_thread.run_sync(...)` you write | same helper, no `limiter` passed | | `iterate_in_threadpool(...)` | one `run_sync` call for every `next()` | | sync tasks in `BackgroundTasks` | Starlette runs them through `run_in_threadpool` | | `FileResponse` | file reads go through anyio's thread-backed file API | | `UploadFile` reads and writes | once the spooled file has rolled over to disk | `asyncio.to_thread` and `loop.run_in_executor` are different: they use asyncio's own default executor, which anyio's limiter does not count. ## Raising the cap safely The limiter belongs to the **running event loop**, so set it from code that runs on that loop. The `lifespan` is the natural place: ```python @asynccontextmanager async def lifespan(app: FastAPI): limiter = anyio.to_thread.current_default_thread_limiter() limiter.total_tokens = 80 yield ``` Calling `current_default_thread_limiter()` at module import time raises `NoEventLoopError`, because no loop is running yet. Because the limiter is per event loop, each worker process has its own, so the effective pool is the per-process value times the number of processes. Before raising it, size against what the threads wait on: - **database connections**: if the connection pool holds fewer connections than there are tokens, extra threads queue inside the driver instead of the limiter, and you have added threads for nothing; - **memory**: each worker thread carries its own stack, and Starlette's docs warn that more threads can cost performance and memory; - **the GIL**: more threads help waiting work, not CPU-bound work. ## Measuring before and after Read the limiter's state rather than guessing: `current_default_thread_limiter().statistics()` returns `borrowed_tokens`, `total_tokens` and `tasks_waiting`. A `tasks_waiting` above zero during the stall confirms the diagnosis; `borrowed_tokens` pinned at the total while CPU is idle says the threads are waiting on I/O. In one in-process run with the limiter set to 2, four requests that each ran a 0.3 s `def` dependency and a 0.3 s `def` endpoint took about 1.2 s (2.4 thread-seconds over 2 tokens), while an `async def` endpoint called at the same moment answered in about a millisecond. ## When raising it is the wrong fix - If the slow work is one blocking client, an awaitable client removes the thread and the token altogether. - If one endpoint should not starve the rest, give its offloaded calls their own `anyio.CapacityLimiter` through the `limiter` argument of `anyio.to_thread.run_sync`, so a burst there cannot take every default token. - If the threads are busy **computing** rather than waiting, more tokens only add contenders for the GIL; that work needs another process, not a bigger thread pool. - If the database connection pool is the real ceiling, tokens beyond its size only move the queue from the limiter into the driver; the two budgets have to be sized together.

  • Why can't the FastAPI thread limit be raised at module import time?
    anyio stores the default limiter per event loop, and `anyio.to_thread.current_default_thread_limiter()` needs a running loop to find it; at import there is none, so it raises `NoEventLoopError`. Setting `total_tokens` inside the `lifespan` changes the limiter of the loop that will serve requests.
  • How do you stop one slow FastAPI endpoint from using all 40 default tokens?
    Keep it `async def` and offload its blocking call with `anyio.to_thread.run_sync(func, limiter=report_limiter)`, where `report_limiter` is an `anyio.CapacityLimiter` with a small `total_tokens`. Its calls then queue on their own limiter and leave the default pool to the other `def` endpoints and dependencies.

  • FastAPI starts a new thread for every def request with no upper limit
  • The thread limit is a Uvicorn setting passed on the command line
  • async def endpoints also queue when the 40 tokens are used up
  • Raising total_tokens always raises throughput, whatever the database pool size
  • asyncio.to_thread draws from the same 40 tokens as def endpoints