skip to content

Under load, def endpoints in a FastAPI orders API using a sync SessionDep fail with QueuePool timeout errors while PostgreSQL looks idle; how do FastAPI's thread pool and the engine's pool interact?

level: seniorimportance: should knowfreq 45%

answer

  1. two limits per worker process
  2. 40 thread tokens versus 15 connections
  3. connection held until teardown
  4. teardown has its own limiter

basics

~20 s

FastAPI runs def endpoints and dependencies in a thread pool of 40 tokens per process, while SQLAlchemy's default pool lends 15 connections. Extra requests wait for connections held until teardown and can fail after 30 seconds.

solid answer

~40 s

Every `def` endpoint and `def` dependency runs through Starlette's `run_in_threadpool`, whose default limiter allows 40 threads per worker process. A default `create_engine` pool allows `pool_size=5` plus `max_overflow=10`, and waits `pool_timeout=30` seconds. A session checks out a connection on its first query and, with the default `scope="request"`, keeps it until teardown after the response and background tasks, so requests beyond the 15th queue and, after 30 seconds without a free connection, fail with `QueuePool limit of size 5 overflow 10 reached` while the database idles. FastAPI runs a sync yield dependency's teardown on its own `CapacityLimiter(1)` so a close never waits for a thread token. Fix it by sizing the pool to the thread budget per worker, times workers within `max_connections`, or by lowering the limiter in `lifespan`, and by shortening hold time.

code

python · 33 lines
python
from contextlib import asynccontextmanager

import anyio.to_thread
from fastapi import FastAPI
from sqlmodel import create_engine

# Per worker process: room for every thread that can hold a session at once.
engine = create_engine(
    "sqlite:///orders.db",
    connect_args={"check_same_thread": False},
    pool_size=20,
    max_overflow=10,
    pool_timeout=10,
)


@asynccontextmanager
async def lifespan(app: FastAPI):
    # Cap sync concurrency (def endpoints and def dependencies) below the 30 pooled connections.
    anyio.to_thread.current_default_thread_limiter().total_tokens = 25
    yield
    engine.dispose()


app = FastAPI(lifespan=lifespan)


@app.get("/limits")
async def limits():
    return {
        "thread_tokens": anyio.to_thread.current_default_thread_limiter().total_tokens,
        "pool_size": engine.pool.size(),
    }

go deeper

for a junior

Recall that def endpoints run in a limited thread pool and that the engine lends a limited number of connections.

for a middle

Explain the two defaults, 40 thread tokens and 5 plus 10 connections, and when a session takes and returns its connection.

for a senior

Diagnose pool timeouts against an idle database from hold time and concurrency, then size pool, limiter and workers together.

for a principal

Set a team rule tying pool size, thread budget and worker count to the database's connection budget, and decide where overload should queue.

## Two pools that do not know about each other A FastAPI worker process running sync endpoints has two independent limits, and the bug lives in the gap between them. | Limit | Where it comes from | Default | What waits on it | |---|---|---|---| | Thread pool tokens | The default thread limiter behind Starlette's `run_in_threadpool`, read with `anyio.to_thread.current_default_thread_limiter()` | 40 per event loop, so 40 per worker process | `def` endpoints, `def` dependencies, response validation for `def` endpoints, iteration of sync streaming generators | | Pooled connections | SQLAlchemy's `QueuePool` on the engine | `pool_size=5`, `max_overflow=10`, `pool_timeout=30` | Every session that runs a query | FastAPI runs every `def` path operation and `def` dependency in that thread pool, so up to 40 requests can be inside sync code at once. The engine, left at its defaults, lends out at most 15 connections. When more than 15 threads hold or want a session's connection, the rest wait up to 30 seconds and, if no connection frees up, fail with `sqlalchemy.exc.TimeoutError: QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30.00`, which FastAPI turns into a 500. ## How long a request holds its connection 1. FastAPI enters `get_session` in the thread pool. `Session(engine)` takes no connection yet. 2. The first query checks out a connection, and the session keeps it until commit, rollback or close. 3. With the default `scope="request"`, the close in teardown runs only after the response is sent, after background tasks, and after the last chunk of a streamed body. So a request that queries, then does slow work such as calling another service or streaming to a slow client, keeps its connection for all of it. ## Why the database looks idle - The waiting happens in the application, in the pool's queue, so the database sees at most 15 connections doing little. - Requests that hold a connection while doing non-database work inflate hold time without adding database load. - Error rates jump at a concurrency threshold rather than growing smoothly, because requests 16 to 40 wait while 1 to 15 run. ## A detail FastAPI already handles FastAPI enters a sync yield dependency through the thread pool, but runs its code after `yield` on a separate `CapacityLimiter(1)` created per call. The source comment explains why: if closing a session had to wait for a thread token while all 40 threads were blocked waiting for connections that only those closes would return, the process would deadlock. Teardown therefore never queues behind the very requests it is trying to unblock. ## Sizing the two together - **Match the pool to the thread budget** per worker: `pool_size + max_overflow` at least the number of threads that can hold a session at once. - **Multiply by processes**: with `fastapi run --workers 4`, each worker has its own engine and its own 40 tokens, so the database must accept four times the per-worker connections within PostgreSQL's `max_connections`. - **Or lower the thread budget**: set `total_tokens` on the default limiter in the `lifespan`, below the pool size, so excess requests queue for a thread instead of timing out on a connection. - **Shorten hold time**: commit before slow work, use `Depends(get_session, scope="function")` where nothing after the return needs the session, and give background work its own session. - **Shorten `pool_timeout`** if you want overload to fail fast instead of piling up 30-second waits. ```python @asynccontextmanager async def lifespan(app: FastAPI): anyio.to_thread.current_default_thread_limiter().total_tokens = 25 yield engine.dispose() ``` ## Confirming the diagnosis Before changing numbers, check that the symptoms match this mechanism: - **Error timing**: failures arrive about `pool_timeout` seconds after the request started, 30 by default, rather than spread out. - **Pool state**: `engine.pool.status()` reports the pool size, the connections in the pool, the current overflow and the checked-out count; under the incident the checked-out count sits at `pool_size + max_overflow`. - **Hold time**: requests that hold connections longest are the ones doing non-database work after their first query, or streaming to slow clients. - **Concurrency**: the errors start when concurrent `def` requests per worker exceed the pool, and disappear below it. If the database itself shows high load or lock waits, the problem is elsewhere; this pattern is specifically an application-side queue. ## async def endpoints `async def` endpoints with an `AsyncSession` do not take thread tokens at all, so nothing caps how many requests reach the pool; there the pool and its timeout are the only limit, and the same arithmetic applies with the thread budget removed.

  • Why does FastAPI run a sync yield dependency's teardown outside the default thread limiter?
    Its `contextmanager_in_threadpool` helper runs `__exit__` on a fresh `CapacityLimiter(1)`. If teardown needed a token from the shared 40, all 40 threads could be blocked waiting for pooled connections that only those pending closes would return, and the worker would deadlock. A separate limiter lets the close always run and hand the connection back.
  • How does the sizing change when the app runs with fastapi run --workers 4?
    Each worker is a separate process with its own engine, its own pool and its own 40-token limiter. The per-worker pool sizing stays the same, but the database must accept four times `pool_size + max_overflow` connections, plus any other clients, within PostgreSQL's `max_connections`. Raising workers without checking that budget moves the timeouts into the database.

A shop lets 40 customers through the door but has 15 fitting rooms. Customers who keep a room while they also queue at the till leave the others waiting outside the rooms, even though nobody is using the mirrors.

  • The database must be overloaded, so add database CPU before touching the app.
  • FastAPI starts a new thread for every def request with no upper limit.
  • Opening Session(engine) in get_session checks out a connection immediately.
  • Teardown frees the connection before the response is sent, so slow clients do not matter.
  • async def endpoints with an AsyncSession share the same 40-thread limit.