In FastAPI, why does a CPU-bound image-thumbnail endpoint slow other requests whether it is declared def or async def?
answer
- computing is not waiting
- the loop thread needs the GIL
- a token held the whole time
- another process, or a queue
basics
~20 sCPU work holds CPython's GIL. Inside async def it runs on FastAPI's event loop and freezes every request on that worker; inside def it runs on a worker thread but shares the GIL with the loop, so other requests still slow.
solid answer
~40 sMoving work to a thread helps code that **waits**; a thumbnail endpoint **computes**. Declared `async def`, the resize runs on the event loop and every other request on that worker waits until it finishes. Declared `def`, FastAPI runs it on an anyio worker thread, so the loop keeps running, but on a standard CPython build only one thread executes Python bytecode at a time: the loop must win the GIL back between slices, so other requests slow down, concurrent thumbnails get no faster than doing them one by one, and each one holds a thread-limiter token throughout. The real fixes take the work off the process's GIL: `anyio.to_process.run_sync` for short jobs, a task queue for long ones, or a library whose C code releases the GIL while it computes.
go deeper
Remember that heavy computation inside async def freezes every other request, and that def only partly helps because of the GIL.
Explain why worker threads help waiting but not computing: the shared GIL, no parallel speedup, and a thread-limiter token held for the whole job.
Choose between anyio.to_process.run_sync, a task queue and a GIL-releasing library from the job's duration, payload size and restart needs, and show the measurement that justifies it.
Decide where CPU-heavy work such as inference or media processing lives in the architecture, so API workers stay responsive and compute scales separately.
## Waiting versus computing FastAPI's choice between the event loop and the thread pool is built for code that **waits**: a network call, a database query, a disk read. While a worker thread waits, it releases CPython's **global interpreter lock (GIL)**, so the event loop thread can run other requests. An image-thumbnail endpoint, or a model inference step, does the opposite: it **computes**. Pure-Python computation holds the GIL, releasing it only when the interpreter forces a switch. That changes what `def` and `async def` buy you. | Declared as | Where the resize runs | Effect on other requests on that worker | |---|---|---| | `async def` | on the event loop | all of them wait until the resize finishes | | `def` | on an anyio worker thread | they keep running, but slower, competing for the GIL | | `def` with a C library that releases the GIL | on a worker thread, mostly outside the GIL | much less contention; the work can use several cores | ## What async def does to CPU work An `async def` endpoint is awaited directly on the event loop. Computation contains no `await`, so the loop cannot switch away: health checks, cheap endpoints and other uploads all wait for the whole resize. This is the same failure as a blocking SDK call, with one difference: CPU usage on that core is **high**, not low. ## What def does to CPU work A `def` endpoint runs through `run_in_threadpool` on a worker thread, so the loop is never frozen outright. Three costs remain: 1. **The GIL is shared.** On a standard CPython build the worker thread and the event loop thread take turns holding the GIL. Other requests still run, but each one waits for its turn. 2. **No parallel speedup.** Several thumbnails on several worker threads still execute one bytecode stream at a time, so total time is about the same as running them in sequence. 3. **A token is held throughout.** Each call borrows one of anyio's default 40 thread-limiter tokens for its whole duration, so a burst of slow resizes also delays `def` endpoints that only wait on I/O. In one in-process measurement, four concurrent requests that each burned about 0.18 s of CPU took about 0.7 s in total either way. An `async def` health check sent at the same moment answered after about 0.15 s when the work was in `def` endpoints, and after about 0.68 s, behind all four, when it was in `async def` endpoints. So `def` is the better of the two declarations for CPU work, but it is damage limitation, not a fix. ## Fixes that take the work off the GIL - **A worker process.** `anyio.to_process.run_sync(make_thumbnail, data)` runs the function in a separate Python process, with its own GIL, and awaits the result without blocking the loop. The function must be importable at module level, its arguments and result are pickled across the process boundary, and anyio caps concurrent processes with its own default process limiter, sized from the CPU count. - **A task queue.** For work measured in seconds, or work that must survive a restart, accept the upload, enqueue a job and return its id. The endpoint then does no heavy work at all. - **A library that releases the GIL.** Some image and numeric libraries release the GIL inside their C code. Called from a `def` endpoint, that work can then run on several cores at once; check the library's documentation rather than assuming it. ```python import anyio.to_process from fastapi import FastAPI, Response, UploadFile app = FastAPI() def make_thumbnail(data: bytes) -> bytes: # stands in for a CPU-bound resize; defined at module level so it pickles return data[:1024] @app.post("/thumbnails") async def create_thumbnail(file: UploadFile): data = await file.read() thumb = await anyio.to_process.run_sync(make_thumbnail, data) return Response(content=thumb, media_type="image/png") ``` ## What does not fix it - **Raising the thread limiter.** More threads mean more contenders for the same GIL, not more CPU work done. - **`BackgroundTasks`.** Background tasks run in the same process after the response, so a CPU-heavy task still competes for the same GIL. How many Uvicorn worker processes to run, and whether the free-threaded CPython build changes the picture, are deployment and Python-runtime questions; within one FastAPI process the choice is loop, thread, or another process.
- Why does a def endpoint doing CPU work in FastAPI still slow an async def health check?On a standard CPython build the worker thread and the event loop thread share one GIL. The loop still runs, but each time it needs to execute Python code it waits for the computing thread to give up the GIL, so the health check answers late without being blocked outright.
- What must be true of a function passed to anyio.to_process.run_sync from a FastAPI endpoint?It runs in another Python process, so the function must be importable by reference, defined at module level rather than as a lambda or a closure, and its arguments and return value must be picklable. Large payloads pay for the copy across the process boundary.
- Declaring a CPU-heavy endpoint def makes it run in parallel on all cores
- A CPU-heavy def endpoint has no effect on async def endpoints
- Raising anyio's thread limiter speeds up CPU-bound endpoints
- async def suits heavy CPU work because it does no I/O
- Background tasks move CPU work out of the server process