Why does a FastAPI API that loads a 3 GB model exhaust its VM's memory after switching to `fastapi run main.py --workers 4`, and what do you change?
answer
- copies, not shared pages
- spawn means a fresh import
- the parent imported it too
- lifespan versus import time
- size workers by memory
basics
~20 sEach Uvicorn worker is a spawned process that imports the app and loads its own model, so four workers hold four copies, plus one in the CLI's parent if loaded at import. Load it in lifespan and size workers by memory.
solid answer
~50 sUvicorn starts workers with `spawn`, so nothing is shared by copy-on-write: each worker imports the module, runs the `lifespan`, and holds its own 3 GB, which is 12 GB for four. If the model is loaded at module level, it gets worse: given a path such as `main.py`, or left to auto-discover, the FastAPI CLI imports the module in the parent process to find the app object, and that parent keeps a fifth copy for as long as it runs. The fix starts with loading the model inside `lifespan`, which the parent never runs, and pinning `[tool.fastapi] entrypoint` so the parent never imports the module at all. Then size the worker count from one worker's measured memory, not from the core count, and if the budget still does not fit, run fewer workers or move the model into a process of its own that the workers call.
code
python · 24 linesfrom contextlib import asynccontextmanager
from typing import Any
from fastapi import FastAPI, Request
def load_model() -> dict[str, Any]:
# stands in for reading the weights from disk
return {"weights": bytearray(1024)}
@asynccontextmanager
async def lifespan(app: FastAPI):
model = load_model() # runs once per worker, never in the CLI parent
yield {"model": model}
app = FastAPI(lifespan=lifespan)
@app.get("/predict")
async def predict(request: Request) -> dict[str, int]:
model = request.state.model
return {"size": len(model["weights"])}go deeper
Remember that every worker process loads its own copy of whatever the FastAPI app loads at startup.
Explain spawn versus fork, why lifespan startup runs in each worker, and how the CLI's app discovery imports the module in the parent.
Diagnose from per-process memory, move loading into the lifespan, pin the entrypoint, and size workers from one measured worker rather than the core count.
Decide where large models live: inside every API worker, in a dedicated model process, or in replicas sized by the scheduler, and make that the team's default.
## What actually happened On one process the API used a bit over 3 GB. After the switch, memory climbs while the workers start and the machine runs out before or shortly after startup completes. Nothing leaked. The deployment now holds **several complete copies** of everything the app loads, because of three facts about how FastAPI's CLI and Uvicorn start workers. ## Three facts that multiply memory 1. **Workers are spawned, not forked.** Uvicorn starts each worker with Python's `spawn` start method. A spawned process is a fresh interpreter that imports your module itself, so nothing the parent loaded is shared through copy-on-write pages. Four workers mean four imports and four independent heaps. 2. **Lifespan runs in every worker.** Each worker runs the app's `lifespan` startup. A model loaded there exists once per worker, and nowhere else. 3. **The CLI's parent may import the module too.** To find the app object, the FastAPI CLI imports the module when it was given a path (`fastapi run main.py`) or had to auto-discover one. That import happens in the parent process, the one that then supervises the workers, and the module stays imported. A model created at module level therefore sits in the parent as well, doing nothing, for the whole life of the server. With an entrypoint taken from `[tool.fastapi]` in pyproject.toml, or from `--entrypoint`, the CLI only builds the import string and never imports the module in the parent. | Where the model is loaded | How the app is named | Copies with `--workers 4` | |---|---|---| | module level | path or auto-discovery | 5 (parent and four workers) | | module level | `[tool.fastapi] entrypoint` | 4 | | inside `lifespan` | any | 4 | | inside `lifespan`, `--workers 2` | any | 2 | At 3 GB a copy, that is 15 GB against 12 GB, before Python, the framework and request buffers are counted. ## What to change 1. **Move the load into `lifespan`** and hand the model to endpoints through the yielded state (`request.state`) or `app.state`. The parent never runs the lifespan, so it never holds a copy, and a load that fails makes startup fail instead of the first request. 2. **Pin the entrypoint** in pyproject.toml so the parent does not import the module just to find the app. It also makes the command line the same everywhere. 3. **Measure one worker.** Start a single worker, wait for `Application startup complete.`, and read its resident memory. That number, not the core count, sets the ceiling: workers times per-worker memory, plus headroom for request buffers and the parent, has to fit the machine. 4. **Choose the count from both limits.** If inference is CPU-bound, workers are your parallelism, so the answer is the smaller of the core-based and memory-based counts. If the endpoint mostly waits on I/O, fewer workers with more concurrency per worker serve the same traffic with fewer copies. 5. **Separate the model when the copies do not fit.** A single model process that the API workers call over a local socket holds one copy, and the API workers become small. Some loaders can also memory-map a weights file read-only, which lets the operating system share those pages between processes; check that yours does before you rely on it. ## What does not fix it - **Gunicorn's `--preload` with uvicorn-worker's `UvicornWorker`.** Preloading imports the app in the Gunicorn master before forking, so module-level objects start out as shared pages. But the lifespan still runs in each worker after the fork, so a lifespan-loaded model is loaded once per worker anyway, and module-level objects lose much of their sharing as Python touches them. - **Raising the swap.** The workers get slower, they do not get smaller. - **Using threads instead.** One process with a bigger thread pool keeps one copy, but on the standard interpreter it also gives up the CPU parallelism for Python code that made you add workers. ## How the same app behaves on a cluster On a container platform that runs one Uvicorn process per container, the copies do not disappear: four replicas still hold four models. The difference is that each container's memory limit is set to one copy plus headroom, and the scheduler places replicas only where that much memory is free. The arithmetic is the same; the cluster does it for you.
- How would you prove that the CLI's parent process holds a model copy?Run `fastapi run main.py --workers 2` with the model created at module level, then list the parent and its children with their resident memory. The parent, which logged `Started parent process`, shows roughly the model's size although it serves nothing. Repeat with `[tool.fastapi] entrypoint` set and no path argument: the parent drops to a few tens of megabytes, while each worker is unchanged.
- Why does each worker's memory stay high even when traffic is low?The model is loaded at startup and held for the life of the worker, whether or not requests arrive. Memory per process stays roughly flat over time while CPU use varies, which is why FastAPI's deployment docs treat memory as the stable constraint on how many processes fit on a machine.
- Does a model loaded in `lifespan` survive a worker restart by the Uvicorn parent?No. A replacement worker is a new spawned process, so it imports the app again and runs the lifespan startup again, loading a fresh copy. Slow model loads therefore lengthen recovery after a crash and every step of a `SIGHUP` rolling restart, which is worth measuring before you rely on fast restarts.
- Uvicorn workers share the parent's model through copy-on-write
- The parent process only supervises, so it can never hold app memory
- Gunicorn --preload makes a lifespan-loaded model load only once
- Set the worker count to the number of CPU cores and ignore memory
- The memory growth must be a leak in the inference code