How do you serve several LoRA adapters from one vLLM server and select one per request?

## What the server is doing A LoRA adapter is a small set of low-rank weight deltas trained against a specific base checkpoint. Serving one does not require materializing a whole new model: vLLM loads the base weights once and applies the selected adapter's deltas during the forward pass, per request. That is what makes many adapters on one replica economically interesting — the expensive part, the base weights in GPU memory, is shared, and each adapter adds only megabytes. From the API's point of view the trick is that adapters are exposed as *models*. `--enable-lora` turns the feature on; `--lora-modules` registers adapters as `name=path` pairs (a JSON form is also accepted when you need to be explicit about the base model name). Each registered name then shows up in `GET /v1/models` next to the base model, and a request routes to it by setting `"model": "sql"`. No new endpoint, no custom header, no separate port. Clients that already know how to pick a model already know how to pick an adapter. ## The two capacity flags `--max-lora-rank` declares the largest adapter rank the server will accept. Rank is the width of the low-rank decomposition and is fixed at training time; the server preallocates buffers sized for this value, so it is not free. Set it below an adapter's actual rank and that adapter is rejected; set it far above (say 64 when every adapter is rank 8) and you have paid memory for nothing. Take the maximum over the adapters you intend to serve, and remember that admitting a higher-rank adapter later is a restart. `--max-loras` is different: it caps how many *distinct* adapters may participate in one batched step. It does not cap how many are registered. With a low value, requests for adapters beyond the cap wait for a step in which a slot frees, which shows up as latency for the unlucky adapters rather than as an error. Raising it lets more adapters share a batch at some throughput cost, because the batched matrix work becomes more heterogeneous. There is also a CPU-side cache (`--max-cpu-loras`) that holds adapters not currently resident on the GPU, so swapping one in does not mean re-reading it from disk. ## Runtime add and remove Registering everything at launch is fine when the adapter set is static, but a per-tenant fine-tuning product ships new adapters continuously and cannot restart a GPU process for each one. vLLM exposes `POST /v1/load_lora_adapter` with a `lora_name` and `lora_path`, and `POST /v1/unload_lora_adapter` to drop one. These routes are gated behind the `VLLM_ALLOW_RUNTIME_LORA_UPDATING` environment variable and are off by default, for good reason: a route that makes the server read a filesystem path and load weights into a running process is an obvious abuse target, so it must never be reachable from untrusted callers. Treat it as a control-plane operation, on a private network path, behind your own authorization. ## The constraints people trip over **Same base, exactly.** An adapter's deltas are shaped for a specific checkpoint. Point a server at a different base — even a different revision or a differently quantized build of the "same" model — and the adapter is at best rejected, at worst semantically wrong. The pairing of adapter and base is a versioned artifact. **One base per process.** All adapters on a server share the one base model. Serving adapters for two different bases means two servers. **LoRA is not free at inference.** Applying deltas adds per-token work and heterogeneity to the batch, so a LoRA-enabled server is measurably slower than the same server without it. That cost is what you pay for not standing up one replica per adapter, and it is usually an easy trade at any meaningful adapter count — but do measure it rather than assuming the overhead is zero. **Name collisions matter.** Adapter names live in the same namespace as the served model name, since both are addressed by `model`. Choose names deliberately, because they are now a public API contract that clients hardcode. ## Operating it Make `/v1/models` your source of truth in health checks — it tells you which adapters this replica currently has, which is the question that matters after a rolling restart or a runtime load. Log the requested `model` value so you can attribute traffic per adapter. And keep adapter artifacts wherever your model weights live, with the base checkpoint id recorded next to them, so a restart re-registers exactly the pairs that were serving before.