OpenAI-Compatible Server

## The server behind the routes `vllm serve <model>` loads the weights onto the GPU and starts an HTTP server (FastAPI on uvicorn) listening on port 8000 by default. The routes it publishes are deliberately the ones a hosted OpenAI-style provider publishes: `POST /v1/chat/completions`, `POST /v1/completions`, `GET /v1/models`, and — when the loaded model is an embedding model — `POST /v1/embeddings`. Beside them sit routes that are vLLM's own and not part of that API surface: `GET /health` for readiness probes, `GET /metrics` for Prometheus scraping, and `POST /tokenize` / `POST /detokenize` for inspecting how text becomes tokens. Compatibility means the JSON shapes match, not just the paths. You send `messages` with `role` and `content`; you get back `choices[0].message.content`, a `finish_reason`, and a `usage` object with prompt and completion token counts. With `"stream": true` you get a `text/event-stream` response whose chunk objects are `chat.completion.chunk` and whose final line is `data: [DONE]`. An SDK written for the hosted API is just an HTTP client for those shapes, so it drives vLLM without a rewrite. That is the whole point of the compatibility layer: migrating off a paid provider becomes a configuration change rather than a code change. ## The three client settings that change **base_url.** The OpenAI SDKs append the route path to whatever base URL you configure, so the value must end at `/v1` — `http://localhost:8000/v1`. Point it at `http://localhost:8000` and the client will request `/chat/completions`, which the server does not serve, and the 404 will look like a broken server rather than a misconfigured client. **model.** The hosted-provider habit of passing any model id and getting a model back does not apply. A vLLM process serves exactly one model, and the `model` field is validated against the names it advertises. Query `GET /v1/models` to see them; the default name is the exact string you passed to `vllm serve` (a Hugging Face repo id or a local directory path). Send anything else and the request is rejected. **api_key.** The SDKs refuse to construct a client with an empty key, so pass a placeholder such as `"not-used"`. vLLM only checks the value if the server was started with `--api-key` (or the `VLLM_API_KEY` environment variable), in which case the client's key must match that shared token exactly. ## The served model name is a real contract `--served-model-name` decouples the public name from the checkpoint on disk. If your application already hardcodes a provider model string, launching with `--served-model-name` set to that string means literally nothing in the application changes except the base URL. It is also how you keep a stable public name while swapping the underlying checkpoint — clients keep calling the same name across a quantized rebuild or a version bump. When LoRA adapters are enabled, their names show up in `/v1/models` too, and are addressed through the same `model` field. ## What does not come along The protocol is compatible; the platform is not. There is no billing, no per-key quota, no organization-level rate limiting, no files, batch or moderation surface, and no managed uptime. You now own GPU capacity, model load time on startup, version upgrades and failure handling. Client code that relied on a provider returning `429` when you exceeded a quota gets a different experience here: an overloaded vLLM queues requests, so the symptom is a growing time-to-first-token rather than a rejection, and client timeouts must be set with that in mind. ## One process, one model A `vllm serve` process holds one model in GPU memory. Serving a second model means a second process (and, usually, a second GPU) behind a router. The one exception is LoRA adapters, which share a base model in the same process and are selected through the `model` field. Embeddings are a related trap: `/v1/embeddings` is only usable when the served model runs as an embedding (pooling) model, so a chat server cannot also answer your retrieval pipeline's embedding calls — run a second server for the embedding model. ## Verifying in thirty seconds `curl http://localhost:8000/v1/models` confirms the server is up and tells you the exact name to send. Then post a one-line chat completion and check that `choices[0].message.content` comes back. Use `/health` for your orchestrator's readiness probe rather than a completion request, so probes do not consume GPU time.