Migrating from a hosted OpenAI endpoint to vLLM, what still breaks despite API compatibility?

## Why the easy part is genuinely easy The compatibility layer earns its keep. Routes, request schemas, response schemas and the streaming chunk format match, so the SDK, the streaming loop, the message-building code and the response parsing all survive the move untouched. Point `base_url` at the server, match the model name, and traffic flows. Any migration plan that budgets significant engineering for *that* is mis-scoped. The work is in everything the API shape does not describe. ## Group one: it is a different model The most under-estimated fact is that you are not migrating an endpoint, you are migrating a model. The open-weight checkpoint has its own tokenizer, so identical text yields a different token count — often by ten percent or more in either direction. Everything downstream of a token count shifts with it: your cost model, your "how many documents fit" retrieval budget, your truncation thresholds, your per-request usage records. Anywhere your code computes a token count against the provider's tokenizer, that computation is now wrong. Quality is a separate question and cannot be inferred from benchmarks. Instruction-following style, verbosity, refusal behaviour, JSON reliability and multilingual coverage all move. Prompts tuned over months against one model are, in effect, overfitted to it; expect a prompt-tuning pass, and expect some prompts to need real rework rather than tweaks. ## Group two: the contract edges **Length limits become hard errors.** The server has a configured maximum context length. If a request's prompt plus its requested output exceeds it, the request is rejected rather than quietly accommodated. Code paths that grew a conversation until the provider complained now fail earlier and differently, so history-trimming logic must be driven by your own accounting rather than by an error you catch. **Parameter support is a per-server fact.** Some fields the hosted API accepts have no equivalent here, and some vLLM extras have no equivalent there. The depth of `logprobs` you may request is bounded by server configuration. Structured-output and tool-calling behaviour depend on what the checkpoint and its configured parser support, not on the protocol. Audit which optional fields your code actually sends; the long tail is where a request starts failing in production for one feature flag. **Error surfaces differ.** Status codes and error bodies are similar in shape but not identical in trigger, so any code branching on a specific error condition needs re-verification. ## Group three: the platform is gone This is the part that catches senior teams. The provider was not only serving a model; it was providing capacity management, quota enforcement, billing, availability and a rate-limit protocol. All of that is now yours. The sharpest instance is **overload behaviour**. A hosted provider rejects excess traffic, and mature clients respond with backoff — that feedback loop is what keeps the system stable. vLLM does not reject; it queues. Excess load turns into a growing time-to-first-token, client timeouts fire, the client retries, and the retries add work to a server that is already behind. A retry policy tuned for a provider's rejections becomes an amplifier. You need client-side concurrency limits, timeouts set against your actual saturation curve, and admission control somewhere in the path. Availability is the other one: a single process on a single GPU has none of the redundancy you were implicitly buying, and losing the node loses the endpoint. ## The principal-level move Design the swap so it is genuinely a configuration change, which means the concerns that used to be the provider's should not be scattered through application code. Put a gateway in front of both the self-hosted endpoint and any hosted one: it owns authentication and per-tenant keys, quotas and rate limits, request-size caps, usage records for chargeback, routing by model name, and fallback to a hosted provider when the self-hosted pool is unhealthy or saturated. With that layer in place, moving a workload between hosted and self-hosted is a routing rule, and you keep the option to move back — which matters, because the first migration rarely covers every workload. ## Validating before you cut over Do not cut over on a smoke test. Capture a representative corpus of real production requests and replay it against both endpoints. Compare on things you can measure automatically: the distribution of `finish_reason` values, the rate of responses that fail to parse against the schema your code expects, tool-call validity, output-length distribution, and refusal rate — alongside whatever task-quality metric you already trust. Then shadow live traffic before it takes user-facing load, and keep the hosted route configured for a fast rollback. The compatibility layer makes that rollback a config flip, which is exactly the property you should protect.