In KServe 0.21, which engine actually serves an InferenceService with modelFormat huggingface, and what does the runtime's --backend argument change?
answer
- one runtime, two engines inside
- default backend is auto
- falls back when architecture unsupported
- what vllm and huggingface override
- OpenAI routes under a prefix
basics
~20 sThe shipped kserve-huggingfaceserver runtime runs it with --backend auto by default: vLLM when vLLM is installed and supports the model's architecture, else the Hugging Face transformers backend. --backend huggingface forces transformers; --backend vllm makes a missing vLLM a startup error.
solid answer
~30 s`modelFormat: huggingface` binds the `kserve-huggingfaceserver` ClusterServingRuntime, which usually loads an `hf://` storage URI. Its `--backend` argument defaults to `auto`, which picks **vLLM** when vLLM is available in the image and supports the model's architecture, and falls back to the Hugging Face (transformers) backend otherwise. `--backend vllm` turns a missing vLLM into a startup error but still falls back for an architecture vLLM cannot run. `--backend huggingface` forces transformers, which is the path for predictive tasks like sequence classification and the workaround when vLLM errors. Generative models answer OpenAI-compatible routes such as `/openai/v1/chat/completions`. Since 0.20 there is also a separate `kserve-vllmserver` runtime.
code
yaml · 16 linesapiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
name: playlist-blurbs
spec:
predictor:
model:
modelFormat:
name: huggingface
args:
- --model_name=blurbs
- --backend=vllm
storageUri: hf://example-org/blurb-7b-instruct
resources:
limits:
nvidia.com/gpu: "1"go deeper
Recall that the Hugging Face runtime defaults to --backend auto and prefers vLLM when it can.
Explain the auto rule (vLLM installed and architecture supported), what vllm and huggingface each change, and the /openai route prefix.
Decide when to pin a backend so failures are loud, and never assume auto chose vLLM without checking the image and the model's architecture.
Choose between the Hugging Face runtime, the standalone vLLM runtime and LLMInferenceService as the platform's LLM path, and keep that choice consistent.
## The Hugging Face serving runtime An InferenceService whose predictor says `modelFormat: {name: huggingface}` is bound to the shipped **`kserve-huggingfaceserver`** ClusterServingRuntime. It lists `huggingface` with `autoSelect: true` and `priority: 1`, and speaks `v2` and `v1`. The model usually comes from the Hub through an `hf://` storage URI, which the storage initializer downloads. A gated repository needs an `HF_TOKEN`, which KServe's docs supply through a `ClusterStorageContainer` that references a Secret. Runtime arguments go in `predictor.model.args`, for example `--model_name=blurbs`. ## What --backend decides The runtime is a single server with two ways to run a model, and its **`--backend`** argument picks one: | Value | Behaviour | |---|---| | `auto` (default) | Use **vLLM** if vLLM is installed in the image **and** vLLM supports the model's architecture; otherwise use the Hugging Face (transformers) backend | | `vllm` | Require vLLM: the server fails to start with `Backend is set to 'vllm' but vLLM is not available` if it is missing. The architecture check still applies, so an architecture vLLM cannot run still lands on transformers | | `huggingface` | Force the transformers backend, even for a model vLLM could run | So for a 7B instruction model with a common architecture on the GPU image, `auto` already means vLLM. That is the 0.21 behaviour, and an answer that says "the Hugging Face runtime runs transformers' generate loop" describes the fallback, not the default. When to override: 1. `--backend huggingface` for predictive tasks such as `sequence_classification`, `token_classification` or `fill_mask`, which KServe serves on the transformers backend. It is also the documented workaround when the vLLM backend errors for a model. 2. `--backend vllm` when a missing engine in the image should fail the start loudly rather than quietly fall back to the slower path. It does not override the architecture check, so it is not a way to push an unsupported model onto vLLM. With the vLLM backend, vLLM's engine arguments can be added to `args` and are passed through. Choosing their values is engine tuning, outside KServe. ## What clients call For generative models the runtime serves an OpenAI-compatible API under the `/openai` prefix: - `POST /openai/v1/chat/completions` - `POST /openai/v1/completions` - `POST /openai/v1/embeddings` - `GET /openai/v1/models` The prefix comes from the `KSERVE_OPENAI_ROUTE_PREFIX` environment variable, which defaults to `/openai`. A client pointed at the service's base URL without it gets a 404. Predictive tasks use KServe's V1 or V2 routes instead. ## Neighbouring options in the same release line - **`kserve-vllmserver`**: since 0.20 KServe also ships a standalone vLLM serving runtime for InferenceService, separate from the Hugging Face runtime. It is a different runtime with its own `supportedModelFormats` entry, chosen by format or by `runtime:`. - **`LLMInferenceService`**: 0.21 also has a separate resource kind (`serving.kserve.io/v1alpha2`) for LLM-specific deployments. It is not an InferenceService and has its own configuration. ## Common mistakes - Assuming the runtime "uses transformers" and tuning for that, when `auto` has already put the model on vLLM. - Leaving `auto` on an image without vLLM and expecting vLLM throughput. `auto` falls back without failing. - Believing `--backend vllm` forces any model onto vLLM. It only requires the engine to be present; an unsupported architecture still runs on transformers. - Pointing an OpenAI client at the bare host without the `/openai/v1` base path. - Mixing the runtimes up: `modelFormat: huggingface` selects the Hugging Face runtime, which may run vLLM internally, and that is a different object from the standalone vLLM runtime added in 0.20. - Putting a Hub token in `args` or the manifest instead of a Secret consumed by the storage container. ## A worked deployment For the blurb writer, a predictor with `modelFormat: huggingface`, `storageUri: hf://example-org/blurb-7b-instruct`, one GPU in `resources.limits`, and `args` holding `--model_name=blurbs` is enough. The storage initializer pulls the weights, the runtime starts with `--backend auto`, and on the GPU image it loads the model on vLLM. Clients then call `/openai/v1/chat/completions` with `"model": "blurbs"`. Adding `--backend=vllm` changes nothing on a healthy image with a supported architecture. It only turns a missing engine into a failed start instead of a slow service. ## Checking what you got - The pod's container is `kserve-container`, running the runtime image with your `args`. - The `args` list on the predictor: only `--backend huggingface` fixes the engine in advance, since `auto` and `vllm` both decide at startup from the image and the model's architecture. - A request to `/openai/v1/models` lists the served model name, which is the `--model_name` you set (otherwise the InferenceService name).
- A client using an OpenAI SDK gets 404s from a KServe Hugging Face runtime at /v1/chat/completions. What is wrong?The runtime mounts its OpenAI-compatible routes under a prefix, `/openai` by default (the `KSERVE_OPENAI_ROUTE_PREFIX` environment variable). The client's base URL must end in `/openai/v1`, so the call goes to `/openai/v1/chat/completions` on the InferenceService host.
- Why might a team pin --backend huggingface on the KServe Hugging Face runtime?Either because the task is predictive, such as `sequence_classification` or `fill_mask`, which KServe runs on the transformers backend, or because the vLLM backend errors for a particular model and the documented workaround is to force transformers. They trade generation throughput for compatibility.
- The Hugging Face runtime always runs transformers' generate loop
- --backend vllm silently falls back when vLLM is missing
- KServe's OpenAI routes sit at /v1 with no prefix
- modelFormat huggingface binds the standalone vLLM runtime
- OpenAI-compatible routes need the standalone vLLM runtime, not this one
Assumes KServe 0.21, whose Hugging Face runtime defaults to --backend auto (vLLM first) and which also ships the standalone vLLM runtime added in 0.20.