skip to content

In KServe 0.21, which engine actually serves an InferenceService with modelFormat huggingface, and what does the runtime's --backend argument change?

level: middleimportance: nice to knowfreq 6%

answer

  1. one runtime, two engines inside
  2. default backend is auto
  3. falls back when architecture unsupported
  4. what vllm and huggingface override
  5. OpenAI routes under a prefix

basics

~20 s

The shipped kserve-huggingfaceserver runtime runs it with --backend auto by default: vLLM when vLLM is installed and supports the model's architecture, else the Hugging Face transformers backend. --backend huggingface forces transformers; --backend vllm makes a missing vLLM a startup error.

solid answer

~30 s

`modelFormat: huggingface` binds the `kserve-huggingfaceserver` ClusterServingRuntime, which usually loads an `hf://` storage URI. Its `--backend` argument defaults to `auto`, which picks **vLLM** when vLLM is available in the image and supports the model's architecture, and falls back to the Hugging Face (transformers) backend otherwise. `--backend vllm` turns a missing vLLM into a startup error but still falls back for an architecture vLLM cannot run. `--backend huggingface` forces transformers, which is the path for predictive tasks like sequence classification and the workaround when vLLM errors. Generative models answer OpenAI-compatible routes such as `/openai/v1/chat/completions`. Since 0.20 there is also a separate `kserve-vllmserver` runtime.

code

yaml · 16 lines
yaml
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: playlist-blurbs
spec:
  predictor:
    model:
      modelFormat:
        name: huggingface
      args:
        - --model_name=blurbs
        - --backend=vllm
      storageUri: hf://example-org/blurb-7b-instruct
      resources:
        limits:
          nvidia.com/gpu: "1"

go deeper

for a junior

Recall that the Hugging Face runtime defaults to --backend auto and prefers vLLM when it can.

for a middle

Explain the auto rule (vLLM installed and architecture supported), what vllm and huggingface each change, and the /openai route prefix.

for a senior

Decide when to pin a backend so failures are loud, and never assume auto chose vLLM without checking the image and the model's architecture.

for a principal

Choose between the Hugging Face runtime, the standalone vLLM runtime and LLMInferenceService as the platform's LLM path, and keep that choice consistent.

## The Hugging Face serving runtime An InferenceService whose predictor says `modelFormat: {name: huggingface}` is bound to the shipped **`kserve-huggingfaceserver`** ClusterServingRuntime. It lists `huggingface` with `autoSelect: true` and `priority: 1`, and speaks `v2` and `v1`. The model usually comes from the Hub through an `hf://` storage URI, which the storage initializer downloads. A gated repository needs an `HF_TOKEN`, which KServe's docs supply through a `ClusterStorageContainer` that references a Secret. Runtime arguments go in `predictor.model.args`, for example `--model_name=blurbs`. ## What --backend decides The runtime is a single server with two ways to run a model, and its **`--backend`** argument picks one: | Value | Behaviour | |---|---| | `auto` (default) | Use **vLLM** if vLLM is installed in the image **and** vLLM supports the model's architecture; otherwise use the Hugging Face (transformers) backend | | `vllm` | Require vLLM: the server fails to start with `Backend is set to 'vllm' but vLLM is not available` if it is missing. The architecture check still applies, so an architecture vLLM cannot run still lands on transformers | | `huggingface` | Force the transformers backend, even for a model vLLM could run | So for a 7B instruction model with a common architecture on the GPU image, `auto` already means vLLM. That is the 0.21 behaviour, and an answer that says "the Hugging Face runtime runs transformers' generate loop" describes the fallback, not the default. When to override: 1. `--backend huggingface` for predictive tasks such as `sequence_classification`, `token_classification` or `fill_mask`, which KServe serves on the transformers backend. It is also the documented workaround when the vLLM backend errors for a model. 2. `--backend vllm` when a missing engine in the image should fail the start loudly rather than quietly fall back to the slower path. It does not override the architecture check, so it is not a way to push an unsupported model onto vLLM. With the vLLM backend, vLLM's engine arguments can be added to `args` and are passed through. Choosing their values is engine tuning, outside KServe. ## What clients call For generative models the runtime serves an OpenAI-compatible API under the `/openai` prefix: - `POST /openai/v1/chat/completions` - `POST /openai/v1/completions` - `POST /openai/v1/embeddings` - `GET /openai/v1/models` The prefix comes from the `KSERVE_OPENAI_ROUTE_PREFIX` environment variable, which defaults to `/openai`. A client pointed at the service's base URL without it gets a 404. Predictive tasks use KServe's V1 or V2 routes instead. ## Neighbouring options in the same release line - **`kserve-vllmserver`**: since 0.20 KServe also ships a standalone vLLM serving runtime for InferenceService, separate from the Hugging Face runtime. It is a different runtime with its own `supportedModelFormats` entry, chosen by format or by `runtime:`. - **`LLMInferenceService`**: 0.21 also has a separate resource kind (`serving.kserve.io/v1alpha2`) for LLM-specific deployments. It is not an InferenceService and has its own configuration. ## Common mistakes - Assuming the runtime "uses transformers" and tuning for that, when `auto` has already put the model on vLLM. - Leaving `auto` on an image without vLLM and expecting vLLM throughput. `auto` falls back without failing. - Believing `--backend vllm` forces any model onto vLLM. It only requires the engine to be present; an unsupported architecture still runs on transformers. - Pointing an OpenAI client at the bare host without the `/openai/v1` base path. - Mixing the runtimes up: `modelFormat: huggingface` selects the Hugging Face runtime, which may run vLLM internally, and that is a different object from the standalone vLLM runtime added in 0.20. - Putting a Hub token in `args` or the manifest instead of a Secret consumed by the storage container. ## A worked deployment For the blurb writer, a predictor with `modelFormat: huggingface`, `storageUri: hf://example-org/blurb-7b-instruct`, one GPU in `resources.limits`, and `args` holding `--model_name=blurbs` is enough. The storage initializer pulls the weights, the runtime starts with `--backend auto`, and on the GPU image it loads the model on vLLM. Clients then call `/openai/v1/chat/completions` with `"model": "blurbs"`. Adding `--backend=vllm` changes nothing on a healthy image with a supported architecture. It only turns a missing engine into a failed start instead of a slow service. ## Checking what you got - The pod's container is `kserve-container`, running the runtime image with your `args`. - The `args` list on the predictor: only `--backend huggingface` fixes the engine in advance, since `auto` and `vllm` both decide at startup from the image and the model's architecture. - A request to `/openai/v1/models` lists the served model name, which is the `--model_name` you set (otherwise the InferenceService name).

  • A client using an OpenAI SDK gets 404s from a KServe Hugging Face runtime at /v1/chat/completions. What is wrong?
    The runtime mounts its OpenAI-compatible routes under a prefix, `/openai` by default (the `KSERVE_OPENAI_ROUTE_PREFIX` environment variable). The client's base URL must end in `/openai/v1`, so the call goes to `/openai/v1/chat/completions` on the InferenceService host.
  • Why might a team pin --backend huggingface on the KServe Hugging Face runtime?
    Either because the task is predictive, such as `sequence_classification` or `fill_mask`, which KServe runs on the transformers backend, or because the vLLM backend errors for a particular model and the documented workaround is to force transformers. They trade generation throughput for compatibility.

  • The Hugging Face runtime always runs transformers' generate loop
  • --backend vllm silently falls back when vLLM is missing
  • KServe's OpenAI routes sit at /v1 with no prefix
  • modelFormat huggingface binds the standalone vLLM runtime
  • OpenAI-compatible routes need the standalone vLLM runtime, not this one

Assumes KServe 0.21, whose Hugging Face runtime defaults to --backend auto (vLLM first) and which also ships the standalone vLLM runtime added in 0.20.