skip to content

KServe

KServe serves a model on Kubernetes as an InferenceService: a predictor on a serving runtime chosen by modelFormat, an optional transformer and canary splits. Interviewers ask how it rolls out.

on this pageshow

explore

questions

6

What does a minimal KServe InferenceService need to serve an XGBoost model stored in S3, and how do clients call it?

level: juniorimportance: must knowfreq 28%

answer

  1. one required component
  2. framework name, not an image
  3. where the files live
  4. init container fills /mnt/models
  5. V1 :predict verb in the path

basics

~10 s

A KServe InferenceService needs a predictor whose model names the format (xgboost) and a storageUri such as s3://bucket/path. KServe picks the serving runtime, downloads the files at pod start, and serves POST /v1/models/<name>:predict.

solid answer

~30 s

The only required component is `spec.predictor`. Inside `predictor.model` you set `modelFormat.name: xgboost` and `storageUri: s3://...`. For a private bucket you also set `serviceAccountName` to an account carrying the S3 secret. KServe matches the format to a serving runtime (the shipped `kserve-xgbserver`), and a **storage initializer** init container downloads the files into `/mnt/models` before the server starts. With the default V1 protocol, clients `POST /v1/models/<isvc-name>:predict` with `{"instances": [[...]]}` and get `{"predictions": [...]}`. A transformer (pre/post-processing) and an explainer (`:explain`) are optional extra components.

code

yaml · 16 lines
yaml
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: skip-model
  namespace: music-ml
spec:
  predictor:
    serviceAccountName: skip-model-s3
    model:
      modelFormat:
        name: xgboost
      storageUri: s3://example-skip-models/v7
      resources:
        requests:
          cpu: "1"
          memory: 2Gi

go deeper

for a junior

Recall the two fields that matter, modelFormat and storageUri under predictor.model, and the V1 predict path with the InferenceService name in it.

for a middle

Explain the sequence: webhook defaults, runtime matched by format, storage initializer downloading into /mnt/models, server started with the model name, status URL.

for a senior

Show you know the files are fetched once at pod start, how credentials reach the downloader, and when an explicit runtime pin is worth adding.

for a principal

Frame the InferenceService as the contract between model owners and the platform: teams hand over a format and a URI, and the platform owns runtimes and rollout.

## What an InferenceService is A **KServe InferenceService** is a Kubernetes custom resource (`apiVersion: serving.kserve.io/v1beta1`, `kind: InferenceService`) that describes a model you want served over HTTP or gRPC. You declare *what* to serve and *where the files live*; the KServe controller creates the pods, the Service and the routing that make it reachable. You do not write a Deployment, you do not build an image around the model file, and you do not pick a port. The only required component is the **predictor**. Two optional components can sit beside it: | Component | Field | What it does | |---|---|---| | Predictor | `spec.predictor` | Loads the model and answers predict calls; required | | Transformer | `spec.transformer` | Runs `preprocess` before and `postprocess` after the predictor call, e.g. turning raw event fields into the feature row | | Explainer | `spec.explainer` | Serves the `:explain` verb next to `:predict` | KServe fills in the predictor's address for a transformer (the `predictor_host` argument), so the transformer's code only has to override its handlers. ## The minimal manifest For a track-skip classifier whose XGBoost booster sits in S3, the predictor needs two things inside `predictor.model`: - **`modelFormat.name`**: the framework, here `xgboost`. KServe uses it to choose a **serving runtime**: a `ServingRuntime` or `ClusterServingRuntime` whose `supportedModelFormats` lists that format with `autoSelect: true`. On a stock install that is the shipped `kserve-xgbserver` ClusterServingRuntime. - **`storageUri`**: where the model files live. The controller knows prefixes such as `s3://`, `gs://`, `pvc://`, `hf://` and `oci://`. Private buckets add one more line: `serviceAccountName` on the predictor, pointing at a service account that carries the S3 secret (with annotations such as `serving.kserve.io/s3-endpoint`). The credentials go to the download step, not into your manifest. ## What happens after kubectl apply 1. KServe's mutating webhook applies defaults. An older manifest written with the framework shorthand (`predictor.xgboost:`) is rewritten into `predictor.model` with `modelFormat: {name: xgboost}`. 2. The controller selects a runtime for `xgboost` and the requested protocol (V1 when `protocolVersion` is unset). 3. The **storage initializer**, an init container, downloads the `storageUri` contents into `/mnt/models` before the server starts. A `pvc://` URI is mounted at `/mnt/models` instead of being copied. 4. The runtime's container starts with `--model_dir=/mnt/models` and `--model_name` set to the InferenceService name, and loads the booster. 5. The InferenceService status gets a URL and `READY` turns `True`. The files are fetched **once, when the pod starts**. A new booster at the same path is not picked up by a running pod; you roll out a new revision by changing `storageUri`. ## Calling it With the default V1 protocol, the routes are: - `POST /v1/models/skip-model:predict` with `{"instances": [[...], [...]]}` (V1 also accepts `inputs` as the root key), answered with `{"predictions": [...]}`. - `GET /v1/models/skip-model`: returns 200 when the model is ready to serve. - `POST /v1/models/skip-model:explain`: answered only when an explainer is defined. The model name in the path is the InferenceService name, because the runtime is started with `--model_name={{.Name}}`. A request to the V2 path `/v2/models/skip-model/infer` works only when the predictor sets `protocolVersion: v2`. ## A worked request A V1 call to the skip model sends one row per listened track and gets one prediction per row back, in the same order: ```json {"instances": [[0.42, 187, 3, 1], [0.05, 31, 0, 0]]} ``` The runtime answers `{"predictions": [...]}` with one entry per instance. What each entry holds, a class or a probability, depends on how the booster was trained and on the runtime, so check one live response instead of guessing. Before sending traffic, poll `GET /v1/models/skip-model` until it returns 200. A pod can be running while the booster is still loading, and the readiness route is what tells the two apart. When the service sits behind a cluster ingress, the request goes to the URL in the InferenceService status. That host already identifies the service, so the path still has to carry the model name: the server inside the pod can host more than one model, and it routes on that name. ## Mistakes that show in interviews - Building a custom image with the booster copied in. That throws away the storage initializer and makes every model change an image build. - Expecting the server to re-read S3 on each request, or to notice a file overwritten in place. - Pinning `runtime:` everywhere "to be safe". It is useful when two runtimes compete, but the format alone is enough for the common case. - Thinking KServe needs a full Kubeflow install. It is its own CNCF incubating project and installs on its own; Kubeflow lists it as an add-on.

  • An old KServe manifest uses predictor.xgboost with a storageUri instead of predictor.model. Does it still work?
    Yes. KServe's mutating webhook rewrites the framework shorthand into `predictor.model` with `modelFormat: {name: xgboost}` and the same storage and resource settings, then removes the `xgboost` block. Runtime selection then proceeds as for any `modelFormat`. New manifests should use `predictor.model` directly, because that is the form the runtime mechanism is built around.
  • The skip model's bucket is private. Where do the S3 credentials go?
    Into a Kubernetes Secret holding the access key pair, annotated with settings such as `serving.kserve.io/s3-endpoint` and `serving.kserve.io/s3-region`, attached to a ServiceAccount. The predictor names that account in `serviceAccountName`. KServe reads the annotations and injects the variables into the storage initializer that does the download, so no credential appears in the InferenceService.

  • The booster must be baked into a custom container image to be served
  • The model server streams the model from S3 on every predict request
  • Every InferenceService must name a runtime image explicitly
  • KServe only runs inside a full Kubeflow installation
  • Overwriting the S3 object makes running pods load the new model
open as a page

On a stock KServe 0.21 install, which deployment mode does an InferenceService without a deploymentMode annotation get, and what does each mode bring?

level: middleimportance: should knowfreq 10%

basics

~20 s

Knative mode. The shipped inferenceservice-config ConfigMap sets defaultDeploymentMode to Serverless, which KServe 0.21 reads as Knative: revisions, scale-to-zero and canaryTrafficPercent. Standard mode, chosen per service by annotation, runs a plain Deployment with HPA or KEDA.

open as a page

A KServe InferenceService for an XGBoost model adds protocolVersion: v2 and names no runtime: which serving runtime does KServe bind, and why?

level: middleimportance: should knowfreq 12%

basics

~10 s

On a stock install it binds kserve-mlserver, not kserve-xgbserver: both auto-select xgboost and speak v2, and kserve-mlserver's priority 2 beats 1. Set runtime: kserve-xgbserver to keep the XGBoost server.

open as a page

Rolling a new skip model out gradually on KServe 0.21, how do canaryTrafficPercent in Knative mode and spec.canary in Standard mode each roll forward and back?

level: seniorimportance: should knowfreq 8%

basics

~20 s

In Knative mode, canaryTrafficPercent sends that share to the new revision; removing it promotes, and 0 rolls back to the last rolled-out revision. In Standard mode (0.20+), spec.canary entries get trafficPercent and fixed replicas; renaming promotes, deleting the entry rolls back.

open as a page

A KServe InferenceService serving a 7B model on GPU nodes keeps one pod running all night with no traffic: which KServe settings stop it scaling to zero?

level: seniorimportance: should knowfreq 9%

basics

~20 s

In KServe 0.21, minReplicas defaults to 1 on every component, and only Knative mode can scale to zero. A service in Standard mode runs under an HPA, KEDA or no autoscaler and keeps its pod; zero needs Knative mode plus minReplicas: 0 everywhere.

open as a page

In KServe 0.21, which engine actually serves an InferenceService with modelFormat huggingface, and what does the runtime's --backend argument change?

level: middleimportance: nice to knowfreq 6%

basics

~20 s

The shipped kserve-huggingfaceserver runtime runs it with --backend auto by default: vLLM when vLLM is installed and supports the model's architecture, else the Hugging Face transformers backend. --backend huggingface forces transformers; --backend vllm makes a missing vLLM a startup error.

open as a page