skip to content

A KServe InferenceService for an XGBoost model adds protocolVersion: v2 and names no runtime: which serving runtime does KServe bind, and why?

level: middleimportance: should knowfreq 12%

answer

  1. candidates filtered, then sorted
  2. supportedModelFormats with autoSelect
  3. protocolVersions must include v2
  4. higher priority wins the tie
  5. runtime: overrides the choice

basics

~10 s

On a stock install it binds kserve-mlserver, not kserve-xgbserver: both auto-select xgboost and speak v2, and kserve-mlserver's priority 2 beats 1. Set runtime: kserve-xgbserver to keep the XGBoost server.

solid answer

~30 s

KServe keeps only runtimes whose `supportedModelFormats` list the model's format with `autoSelect: true` and whose `protocolVersions` include the requested protocol, then sorts by `priority`, highest first, with namespace `ServingRuntime`s ahead of `ClusterServingRuntime`s. With the default V1, only `kserve-xgbserver` qualifies. With `protocolVersion: v2`, `kserve-mlserver` (xgboost, priority 2, v2 only) outranks `kserve-xgbserver` (priority 1), so the pod quietly runs a different server. Naming `runtime: kserve-xgbserver` overrides priority. Clients also move from `/v1/models/<name>:predict` with `instances` to `/v2/models/<name>/infer` with named, typed `inputs`.

code

yaml · 12 lines
yaml
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: skip-model
spec:
  predictor:
    model:
      modelFormat:
        name: xgboost
      protocolVersion: v2
      runtime: kserve-xgbserver
      storageUri: s3://example-skip-models/v7

go deeper

for a junior

Remember that the runtime is chosen from the model format, and that runtime: pins one by name when you need to.

for a middle

Walk through the filter (format, autoSelect, protocol) and the sort (priority, namespace before cluster), and show why v2 flips the XGBoost case.

for a senior

Pin runtimes on production services whose format several runtimes serve, and check the bound runtime after any protocol or runtime-catalogue change.

for a principal

Treat the runtime catalogue as a platform contract: decide who may add auto-selectable runtimes and how priorities are assigned so teams are not silently moved.

## How KServe binds a model to a runtime A KServe InferenceService never names a server image directly in the usual case. It names a **model format** (`predictor.model.modelFormat.name`, optionally `.version`) and, optionally, a **protocol** (`protocolVersion`: `v1`, `v2` or `grpc-v2`; unset means `v1`). The controller then picks a **serving runtime**: a namespaced `ServingRuntime` or a cluster-wide `ClusterServingRuntime`, which is a pod template plus a declaration of what it can serve. The declaration lives in `supportedModelFormats`, one entry per format, each with: - `name` and optional `version` (`"2"`, `"1.0"`): the format it serves. - `autoSelect`: whether the runtime may be chosen **when the InferenceService names no runtime**. - `priority`: a positive integer that breaks ties between auto-selectable runtimes; **higher wins**, and it is ignored unless `autoSelect` is true. The runtime also lists `protocolVersions` it speaks. ## The selection steps 1. List namespace `ServingRuntime`s and `ClusterServingRuntime`s, skipping disabled ones. 2. Keep a runtime only if one of its formats matches the model's format (and version, if the model gives one) **and** `autoSelect` is true for it, unless `runtime:` was set, in which case `autoSelect` does not matter. 3. Keep it only if its `protocolVersions` include the model's protocol. 4. Sort each group by `priority`, highest first; a runtime with a priority beats one without. Namespace runtimes come **before** cluster runtimes in the final list. 5. Use the first one. ## The XGBoost case on a stock install KServe ships three ClusterServingRuntimes that list `xgboost`, and two of them may be auto-selected for it: | Runtime | xgboost versions | autoSelect | priority | protocolVersions | |---|---|---|---|---| | `kserve-xgbserver` | `"2"` | true | 1 | `v1`, `v2` | | `kserve-mlserver` | `"1"`, `"2"` | true | 2 | `v2` | | `kserve-predictiveserver` | `"2"` | false | 3 | `v1`, `v2`, but used for xgboost only when named in `runtime:` | The third row shows why `autoSelect` comes first: its priority 3 is the highest, but it never takes part in auto-selection. So the outcome depends on the protocol: - **No `protocolVersion` (V1):** only `kserve-xgbserver` speaks V1, so it wins. - **`protocolVersion: v2`:** both qualify, and `kserve-mlserver` wins on priority 2 over 1. The pod now runs a different server image with different arguments, even though the manifest only added one line. - **`modelFormat.version: "1"`:** `kserve-xgbserver` lists only version `"2"`, so it is not eligible at any protocol; `kserve-mlserver` is, provided the protocol is `v2`. KServe's own XGBoost example sets `protocolVersion: v2` *and* `runtime: kserve-xgbserver` for exactly this reason: naming the runtime overrides priority. If nothing qualifies, no runtime can be bound and the InferenceService does not become ready; the fix is in the format, the version, the protocol or the runtime list, not in the model files. ## What the protocol changes for clients | | V1 | Open Inference Protocol (V2) | |---|---|---| | Predict route | `POST /v1/models/<name>:predict` | `POST /v2/models/<name>/infer` | | Request body | `{"instances": [...]}` or `{"inputs": [...]}` | `{"inputs": [{"name", "shape", "datatype", "data"}]}` | | Response | `{"predictions": [...]}` | `{"model_name", "outputs": [...]}` | | Readiness | `GET /v1/models/<name>` | `GET /v2/models/<name>/ready`, `GET /v2/health/ready` | | gRPC | no | yes (`grpc-v2`) | V2 carries typed, shaped tensors and a gRPC variant, which is why teams move to it; the same protocol is spoken by several runtimes, so clients stop caring which server sits behind the URL. ## A worked diagnosis The skip model's owners add `protocolVersion: v2` so their batch client can send typed tensors. The next revision fails its smoke test even though the booster has not changed. The trail runs like this: 1. Compare the new pod's image with the old one. The server image is different, which means the runtime changed. 2. List the ClusterServingRuntimes that mention `xgboost` and read their `protocolVersions` and `priority`. Two qualify for v2, and the one with priority 2 won. 3. Decide which server you want. If the smoke test was written against the XGBoost server's behaviour, pin `runtime: kserve-xgbserver`. If the other runtime is acceptable, update the test. 4. Write the decision into the manifest, not into a runbook. The pin is what stops the next catalogue change from repeating the incident. ## Operating advice - Pin `runtime:` on production InferenceServices whose format several runtimes serve. An admin who installs a new runtime with a higher priority otherwise changes what your next revision runs. - A team can shadow a cluster runtime by creating a namespace `ServingRuntime` for the same format: namespace runtimes are tried first. - After a change, check the server image the predictor pod actually runs instead of assuming which runtime won.

  • A platform team adds its own namespace ServingRuntime for xgboost with autoSelect true and no priority. Which runtime does a new skip-model revision get?
    The namespace runtime, provided it speaks the requested protocol. KServe sorts namespace runtimes and cluster runtimes separately by priority, then puts every namespace runtime ahead of every cluster runtime, so the shipped ClusterServingRuntimes are reached only when no namespace runtime qualifies. An explicit `runtime:` on the InferenceService still overrides this.
  • What does a V2 request to the skip model look like compared with V1?
    V1 sends `{"instances": [[0.4, 12, 3]]}` to `/v1/models/skip-model:predict`. The Open Inference Protocol sends `{"inputs": [{"name": "input-0", "shape": [1, 3], "datatype": "FP32", "data": [0.4, 12, 3]}]}` to `/v2/models/skip-model/infer`, and the answer comes back as named `outputs` with their own shape and datatype.

  • modelFormat alone always maps to one fixed server image
  • Changing protocolVersion only changes the URL clients call
  • A lower priority number wins the runtime selection
  • priority is honoured even when autoSelect is false
  • Cluster runtimes are always preferred over namespace runtimes