What does a minimal KServe InferenceService need to serve an XGBoost model stored in S3, and how do clients call it?
answer
- one required component
- framework name, not an image
- where the files live
- init container fills /mnt/models
- V1 :predict verb in the path
basics
~10 sA KServe InferenceService needs a predictor whose model names the format (xgboost) and a storageUri such as s3://bucket/path. KServe picks the serving runtime, downloads the files at pod start, and serves POST /v1/models/<name>:predict.
solid answer
~30 sThe only required component is `spec.predictor`. Inside `predictor.model` you set `modelFormat.name: xgboost` and `storageUri: s3://...`. For a private bucket you also set `serviceAccountName` to an account carrying the S3 secret. KServe matches the format to a serving runtime (the shipped `kserve-xgbserver`), and a **storage initializer** init container downloads the files into `/mnt/models` before the server starts. With the default V1 protocol, clients `POST /v1/models/<isvc-name>:predict` with `{"instances": [[...]]}` and get `{"predictions": [...]}`. A transformer (pre/post-processing) and an explainer (`:explain`) are optional extra components.
code
yaml · 16 linesapiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
name: skip-model
namespace: music-ml
spec:
predictor:
serviceAccountName: skip-model-s3
model:
modelFormat:
name: xgboost
storageUri: s3://example-skip-models/v7
resources:
requests:
cpu: "1"
memory: 2Gigo deeper
Recall the two fields that matter, modelFormat and storageUri under predictor.model, and the V1 predict path with the InferenceService name in it.
Explain the sequence: webhook defaults, runtime matched by format, storage initializer downloading into /mnt/models, server started with the model name, status URL.
Show you know the files are fetched once at pod start, how credentials reach the downloader, and when an explicit runtime pin is worth adding.
Frame the InferenceService as the contract between model owners and the platform: teams hand over a format and a URI, and the platform owns runtimes and rollout.
## What an InferenceService is A **KServe InferenceService** is a Kubernetes custom resource (`apiVersion: serving.kserve.io/v1beta1`, `kind: InferenceService`) that describes a model you want served over HTTP or gRPC. You declare *what* to serve and *where the files live*; the KServe controller creates the pods, the Service and the routing that make it reachable. You do not write a Deployment, you do not build an image around the model file, and you do not pick a port. The only required component is the **predictor**. Two optional components can sit beside it: | Component | Field | What it does | |---|---|---| | Predictor | `spec.predictor` | Loads the model and answers predict calls; required | | Transformer | `spec.transformer` | Runs `preprocess` before and `postprocess` after the predictor call, e.g. turning raw event fields into the feature row | | Explainer | `spec.explainer` | Serves the `:explain` verb next to `:predict` | KServe fills in the predictor's address for a transformer (the `predictor_host` argument), so the transformer's code only has to override its handlers. ## The minimal manifest For a track-skip classifier whose XGBoost booster sits in S3, the predictor needs two things inside `predictor.model`: - **`modelFormat.name`**: the framework, here `xgboost`. KServe uses it to choose a **serving runtime**: a `ServingRuntime` or `ClusterServingRuntime` whose `supportedModelFormats` lists that format with `autoSelect: true`. On a stock install that is the shipped `kserve-xgbserver` ClusterServingRuntime. - **`storageUri`**: where the model files live. The controller knows prefixes such as `s3://`, `gs://`, `pvc://`, `hf://` and `oci://`. Private buckets add one more line: `serviceAccountName` on the predictor, pointing at a service account that carries the S3 secret (with annotations such as `serving.kserve.io/s3-endpoint`). The credentials go to the download step, not into your manifest. ## What happens after kubectl apply 1. KServe's mutating webhook applies defaults. An older manifest written with the framework shorthand (`predictor.xgboost:`) is rewritten into `predictor.model` with `modelFormat: {name: xgboost}`. 2. The controller selects a runtime for `xgboost` and the requested protocol (V1 when `protocolVersion` is unset). 3. The **storage initializer**, an init container, downloads the `storageUri` contents into `/mnt/models` before the server starts. A `pvc://` URI is mounted at `/mnt/models` instead of being copied. 4. The runtime's container starts with `--model_dir=/mnt/models` and `--model_name` set to the InferenceService name, and loads the booster. 5. The InferenceService status gets a URL and `READY` turns `True`. The files are fetched **once, when the pod starts**. A new booster at the same path is not picked up by a running pod; you roll out a new revision by changing `storageUri`. ## Calling it With the default V1 protocol, the routes are: - `POST /v1/models/skip-model:predict` with `{"instances": [[...], [...]]}` (V1 also accepts `inputs` as the root key), answered with `{"predictions": [...]}`. - `GET /v1/models/skip-model`: returns 200 when the model is ready to serve. - `POST /v1/models/skip-model:explain`: answered only when an explainer is defined. The model name in the path is the InferenceService name, because the runtime is started with `--model_name={{.Name}}`. A request to the V2 path `/v2/models/skip-model/infer` works only when the predictor sets `protocolVersion: v2`. ## A worked request A V1 call to the skip model sends one row per listened track and gets one prediction per row back, in the same order: ```json {"instances": [[0.42, 187, 3, 1], [0.05, 31, 0, 0]]} ``` The runtime answers `{"predictions": [...]}` with one entry per instance. What each entry holds, a class or a probability, depends on how the booster was trained and on the runtime, so check one live response instead of guessing. Before sending traffic, poll `GET /v1/models/skip-model` until it returns 200. A pod can be running while the booster is still loading, and the readiness route is what tells the two apart. When the service sits behind a cluster ingress, the request goes to the URL in the InferenceService status. That host already identifies the service, so the path still has to carry the model name: the server inside the pod can host more than one model, and it routes on that name. ## Mistakes that show in interviews - Building a custom image with the booster copied in. That throws away the storage initializer and makes every model change an image build. - Expecting the server to re-read S3 on each request, or to notice a file overwritten in place. - Pinning `runtime:` everywhere "to be safe". It is useful when two runtimes compete, but the format alone is enough for the common case. - Thinking KServe needs a full Kubeflow install. It is its own CNCF incubating project and installs on its own; Kubeflow lists it as an add-on.
- An old KServe manifest uses predictor.xgboost with a storageUri instead of predictor.model. Does it still work?Yes. KServe's mutating webhook rewrites the framework shorthand into `predictor.model` with `modelFormat: {name: xgboost}` and the same storage and resource settings, then removes the `xgboost` block. Runtime selection then proceeds as for any `modelFormat`. New manifests should use `predictor.model` directly, because that is the form the runtime mechanism is built around.
- The skip model's bucket is private. Where do the S3 credentials go?Into a Kubernetes Secret holding the access key pair, annotated with settings such as `serving.kserve.io/s3-endpoint` and `serving.kserve.io/s3-region`, attached to a ServiceAccount. The predictor names that account in `serviceAccountName`. KServe reads the annotations and injects the variables into the storage initializer that does the download, so no credential appears in the InferenceService.
- The booster must be baked into a custom container image to be served
- The model server streams the model from S3 on every predict request
- Every InferenceService must name a runtime image explicitly
- KServe only runs inside a full Kubeflow installation
- Overwriting the S3 object makes running pods load the new model