How does TF Serving pick which SavedModel version to serve, and how do you roll one out?

## The directory contract TF Serving does not track models in a database. The model base path is a directory whose **children are integers**, and each integer directory is a complete SavedModel: ``` /models/my_model/ 1/saved_model.pb variables/ 2/saved_model.pb variables/ ``` Non-numeric children are ignored. Pointing `--model_base_path` at the SavedModel directory itself, rather than its parent, is the classic first-run error: the server starts, finds no numeric children, and reports no servable versions. ## Polling and the swap The server periodically re-lists the base path — `--file_system_poll_wait_seconds` controls the interval, and setting it to 0 disables polling so the set of versions is fixed at startup. When a new numeric directory appears and the policy selects it, the server: 1. loads the new version (reads the graph, restores variables, allocates buffers) while the old one keeps serving, 2. flips the servable handle so new requests route to it, 3. unloads the previous version once it is no longer selected and its in-flight requests drain. The consequence people forget is step 1 and 2 overlapping: **two versions are in memory at once**. On a large model that is the difference between a comfortable node and an OOM kill during every deploy. The other consequence is that the new version's first requests are cold — buffers not warm, no cached shapes — which shows up as a p99 spike right after each rollout. TF Serving addresses this with model warmup: a file named `tf_serving_warmup_requests` under the version's `assets.extra/` directory, holding serialized `PredictionLog` records that the server replays at load time before marking the version available. ## Publishing atomically An export writes many files. If the poller lists the directory mid-copy, it may try to load an incomplete SavedModel. The standard discipline is to write the version somewhere else on the **same filesystem** and rename it into the base path — rename is atomic, so the directory either is not there or is complete. On object storage, prefer writing to a temporary prefix and then a single copy/rename that the server observes as one step. ## The version policy Without a config file, the policy is "serve the latest one version". A config file makes the choice explicit: - `latest { num_versions: 2 }` — keep the two highest versions loaded. This is how you keep the previous version hot for instant rollback, at the cost of holding both in memory permanently. - `specific { versions: 42 }` — pin exactly. Nothing new is picked up automatically; deploys become deliberate config changes rather than file drops. - `all {}` — serve every version present. Rarely wise for large models. The file is re-read on an interval set by `--model_config_file_poll_wait_seconds`, so a policy change can be applied without a restart. The same config file is how one server hosts several models, each with its own base path and policy. ## Labels and canaries Hard-coding version numbers into clients is brittle. `version_labels` maps a name to a number: ``` version_labels { key: "stable" value: 1 } version_labels { key: "canary" value: 2 } ``` Clients then call `/v1/models/my_model/labels/canary:predict`, or set the label on the gRPC `model_spec`. Promoting a canary is repointing the `stable` label at the new number — clients change nothing. By default a label may only point at a version that is already loaded, which prevents pointing traffic at something that does not exist; `--allow_version_labels_for_unavailable_models` relaxes that for the assign-then-load ordering. Note what labels do and do not do: they let a caller *address* a version. They do not split traffic by percentage — that shaping belongs to whatever routes requests in front of the server, which sends some share to the canary label and the rest to stable. ## Rollback Because selection is a function of the directory contents and the config, rollback has two shapes. With `latest { num_versions: N }`, deleting or moving the bad version out of the base path makes the server fall back to the next highest — fast, but destructive to the artifact. With pinned versions or labels, rollback is repointing `stable` (or the `specific` list) at the old number and letting the config poll pick it up — non-destructive and auditable. Teams running real traffic generally choose the second, and keep the previous version loaded so the switch costs no load time. ## What to watch Memory during overlap, load duration for the new version, p99 immediately after a swap (the warmup signal), and the model-status endpoint reporting the expected version. A deploy is not done when the file lands; it is done when `GET /v1/models/my_model` shows the intended version `AVAILABLE` and a real predict against it succeeds.