A provider offers both a floating alias for a chat model and a pinned snapshot identifier of the same model. Which of the two should a recurring safety-benchmark suite be pointed at, and what does each choice cost you?

### The idea underneath: separate the instrument from the subject A recurring safety suite has two jobs that pull against each other. One is to detect change in the thing being measured — the endpoint your users actually reach. The other is to prove the measuring apparatus still reads true, so that when the number moves you can say the *subject* moved. A single target cannot do both, and that, rather than any property of the provider's naming scheme, is why the answer is usually "both, sized differently". ### The floating alias — the risk measurement A floating alias is the product-level name production calls. Whatever the provider is currently routing that name to answers the request, including any serving-side filtering — an input or output classifier in front of the model — that the account has enabled. It is therefore the only run whose number describes deployed exposure, and the only run in which a silent model update or a newly enabled filter can show up at all. Its weakness is that it is a moving target: if the alias run moves and nothing else is held fixed, you cannot say whether the subject changed or your own instrument did. ### The pinned snapshot — the control A pinned snapshot identifier holds the weights fixed across quarters. Its rate should therefore be stable up to sampling noise. If it moves, you changed something: the harm judge's revision or threshold, the prompt template, temperature or `max_tokens`, the behaviour subset, the error-counting convention. That makes it a **control run**, an instrument check, not a statement about risk. It is also the target a published result should cite if the result is meant to be reproducible by anyone else. ### How to size the pair | | target | scope | what it answers | |---|---|---|---| | risk run | floating alias | full behaviour set | what is our current exposure | | control run | pinned snapshot | small fixed subset | is the instrument still reading true | A control does not need coverage; it needs sensitivity to harness change. A tenth of the behaviours, fixed and never edited, run in the same job with the same judge, is usually enough. The operating rule: **if the control moved, stop and fix the instrument before reading the alias result at all** — otherwise you will spend the quarter explaining a number the harness invented. ### What it costs Running both roughly doubles the query budget only if the control is full-size, which is why it should not be. Concretely, a 200-behaviour alias run at five attempts is about 1,000 target completions plus 1,000 judge calls; a 20-behaviour control adds about 100 of each, a ten percent surcharge, plus a few minutes of wall clock. Against that, the control buys you the ability to attribute movement at all, which otherwise costs an analyst days of retrospective archaeology. The larger recurring cost is human triage of alias-run hits, and the control does not add to it. ### Where each choice misleads - **Reporting the pinned run as production exposure.** The most common error. Users do not call the pin; whatever the alias resolves to today answers them, filters and all. A pinned number presented as risk is a statement about a build nobody in production is talking to. - **Reading alias movement as model change.** It bundles model change, filter change, judge drift and harness drift. Without a control you have no way to peel any of them off. - **Believing a pin freezes everything.** It freezes weights, not the serving stack. Provider-side filtering can be changed in front of a frozen model, so a moving control is not automatically your bug — check your own manifest diff first, then consider this. - **Splicing across a retired pin.** Providers retire snapshots on their own schedule, and the control will eventually break. Re-baseline deliberately and record the discontinuity in the tracker rather than drawing one continuous line through it. ### What you would check That the manifest names which target each number came from — "we tested the endpoint" is not reproducible. That the control subset has not been quietly edited (edit it and it stops being a control). That any served-revision identifier the provider returns is captured for both runs. And where no pinned option exists at all, that a locally held open-weights model, whose artefact you can hash, is standing in as the harness control: it says nothing about the hosted endpoint, but it keeps the instrument honest.