Regression Suites & CI Gating

## What temperature 0 actually buys you Temperature scales the logits before sampling. At 0 the sampler degenerates to argmax: always take the highest-probability token. That removes *sampling* randomness, which is the largest and most obvious source of run-to-run variation, and it is the right default for an eval run. But it does not make the pipeline a pure function, and treating it as if it did is the most common mistake in eval engineering. ## Sources of variation that survive greedy decoding **Serving-stack nondeterminism.** Inference runs on accelerators in batches assembled from whatever requests arrive together. Floating-point addition is not associative, so reduction order changes the last bits of a logit; kernel selection and tensor parallel splits vary with batch shape; mixture-of-experts architectures route tokens to experts in a way that can depend on the composition of the batch. None of this matters until two candidate tokens are nearly tied — then a bit-level difference flips the argmax, and because generation is autoregressive, one flipped token sends the rest of the completion down a different path. This is why identical requests at temperature 0 occasionally return materially different text. **Seeds are best-effort.** Where a provider exposes a seed, it constrains the sampler, not the arithmetic underneath it. Vendors typically document it as best-effort and pair it with a fingerprint of the backend configuration precisely because they cannot promise reproducibility across serving changes. A seed is worth setting; it is not a determinism guarantee. **Model version drift — the expensive one.** If your configuration names a floating alias rather than a dated snapshot, the provider decides when you change models. The classic incident: a suite sits green for weeks, then overnight drops six points with no commit in the diff. Hours go into bisecting prompt changes that are not there before someone checks the response metadata and finds the alias now resolves to a new snapshot. Pinning a dated snapshot converts that from a mystery into a scheduled, reviewable upgrade: you bump the pin in a pull request, the eval runs, and you see the delta before it reaches production. **Everything around the model.** Retrieval indexes get re-embedded and re-ranked; tool endpoints return live data; any case whose expected answer depends on the current date rots; a judge model has all of the above problems plus its own version drift. A prompt containing a timestamp or a randomly ordered document list changes the input on every run. ## The pin set A disciplined suite records, and asserts on, a fixed set of pins for every run: - exact **dated model snapshot** (never a floating alias), and the backend fingerprint if the provider returns one; - **decoding parameters**: temperature, top-p, max output tokens, stop sequences, and any reasoning-effort setting, since effort level changes both output and cost; - **prompt template version or content hash**, so a whitespace edit is visible in the run record; - **fixtures**: retrieval index snapshot, recorded tool responses, frozen conversation histories, a frozen clock for anything date-dependent; - **judge model snapshot and rubric version**, treated with the same rigour as the system under test. Writing these into the run's output turns an unexplained score move into a diff: you can see which pin changed. A run whose pins differ from the baseline's pins is not comparable, and a good harness says so loudly rather than reporting a delta as if it meant something. ## Living with the residue After pinning, some variance remains — irreducibly, because it lives in someone else's serving infrastructure. The engineering response is not to chase bitwise reproducibility but to *measure* the residue and design around it. Run the unchanged suite several times and record the spread of the aggregate score; that spread is your noise band. Anything smaller than the band is not evidence of a regression, and the gate's decision rule has to reflect that — a margin sized above the band, and repeated runs with a majority rule rather than a single pass or fail. A useful mental frame: the eval is a measurement instrument with a known error bar, not a unit test. Pinning shrinks the error bar; it never removes it. Teams that understand this ship confident gates. Teams that assume temperature 0 means determinism spend their time re-running builds and eventually turn the gate off.