Why is held-out loss a poor yardstick for whether a fine-tune helped?

## What held-out loss actually measures During supervised fine-tuning the objective is the negative log-likelihood of the reference completion given the prompt, usually with the loss masked to the completion tokens only. Held-out (validation) loss is that same quantity computed on examples the optimiser never updated on. It is a measure of *how surprised the model is by one specific reference wording*, averaged over tokens. That definition explains both its usefulness and its blindness. It is dense - every token contributes - so it is statistically stable on small validation sets and can be computed every few hundred steps for almost no cost. And it is exactly the quantity being optimised, which makes the gap between training and validation loss the cleanest early warning that the model has started fitting the training set rather than the task. ## What it is blind to **Paraphrase.** If the reference says "apply a protectant fungicide before the next rain event" and the model says "spray a protectant fungicide ahead of the next rainfall," the answers are equivalent and the loss is worse. Held-out loss systematically rewards matching the training set's phrasing rather than being right. **Correctness.** Loss has no notion of truth. A confidently-worded diagnosis that names the wrong disease can carry lower loss than a hedged answer that names the right one, because fluent domain-shaped text is what the model learned to make probable. **Utility.** Whether the answer is actionable, correctly formatted, appropriately cautious, or the right length are all invisible to a per-token likelihood. **One-reference-per-input.** Most open-ended tasks admit many good answers. Loss scores against exactly one, so its ceiling is "reproduce this annotator," not "do the task well." A practical consequence: held-out loss can move in the *opposite* direction from quality. A checkpoint trained a little longer often produces more stylistically training-like text - lower loss - while becoming more rigid and less accurate on inputs outside the training distribution. ## The three-yardstick ladder **1. Held-out loss - run health.** Cheap, frequent, per-step. Its job is to answer "is this training run behaving?" and to flag the step where validation loss turns up while training loss keeps falling. Never quote it as evidence that the product improved. **2. Task metric - the automated gate.** Something that scores the *answer*, not the tokens. The form depends on the task: exact match or F1 for extraction, a unit test or compiler for code, a schema validator for structured output, a programmatic grader for anything with a checkable property, or a rubric scored by a model for open-ended text. This is what runs on every candidate checkpoint in CI, because it is deterministic enough to compare across runs. Its blind spot is its own definition: whatever the rubric or grader does not check, the model is free to get worse at, and models tuned against a fixed grader will find its gaps. **3. Human or judge preference - the ship decision.** Blind pairwise comparison on a sample of held-out inputs, arms presented in randomised order. This is the yardstick closest to the user experience, and the only one that catches "technically correct but useless." It is also the most expensive and the noisiest: raters disagree, and both people and models tend to prefer longer, more confident answers. So it is used on samples at decision points, not on every checkpoint. ## How they are combined in practice The usual arrangement is a funnel. Loss curves run continuously and stop or flag a bad run. The task metric gates every checkpoint automatically and picks the candidate. Preference evaluation runs once on the candidate against the baselines, and decides whether it ships. Crucially, when the yardsticks disagree, the *later* one wins - a checkpoint with slightly worse held-out loss but better graded task accuracy and better human win rate is the better model, full stop. A useful discipline is to write down which yardstick decides *before* the run. Teams that decide afterwards tend to quote whichever number improved, which is how a fine-tune that made the product worse gets shipped on the strength of a loss curve. ## What to say when asked The strong answer names the mismatch explicitly: loss is a proxy for the training objective, not for the task, and the two only correlate in the region where the model is still learning the task's general shape. Once it starts learning the annotators' phrasing, the correlation inverts. That is precisely why every serious fine-tuning workflow carries a task metric and a preference protocol alongside the loss curve rather than instead of it.