What baselines must a fine-tuned model beat before you ship it?

## Why baselines are the whole question A fine-tune is never evaluated in isolation. The number that matters is not "the fine-tune scores 78%" but "the fine-tune scores 78% where the alternative you could have shipped in an afternoon scores 74%." Fine-tuning costs a training run, a dataset somebody must keep curated, a model artifact you must version and serve, and a standing commitment to retrain whenever the base model or the task drifts. That bill is only justified by a margin over the cheapest thing that could have worked, and the cheapest thing is a well-written prompt. ## The arms of the comparison A defensible evaluation has at least three arms, and often four. 1. **Base model, strong system prompt.** The instructions a competent engineer would write after a day of iteration: role, output format, the domain rules the fine-tune was supposed to teach. 2. **Base model, few-shot.** The same prompt plus a small number of high-quality demonstrations drawn from the training pool - never from the held-out set. 3. **The fine-tuned model**, served with the chat template and system prompt it was trained under. 4. **The incumbent**, if something is already in production. Beating a prompt is academic if you cannot beat the system you are proposing to replace. A retrieval-augmented arm belongs in the list whenever the task needs facts rather than form, because retrieval and fine-tuning solve different problems and are frequently combined rather than chosen between. ## Holding everything else constant The most common way a fine-tune "wins" is that the comparison was not controlled. Every arm must share: - **The same held-out inputs.** One eval set, drawn once, used by all arms. - **The same decoding settings.** Temperature, top-p, max tokens and stop sequences change output quality on their own. Comparing a fine-tune at temperature 0 with a prompted baseline at temperature 1 measures decoding, not training. - **The same parsing and scoring code.** If the fine-tune emits clean JSON and the baseline emits JSON inside prose, and your parser only handles the former, you are measuring your parser. - **The same effort spent on the prompt.** A straw-man baseline prompt is the easiest way to manufacture a win. Have someone who wants the prompt to succeed write it, ideally with the same time budget the fine-tune consumed. ## Reporting a result you can defend Averaged scalar scores hide the interesting structure. Two habits fix that. **Pair the comparisons.** Each held-out input is answered by every arm, and you compare arm-to-arm on the same input. Paired comparison removes item difficulty as a source of noise, so the same sample size buys a much tighter estimate. **Report an interval, not a point.** With 300 items and a small margin, the difference between 76% and 78% is frequently indistinguishable from noise. Bootstrap the paired differences and report the interval. If it straddles zero, the honest statement is "no measurable difference at this sample size," which is a real and useful finding. For open-ended output, blind pairwise preference is the strongest signal available: present two anonymised answers per input, randomise which side each arm appears on, and report a win rate. In the crop-disease example, agronomists judging 300 field reports pairwise gives a directly interpretable number - "the fine-tune was preferred on 61% of reports where the raters expressed a preference." ## Slicing before deciding An aggregate win can hide a regression that matters more than the average. Slice the eval set by the dimensions your users care about - report type, crop, severity, report length, unfamiliar pathogens - and look for slices where the fine-tune is worse. A model that gains four points overall while losing ten on rare-but-severe cases is not a shipping candidate; it is a retraining brief. ## Turning the numbers into a decision The last step is not statistical. A fine-tune that wins by a clear margin still has to justify its lifetime cost against a prompt that is free to change, needs no retraining when the base model is upgraded, and can be debugged by reading it. The cases where a fine-tune reliably clears that bar are consistent behaviour that no prompt reliably enforces, an output format the base model keeps drifting away from, latency and cost gains from moving work onto a smaller tuned model, and tone or style that is easier to demonstrate than to describe. When the prompt gets you most of the way, ship the prompt and revisit later - that decision is much easier to reverse than a training pipeline nobody owns.