Why does an eval that scores blank answers like wrong answers push a model to guess?

## The arithmetic that creates the behaviour Take the standard scoring rule: one point for a correct answer, zero for a wrong one, zero for an abstention. Now put the model in front of a question it half-knows, with subjective probability p of getting it right. Answering has expected value p. Abstaining has expected value 0. For any p above zero — including a wild guess — answering wins. There is no confidence so low that silence is the better play. That is the whole mechanism. It requires no bad intent, no faulty training, no exotic failure mode. Any process that selects for score under that rule selects for a model that always answers. ## Where the pressure actually enters This matters because the scoring rule is not confined to a report at the end. It is the selection signal at several stages: which post-training checkpoint gets shipped, which reward the model is optimized against on tasks with a checkable answer, which prompt variant wins your A/B, which fine-tune you keep. Each of those is a filter, and the filter has the same shape. A candidate model that correctly says "the handbook I was given does not cover that" loses to a candidate that invents a plausible clause and happens to be right some of the time. The concrete case is worth holding onto. An internal HR policy assistant is asked about a parental-leave rule that genuinely is not in its document snapshot. The correct behaviour — "I don't have that policy" — scores zero on an accuracy-only eval, identical to fabricating a leave entitlement. Run that eval as your selection criterion and you will systematically ship the bot that invents entitlements, then be surprised by the incident. ## Repricing the rubric The fix is to make the scoring reflect what you actually want, which is almost never raw accuracy. **Put a price on being wrong, and state it in the prompt.** A rule such as plus one for correct, minus three for wrong, zero for abstaining implies a threshold: answering is worth it only above 75% confidence. Crucially, the threshold must be stated to the model as part of the task, not just applied silently at grading time — otherwise you are asking it to guess your risk appetite. This is behavioural calibration: you are not asking for a probability, you are asking for a decision under a stated payoff, which is far easier to elicit and to verify. **Report two numbers instead of one.** Coverage — the share of questions answered — and accuracy on answered items. A single accuracy figure collapses two very different systems: one answering 60% of questions at 97%, another answering 100% at 70%. For a policy bot or a clinical summarizer the first is obviously better, and the collapsed metric cannot see it. Sweeping the abstention threshold and plotting accuracy against coverage gives you a risk-coverage curve, which is the honest way to compare and the right artefact to take to a product decision about where the threshold should sit. **Keep unanswerable items in the set.** An eval built only from questions your corpus covers cannot measure abstention at all. Deliberately include questions outside the snapshot, and score the refusal as the correct answer. This is the single cheapest change most teams have not made. ## The honest limits Repricing your own eval does not fix the models you buy. Public leaderboards still overwhelmingly report plain accuracy as of mid-2026, so the pretrained candidates you choose among have already been shaped by that pressure, and no rubric of yours undoes it. What you can do is select among them on your rubric, and make abstention safe in your own product surface. There is also a real cost to over-correcting. A penalty set too high produces a model that refuses constantly, which users experience as useless and which quietly pushes them to a less careful tool. Over-abstention is a failure mode with its own incident report; it is just a quieter one. The threshold is a product decision — how bad is a wrong parental-leave answer versus a shrug — and should be argued in those terms, with the risk-coverage curve on the table, rather than picked by whoever is writing the eval that week. Finally, an abstention is only as good as what surrounds it. "I don't have that policy" is a correct answer and a poor experience if it dead-ends; pair it with the escalation path, the document it searched, or the person to ask. ## What an interviewer is listening for They want the expected-value argument stated plainly, the recognition that the eval is a selection pressure rather than a passive measurement, and a concrete rubric change with numbers in it. Mentioning coverage and accuracy-on-answered as a pair, and naming over-abstention as the opposing failure, is what separates the senior answer from a recital of "the model is rewarded for guessing".