Capabilities and Limits

## What the word means A hallucination is output that a language model presents as fact but that is either false about the world or unsupported by the material the model was given. The defining property is not that the model is sometimes wrong — every system is sometimes wrong — but that the wrong output arrives in exactly the same fluent, confident register as a correct one. There is no stylistic tell, no hedge, no error code. An invented court case is formatted like a real one, complete with a plausible reporter volume and page. An invented internal-policy clause reads like the rest of the handbook. That uniformity is what makes hallucination a product problem rather than a curiosity. A system that failed loudly would be easy to wrap in a check. A system that fails in the same voice it succeeds in pushes the entire burden of detection onto the reader, and readers do not audit fluent text. ## Why the model has no "I made that up" flag A base model is trained to predict the next token of text. It learns which continuations are likely given everything before them. Nothing in that objective builds a separate store of "facts I hold" that could be consulted and reported empty. Recalling a fact is not a lookup in a table; it is reconstruction from distributed parameters, and reconstruction degrades smoothly. When the evidence in the weights is strong, the reconstruction is right. When it is thin — a name seen once, a number that never appeared at all — the same machinery still produces a fluent, well-shaped answer, because producing fluent well-shaped answers is what it was trained to do. The model does not experience the second case as different from the first. This is why "just ask it if it is sure" is weak. A confidence statement is itself generated text. It is produced by the same process, conditioned on the same context, and correlates with correctness only loosely unless the model has been specifically trained and calibrated for that behaviour. ## Why the instruction helps a little, and only a little An instruction like "answer only if you are certain; otherwise say you don't know" is not useless. It shifts the output distribution toward hedging and abstention, and on questions the model half-knows it will refuse more often. Two limits bite quickly. First, it supplies no new information. The instruction cannot tell the model which claims are shaky, because the model's own uncertainty is not legible to it in a form the instruction can act on. So abstention becomes roughly untargeted: the model may refuse things it actually knows and still assert things it does not. Second, it competes with everything else the model was trained to be. Post-training rewards answers that are helpful, complete and confident; users and graders alike prefer a direct answer to a shrug. A single line in a system prompt is a weak counterweight to that pressure, which is why models trained on accuracy-only scoring keep guessing even when told not to. ## What actually reduces it Four levers do real work, roughly in order of payoff: **Ground the answer.** Put the relevant source text into the context and instruct the model to answer only from it. This converts an open recall problem, where the model must dredge a fact out of its weights, into a reading problem, where the fact is on screen. It does not eliminate error, but it changes the failure into one you can check mechanically. **Make abstention a legitimate output.** If the product has no path for "the documents I was given do not cover this", the model will fill the gap. Design the refusal, write it into the prompt, and — crucially — make sure your own evaluation gives credit for it rather than scoring it as a miss. **Hard-check the checkable.** Anything with an authoritative index — case citations, SKUs, ticket numbers, API names, drug identifiers — should be resolved against that index before the answer ships, and the answer failed closed if it does not resolve. This is ordinary software, not model behaviour, and it catches the most damaging class of fabrication. **Verify claims against sources.** A second pass that re-reads the cited passages and labels each claim supported, unsupported or not addressed catches drift that grounding alone lets through. ## Saying it in an interview Give the definition in one line, then immediately say why prompting is a weak control: the instruction changes willingness to answer, not knowledge of which answers are safe. Interviewers are listening for whether you reach for prompt wording as the fix — the junior answer — or for grounding, abstention and verification, which is the answer of someone who has shipped this.