How do you write a RAG insufficient-context refusal rule that does not over-refuse?

## Abstention is a prompt-side contract Retrieval decides what evidence exists; the generation prompt decides what the model does when the evidence is not there. Left unspecified, a capable model will fill the gap from its parametric knowledge, and the result is an answer that reads exactly like a grounded one — same tone, same confidence, frequently no citation. That is the highest-severity failure in a grounded assistant, because the whole value proposition was "this comes from your documents". ## Say what to do, not only what to avoid "Do not hallucinate" is not an instruction; it names an outcome, not a behaviour. A usable rule specifies a condition, an output, and a shape. For a clinical-guideline assistant: if the provided guideline sections do not address the drug asked about, reply with the fixed sentence "Not covered by the provided guidance", state which sections were provided, and stop. The three parts matter — the condition is testable, the output is detectable, and telling the user what *was* retrieved converts a dead end into a next query. ## Define insufficiency operationally Models are poor at introspecting on whether they "know enough". They are much better at a textual test. Useful formulations: no passage states the requested fact; the passages concern a different entity, drug, plan year or jurisdiction than the question; the passages state the fact but disagree with each other. The last one deserves its own branch — the right behaviour for conflict is usually to surface both positions with their citations rather than silently pick one or refuse outright. ## Make the abstention machine-detectable Free-form apologies are unparseable. Either fix the wording exactly, or better, have the model emit a structured field alongside the answer indicating whether the question was answerable from the context. Detection is what lets you route: offer a broader search, escalate to a human, log the query as a corpus gap. It is also how you measure the abstention rate at all, which is one of the most informative operational metrics a RAG system has — a sudden spike almost always means retrieval broke, not that users started asking harder questions. ## Allow partial answers All-or-nothing rules are the main driver of user-perceived uselessness. If four of five sub-questions are covered, the useful behaviour is to answer those four with citations and explicitly flag the fifth as not covered. Write this into the rule; otherwise a strict instruction pushes the model to refuse the whole response over one uncovered detail. ## The recall cost, and why it is usually retrieval's fault Tightening the abstention rule always trades one error for another. Push hard and the model starts abstaining when the answer *is* in the corpus but is phrased differently from the question, split across two chunks, or stated implicitly in a table. From the user's seat that is indistinguishable from "your product doesn't know", and it is worse than a hedged answer because there is nothing to correct. Crucially, most of those cases are retrieval or chunking defects surfacing at the generation stage. The diagnostic is simple: for each false refusal, check whether the gold passage was actually in the injected set. If it was, the prompt or model is too strict. If it was not, stop tuning the instruction and go fix retrieval. ## Measure both sides You need two evaluation sets. An unanswerable set — questions whose answers are genuinely absent from the corpus, including near-miss distractors that retrieve plausible but non-answering passages — measures how often the system answers when it should not. An answerable set with known gold passages measures false refusals. Report both. Teams that track only hallucination rate will happily ship a system that refuses half the time and call it safe; teams that track only helpfulness ship the opposite. A single "accuracy" number hides the trade entirely. ## The asymmetry is a product decision Where the two errors sit is not an engineering choice. In a clinical setting an unsupported answer can cause harm and a refusal costs a clinician thirty seconds, so the rule leans hard toward abstention and the escalation path must be excellent. In an internal search assistant, over-refusal drives users straight back to grepping the wiki and the product dies of irrelevance. Name the asymmetry explicitly, set the target rates before tuning, and revisit them with real usage data rather than intuition. ## Common pitfalls Burying the rule mid-prompt where it competes with a dozen other instructions. Writing an abstention condition the model cannot test. Treating every refusal as correct behaviour and never sampling them. Letting refusals be silent — logged as a successful response — so the corpus gaps they reveal are never collected and fixed.