What is a step-back query in RAG, and when does it beat searching the literal question?

## The retrieval gap it closes Retrieval matches text to text. A question about one instance retrieves documents about that instance. But many real questions are instance-shaped while their answers are rule-shaped: the user names a project, a customer, an incident or a ticket, and what actually determines the answer is a policy, a definition, or a general principle written in a document that never mentions that name at all. When that is the case, the top-k fills with chunks that mention the named entity and are useless for reasoning — a project status page, a budget spreadsheet extract, a meeting note. The generator then answers from those, either hedging or inventing the rule it was never given. The retrieval scores look healthy; the answer is wrong. This is one of the most confusing RAG failure modes to diagnose because nothing in the pipeline reports an error. ## What step-back prompting does Before retrieving, you ask a model to *abstract*: given the user's question, what is the more general question whose answer would let you answer this one? The output is a second query at a higher level of generality. Both queries are then embedded and searched, and their results are merged into one context. The accounting example is the clean one. "Can we capitalize the R&D spend on Project Atlas in Q3?" steps back to "what are the general rules for capitalizing R&D spend?" The specific query is still worth running — you do need the Project Atlas facts. The abstract query is what supplies the criteria those facts must be evaluated against. Neither alone produces a defensible answer; together the generator has both a rule and a case. ## Why the abstraction has to be generated, not hardcoded You cannot precompute a step-back query per question because you do not know in advance which axis to abstract along. "Can we capitalize the R&D spend on Project Atlas in Q3?" could step back toward the accounting treatment, toward the fiscal-period rules, or toward what counts as R&D at all. A model reads the question and picks a plausible axis — usually the one carrying the most domain vocabulary. That is also where it fails: abstract too far and you get "what is accounting?", which retrieves nothing useful and dilutes the context; abstract too little and the query is a paraphrase of the original, retrieving what you already had. Giving the rewriter one or two worked examples in the prompt controls this better than any instruction about "the right level". Examples pin the intended granularity in a way adjectives do not. ## Where it fits among query transformations Step-back sits between paraphrasing and decomposition. Paraphrases hold the level of specificity constant and vary the wording. Decomposition holds the level constant and splits the information need into parts. Step-back deliberately *changes the level*, moving up the abstraction ladder, and keeps the original alongside it. That is why it is usually additive rather than a replacement: you issue two queries and merge, rather than substituting one for the other. It composes naturally with decomposition — a compound question may deserve a step-back query for the shared background plus a sub-question per part — but each composition multiplies calls, so reserve it for query classes where you have measured the gain. ## When not to use it Skip it when the specific document is the answer. "What is the incident number for last Tuesday's payments outage?" has no useful abstraction; the general query returns incident-management policy, which is noise crowding out the one chunk that mattered. The rough test is whether answering requires *applying* something to the named entity, or merely *finding* it. Application benefits from the step-back; lookup does not. Also be careful about how you allocate context. If you retrieve five chunks for each of the two queries and pass all ten, the general chunks compete with the specific ones for the generator's attention, and background prose is often longer and more confident-sounding than the sparse factual chunk that names the entity. Reserve slots explicitly — for instance the top few from each side rather than a single blended top-k — so the specific evidence cannot be squeezed out entirely. ## Signals that you need it In production, the tell is a class of questions where retrieval metrics look fine but answers are hedged or wrong, and where a human answering the same question would first go and read the policy. If your evaluation set contains questions whose gold documents never mention the entity in the question, that is direct evidence: the retriever cannot find those documents from the literal query, and a step-back query is one of the few pre-retrieval fixes that can.