In Self-RAG, what do the retrieval and critique tokens actually control?

## The idea in one sentence Self-RAG moves the decisions "should I retrieve?", "was that passage relevant?" and "is what I just wrote actually supported?" out of surrounding pipeline code and into the model's own output stream, as reserved tokens the model has been trained to emit. ## Why that is a real architectural change In a conventional pipeline those three decisions live in different places, or nowhere. Retrieval is unconditional: every question triggers a vector search whether or not it needs one. Relevance is implicit: whatever the retriever ranked highest is trusted. Grounding is unchecked: the generator writes prose and nobody verifies that the prose follows from the passages. Self-RAG makes each decision explicit and per-segment. The unit is not the whole answer but a span of it, so an answer can mix a segment written from parametric knowledge with a segment grounded in a retrieved passage, each labelled. ## The token families **Retrieve.** Emitted before generating a segment, it decides whether to fetch passages for that segment. Its values distinguish not retrieving, retrieving now, and continuing with evidence already in hand. This is what makes the architecture *adaptive*: cost scales with how much of the answer actually needs grounding, and a question the model can answer directly does not pay for a search. **Relevance.** Given a retrieved passage, does it bear on the question? Passages that fail are not allowed to drag the continuation around. This is the model grading its own retriever. **Support.** Given a generated segment and the passage it was written against, is the segment fully supported, partially supported, or unsupported by that passage? This is an entailment judgement, and it is the token that most directly attacks hallucination, because it asks about the relationship between two texts rather than about the truth of a claim in the abstract. **Utility.** Independent of grounding, is the segment a useful response to what was asked? A perfectly supported sentence can still be a non-answer. ## How the tokens are used at inference Self-RAG generates candidate continuations, typically one per retrieved passage, and scores each candidate segment using the probabilities the model assigns to its own critique tokens. A weighted sum decides which continuation survives, and the weights are set at inference time. Turning up the weight on support biases the system toward answers that stay close to the citations; turning it down lets it be more fluent and more willing to speak from parametric knowledge. That knob without retraining is one of the design's most quoted benefits, because it lets one deployed model serve a strict compliance surface and a looser exploratory surface. ## The cost nobody mentions in the summary The tokens are not magic annotations you can request in a prompt. In the published design they are learned: a stronger critic model labels training data, and those labels are distilled into the generator so that at inference there is no separate critic to call. That means adopting Self-RAG literally means owning a fine-tune, a labelled dataset and a retraining loop whenever the base model changes. As of mid-2026 the far more common deployment is the pattern rather than the paper: a separate, cheap grader model is prompted to answer the same questions — is retrieval needed, is this passage relevant, is this sentence supported — and pipeline code branches on the answers. You lose the elegance and the single-pass efficiency, you gain the ability to swap the base model on a Tuesday. Interviewers who know the paper often ask exactly this, so distinguish the mechanism from the pattern. ## Where it fits and where it fails A municipal permitting help desk is a good fit: many questions are procedural and answerable directly ("how long is the comment period?" from a summary the model already carries), while others turn on the exact text of an ordinance and must be grounded and cited. Per-segment retrieval decisions and support grades let one answer mix both honestly, and the support grade gives the operator a signal to surface: sentences graded partially supported can be shown with weaker language or withheld. The failure modes are equally concrete. Self-assessment is still assessment by the same model family, so a confidently wrong reading of a passage can be graded supported. Support is entailment against the retrieved passage, not truth: if the passage is out of date, a well-grounded answer is still wrong. And segment-level branching multiplies decoding cost, since several continuations are explored per retrieved passage. ## What to say in an interview Name the two token families and what each decides, say that adaptivity comes from the retrieve decision and hallucination reduction from the support grade, then volunteer the honest caveat: the published architecture requires a trained model, most production systems approximate it with an external grader, and either way the critic shares the generator's blind spots, so an independent verifier is stronger evidence than self-critique when one exists.