Changing the Answer Instead

## Two attacks that share one vocabulary **Prompt injection** aims at the *application's* instructions. The attacker gets text in front of the model - directly in their own turn, or indirectly through content the application retrieves, fetches or is handed - and that text tries to change what the assistant is doing. **Jailbreaking** aims at something else again: the model's *trained* refusal. Both are attacks on behaviour, and both leave a characteristic trace, because a directive has to read like a directive to work. This leaf is about the third thing, and it is the one that gets confused with the first. The attacker does not tell the assistant to do anything. The attacker changes what the assistant will read as true, and then lets the ordinary pipeline do the rest. ## What the screen actually inspects A screen placed over retrieved text before it reaches the model - whether it is a small classifier, a set of patterns, or a generative screening model asked to judge a span - is scoring for the shape of an instruction. The signals available to it are things like: - imperative verbs addressed to an assistant rather than to a human reader - an assertion that earlier guidance is superseded, or that the task has changed - role or speaker markers embedded inside body text - structure that mimics a system or developer message - content that is off-topic for the passage it appears in A false declarative sentence has none of these. It is on-topic by construction - it has to be, or nobody would retrieve it for the question it is meant to answer - and its register is identical to every other passage in the corpus, because it was written to sit beside them. | Stage | What it can see | What it cannot see | |---|---|---| | Screen over retrieved text | Whether the span looks like a command | Whether the span is true | | The assistant | The chunks it was handed | Which of them anybody vouched for | | A comparison of answer to retrieved text | Whether the answer follows from the source | Whether the source follows from reality | That middle row is the whole point. Similarity ranking is a distance computation; it is not a judgment about authority, provenance or truth. Being retrieved is not being vouched for. ## Why the model is behaving correctly This matters for how the failure gets classified. When an assistant repeats a planted claim it has not disobeyed anything. It was asked to answer from the retrieved material, and it did. The answer is faithful to its source. The defect is upstream: a store that renders anyone's edit into an answering context treated that edit as material worth grounding on. So the failure surfaces as a wrong answer with no anomalous features anywhere in the trace - no blocked span, no refusal, no odd instruction sitting in the transcript. It looks exactly like a model getting something wrong on its own. ## What the attacker pays, and what they get The cost is write access to one field that gets rendered into prompts, plus knowing which question people ask routinely. There is no wording to tune, no screen to probe, no refusal to work around - which also means there is nothing about the construction that becomes obsolete when a model version changes or a screening layer is retrained. A directive-shaped span is in an arms race with whatever inspects text; a false sentence is not in that race at all, because the thing that would have to inspect it is a fact check, and no stage of the pipeline is one. The payoff in an ask-the-data setting is a false operational answer that a person carries into a document, a decision or a number somebody else quotes. And there is a second, quieter payoff: when the error is noticed, the natural diagnosis is that the model hallucinated. Triage that closes it as a model error leaves the artefact in place, so the same answer comes back tomorrow. ## Where it stops working It stops working where somebody actually reads the source and disagrees with it - which is why plausibility, not cleverness, is the binding constraint on this construction. It stops working where the corpus is small enough that a contradiction with a neighbouring passage is obvious. And it stops working where the claim is checkable against something other than text, because the moment a reader recomputes the number the sentence describes, the sentence loses. None of that is detection of an attack. It is somebody noticing a fact is wrong - which is why this class of finding is so frequently filed against the wrong owner.