Why is a prompt-injection detection classifier not a security boundary?

## What a detector actually gives you An injection detector — a fine-tuned classifier, a small guard model, a heuristic over suspicious phrasing — outputs a score. You pick a threshold and trade false negatives against false positives. On a benign corpus with a fixed attack suite this looks excellent: reported detection rates in the high nineties are normal. The number is real, and it is also the wrong number to design around. ## Why the attacker moves second Security evaluations of defenses are usually run against a **static** set of attacks that existed before the defense did. The attacker in production does not work that way. They see the deployed system, probe it, and adapt — rephrasing, encoding, splitting an instruction across documents, burying it in a format the classifier scores as benign. The 2025 study "The Attacker Moves Second" made this concrete by re-testing twelve published defenses with attacks optimised specifically against each, using human red teams and automated search. Most defenses that had reported robustness collapsed past 90% attack success. The generalisable lesson is not that those twelve were bad work; it is that a static-suite number is an upper bound on robustness, and adaptive success rate is the number that matters. ## The arithmetic of retries Suppose your stack fails on 0.1% of attempts — an excellent per-attempt figure. An attacker who can send a hundred emails, open a hundred issues, or publish a hundred pages faces roughly a 10% chance overall, and they only need one. Vendor containment write-ups report exactly this shape: single-attempt success near a tenth of a percent, rising to several percent under a hundred adaptive attempts. Wherever the attacker controls the retry budget, per-attempt probabilities are close to meaningless. ## Two structural gaps detectors cannot close First, **direct injection through the user's own input** bypasses model-layer training entirely: instruction-hierarchy training teaches the model to privilege the developer's instructions over content, but the user turn is a legitimate high-trust channel, and a malicious user is inside it. Second, **the detector sees text, while the damage is done by actions**. A payload can be split so that no single document looks hostile, and the harmful behaviour only emerges from the composition of several benign-looking inputs across a long session. ## What a boundary looks like instead The distinguishing property of a boundary is that it holds *when the model has already been fooled*. Concretely: - **Absent capability.** The session that reads untrusted content has no credential for the private store and no outbound path. Nothing the model decides can change that. - **Fixed control flow.** The sequence of actions is committed before untrusted content enters, so injected text cannot introduce a step that was not planned. - **Policy on values.** Every value carries provenance and a permitted audience, and the check happens at the point of action, not in the model. - **Isolation.** Untrusted items are processed in separate contexts whose outputs are constrained to a narrow, typed shape. All four are deterministic: they are code, not inference. The useful formulation is that the deterministic boundary is what gets hit when everything probabilistic misses — so the design question is never "is the filter good?" but "what happens on the run where the filter is wrong?" ## Where filters still belong None of this means deleting your classifier. Detection is genuinely valuable as (a) noise reduction against the large volume of low-effort attacks, (b) a telemetry signal that tells you an attacker is probing, and (c) a rate-limiting trigger that raises the cost of the retry loop that makes adaptive attacks work. What you must not do is let a detector license a design that would be unsafe without it. The test is simple and worth asking in review: if this classifier returned "benign" for every input, what would the worst outcome be? If the answer is "our customer database leaves the building," the classifier was carrying weight it cannot carry. ## How to talk about this in an interview Do not overclaim in either direction. "Prompt injection is unsolved at the model layer" is the current honest position; "therefore nothing helps" is wrong. Layer the probabilistic defenses, expect them to be partially bypassed, and size the blast radius so that being bypassed is survivable. Interviewers are listening for whether you treat a percentage as a mitigation or as a boundary.