A deployment screens each user message with an input classifier and then screens the model's reply with a separate output classifier. You run an attack corpus end to end against that live path and every case is blocked. Why does that green result not tell you which of the two classifiers earned the block, and what test design does?

### What the green run actually licenses An **end-to-end run** means pushing every case of a fixed attack corpus — a hand-labelled set of prompts you expect the product to refuse — through the deployment the way a user would, and recording only what a user could see: did the request complete, or was it stopped. That observation licenses exactly one claim: *this corpus, against this whole configuration, on this day, produced no visible failure.* It licenses none of the three claims a defence owner actually wants: which layer stopped each case, what dropping a layer would cost, and whether some layer is contributing nothing at all. ### The mechanism: a screening stack short-circuits A layered defence is an OR over block decisions evaluated in a fixed order, and evaluation stops at the first block. Make it concrete with the two layers in the question. The **input classifier** scores the user's message before the LLM is called — a hosted moderation endpoint, a small classifier model, or a rule set. The **output classifier** applies the same idea to the generated reply. When the input classifier refuses a case, the LLM is never invoked, no reply is ever produced, and the output classifier is never called. Its verdict for that case is not `pass` and not `fail`; it is a **missing cell**. So a 100% end-to-end block rate is fully compatible with layer one doing all the work while layer two is a no-op — misconfigured, pointed at a stale endpoint, or disabled by a flag nobody has read in six months. It is also compatible with the reverse. The run cannot distinguish these worlds, because the evidence that would distinguish them was never generated. Order matters for the same reason: reverse the two layers and the same corpus can produce a completely different attribution, with identical end-to-end output. ### The design that attributes: one run per boundary Hold the corpus byte-identical across runs, then: 1. Run the baseline end to end. This is the only observation of the real deployed path and everything else has to reconcile with it. 2. For each layer in turn, arrange for that layer to be the only one permitted to stop a request. Every other layer still evaluates and records its verdict but does not block — vendors call this record-only, log-only, shadow, audit, or `detect` rather than `block`. 3. Store a matrix: rows are corpus cases, columns are layers, and each cell takes one of **three** values — `blocked`, `not blocked`, `not reached`. From that matrix you can compute per-layer catch, per-layer *unique* catch (cases only that layer stopped), and the residual set nothing caught — which is the finding, and which an end-to-end run also hides whenever a later layer happens to cover it. ### What it costs N layers means roughly N+1 passes over the corpus. Each pass with a live target is one generation call per case plus one call per classifier per case. A 500-case corpus against a two-layer stack is ~1,500 target generations across three passes, plus classifier calls; if either classifier is a metered hosted service, that line multiplies too. Wall-clock usually hurts more than spend: a serialized pass at a couple of seconds per case is tens of minutes, and production rate limits often cap the concurrency that would fix it. The largest line item is normally engineer time — the harness ability to put one layer in record-only *without altering what the next layer receives* is real work, and on a client's production stack it is a config change, which means a change ticket, not an afternoon. ### Where the number misleads Three readings to refuse. First, **"the guardrails held"**: the *stack* held, on this corpus; any per-layer claim drawn from an end-to-end run is invented. Second, **recording an unreached layer as a pass** — the natural data shape in a home-grown harness is one boolean ("was the request blocked?") broadcast to every column, which silently pushes every short-circuited layer's measured catch toward 100%. The mirror error, recording it as "not blocked", deflates the layer instead; both come from having two values where the measurement needs three. Third, even a clean isolated block rate is not that layer's contribution to the deployed system: two layers at 90% each may be catching the same cases, so neither is worth ten points on removal. ### What you would check before believing it Send a case you know a layer blocks and confirm that in record-only mode it actually reaches the next stage — quiet is not the same as non-blocking. Confirm no record-only layer redacts, normalises or wraps the text, because a rewriting layer changes the prompt and you are now measuring a path that does not exist in production. Keep the benign half of the corpus in every run, since isolating a layer also changes its false-positive count. Count model self-refusals separately from classifier blocks. Finally, recompose the per-layer columns with an OR and compare against the baseline: if they disagree, your isolation changed the system's behaviour and the matrix is describing something other than the deployment.