A probe you send at an application fronted by a rail framework such as NeMo Guardrails comes back as a short generic refusal. Why can that refusal text not tell you which rail blocked the turn, and what do you inspect instead?

### Why the string cannot name the layer In a rail framework the refusal a caller sees is a bot message defined **once for the whole deployment**. In NeMo Guardrails that is a Colang `define bot refuse to respond` block — one canned sentence — and every blocking path converges on it: a self-check flow listed under `rails.input.flows` refusing on the way in, a dialog flow that matched a disallowed user intent, a flow under `rails.output.flows` rejecting the generated answer, or a custom action returning False. That convergence is usually deliberate. Per-layer refusal strings would let anyone probing the public endpoint fingerprint the stack for free, so the product decision is to say as little as possible. As a tester you are on the wrong side of that decision, and no amount of re-reading the wording moves you to the right side. There is also a fourth source that is not a rail at all: **the application model may have refused on its own**, with every rail passing. That is the misattribution that quietly voids a finding — no configuration edit reproduces or removes it, and the next model swap deletes your result without anyone noticing. ### The mechanism you inspect instead NeMo Guardrails' `LLMRails.generate` accepts an `options` argument. Ask it for the log — `options={"log": {"activated_rails": True, "llm_calls": True}}` — and the response carries `response.log.activated_rails`: one entry per rail that actually ran, each with its `type` (`input`, `dialog`, `generation`, `output`), its name, the decisions it took, and whether it stopped the turn. The parallel `llm_calls` list shows every model request the turn made, with prompt, completion and token counts, which is how you tell a judge-backed rail from a pattern rule without reading the config at all. The interactive equivalent is `nemoguardrails chat --config=... --verbose`. That trace is the only source that settles attribution **inside a single request**, which is why trace or log access for your test traffic belongs in engagement scoping rather than being discovered as a gap on day three. Where no trace is available, **differential configuration runs** prove causation instead: copy the config, remove one entry from `rails.input.flows` or one `define user` block, replay the byte-identical payload, and see whether the verdict moves. Slower, needs config access, and it changes the system under test — but it is causal where a trace is only descriptive. ### What it costs Logging itself is free; the rails are not. A config with an input self-check and an output self-check makes **three model calls per turn** — judge in, application generation, judge out — plus an embedding call whenever dialog rails are active. So a 500-payload sweep against a fully railed app is roughly 1,500 completions, not 500, and the input judge is frequently the same expensive model as the application. Differential runs multiply again: (L+1) configs × N payloads. Budget a run in **model calls, not probes**, or you will hit a provider rate limit a third of the way through and — worst case — read the resulting error responses as blocks. ### Where the number misleads The tempting metric is a block rate: *"the guardrails stopped 84 of 200 probes."* That figure silently pools rail blocks with plain model refusals, and the model's share of it is not a control anyone owns; it moves the day the vendor ships a new checkpoint, with the config untouched. The symmetric error is worse: **a normal-looking answer does not prove no rail fired.** An output rail can rewrite or truncate a response rather than refuse, and a rail can pass a turn while logging it as suspicious. "The app answered, so the guardrail missed it" is a claim about the trace, not about the reply. Latency is a hint, never a verdict — a judge rail does add a call, but caching, queueing and network jitter cover that difference easily. ### What you would check Send a control payload you know matches one named rail, and confirm the trace shows exactly that rail stopping the turn: that proves your logging is wired to the config you believe you are testing. Then send the same payload straight at the raw generator with rails disabled — if it still refuses, you have a model refusal and there is no guardrail defect to file. Finally, write the **layer**, not the symptom. "A refusal was returned" is not a result; "the input self-check passed this payload and the block came from the output rail" is, because it names the part of the config the owner has to change and it survives the next model change.