A PyRIT run against your deployed chat assistant finishes with no successful attacks recorded. What has that run actually tested, and what has it not?

### What the framework actually does PyRIT is an orchestration library, not a scanner. You write a short script that constructs three kinds of object and hands them to an attack orchestrator: a **prompt target** — an object that wraps exactly one send path, such as a hosted chat-completion endpoint, an HTTP endpoint of your own application, or a local model; a set of **seed prompts**, the attack content, usually loaded from a dataset file; and one or more **scorers**, the objects that read a response and decide whether it counts as a hit. The orchestrator then loops: take a prompt, optionally push it through converters, send it through the target, write the request and the response into PyRIT's memory store (a local DuckDB file by default), score the response, move on. Nothing in that loop performs discovery. There is no crawl, no route enumeration, no step that goes looking for the other places your model can be reached. The set of surfaces probed is precisely the set of prompt-target objects your script constructed — no more, and never implicitly more. ### The sentence a clean run actually supports "Zero successful attacks" unpacks to: *of N attempts, drawn from prompt set P, delivered through target set T, none was labelled a hit by scorer S.* Four qualifiers, and a report typically preserves only the first. Candidates habitually collapse the target bound into the prompt bound — they will discuss whether the prompt set was strong enough and never ask how many places it was sent to. ### Three ways a surface stays out of the denominator 1. **Never wired.** No target object exists for it: a nightly batch summariser, a webhook or mail consumer, a second tenant, a partner integration, an internal console that talks to the same deployment, an older API version still routed for legacy clients. It cannot appear in any result, and its absence produces no artefact at all. 2. **Wired at the wrong layer.** The target points at the raw provider API while real users hit an application endpoint that adds a system prompt, retrieval and a guard — or the reverse. The run characterises a system that is not the deployed one. 3. **Wired but inert.** Every attempt failed on authentication, TLS, a rate limit or a timeout. If those failures are not separated from genuine refusals, the target appears in the run, occupies rows in memory, and tested nothing. ### What it costs, and why teams under-wire Cost per attempt is metered calls, not requests: the target call, the scorer call when the scorer is itself an LLM judge, plus an adversarial model call on every turn of a multi-turn strategy. A single-turn sweep of a couple of hundred seed prompts against one chat endpoint is therefore several hundred metered calls — minutes to an hour of wall clock and small money. Engineer time is the dominant cost and it is wildly uneven: a second chat-shaped target is an hour of work, while a target for an asynchronous ingestion path that must deposit content and later read back the effect is days. That asymmetry, not risk, is what usually decides which surfaces get wired — the wired set skews toward what is cheap to wire. ### Where the number misleads Attack-success rate is hits over attempts, and attempts are counted only over wired targets. Adding a genuinely hardened target lowers the rate. Forgetting a vulnerable surface also lowers the rate. The two are arithmetically indistinguishable in the aggregate, so an improving trend line can mean better defences or a shrinking denominator. Errored attempts make it worse: an empty body carries no harmful content, so a scorer labels it a non-hit, and absent data sits in the denominator behaving like a survived attack. Two runs are comparable only if the target set and the prompt set were identical; nothing in the artefact tells you whether they were. ### What you check before believing it Pull per-target attempt counts out of PyRIT's memory rather than reading the summary. Classify each attempt into scored response, refusal, transport error, auth rejection, truncation, timeout — a target dominated by anything but the first two contributed no coverage. Confirm the target's base URL, credentials class and deployment identifier match what production serves. Then fire a **positive control**: a benign probe whose production response you can recognise, sent through the same target and the same scorer, proving end to end that requests landed and results came back. Finally, publish the target list and its attempt counts beside the hit count. A clean run is a coverage statement about a named set of surfaces; presented without that set, it is not evidence about the application at all.