Your team re-runs the same PyRIT target set against a fast-moving product every release. How do you keep the wired target set from silently drifting behind the surfaces the product actually ships?

### The two numbers, one of which nobody publishes Every run produces two facts: what it found, and what fraction of the model-reaching surface it looked at. Teams publish the first and carry the second in their heads. That is the whole mechanism of the drift: a hand-maintained target list decays at exactly the rate the product ships, and its decay is invisible, because a missing target and a passing target produce the same thing in the report — nothing. ### The structural fix: reconciliation Stop remembering the target set; derive it. Take the artefacts the platform already deploys from — route manifests, gateway or ingress configuration, the service registry, the data-flow or data-map inventory — and compare them, on a cheap and frequent cadence, against the set of wired PyRIT prompt targets. The comparison runs both ways: a model-reaching path with no target is a coverage gap; a target for a route that no longer exists is a stale target quietly inflating the wired count. Three properties make this work rather than become shelfware: - **It must not execute attacks.** Reconciliation compares two lists. It can run nightly at zero model cost, on infrastructure that has no credentials to send a prompt anywhere, while the metered run stays on release cadence and inside its budget. Coupling the check to the run is what makes teams run it quarterly. - **Waivers carry an owner and an expiry.** A gap you have consciously accepted is fine; a gap that is silent is not. Expiry is the mechanism that forces the list to be re-argued rather than inherited. - **Registration sits at the point of change.** The red team is a poor sole owner of a list that grows from other people's work. If adding a model-reaching intake path includes registering a target, drift is bounded by the release process. Keep the reconciliation and the escalation with the red team; put the registration step with the people shipping. ### Design trade-offs worth arguing **Derived versus curated.** Deriving scales and cannot be forgotten, but it is noisy — most routes never touch a model, and a check that fires constantly gets ignored. Narrow it with an explicit annotation on model-reaching paths, or a classifier over the route metadata. A curated list is precise on the day it is written and wrong by the next quarter. **Blocking versus reporting.** Failing a deploy on an unregistered model-reaching path forces attention, but if it fires often, waiving becomes reflex and the signal is gone. A workable middle: report continuously, block only when an unwaived gap persists past a window, and never remove the waiver mechanism — an unwaivable check on a noisy signal gets routed around or switched off entirely. **Structural versus configuration drift.** This is the limit worth stating out loud, because it is the failure the inventory cannot see. The route still exists and still has a target, but it is no longer the thing you tested: the guard was disabled for latency, the tenant policy was loosened, the model deployment moved to a new version, the system prompt was rewritten. Reconciliation sees no change at all. Cover it by having each target **assert the properties it depends on at run start** — the guard rejects a known-rejected case, the deployment identifier matches, retrieval returns from the production index — and fail loudly when an assertion breaks. Those assertions are also what keep results comparable across releases. ### What it costs The reconciliation itself is nearly free: two list pulls and a diff, plus the one-off engineering to get at the deployment artefacts and to annotate which paths reach a model. The real spend is elsewhere — each newly registered target costs build and credentialing time, and then costs metered calls on every run thereafter, because each attempt bills the target and the scorer, and each turn of a multi-turn strategy bills the adversarial model as well. So a growing inventory drives a growing run bill, and the honest conversation is which targets get the full prompt set and which get a sampled subset, not whether the surface exists. ### Where the number misleads A flat hit count across four releases while the wired set quietly shrank — targets broke, were disabled, were never replaced — reads as stable security and is the opposite. Hit counts and success rates are only comparable release to release if the denominator is stated and unchanged; otherwise a shrinking denominator holds the headline steady while exposure grows underneath it. The measure of success is not a perfect inventory, which nobody has. It is that every release's result is published with its denominator attached, and that a shrinking denominator is visible on the page rather than inferred after an incident.