PyRIT

Each stored turn carries identifiers tying it to its conversation and run, so a report item can point at the exact exchange that produced it. Keep those identifiers in the finding. The transcript proves what happened once against that configuration - not that it reproduces, since the target samples and may have changed.

Because the run stored every prompt and response, you can point a different scorer at the saved transcripts and relabel them without spending another target call. What that cannot tell you is what the attack would have done under the new judgment: the strategy branched on the old verdicts, so the turns it never explored simply do not exist.

Use PyRIT's generic HTTP target: give it the request template with a placeholder where the prompt goes, plus logic that pulls the assistant text out of the reply. You then own auth refresh, retries and rate limiting yourself. Anything the API never returns - system prompt, tool calls, retrieved context - stays untested.

Concurrency changes arrival rate, not work. The same calls are made, so spend is unchanged; you only reach the quota sooner. Wall clock improves until the tightest-quota of the three endpoints saturates, then stops improving, and past that point the other two legs idle behind the bottleneck.

The target cannot change — it is the system under test, and swapping it invalidates the run. The adversarial model and the scorer can. A weaker adversarial model finds fewer breaches, so results understate risk. A weaker scorer misjudges responses, which corrupts the number the whole run reports.

Stopping at first success gives an existence proof and nothing else: no severity, no repeatability, no sense of whether it took two turns or nine. One run is a single sample of a stochastic search. Repeat each objective several times, record turns-to-success across runs, and hold the budget fixed across targets you compare.

Check reachability before blaming the prompts. Confirm the incident path was in the target list at all, replay a positive control through each target, read the stored attempts for empty bodies, errors, auth rejections and timeouts, and compare the target's endpoint, tenant and deployment identifier against what production served.

Any path whose content reaches the same model: document or ticket ingestion, file and attachment parsing, transcribed voice or extracted image text, scheduled and batch jobs, webhook consumers, admin or internal endpoints, other tenants and older routed API versions. Each needs its own send path, matching credentials, and somewhere the effect can be observed.

Re-scoring recovers verdicts on responses you already collected: single-turn runs are fully repaired and missed hits reappear. It cannot recover the run itself. In multi-turn attacks the old scorer chose when to escalate and when to stop, so the transcripts hold only the paths it steered into; turns never taken stay unexplored.

Spend the first slice on broad single-send runs to find where the target is soft, then concentrate deep multi-turn runs on those objectives. Set the turn cap from observed successes, not from a guess, and reserve a slice for repeats, because one run against a non-deterministic endpoint is a sample rather than a result.

Size the matrix by distinct transform classes, not by count: variants multiply target and scorer calls linearly and mostly re-test the same input path. A per-converter number measures which surface form got past the input handling and the judge — not distinct model weaknesses, and not attack-surface coverage.

A local per-operator store is simplest and keeps harmful transcripts on one controlled machine. A shared store lets a team resume each other's runs, deduplicate findings and report across the whole engagement, but concentrates every operator's harmful text in one place that now needs access control, retention rules and a named owner.

Let the scorer decide only inside the run: when to stop a conversation and how to rank what gets read first. Anything leaving the team as a finding needs a human who read the transcript. Keep a fixed labelled transcript set as the scorer's own regression test, re-run whenever the scorer changes.

Decide from the question being asked. The raw endpoint measures the model's own behaviour with no product defences; the product API measures what a user can actually reach, defences included. Each report claims only its own surface. Where budget allows, wrap both and read the gap between them as the value of the defences.

Split the budget: a broad shallow pass to find which objectives show movement, then deep turn budgets spent only on those. Cap spend per objective so one runaway conversation cannot eat the pass. Report cost per objective and per confirmed finding, and state which objectives were cut short rather than cleared.

Do not spread it evenly. Run a short screening pass over all objectives, then spend deep budgets only on the ones whose transcripts showed the target trending rather than refusing flat. Report every negative with the budget that produced it, treat shallow negatives as untested rather than safe, and keep budgets fixed between retests.

Derive the target set instead of remembering it. Reconcile wired targets against the artefacts the platform deploys from — routes, service registry, data-flow records — and flag any model-reaching path with no target and no expiring waiver. Make registering a target part of shipping a new intake path, and track wired-over-deployed as a first-class number.

Require every adapter to pass one shared conformance check: a known refusal, a known compliance, a forced error and a forced timeout, each landing in PyRIT's memory as a distinct expected outcome. Also require the same prompt set, scorer and turn budget. Otherwise the differences measure adapters, not services.