PyRIT

Your target owns everything endpoint-specific: authentication, request and response shape, timeouts and retries, and turning the reply into one response PyRIT can store. Everything else is unchanged. The attack strategy still chooses prompts, converters still transform them, scorers still judge them, and the memory store still records each exchange.

You build the strategy once with its target, scorer and any converters, then execute it in a loop, once per objective. The wiring is shared across all twenty. Each execution carries its own objective and its own conversation, so the transcript and the verdict stay separate per objective.

A PyRIT prompt converter rewrites the outgoing prompt text after the seed prompt is chosen and before it is sent. It changes the surface form, not the request underneath. On its own it is just a transform: it tests whether that form gets through, never whether the model will actually comply.

PyRIT's memory records every prompt sent and every response received, turn by turn, with the scores attached, so a run can be resumed, audited and re-scored later. The in-memory option keeps that only for the process lifetime: when it exits, the transcripts are gone and nothing can be re-read or re-scored.

In PyRIT, the scorer reads the target's response and decides whether the attack objective was met. In a multi-turn run that verdict is control flow: a positive verdict ends the run early and marks it a success. So the scorer, not you, decides when testing stops.

A PyRIT prompt target is the adapter that knows one endpoint: it sends a prompt there and returns the response. A run usually needs several of them, because the system under test, the attacker model that drafts prompts, and any model that scores replies are three different endpoints.

Three. A PyRIT turn bills the adversarial model that writes the next prompt, the target under test that answers it, and the scorer that judges the reply. Every turn pays all three, so a fifty-turn objective is roughly one hundred fifty billed calls, not fifty.

You set four things: an objective describing what the system under test should be made to do, an adversarial chat model that writes each attacker turn, the target endpoint being tested, and a scorer that judges the target's reply. The loop ends when that scorer says the objective was met, or when the turn budget runs out.

Only the targets you configured. PyRIT sends prompts to the prompt targets you wired and scores the replies; anything it was never pointed at, such as another endpoint, an upload path or an internal service, is untested rather than proven safe. A clean result describes your target list, not the application.

A false positive is caught: a human reads the flagged transcript during triage and drops it. A false negative is never read, because the transcript looks like an ordinary refusal and nobody opens it. In a multi-turn attack the miss also tells the strategy to keep pushing or to stop, so it changes the run itself.

A single-send strategy delivers each prompt once, scores the reply and stops; nothing feeds forward. A multi-turn strategy loops: it reads the last reply, composes the next prompt with an attacker model, and scores each turn until the scorer says achieved or the turn cap runs out. Single-send on a multi-turn goal returns not-achieved, having proved nothing.

The chain runs in series over a single prompt: the first converter's output becomes the second's input, and only the final string is sent. So the converters compose, and composition is not commutative — rewriting an already-transformed string gives a different result than transforming a rewritten one.

By default PyRIT writes the run to a local database file on the machine you launched it from, unencrypted. That file holds attack prompts and the target's worst answers verbatim. Treat it as sensitive evidence: put it on encrypted storage you control, keep it off shared drives and backups, and delete it on schedule.

A true/false scorer gives the loop the yes-or-no it needs to stop, so it is the default for objective-driven runs. A scaled scorer returns a degree of compliance and forces you to pick the threshold that counts as success. A category scorer says what kind of harm, not whether the attack worked.

For a chat-completion style endpoint the target replays the stored turns: it reads the conversation out of memory and sends the whole message list each call. If the service keeps its own session and accepts only the newest message, the history lives on the server, so the tool's copy and the real context can drift apart.

Multiply, do not add. Target calls are objectives times turns times converter variants. Scoring calls are target responses times the number of scorers attached. Adding one converter and one scorer roughly doubles two legs at once. Work the product out on paper first, then dry-run one objective to check it.

It only says the adversarial model did not get there within the turns you allowed. The budget is part of the result, not a property of the target. A longer run, a different attacker configuration, or another attack strategy can still breach the same endpoint. Report the negative together with the turn budget that produced it.

The raw API tests the model alone, with no production system prompt, retrieval, tools or guard. The application endpoint tests the whole deployed stack. A clean run at one says nothing about the other: a guard can hide model weakness, and a bare-model result ignores every layer users actually pass through.

Hand-label a sample and compare. Sample from the 388 the scorer called non-hits, not only the 12 hits, because misses are where uncounted findings sit. Read the hits too. Then report the count together with the measured miss rate and false-alarm rate on that sample, so a reader knows what the 12 excludes.

Do not trust the summary number yet. Open the responses PyRIT stored: a hardened service returns real refusal text, a broken adapter stores empty strings, error bodies, truncated replies or the same string every time. Then replay two or three of those exact prompts by hand and compare against what the adapter recorded.

Treat the jump as a scoring hypothesis first. Pull the stored exchanges, decode the replies, and read whether the content is really there. A transform that changes the reply's form also changes what a text-based PyRIT scorer sees, so refusals stop matching and garbled output can read as compliance.

Pull stored conversations from earlier runs, label the responses yourself, then re-score them offline with the candidate scorer and compare. Load the set with the hard cases: refusal-then-comply, partial compliance, in-fiction compliance, and confident nonsense. Only then let it stop runs. Replaying stored text costs no target turns.

The scorer is firing on responses that are not real compliance, and because a positive verdict ends the run, every false hit is also a truncated attack. False positives inflate the success rate and delete the turns you never sent. False negatives cost the whole turn budget and undercount. Read the stop-turn responses.

Write a deterministic rule when the success condition has a checkable surface: a specific string, a schema, a tool call, a leaked identifier you planted. Back it with a model only when success is a judgment about meaning. The rule is cheap, reproducible and re-runnable over stored transcripts; the model call adds cost, latency and run-to-run variation.

Consume the whole stream and assemble one complete response before returning, because scorers see a single stored response, not chunks. If you return on the first chunk or bail at a client timeout, a truncated answer is stored as the target's real reply, and a response that was about to comply can score as a refusal.

Each model-backed link bills one call per prompt, so two links over 200 prompts add roughly 400 calls before anything is sent anywhere. On top of that the run still pays the target call and the scorer call per attempt, plus the attacker model's calls if the strategy uses one.

The same sentence steers the attacker and defines success. If it names no observable artefact in the target's reply, the adversarial model has nothing concrete to drive toward and drifts, while the scorer has nothing concrete to check and can mark a hedged, harmless answer as met. Write objectives a reviewer can verify from the transcript.

PyRIT already tracks conversation identity, so reusing one server session makes history arrive twice or bleed between attempts. Mint a fresh session per PyRIT conversation, bind it to that conversation, and tear it down at the end. Otherwise a result depends on which attempts ran before it and stops being reproducible.

It ends three ways: the scorer returns an achieved verdict and the loop stops early, the turn cap is exhausted, or something raised — target error, rate limiting, the attacker side declining to continue. Not-achieved conflates all of the non-success endings, so it says the run stopped, not that the target held.