roadmap
- 01
- 02
- 03
- 04
- 05
- 06
- 07
- 08
Use PyRIT to red-team the generative AI systems your team ships, with repeatable runs and evidence you can defend. Covers attacks, targets, converters, scorers, memory, run budgets and custom parts.
151 of 151 questions in your plan: 0 known, 0 in progress, 0 due, 0 skipped, 151 new. 0 cut, outside your plan.
PyRIT usually surfaces in AI red team and AI security loops, and in AI engineering interviews for teams that own a model-backed product and must test it before release. It rarely gets a round of its own. It arrives inside a scenario: your company's support assistant ships next month, how would you assess it, and what would the evidence look like? Candidates who have used the framework are then pushed on its parts, the prompt target that wraps the endpoint, the converters that rewrite each prompt, the scorer whose verdict decides when a run stops. The follow-ups test whether you know what a run can claim. A multi-turn attack that ends with its objective not achieved has only shown that this turn budget, this adversarial model and this scorer found nothing. A batch whose success rate jumps after an encoding converter is added may be counting garbled replies the scorer misread as compliance. A near-zero rate behind a hand-written target can mean a hardened service or an adapter that returned before the stream finished. A junior answer wires the pieces together and reads the score. A senior answer audits the score: it labels transcripts by hand to measure the scorer, prices each turn across the adversarial model, the target and the judge, and says which surfaces the run never touched.
It begins with why you test your own system at all. [Safety evaluation](/topics/found-llm-safety-security-safety-evaluation) frames a red-team run as evidence about something you ship, and [scope and authorisation](/topics/sec-ai-redteam-tools-reporting-engagement-scope-authorization) settles what you may point it at, which matters when the model behind your product is hosted by someone else. [Jailbreak families](/topics/sec-ai-llm-attacks-jailbreaks) follow, because they are the behaviour a run tries to provoke, and PyRIT automates that behaviour without defining it. The framework's own parts come next, each after the idea it leans on. [Attack strategies](/topics/sec-ai-redteam-tools-pyrit-object-model-attack-driver) and the [prompt targets](/topics/sec-ai-redteam-tools-pyrit-object-model-target-adapter) they drive come first, since every later piece hangs off that pair. [Input screens and how prompts slip past them](/topics/sec-ai-llm-attacks-guardrail-evasion-input) precede [converters](/topics/sec-ai-redteam-tools-pyrit-object-model-converters), so a transform reads as a hypothesis about a filter rather than a trick. [Judge-model validity](/topics/found-ai-engineering-evaluation-llm-as-judge) precedes [scorers](/topics/sec-ai-redteam-tools-pyrit-object-model-scorers) for the same reason. [Conversation memory](/topics/sec-ai-redteam-tools-pyrit-object-model-memory) closes the stretch beside [handling the harmful output](/topics/sec-ai-redteam-tools-reporting-engagement-harm-handling) it stores and [reproducing an attack](/topics/sec-ai-redteam-tools-reporting-findings-reproducibility) from it. Then the plan turns from parts to running them. [Attacker-model loops](/topics/sec-ai-redteam-tools-auto-jailbreak-attacker-loops) explain the propose, score and rewrite cycle that PyRIT's [multi-turn runs](/topics/sec-ai-redteam-tools-pyrit-running-attacks-multi-turn-loop) put into practice. [Client-side concurrency](/topics/found-ai-engineering-cost-latency-batching-concurrency) sits under [what a run costs](/topics/sec-ai-redteam-tools-pyrit-running-attacks-concurrency-cost), and the [surfaces nobody wired](/topics/sec-ai-redteam-tools-pyrit-running-attacks-target-hardening-blind-spot) come with [declaring coverage](/topics/sec-ai-redteam-tools-reporting-engagement-coverage-plan) in the report. [Writing your own scorers and targets](/topics/sec-ai-redteam-tools-pyrit-extending) comes last, because a custom piece is only as trustworthy as the checks the earlier modules taught.
New to red-teaming but comfortable with Python and model APIs? Switch study order to by topics, work from the top, and run a single-send attack against a model endpoint you control before reaching the multi-turn modules; the questions about budgets and stop rules make more sense once you have watched a run end. A security engineer who already knows the attack families can move faster: with importance raised to must know and level narrowed to the seniority on the posting, most of the hours belong to scorers, memory and the coverage modules, where interviews expect judgment rather than vocabulary. The plan stays with model endpoints and the chat applications built on them. Agents whose attacks land as tool calls and changed records, adversarial attacks on classical models, and the rest of an engagement's reporting belong to the [AI Red Teaming roadmap](/roadmaps/ai-red-teaming), the role that carries this tool among the others. For a scanner that sweeps a model for known failure modes, see the [garak roadmap](/roadmaps/garak).
PyRIT usually surfaces in AI red team and AI security loops, and in AI engineering interviews for teams that own a model-backed product and must test it before release. It rarely gets a round of its own. It arrives inside a scenario: your company's support assistant ships next month, how would you assess it, and what would the evidence look like? Candidates who have used the framework are then pushed on its parts, the prompt target that wraps the endpoint, the converters that rewrite each prompt, the scorer whose verdict decides when a run stops.
The follow-ups test whether you know what a run can claim. A multi-turn attack that ends with its objective not achieved has only shown that this turn budget, this adversarial model and this scorer found nothing. A batch whose success rate jumps after an encoding converter is added may be counting garbled replies the scorer misread as compliance. A near-zero rate behind a hand-written target can mean a hardened service or an adapter that returned before the stream finished. A junior answer wires the pieces together and reads the score. A senior answer audits the score: it labels transcripts by hand to measure the scorer, prices each turn across the adversarial model, the target and the judge, and says which surfaces the run never touched.
It begins with why you test your own system at all. Safety evaluation frames a red-team run as evidence about something you ship, and scope and authorisation settles what you may point it at, which matters when the model behind your product is hosted by someone else. Jailbreak families follow, because they are the behaviour a run tries to provoke, and PyRIT automates that behaviour without defining it.
The framework's own parts come next, each after the idea it leans on. Attack strategies and the prompt targets they drive come first, since every later piece hangs off that pair. Input screens and how prompts slip past them precede converters, so a transform reads as a hypothesis about a filter rather than a trick. Judge-model validity precedes scorers for the same reason. Conversation memory closes the stretch beside handling the harmful output it stores and reproducing an attack from it.
Then the plan turns from parts to running them. Attacker-model loops explain the propose, score and rewrite cycle that PyRIT's multi-turn runs put into practice. Client-side concurrency sits under what a run costs, and the surfaces nobody wired come with declaring coverage in the report. Writing your own scorers and targets comes last, because a custom piece is only as trustworthy as the checks the earlier modules taught.
New to red-teaming but comfortable with Python and model APIs? Switch study order to by topics, work from the top, and run a single-send attack against a model endpoint you control before reaching the multi-turn modules; the questions about budgets and stop rules make more sense once you have watched a run end. A security engineer who already knows the attack families can move faster: with importance raised to must know and level narrowed to the seniority on the posting, most of the hours belong to scorers, memory and the coverage modules, where interviews expect judgment rather than vocabulary.
The plan stays with model endpoints and the chat applications built on them. Agents whose attacks land as tool calls and changed records, adversarial attacks on classical models, and the rest of an engagement's reporting belong to the AI Red Teaming roadmap, the role that carries this tool among the others. For a scanner that sweeps a model for known failure modes, see the garak roadmap.