You have a fixed pool of adversarial turns to spend across roughly 40 objectives in a PyRIT engagement. How do you allocate the turn budget between deep runs on a few objectives and shallow runs on many, and how do you report the choice?
answer
- screen wide, confirm deep
- even split serves neither goal
- shallow negative = untested, not safe
- hold a reserve for surprises
- tier budgets fixed across retests
basics
~20 sDo not spread it evenly. Run a short screening pass over all objectives, then spend deep budgets only on the ones whose transcripts showed the target trending rather than refusing flat. Report every negative with the budget that produced it, treat shallow negatives as untested rather than safe, and keep budgets fixed between retests.
solid answer
~50 sEven allocation is the worst option: it produces forty negatives that are all too shallow to defend and no positives deep enough to act on. **Two-tier instead.** A screening tier gives every objective a small budget, several repeats, and one job — separate objectives where the target holds flat from objectives where it visibly moves. A confirmation tier takes only the movers and spends a much larger budget and higher repeat count on them, producing findings with reproducibility and turns-to-success attached. **The reporting rule matters more than the split.** Screening-tier negatives are *not tested to depth*, and must appear in the report under that heading, never merged into a single safe list with the confirmation-tier negatives. Otherwise the shallow tier's job — cheap triage — silently becomes an unearned assurance claim. **Fix the tiers.** Once the tier budgets are published, hold them across retests and across targets, so a change in results is a change in the system rather than a change in how hard you looked.
go deeper
Recognises that turn budget is a knob and that a small one produces weak negatives.
Proposes spending more turns where hits occur and less elsewhere, and knows the budget belongs in the report.
Designs the screen-then-confirm structure with explicit promotion criteria and holds budgets constant for comparisons.
Owns the reporting contract as well as the split — tiered results never aggregate, budgets ratchet only upward across retests, and tier definitions are standard across engagements so numbers travel.
**Frame it as evidence allocation, not scheduling.** A turn is the atomic unit of spend in a PyRIT engagement, and it buys two incompatible things: breadth of coverage across objectives, and confidence per objective. You cannot buy both with the same turn. The instinctive move — divide the pool by forty — buys neither: forty negatives too shallow to defend, and any positives too shallow to characterise. **The arithmetic that makes this real.** One turn bills three model calls: the adversarial chat model writes the attacker turn, the target answers, the objective scorer judges. Turns inside a conversation are strictly sequential, so wall clock scales with turns, and only whole runs can be parallelised. A worked example on a 40-objective engagement: ```text screen: 40 objectives x 3 repeats x 5 turns = 600 turns ~ 1,800 model calls confirm: 6 promoted x 10 repeats x 20 turns = 1,200 turns ~ 3,600 model calls reserve: ~15% of the pool held back ``` The same pool spent evenly would be 45 turns per objective — about nine turns each at five repeats, which is too shallow for a defensible negative and too few repeats for a hit fraction anyone would act on. **The structure.** 1. **Screen.** Small budget, modest repeat count, every objective. The output is explicitly *not* a verdict; it is a triage signal answering one question — does the target hold flat, or do its replies move across turns? 2. **Confirm.** Large budget, high repeat count, only the objectives that hit or that were still trending when the screen's cap fired. The output is a finding with a hit fraction and a turns-to-success distribution attached. 3. **Reserve.** Hold back a slice. Something in the screen will be surprising, and an engagement with no slack cannot chase it without cannibalising the confirmation tier. **Promotion is the decision that matters.** Promote on any hit, and also on any screening transcript whose target replies were still softening at the cutoff. Promoting on scorer hits alone is the classic error: it discards precisely the objectives most likely to break at depth, because a short budget is where a slow build-up is guaranteed to look like resistance. **Where the number misleads, and why the fix is structural.** A page of shallow negatives is indistinguishable, to anyone outside the team, from a page of tested-and-safe. Thirty screened negatives plus ten confirmed negatives becomes "the system resisted 40 of 40 objectives" in one summarisation step, and cheap triage has been promoted into an assurance claim the runs never supported. Editorial caution does not survive that step; only structure does. Name the tier in every row. State the turn budget and the repeat count in every row. Refuse to publish an aggregate percentage that mixes tiers — if leadership wants a single number, it must be one tier's number, with its budget and N stated beside it. And describe screening-tier negatives as *not tested to depth*, which is what they are. **Retests invert the shape.** After a mitigation ships, most of the pool goes to the handful of previously-hitting objectives, at the same budget, the same attacker configuration and the same N as the original run — otherwise the improvement you report may be entirely a budget difference. Budgets ratchet upward across retests, never down. New objectives get a screening pass with whatever is left. Note also that a clean retest at a small N is weak evidence: at a 2/10 baseline, an unchanged system still returns 0/10 about eleven percent of the time by luck alone. **What I would standardise across engagements.** Fixed tier budgets and repeat counts, published; a shared objective library so the same wording is used everywhere and results travel between engagements; the ratchet rule on retest budgets; and a review step in which the screening transcripts of anything promoted *and anything not promoted* can be inspected. That last one is the audit that catches the failure this whole allocation exists to avoid — an objective quietly filed as resistant because the screen ran out while the target was still moving.
- What promotes an objective from the screening tier to the deep tier?Any hit, plus any transcript where the target's replies were still moving when the screen's budget expired. Promoting on scorer hits alone throws away the objectives most likely to break at depth.
- Leadership asks for a single 'percentage of objectives the system resisted'. What do you give them?A number scoped to one tier and one budget, stated as such, plus the count of objectives that were only screened. Aggregating across tiers converts cheap triage into an assurance claim the runs do not support.
- On a retest after a mitigation, how does the allocation change?It inverts: most of the pool goes to the previously-hitting objectives, at the same budget and repeat count as the original run, so the comparison is valid. New objectives get a screening pass with whatever is left.
saying these in an interview costs you the question
- Splitting the pool evenly across objectives to look thorough.
- Publishing one resistance percentage that mixes screened and deeply tested objectives.
- Lowering the budget on a retest and reporting the improvement.
- Promoting to the deep tier only on scorer hits, ignoring transcripts that were still trending at the screen's cutoff.
- Leaving no reserve, so nothing surprising found mid-engagement can be pursued.