You have a fixed token spend for an agent red-team harness sweep against a metered hosted agent and about 40 candidate attack payloads, and each trial is a whole multi-turn episode rather than one call. How do you split that spend between covering more payloads and repeating each payload enough times to quote a rate?

## Frame it as a search followed by a measurement **Screening** and **rating** are different problems with different optimal spends. - **Search** wants many cheap draws spread across many payloads, because early in an engagement a hit anywhere is worth more than a precise rate for a payload you have not found yet. - **Measurement** wants many draws concentrated on one payload, because concentration is the only thing that narrows an interval. Running a uniform twenty trials across all forty payloads spends most of the budget establishing, to three significant figures, that things which never work never work. ## A two-stage allocation 1. **Stage one** screens every candidate at a small fixed depth — three trials is a common choice — purely to separate "never landed" from "landed at least once". 2. **Stage two** spends the bulk of the remaining budget deepening only the survivors, adding trials until the interval answers the decision in front of you. Hold back a **reserve**, roughly a fifth, for two things teams reliably forget: the variants the screen suggests once you can see which family of payloads lands, and the fix-verification runs you will owe after filing. Verification costs on the order of the original measurement; unbudgeted, it silently shrinks to two runs and proves nothing. ## Price a trial correctly, because it is not a call In an agent harness a trial is a whole episode: several model turns, a tool round-trip per action, and any judge-model call the scoring object makes on top. The spend multiplies as payloads x trials x turns per episode x tokens per turn, plus tool and judge calls. Forty payloads at twenty trials and eight turns is 6,400 model turns before tools, and wall-clock is usually set by tool latency and the endpoint's rate limit rather than by token count, so a sweep that looks affordable in dollars can still be a two-day run. The **turn budget** is the sneaky lever: halving it roughly halves the cost and buys twice the trials, and it also removes the long-horizon paths where a slow escalation lands. If you cut turns to afford trials, the number you quote is a rate under that turn budget and is not comparable to the earlier sweep — say so, rather than letting two incomparable numbers sit in one table. ## Sequential stopping Decide the question before you spend: "is this above the ten-percent line that makes it a release blocker?" Then deepen a payload only while its interval still straddles ten percent. Payloads that resolve early release budget to the ones that do not, which is where an **adaptive allocation** beats any fixed n. ## Where this allocation misleads - The **screen's misses** are the big one. A payload that lands five percent of the time shows zero hits in three trials about six times in seven, so "34 of 40 payloads did not land" is a statement about your screen depth, not about the target. Those payloads are unmeasured, and writing them up as clean is the most common way a two-stage sweep manufactures false assurance. - Second, selecting survivors on a first hit biases the deepened rates slightly upward, because you kept the payloads that happened to land early; the effect is small next to the interval at these trial counts, but it is real and deserves a sentence. - Third, a reserve quietly spent on more screening leaves the engagement with no verification budget, which is how a mitigation ships unverified. - Fourth, the plan assumes trials are independent: if the environment is not reset between them, later trials in a payload's block are correlated with earlier ones and your effective n is smaller than the number you quote. ## What to check before believing the allocation worked - That the hit criterion was byte-identical in both stages, since a screen and a deepening scored differently are not two stages of one experiment. - That the environment reset actually ran between trials rather than being assumed. - That aborted episodes — rate limits, tool errors, timeouts — are excluded from denominators and disclosed as a count. - And that the report shows the allocation itself: screen depth, which payloads were deepened and why, and the trial count and turn budget behind every quoted rate. A reader who cannot see the allocation cannot distinguish a payload that was tested hard and stayed clean from one that was barely touched.