How do you split an agent eval suite across CI tiers under a fixed cost budget?

## The cost model Eval spend is roughly *tasks x rollouts x tokens-per-rollout*, plus fixed per-rollout overhead (sandbox startup, fixture restore) and any judge calls, which are themselves *tasks x rollouts x judge tokens*. Agentic tasks are token-hungry because every tool result re-enters the context on the next turn, so a single long-horizon rollout can cost far more than a chat-style eval. Wall clock is a separate, partly independent constraint: it is bounded by rollout latency divided by achievable concurrency, and concurrency is bounded by sandbox isolation slots and provider rate limits, not by ambition. With a fixed budget, the design question is where each of those three factors buys the most information. ## What each factor buys **More tasks buys resolution.** The sampling error on a suite-level pass rate falls roughly as one over the square root of the number of tasks. At 30 tasks, the standard error is around 8 points: that suite can tell you the agent broke, but it cannot tell you 80% from 85%. Distinguishing small differences is a task-count problem, and no amount of repetition fixes it. **More rollouts buys reliability information.** k>1 is how flaky tasks become visible and how pass^k becomes computable. It says nothing about tasks you did not include. **More tokens per rollout — bigger contexts, more turns, higher effort settings — buys capability headroom** but makes every other axis more expensive. ## A tiered allocation **Tier 1, per change.** A curated subset — say 30 tasks chosen to span the distinct capabilities and tool paths, not the 30 easiest — at k=1, stubbed or replayed tools, programmatic verifiers only, no judge. Target: single-digit minutes, a few dollars, and no flakiness. Its job is breakage detection: a malformed tool schema, a prompt edit that destroys instruction following, a broken fixture. Be explicit that this tier is not a quality measurement, or people will over-read its number. **Tier 2, nightly.** The full suite — for a support agent, perhaps 120 golden tasks over a seeded database — at k=3 to 5, judged tasks included, run under an explicit ceiling such as a fixed spend cap and a wall-clock cap. Parallelize to fit, allocating one isolated sandbox per worker, and throttle to stay under provider limits. This is the tier that produces the numbers you actually reason about. **Tier 3, pre-release or weekly.** High k (8 or more) on the reliability-critical subset, plus live contract checks against real integrations. Expensive, infrequent, and the only tier that reports pass^k you would quote externally. ## Enforcing the ceiling A budget that is not enforced is a wish. Three controls matter. *Per-rollout caps.* An agent that loops can consume a disproportionate share of the night. Cap tokens, turns and wall clock per rollout, and record the exhausted runs as a distinct outcome rather than silently as failures — the exhaustion rate is itself a signal. *Suite-level abort.* Track cumulative spend during the run and stop cleanly when the ceiling is hit, reporting partial results with the number of tasks completed. A truncated report you can trust beats an overrun. *Cheapening the tokens.* Stable prefixes (system prompt, tool definitions, fixture description) are identical across every rollout of every task, which makes prompt caching unusually effective for evals. Stubbed tools also cut tokens by returning tight payloads instead of sprawling real ones. ## Choosing the subset The cheap tier's task selection is the highest-leverage decision in the whole scheme, and it is not random sampling. Cover each distinct tool path, each major capability, and the failure modes that have actually bitten you in production — a regression that has happened once will happen again. Keep a couple of tasks that are known-hard and expected to fail; if they suddenly pass, either the agent improved or a verifier broke, and both are worth knowing. Rotate the subset periodically so the fast tier does not become the only thing anyone optimizes for. ## Honest limits None of this is settled practice, and the right numbers are workload-specific: a suite gating an agent that moves money deserves a budget an internal drafting assistant does not. State the principle — the cheap tier detects breakage, the expensive tier measures reliability, and each tier's budget is enforced rather than hoped for — and then say the constants come from measuring your own rollout cost and latency, not from a rule of thumb. ## How to answer Start from the cost identity, explain what each factor buys, propose the tiering, and be specific about enforcement and subset selection. The weak answer runs the whole suite on every commit and is surprised by the bill, or shrinks k and task count together until the suite is cheap and meaningless.