How do you split an LLM eval suite between a per-PR gate and a nightly run on a fixed budget?

## The constraint is real and it is two-dimensional An LLM eval suite costs money per run and takes wall-clock time. Both scale with case count, both are paid on every execution, and they pull in opposite directions from coverage. A 900-case suite at around fourteen dollars per run is a rounding error once a night and a serious line item at forty merges a day; twenty minutes of wall clock is fine overnight and is a productivity tax on every pull request. Engineers respond to slow gates predictably — they batch changes, they context-switch away, and eventually they look for the flag that skips it. A gate that people route around protects nothing, so latency is a correctness property of the gate design, not a nicety. ## Tiering The standard shape is three tiers with different jobs: **The pull-request tier** is small, fast, and blocking. Its job is to catch the obvious, high-frequency regressions in the time a reviewer spends reading the diff. Target a few minutes end to end. **The nightly tier** is the full suite, run on the main branch. Its job is coverage. It does not block anyone's merge; it opens a ticket, pages the owning team, and — crucially — blocks the release candidate. **The release tier** is the full suite plus anything too slow or expensive for either of the above: long multi-turn scenarios, large-context cases, expensive judge rubrics. It runs on a candidate build before promotion. ## Choosing the pull-request subset The subset is not a random sample. Stratify it: - **Cover every failure mode.** One or more cases per known failure category, so no category is invisible at merge time. - **Cover every critical slice.** Anything with a hard guard — safety, legal, schema compliance — belongs in the fast tier by definition, because those must never merge broken. - **Weight by historical discrimination.** Some cases move whenever anything breaks; others have passed unchanged for a year. Score cases by how often they have flipped alongside a genuine regression and keep the discriminating ones. - **Prefer deterministic graders.** Cases graded by code are faster, cheaper, and quieter than judged ones, so they earn their place in a latency-constrained tier at a much better rate. - **Exclude the quarantined.** Advisory cases do not belong in a blocking tier. ## Engineering the wall clock down Most of the four minutes is waiting on the network, so concurrency is the main lever: run cases in parallel up to the provider's rate limit, with bounded workers so a burst does not trigger throttling that costs more time than it saves. Beyond that, share a cached prompt prefix across cases where the system prompt is common, keep reasoning effort at the level the case actually needs rather than the maximum, and run deterministic graders locally rather than as extra model calls. Report the run's cost and duration on every build so the budget stays visible instead of surfacing as a surprise invoice. ## Measure the gate, not just the product The question that makes this a design problem rather than an arbitrary trim is: **what does the fast tier miss?** You can answer it empirically. Every regression the nightly catches is a labelled example; replay it against the pull-request subset and ask whether the subset would have caught it. The resulting recall — say, the subset catches 80% of what the full suite catches — is the honest description of the gate. If it is low, the subset is mis-weighted, not merely too small, and the fix is usually to swap in cases from the categories that keep slipping through rather than to double the case count. Re-derive it periodically, because as failure modes shift, yesterday's discriminating cases become today's permanent passes. ## The trade to state out loud A fast blocking tier plus a comprehensive non-blocking tier means some regressions land on the main branch and are caught hours later. That is a deliberate choice: the cost is a revert or a follow-up fix; the benefit is a merge loop people actually use and a bill that scales sub-linearly with team size. The alternative — full coverage on every pull request — buys strictly earlier detection at a price that rises with headcount and a wait that erodes the gate's legitimacy. State the trade explicitly, and pick the split from your actual regression frequency and revert cost rather than from a preference for thoroughness. The teams that get this wrong usually do so by being too thorough at the merge point and then quietly disabling the whole thing three months later.