When does an agent eval task need a judge model instead of a programmatic verifier?

## The verifier is the task A harness answers one question per rollout: did the agent succeed? Everything else — fixtures, sandboxes, tool fidelity — exists to make that question answerable repeatedly. So the verifier is not a detail bolted on afterwards; it constrains which tasks belong in the suite at all. A useful discipline is to write the verifier first. If you cannot state success as a check, either the task is under-specified or it does not belong on the deterministic tier. ## Programmatic verifiers A programmatic verifier is code that inspects the end state or the produced artifact. - **End-state assertions.** After the rollout, query the fixture: does the refund row exist, with the right amount, status and audit entry, and were no other orders touched? The last clause matters — a good verifier checks both the intended change and the absence of collateral damage. - **Executable checks.** Run the repository's test suite, compile the patch, validate the emitted JSON against a schema, execute the generated SQL. These are strong because the check is independent of how the agent phrased anything. - **Deterministic string or structural matching.** Weakest form, and brittle for agent output; prefer parsing to matching. *Why they are the default.* They cost essentially nothing, so they scale with k. They are stable: the same end state scores the same way on every rerun, which means a change in the score is a change in the agent, not in the grader. And they are path-agnostic — the agent may reach the correct end state by a different route than your reference, and a state assertion still passes where a path comparison would not. *Their limits.* They only see what they were told to look at. An agent can satisfy the assertions and still have done something unacceptable that no assertion covers, so state checks need to include negative conditions. And many real deliverables genuinely have no canonical form: a drafted apology email, a triage summary, an explanation of a policy decision. ## Judge models A judge model reads the agent's output against a rubric and scores it. It is the right tool when the artifact is open-ended and quality is the thing being measured. The costs are concrete and often understated: - **Spend and time.** Every rollout invokes the judge; k repeats multiply that. On a suite of 120 tasks at k=5, that is 600 judge calls per run, on top of the agent's own tokens. This is frequently what pushes a suite past its CI budget, and it is the main reason the pre-merge tier tends to be programmatic-only. - **A second variance source.** The judge samples too. Two identical agent outputs can receive different scores, which muddies exactly the reliability signal pass^k exists to measure. Judges are usually run at low temperature with a tight rubric for this reason. - **Unvalidated scores are not evidence.** A judge is a measuring instrument and must be calibrated against human labels on a sample before its numbers are used to make decisions, then re-checked when you change the judge model or the rubric. A judge nobody has checked can be confidently wrong for months. ## The hybrid pattern In practice most tasks decompose. "Handle this return request" has hard requirements (a refund of exactly this amount was issued, the return label was created, no other order was modified) and soft ones (the customer message was accurate, apologetic and did not promise anything outside policy). Assert the hard requirements in code as a gate; if the gate fails, the task fails and you never pay for a judge call. Score the soft residue with a judge only on the tasks that passed the gate. This keeps the expensive, noisy grader off the majority of rollouts and keeps the pass/fail decision anchored on something deterministic. A second useful split is by tier: programmatic-only on the fast pre-merge suite, judge-assisted on the nightly run where the budget is larger and latency does not block a developer. ## Designing tasks to be checkable Much of the skill is upstream. Prefer tasks whose goal terminates in a state change over tasks whose goal is a conversation. Specify the target precisely enough that a check can exist ("refund the full amount and set status to REFUNDED", not "make the customer happy"). Where a free-text artifact is unavoidable, extract the checkable parts — did the message contain the correct order number and refund amount — and judge only what remains. ## How to answer Lead with the default: code, because it is free, stable and path-agnostic, and it is why test-based coding suites work. Name what code cannot see, and admit the judge's three costs — spend times k, its own variance, and the need for validation against human labels. Then give the hybrid gate as the design you would actually ship. The weak answer defaults to a judge for everything because it is easy to write, and never mentions that the grader itself is an unvalidated model.