A vendor memo ranks models for a permit-review assistant using SWE-bench Verified and Terminal-Bench — what is wrong, and what would you measure instead?

## Naming the defect **Construct validity** asks whether a measurement captures the thing you intend to measure. SWE-bench Verified measures whether an agent can resolve real issues in real code repositories, verified by running the repository's tests. Terminal-Bench measures whether an agent can complete multi-step tasks in a command-line environment. Both are good benchmarks. Neither measures: retrieving the correct section of a municipal building code, reading a zoning table, distinguishing a rule that applies to a detached garage from one that applies to an accessory dwelling, citing the section that governs, or refusing when the code is silent. So the memo's ranking is not wrong-in-detail, it is wrong-in-kind. It answers a question nobody asked. This is the single most common eval mistake in a procurement decision, and it survives because agentic benchmark scores *feel* more product-relevant than knowledge quizzes do. ## Is there any transfer at all? Be honest here rather than absolutist, because a good interviewer will push. Strong agentic coding scores do correlate loosely with instruction-following, long-context handling and multi-step reliability, all of which the permit assistant uses. Coding benchmarks are also among the few with objective end-state verification, so they are less noisy than most. But *loosely correlated* is not *sufficient for a ranking decision*, and the correlation is weakest exactly where these two models differ — that is what makes the top of any leaderboard a near-tie. The defensible framing: benchmark-to-product transfer is an empirical question. If you believe SWE-bench Verified predicts permit-answer quality, that is a hypothesis you can test by running both models on your own suite and checking whether the ranking survives. Usually it does not. ## What to measure instead Start from the failure modes the product actually has, then write items that expose them: - **Citation correctness.** Does the response point at the section that genuinely governs the applicant's situation? This is scoreable objectively against a reference section number. - **Fabrication.** Does the model invent a clause, a setback distance or a permit class that does not exist in the corpus? A single hallucinated code reference in a permit context is a compliance incident, not a quality ding. - **Safe abstention.** For questions the corpus does not answer — a jurisdiction you do not hold, a case requiring a variance — does the assistant escalate to a human reviewer or does it produce a confident guess? - **Case discrimination.** Near-miss pairs where two similar structures fall under different rules. These are where models separate. - **Grounded reasoning over tables.** Zoning and setback data live in tables; getting the right row and column is a distinct, testable skill. Run candidates through the **whole assistant** — the retrieval layer, the prompt, the citation formatter — not the bare model. A weaker model with better retrieval routinely beats a stronger model with worse retrieval, and a model-only comparison would attribute that entirely to the model. ## Why a small suite beats a big leaderboard here A few dozen carefully chosen permit cases carry far more decision value than any public benchmark, for one reason: every item is on the axis you care about. Public suites spend nearly all their resolution on capabilities you do not exercise. When two models sit within noise of each other on MMLU-Pro, GPQA-Diamond and LMArena — which top models routinely do — a targeted internal suite is the only instrument that can still tell them apart on the thing you are buying. Expect to find that one model reliably abstains and the other confabulates a plausible section number, a distinction that appears nowhere on any leaderboard and completely determines which one you can ship. One caveat to state out loud: a small suite resolves large differences, not marginal ones. If two models come out two items apart, treat that as a tie on quality and decide on cost, latency or operational factors instead of over-reading it. ## Where public benchmarks still belong in this decision Use them upstream. They cheaply eliminate models that are far off the frontier, and they surface newly released candidates worth testing. Occupational suites like GDPval, which score deliverables on real professional tasks against human-produced work, sit closer in spirit to a permit assistant than a coding benchmark does — but still not on your corpus, your jurisdiction or your citation rules. The funnel is: leaderboards narrow the field, your suite decides, and the suite is the artifact you re-run when the vendor ships the next model. ## How to say it in an interview Name the concept (construct validity), say concretely what the cited benchmarks measure and what the product needs, concede the partial transfer honestly, then describe the replacement measurement in terms of scoreable failure modes. Candidates who jump straight to "just build an internal eval" without diagnosing *why* the memo's evidence is invalid give a much weaker answer.