How much weight should an engineering org give vendor-reported benchmark scores?
answer
- shortlist, not decision record
- reported by the party selling it
- scaffold and effort setting undisclosed
- cost and latency appear nowhere
- capability class yes, ranking no
basics
~20 sEnough to build a shortlist, never enough to close a decision. Reported scores come from the party selling the model, often under undisclosed scaffolding and effort settings, and they omit cost, latency and reliability. Internal suites hold the deciding vote.
solid answer
~50 sTreat public and vendor-reported scores as a **prefilter with a shelf life**. They are cheap, they show where the frontier has moved, and a model far off the pack can be eliminated without further work. What they cannot do is settle a choice between near-tied frontier models, for structural reasons: the number is reported by the seller; agentic benchmarks depend heavily on undisclosed scaffolding, retries and tool access; reasoning-effort settings move both score and cost by large factors and are frequently unstated; subsets and run counts can be chosen after the fact; and preference rankings reward style. So the org-level policy is: shortlist from public scores, require that any quoted figure carry the harness, the effort setting, the date and the cost per task, and give the final vote to an internal suite per product surface. That suite is the durable asset — it outlives every model version and turns each new release into a re-run rather than an argument.
go deeper
Know that benchmark scores in a vendor's material are marketing-adjacent and should send you to test the model yourself rather than settle the choice.
Be able to list what a reported score omits — harness, effort setting, run count, cost, latency — and explain why two published numbers for the same model on the same benchmark can legitimately differ.
Show you re-run open benchmarks under one harness where feasible, insist on cost-paired quality numbers, and can defend a model choice with evidence from your own suite rather than a leaderboard screenshot.
Own the evidentiary standard for the organisation: what may be cited, what must be re-measured, who owns the per-surface suite, and how model migrations stay cheap so no vendor's scores can lock you in.
## Why this is a policy question, not a measurement question Any individual engineer can be careful about a benchmark score. What breaks organisations is the *default*: a vendor deck lands, a number is quoted in a planning doc, and three months later the model choice is treated as settled because it was written down. Setting a weight policy is about making the evidentiary standard explicit before the deck arrives. ## Where reported numbers lose their grip **Reporting incentives.** The scores that reach a procurement conversation are usually produced and selected by the party selling the model. Nothing improper needs to happen for this to bias the picture: selective publication alone does it, because a lab runs many internal variants and publishes the ones that look good. The critique published as "The Leaderboard Illusion" made this concrete for community arenas — private pre-testing of multiple variants with selective disclosure, plus uneven sampling and model deprecation, systematically advantage the labs that can afford to play. **Undisclosed scaffolding.** On agentic benchmarks, the harness matters as much as the model: how many attempts, which tools were exposed, whether a scratchpad or a retrieval step was allowed, how the end state was verified. Two reported SWE-bench Verified numbers for the same model can differ substantially on scaffolding alone. A score without its harness described is not a comparable measurement. **Effort and cost settings.** Providers now expose a reasoning-effort dial, and thinking tokens can balloon several-fold between its low and high settings. A headline score achieved at maximum effort and a production deployment at a moderate setting are not the same system, and the price difference can be an order of magnitude. Any quoted quality number without its effort setting and cost per task is half a datum. **Per-benchmark tuning.** Prompts, formatting and output parsers can be tuned per benchmark. That is legitimate engineering for a benchmark run and completely non-transferable to your product, where nobody will hand-tune the prompt per user request. **Style, not substance, in preference rankings.** Community preference rankings measure what raters prefer, which is entangled with formatting, length and tone. Ranking movements driven by presentation are real for chat products and misleading for a system whose output is a cited answer or a structured record. **What is missing entirely.** No leaderboard reports your cost per request at your token profile, your p95 latency, provider rate limits, deprecation history, data-handling terms, or how the model behaves on your corpus. Those routinely dominate the decision. ## A workable policy 1. **Shortlist from public scores; decide from internal suites.** Public numbers determine which two or three models are worth integrating. A per-surface internal suite determines which one ships. Write that split down so it is not renegotiated per project. 2. **Set a citation standard.** Any external number entering a decision doc carries: benchmark and version, harness or scaffold description, effort setting, run date, and cost per task. Numbers that cannot carry that are marked as directional, not evidentiary. 3. **Re-run what you can.** For open benchmarks, running the candidates yourself under one harness removes most of the comparability problem, at modest cost. Where re-running is infeasible, prefer benchmarks with private held-out splits. 4. **Discount by age and popularity.** Older, heavily cited suites carry the most saturation and contamination risk. Weight recent and agentic or occupational suites higher for agent-shaped products, while remembering they still are not your task. 5. **Fund the internal suite as infrastructure.** It is the only asset that survives model deprecations, price changes and provider outages. Each new release becomes a re-run with a number, instead of a debate. Owning that is the difference between an org that can switch models in a week and one that cannot switch at all. 6. **Report quality against cost, never alone.** The decision is a point on a quality-cost frontier. A model two points better at four times the price is a different decision at ten requests a day than at ten million. ## Being honest about the contested part There is no settled formula for how much weight leaderboards deserve, and a good candidate says so. The defensible position is directional: public scores are strong evidence about *capability class* — whether a model can plausibly do multi-step tool use at all — and weak evidence about *ranking within a class*. Buy the capability-class signal, ignore the ranking, and let your own data break the tie. State that as a judgment with a rationale rather than pretending consensus exists.
- Two frontier models are within a point of each other on every public benchmark. What decides it?Your own suite first; if that comes out a tie too, then the operational axes — cost per request at your real token profile, p95 latency including thinking time, rate limits and burst headroom, deprecation and versioning history, data-handling terms, and how painful a future migration would be. Treat near-tied quality as genuinely tied and optimise the things you can measure precisely.
- When is it reasonable to act on a public benchmark result alone?When the signal is about capability class rather than ranking, and the stakes of being wrong are low. A large jump on an agentic or occupational suite is good reason to spend a day testing a new model, and a model far below the pack can be eliminated without further work. What you should never do on a public score alone is pick between near-tied candidates or justify a migration.
- What does requiring cost per task alongside a quality score actually prevent?It prevents comparing a maximum-effort benchmark configuration against a production-grade deployment as if they were the same system. Reasoning effort can multiply token spend several-fold for a few points of quality, so a score reported without its effort setting and price is not actionable. Pairing the two also forces the discussion onto the quality-cost frontier, which is where the decision actually sits.
saying these in an interview costs you the question
- Treats a vendor deck's benchmark table as a decision record
- Compares agentic scores without asking what scaffolding produced them
- Ignores that reasoning-effort settings change both score and cost
- Assumes preference-arena movement reflects task correctness
- Picks a model on quality alone without cost or latency