Why do MMLU, GSM8K and HumanEval no longer separate frontier models?

## What saturation means A benchmark discriminates when the models under test are spread out across its range. As scores approach the maximum, the spread collapses: every candidate answers the easy majority correctly, and the only items left are the hard tail — which, in an aged benchmark, overlaps heavily with the *broken* tail. MMLU, GSM8K and HumanEval have all reached that state. Frontier models cluster roughly between 88% and 99% depending on the suite, and the visible differences between them are within the range that dataset defects alone can produce. This is a **ceiling effect**, and it has a specific consequence: the benchmark's remaining resolution is spent on the wrong thing. If four percent of MMLU items are mislabelled or ambiguous — and audits of these older sets have found errors at roughly that scale — then no model can exceed about 96% honestly, and a model that scores 97% is being rewarded for reproducing the label errors, which is a memorisation signal, not a reasoning one. Ranking on the last two points therefore inverts what you wanted to measure. ## Why the residual errors stop being informative Three things pile up in an old benchmark's tail: - **Label noise.** Reference answers that are wrong, or defensible alternatives marked wrong. - **Ambiguity.** Questions with underspecified context where two readings both work. - **Format artefacts.** Items where the score depends on parsing, option ordering or answer formatting rather than the content of the response. None of these track capability. So the correlation between a benchmark's top-of-range score and any downstream ability weakens exactly where you are trying to read it. ## Contamination is the second half of the story Saturation and **contamination** are distinct but they arrive together. GSM8K, HumanEval and MMLU have been in public repositories, papers, blog posts and tutorials for years. Whatever filtering a lab does, the items and their solutions have been reproduced, paraphrased and discussed all over the open web. A high score can therefore reflect recall of a seen item rather than the ability the benchmark was designed to probe. The evidence for this is indirect but consistent: when researchers built fresh grade-school-maths items in the same style as GSM8K (the GSM1k work) and when they generated symbolic variants of the same problems (GSM-Symbolic), several models dropped noticeably relative to their GSM8K score — the signature of partial memorisation. ## What replaced them, and why The successor suites were built specifically to restore headroom, and they attack different axes: - **MMLU-Pro** keeps the multiple-choice knowledge format but widens the option set and filters for items that require reasoning rather than recall, pushing scores back down the range. - **GPQA-Diamond** uses graduate-level science questions written so that a non-expert with a search engine still fails them. - **ARC-AGI-2** targets abstract pattern induction that humans find easy and models find very hard, deliberately resisting a knowledge-recall shortcut. - **Humanity's Last Exam** aggregates expert-authored questions across many fields at the top of the difficulty range. - **SWE-bench Verified** (a human-validated subset of SWE-bench), **SWE-bench Pro** and **Terminal-Bench** moved the format from a single completion to multi-step agentic work in a real environment, where the score depends on end-state verification rather than string match. - **GDPval** goes further out, scoring deliverables on real occupational tasks by expert human comparison against human-produced work. Note the trend across that list: away from short recall items, toward tasks that are hard to memorise, verified on an end state, and closer to work someone would pay for. ## The practical lesson Benchmarks have a lifecycle. A new one is informative; a mature one is a summary statistic; a saturated one is a historical artefact. Two habits follow. First, when reading a score, look at where it sits in the range — a benchmark where every candidate is above ninety tells you nothing about which candidate to pick. Second, expect your own internal suite to saturate too. If every configuration you try passes 58 of 60 items, the suite has stopped doing its job and needs harder items drawn from where the system currently fails, not another round of tuning against a ceiling. The interview version of this answer is compact: saturation means no headroom, no headroom means the surviving gap is dataset noise plus memorisation, and that is why the field moved to harder, newer and more agentic suites — and why your own eval set needs the same treatment over time.