For a few years, a handful of benchmarks defined progress in language models. A new model would be announced with its score on general knowledge, grade-school maths and short coding problems, and those numbers told you roughly where it stood. Then scores crowded into the high 80s and 90s, the differences between leading models shrank to a point or two, and the benchmarks stopped telling anyone much.
This is benchmark saturation: the point at which a test no longer separates strong systems from each other. This article explains why benchmarks saturate, how to tell a benchmark has stopped being useful, what kinds of evaluation are replacing saturated ones, and what this means for teams choosing models.
Why benchmarks saturate
Three forces drive saturation, usually together.
Models genuinely improve. A benchmark designed to be hard for 2020-era models may simply be easy for later ones. That is success, not failure.
Contamination. Popular benchmarks spread across the web and into training data, inflating scores beyond real capability; see benchmark contamination: how to check whether a model has seen the test.
Optimisation pressure. When a benchmark becomes the headline number, model developers tune towards it, consciously or not: training data choices, prompt formats and evaluation settings drift towards what scores well. This is Goodhart's law: a measure that becomes a target stops being a good measure.
There is also a ceiling effect from label noise. Every large benchmark contains some questions with wrong or ambiguous reference answers. Gema et al. (2024), "Are We Done with MMLU?", manually re-annotated a sample of the widely used MMLU benchmark and found a notable share of questions with errors, with error rates much higher in some subject areas. Once models approach the noise ceiling, further score gains are meaningless.
Signs that a benchmark has saturated
- Top scores cluster within a few points of each other and of the apparent ceiling.
- Differences are smaller than the confidence interval for the benchmark size; see why your 2-point improvement might be noise.
- Small models score close to large ones, while on harder or newer tests the gap is large.
- Scores stop correlating with user experience: models that rank differently in real use score the same.
- Remaining errors are mostly label noise or ambiguous questions.
What replaces saturated benchmarks
The field has responded in several ways, each with different strengths.
| Approach | Example | Strength | Weakness |
|---|---|---|---|
| Harder versions of the same task | MMLU-Pro, with more options and harder questions | Continuity with the original | Saturates again, eventually |
| Expert-written, search-resistant questions | GPQA (Rein et al., 2023), Humanity's Last Exam (2025) | Very hard; tests deep expertise | Expensive to build; narrow coverage |
| Real-world task benchmarks | SWE-bench (Jimenez et al., 2023): fixing real GitHub issues | Closer to actual work | Complex harnesses; environment noise |
| Regularly refreshed benchmarks | Benchmarks that add new questions over time, such as LiveBench | Resists contamination | Comparisons across time are harder |
| Human preference arenas | Large-scale pairwise votes by users | Captures overall preference | Biased towards style and length; not task-specific |
| Novel reasoning tests | ARC-AGI (Chollet, 2019) and its successors | Targets generalisation to new problems | Narrow puzzle format |
| Agentic benchmarks | Multi-step tasks with tools and environments | Tests what agents actually do | Costly, high variance, hard to reproduce |
Two trends stand out. First, a shift from knowledge recall to doing work: completing software tasks, operating tools and navigating environments. Second, a shift from fixed tests to living evaluations that refresh, add tasks or rely on new human judgements to stay ahead of training data.
The limits of the replacements
Every successor has costs. Hard expert benchmarks are small, which makes their scores noisy. Real-world task benchmarks depend on complex environments where infrastructure problems can look like model failures. Preference arenas reward responses that people like, which is not the same as correct. Agentic benchmarks vary widely between runs; reliability measures such as pass^k help, as discussed in tool-use evals are broken.
And every public benchmark will eventually face the same pressures that saturated the old ones. That is not a reason for cynicism; it is a reason to treat public benchmarks as a moving, imperfect signal.
What this means for choosing models
- Ignore saturated benchmarks when comparing frontier models. A one-point gap on an old test is not evidence.
- Look at benchmarks close to your task (coding, retrieval, tool use, a specific domain) and check their age and contamination risk.
- Read the settings. Harder benchmarks are especially sensitive to reasoning settings, number of attempts and scaffolding; see how to read a model card critically.
- Trust trends across several benchmarks more than any single number.
- Build a private eval. Your own task set cannot saturate in the same way, because you can always add harder cases from real usage, and it cannot be contaminated by public training data.
How to read a new benchmark announcement
New benchmarks appear constantly, each claiming to measure what older ones miss. A few questions help you decide how much weight to give one:
- Who built it, and do they also build models? Developer-built benchmarks are not invalid, but check whether their own models are compared under the same conditions as others.
- How large is it? A benchmark with a few hundred questions has wide confidence intervals; small differences between models on it are noise.
- How are answers graded? Exact-match grading is reproducible but brittle; model-graded benchmarks inherit the grader's biases.
- Is there a private test set? Benchmarks that keep part of their data private are more resistant to contamination.
- Does it correlate with something you care about? A benchmark that tracks real usage in your domain is worth more than a harder but irrelevant one.
Designing your own eval so it does not saturate
Your internal eval set can saturate too: once your system passes 98% of it, it stops guiding decisions. Keep it useful:
- Add every production failure as a new case.
- Retire easy cases that every candidate passes, or keep them as a separate regression suite.
- Maintain a "frontier" slice of hard cases that current systems fail, and track progress on it separately.
- Refresh periodically with new real inputs, so the set reflects current usage.
- Measure consistency, not just single-attempt success: requiring several successful runs per case raises the bar without new questions.
Scaling laws suggest models will keep improving, with diminishing returns per unit of compute; see scaling laws in plain English. That guarantees today's hard benchmarks will be tomorrow's saturated ones. Plan for it.
Frequently asked questions
What does benchmark saturation mean in AI?
A benchmark is saturated when leading models score so close to its maximum, or to each other, that the remaining differences are within noise. At that point it no longer distinguishes stronger systems from weaker ones.
Why do AI benchmarks become saturated?
Models improve, benchmark data leaks into training sets, developers optimise towards headline benchmarks, and label errors put a ceiling on meaningful scores. Together these push scores to the top of the range.
Is MMLU still a useful benchmark?
For comparing frontier models, much less than it was: top scores are close together and the benchmark contains a share of erroneous questions. It can still be useful for evaluating smaller models, alongside harder successors such as MMLU-Pro.
How do I keep my internal eval from saturating?
Continuously add real failure cases, retire cases every system passes, keep a separate slice of hard cases, and measure consistency across multiple runs rather than single attempts.
