Large language models are trained on enormous scrapes of the public web. Public benchmarks live on the public web: in GitHub repositories, papers, blog posts, forum threads and leaderboard discussions. It would be surprising if none of that test material ended up in training data, and the evidence suggests a lot of it does.
Benchmark contamination (also called data leakage or test-set contamination) happens when a model has seen benchmark questions, answers or close paraphrases during training. The result is a score that measures recall instead of capability. This article explains how contamination happens, why it is hard to rule out, and four practical methods you can use to check for it.
Why contamination matters more than it seems
A contaminated benchmark does not simply add a few points. It changes what the number means. On a clean test, a model that scores 85% can solve roughly 85% of problems *like these*. On a contaminated test, it may solve far fewer new problems of the same kind, while still looking excellent on the published ones.
The effect is uneven, which makes it worse. Contamination tends to hit the most popular, oldest and most discussed benchmarks hardest, which are exactly the ones used in launch announcements. It also varies by model, so comparisons between models can be distorted in either direction.
A well-known demonstration came from researchers at Scale AI, who in 2024 wrote GSM1k, a new set of grade-school math problems designed to match the style and difficulty of the popular GSM8K benchmark. Several model families scored noticeably lower on the fresh problems than on the original set, a pattern consistent with overfitting to the public benchmark, while some frontier models showed little or no gap. The lesson is not that every score is fake; it is that you cannot tell which ones are inflated without checking.
How test data leaks into training
- Direct inclusion. Benchmark files are copied into public repositories and scraped.
- Paraphrase and discussion. Questions and worked answers appear in blogs, tutorials and Q&A sites.
- Synthetic data. Training data generated by other models may reproduce benchmark-like items those models memorised.
- Tuning on the test. Repeated prompt and hyperparameter selection against a public test set leaks information even if the items never enter the weights.
The last one applies to you as well. If your team tunes prompts against a fixed eval set for months, your own scores drift upwards for the same reason.
Method 1: canary strings
Some benchmark authors embed a unique canary string, a random identifier, in their data files and ask that documents containing it be excluded from training. BIG-bench popularised this. If you can prompt a model to complete the canary string, the data was very likely in its training set.
Canaries are cheap to test but limited: only some benchmarks have them, filtering is voluntary, and a model can see the questions through a blog post that never included the canary.
Method 2: n-gram overlap
If you have access to the training data, which is mostly true for open-data models and research settings, you can measure how many benchmark items share long word sequences with it. The GPT-3 paper, for example, flagged items with 13-word overlaps against the training corpus and reported results on clean and dirty subsets separately.
For closed models, you can approximate this with completion probes: give the model the first half of a benchmark question and see whether it reproduces the second half verbatim, or give it a question and ask for the multiple-choice options in order.
def completion_probe(model, item, split=0.5):
words = item.split()
cut = int(len(words) * split)
prefix, target = " ".join(words[:cut]), " ".join(words[cut:])
out = model.complete(f"Continue this text exactly:\n{prefix}", temperature=0)
# exact or near-exact continuation of an obscure item is a strong memorisation signal
return similarity(out[: len(target)], target)High similarity on many items is strong evidence of exposure. Low similarity does not prove the model is clean, because models can learn answers without memorising wording.
Method 3: perturbation tests
Change the surface of each benchmark item while keeping the underlying problem identical: rename entities, change numbers, reorder the options, rephrase the question. A model that genuinely solves the task should be nearly as accurate on the perturbed version. A large drop suggests the original score leaned on memorised surface features.
| Perturbation | Example | A large score drop suggests |
|---|---|---|
| Shuffle answer options | Move the correct answer from B to D | Memorised option positions |
| Change numbers | Replace 12 apples with 17 apples | Memorised final answers |
| Rename entities | "Alice" becomes "Priya" | Memorised specific items |
| Paraphrase the question | Same problem, new wording | Memorised phrasing |
Perturbation is the most practical method for teams without training-data access, and it doubles as a robustness check.
Method 4: fresh held-out sets
The strongest defence is a test the model could not have seen: written after the model's training cutoff, or written privately and never published. Some benchmarks now refresh questions on a schedule for this reason. For product decisions, your own private eval set built from real user tasks is the gold standard: it is fresh, relevant and invisible to training crawlers. Our guide to building an eval set in an afternoon shows how.
Keep two rules. Never paste the private set into public tools that may train on inputs. And retire items once they have been used heavily for tuning, replacing them with new ones.
A contamination checklist for your own evals
The same mechanisms that inflate public leaderboards can quietly inflate your internal numbers. Before you trust an internal score, run through this list:
- Where did the items come from? Items copied from public tutorials, documentation examples or popular datasets may already be in the model's training data.
- Who has seen the answers? If reference answers were drafted with the help of a model, that model and its relatives may share the same blind spots and agree with themselves.
- How often have you tuned against the set? Every prompt iteration evaluated on the full set leaks a little information. Keep a held-back slice that you only run when making a final decision.
- Are the items still representative? Products change. An eval set built a year ago may over-represent problems you no longer have and miss new ones.
- Do perturbed versions score the same? Run a quick perturbation pass on a sample. A big gap is a warning sign whether the cause is contamination or brittle prompting.
None of this requires access to training data, and an hour spent here can save a wrong model decision.
Reading published scores in light of contamination
When you read a model announcement or leaderboard:
- prefer results on benchmarks released after the model's training cutoff;
- look for reported decontamination methods in the model card (see reading a model card critically);
- distrust very large jumps on old, saturated benchmarks, especially when newer, harder benchmarks show smaller gains;
- treat small differences as noise unless intervals are reported; our guide to confidence intervals explains why.
Contamination is also one reason benchmarks saturate and need replacing, a topic we cover in what should replace a saturated benchmark.
Frequently asked questions
What is benchmark contamination in machine learning?
Benchmark contamination occurs when test questions, answers or close paraphrases appear in a model's training data. The model's score then partly reflects memorisation rather than its ability to solve new problems of the same type.
How can I tell if an LLM has memorised a benchmark?
Use completion probes (does it reproduce the rest of a question verbatim?), canary-string checks, perturbation tests that change surface details, and comparisons against a fresh held-out set written after the model's training cutoff.
Are all public benchmark scores unreliable?
No. Many scores are informative, especially on newer or regularly refreshed benchmarks. But the risk is highest on old, popular benchmarks, so use them for shortlisting and confirm decisions on private, task-specific evals.
Can contamination happen with my own eval set?
Yes, indirectly. Repeatedly tuning prompts and settings against the same fixed set overfits to it. Rotate items, hold back a portion you never tune on, and refresh the set as your product changes.
