Skip to content
AI.info

Evaluation

Benchmark Design, Contamination, and Saturation

Learn how to construct, interpret, and retire benchmarks while managing contamination, annotation quality, saturation, and leaderboard adaptation.

By the end you can

A leaderboard can become a target rather than a test

Once a benchmark becomes prestigious, teams adapt to it. Models, prompts, preprocessing and data collection all bend toward its quirks. The number stays consistent. What it measures drifts. Good benchmark governance treats the dataset, the scoring code, the submission history and the known exposure as parts of one instrument.

GLUE showed how fast that can happen. SuperGLUE arrived in 2019, “a little over one year” after GLUE, because performance on the older benchmark “has recently surpassed the level of non-expert humans, suggesting limited headroom for further research”. The new benchmark kept GLUE's styling and raised the difficulty of the tasks: “a new benchmark styled after GLUE with a new set of more difficult language understanding tasks”. The instrument had not broken. It had been solved.

A benchmark can be valid at launch and weak after years of optimization.

Visual

The parts of a benchmark instrument

Each layer can introduce uncertainty or bias, and each one is attacked separately in what follows. The reference process is attacked by the label audits of ImageNet and MMLU. The sampling frame is attacked by the teams who went out and collected fresh test items. The scoring protocol is attacked by the arithmetic of a leaderboard read down to a tenth of a point. A benchmark rarely fails as a whole. It fails at one layer, and the single published number carries the failure forward unmarked.

FigureHierarchy · 5 levels

  • Construct definition

    The capability or behavior the benchmark intends to measure.

    • Task design

      Inputs, outputs, instructions, context, and allowed tools.

      • Reference process

        Labels, adjudication, ambiguity handling, and quality checks.

        • Sampling frame

          Domains, languages, difficulty, recency, and exclusions.

          • Scoring protocol

            Metrics, aggregation, uncertainty, submissions, and hidden tests.

Comparison

Contamination has several forms

Exposure can be direct or indirect. Direct overlap is not a hypothetical that careful laboratories avoid. The best-documented case is a lab reporting it about itself. OpenAI's de-contamination filter for GPT-3 failed, and the paper announcing the model says so, in the section headed “Measuring and Preventing Memorization Of Benchmarks”: “Unfortunately, a bug in the filtering caused us to ignore some overlaps, and due to the cost of training it was not feasible to retrain the model.” The 175-billion-parameter model was too expensive to train again. The affected results were flagged with an asterisk rather than removed; PIQA and Winograd were kept and asterisked. The evaluation shipped contaminated, in the open, in the paper that announced the model.

Near-duplicate exposure and provenance checking sound like housekeeping until someone counts. Dodge and colleagues counted, in 2021. Their subject was C4, the web corpus used to pre-train T5. Their abstract states the finding plainly: “Then we explore the content of the text itself, and find machine-generated text (e.g., from machine translation systems) and evaluation examples from other benchmark NLP datasets.” Table 2 puts numbers on it — “The number of exact matches from test sets of various benchmarks in C4.en”. Exact test-set matches ran to 1.87% of TIFU-long target texts, 15.49% for XSum and 24.88% for TIFU-short. For GLUE inputs the range runs from 1.8% of QNLI questions to 53.6% of QNLI sentences. More than half of one GLUE component's test inputs were sitting inside a public pre-training corpus.

Protocol adaptation needs no item overlap at all. Teams optimize to known task formats and scoring rules. The gains accumulate over years of submissions, and nothing in the dataset has been touched. That is why provenance matching, the only check with a hard number attached, cannot clear a benchmark on its own.

FigureComparison · 3 columns

Direct overlap

Exact benchmark items appear in training or tuning data.

  • Enables memorization
  • Can arise from web crawls
  • Detected through provenance or matching
  • Invalidates clean generalization claims

Near-duplicate exposure

Paraphrases, templates, translations, or derived examples appear elsewhere.

  • Harder to detect
  • Can preserve answer cues
  • Inflates familiar formats
  • Needs semantic and provenance checks

Protocol adaptation

Teams optimize to known task formats and scoring rules.

  • No item overlap required
  • Can exploit metric weaknesses
  • Often accumulates over time
  • Reduces transfer evidence

Example

Signs that a benchmark is saturated

Saturation is not only a high average score.

  • Leaderboard compression: Many systems differ by less than the uncertainty of the estimate, so the published order is finer than the evidence behind it.
  • Ceiling concentration: Remaining errors are dominated by ambiguous, mislabeled, or out-of-scope items. The audited floor is an average of at least 3.3% label errors across ten standard test sets, and at least 6% of the ImageNet validation set.
  • Protocol gaming: Small formatting tricks improve scores without improving the intended capability.
  • Weak transfer: Ranking on the benchmark does not predict results on fresh tasks or deployments. Rebuilt test sets cost models 11–14% accuracy on ImageNet, and freshly commissioned grade-school arithmetic cost up to 8 points.
  • Repeated exposure: Public examples, solution discussions, and generated training sets circulate widely. Exact GLUE test inputs turned up inside C4 at rates as high as 53.6%.

Analogy

A lock used to certify every locksmith

Ten years of certifying locksmiths against the same public lock does two things at once. Skill improves. Participants also learn that lock’s exact tolerances and common shortcuts.

The first is real, which is what makes the second so hard to see. What ten years of rising scores no longer supports is a claim about locks nobody has opened yet.

In 2024 a team had that lock cut fresh. GSM1k is a set of new grade-school maths problems, matched to GSM8k on human solve rate, number of solution steps and answer magnitude — the same specification, never published for training. Zhang and colleagues commissioned it, and their abstract reports what happened: “When evaluating leading open- and closed-source LLMs on GSM1k, we observe accuracy drops of up to 8%, with several families of models showing evidence of systematic overfitting across almost all model sizes.” And the drop was not random. Across models, a Spearman r² of 0.36 links a model's probability of generating a GSM8k example to the size of its GSM8k-to-GSM1k gap. The more of the old lock a model had memorised, the worse it did on the new one.

Mastering one instrument is not identical to mastering the underlying capability.

Key idea

Small leaderboard gains need context

A difference of 0.3 points can be meaningful on a large, stable benchmark. It can also be meaningless under label ambiguity and correlated items. So report confidence intervals, paired comparisons, subgroup changes and compute cost — and whether the gain survives a fresh set.

The same discipline applies to gaps far larger than 0.3, including the ones that look like proof. Models lost 11–14% accuracy against a rebuilt ImageNet test set, and the gap itself then became the claim under examination. Engstrom and colleagues showed in 2020 that the standard way of replicating a dataset introduces statistical bias of its own: “We show that after remeasuring selection frequencies and correcting for statistical bias, only an estimated 3.6% of the original 11.7% accuracy drop remains unaccounted for.” The preprint carries the intervals: 3.6% ± 1.5% of an 11.7% ± 1.0% drop. Two thirds of a headline result was a property of how the second test set had been assembled. Finding that out required modelling the measurement process rather than the models.

The best-performing system may also be worse for the deployment if it sacrifices latency, calibration, robustness, or maintainability.

Leaderboard order is not a complete decision rule.

Case

The label-error floor under ten standard benchmarks

Ten of the most commonly used vision, language and audio datasets carry errors in their own test labels. Northcutt and colleagues audited “the test sets of 10 of the most commonly-used computer vision, natural language, and audio datasets” in 2021. They “estimate an average of at least 3.3% errors across the 10 datasets”. For ImageNet, label errors make up at least 6% of the validation set. A 0.3-point difference is small next to that.

The floor did not stay in 2021, and it did not stay in vision. Gema and 15 co-authors re-annotated MMLU by hand and published the result in 2025. The ground truth of a benchmark quoted in every model release turned out to be wrong at a measurable rate: “For example, we find that 57% of the analysed questions in the Virology subset contain errors,” their abstract says. Across the benchmark they estimate that 6.49% of MMLU questions contain errors. Their corrected subset, MMLU-Redux, comprises 5,700 manually re-annotated questions across all 57 subjects. Those are the reported scores' own units. A subject where more than half the analysed items are defective still contributes to the single MMLU number that gets compared between models at one decimal place.

Position

A ranking is only as fine as the labels underneath it

The tenths on a leaderboard rest on labels that are themselves wrong at a measurable rate. Northcutt and colleagues audited the test sets of ten of the most commonly used vision, language and audio datasets and “estimate an average of at least 3.3% errors across the 10 datasets”. For ImageNet, label errors make up at least 6% of the validation set. Crowdworkers confirmed 51% of the algorithmically flagged candidates as genuinely mislabelled. Leaderboards built on those datasets are still read down to a tenth of a point.

The order does not merely wobble under that pressure. It inverts. “For example, on ImageNet with corrected labels: ResNet-18 outperforms ResNet-50 if the prevalence of originally mislabeled test examples increases by just 6%,” their abstract reports. On CIFAR-10 the same effect puts VGG-11 above VGG-19 at just 5%. In both cases the smaller, older model wins, and it wins without changing. What changed was the test set being right.

The honest version of this argument is narrower than the cynical one. A difference of 0.3 points can be meaningful on a large, stable benchmark. Under label ambiguity and correlated items it can be meaningless. What the ranking cannot do is tell a reader which of the two cases they are in, and that burden belongs to whoever wants to make the claim. GLUE is the other half of it. Within a little over a year of launch, performance on the benchmark had surpassed the level of non-expert humans. The response was a harder benchmark, not an announcement that language understanding was finished. Position on a leaderboard is evidence about submissions to one instrument. A claim about capability costs more than that, and usually means a fresh set.

A rank gives the order. It does not give the size of the gap.

Steps

Maintain a living benchmark program

Refresh evidence without destroying comparability. The steps below read as good practice, which is how a program of this kind usually survives — until someone stops paying for it. One jurisdiction has converted parts of it into an obligation. The EU AI Act, published in the Official Journal of 12 July 2024, puts the development of the instruments themselves on the Commission. Article 15(2) reads: “To address the technical aspects of how to measure the appropriate levels of accuracy and robustness set out in paragraph 1 and any other relevant performance metrics, the Commission shall, in cooperation with relevant stakeholders and organisations such as metrology and benchmarking authorities, encourage, as appropriate, the development of benchmarks and measurement methodologies.” Article 15(3) then moves the number itself out of the leaderboard. The accuracy levels and the relevant accuracy metrics of a high-risk AI system must be declared in its instructions for use. Under that rule an accuracy figure travels with the product, in writing, to whoever deploys it. A declared metric is a claim someone can hold you to. A rank can quietly stop being updated.

FigureProcess · 5 steps

  1. 1. Track exposure

    Record public release, suspected training overlap, submission count, and adaptation history.

  2. 2. Audit references

    Re-adjudicate ambiguous items and measure annotator disagreement.

  3. 3. Add fresh cohorts

    Collect new domains, periods, languages, and failure modes under the same construct.

  4. 4. Preserve anchors

    Retain a limited stable subset to measure longitudinal change.

  5. 5. Retire claims

    Stop using saturated scores as broad evidence when transfer no longer holds.

Benchmarks should predict something beyond themselves

A strong benchmark has external validity. Improvements on it should correspond to better behavior on fresh examples, related tasks, or operational outcomes.

That is a testable proposition, and it has been tested at the cost it deserves. Recht and colleagues rebuilt the CIFAR-10 and ImageNet test sets by following the original collection procedures. Then they ran the existing models against images none of them had ever seen: “We evaluate a broad range of models and find accuracy drops of 3% - 15% on CIFAR-10 and 11% - 14% on ImageNet,” their 2019 abstract reports. No model changed. Only the test set was new. How much of that gap is a real generalization failure remains contested — Engstrom and colleagues later attributed most of it to bias in the replication procedure itself — but the contest is being conducted in fresh data and explicit error bars, which is the standard.

When the relationship between benchmark score and behavior beyond the benchmark disappears, the remedy is not another decimal place. The measurement program needs new evidence, collected the way those teams collected it.

The value of a benchmark lies in the claims it predicts, not the attention it attracts.

Key takeaways