Benchmarks, Contamination, and Capability Claims

The layers address different ways a benchmark can mislead — a protected task set, behavioral contrasts, slice analysis, workflow evaluation, production follow-up. The first is the one most often asserted and least often specified.

FrontierMath is the worked case. Existing models solved under 2% of its problems, the benchmark paper reported in 2024. That is exactly the property that makes a benchmark worth protecting.

Then the arrangement behind it became public. An o3 result on FrontierMath had been announced, and on 23 January 2025 Epoch AI published a clarification. “OpenAI commissioned Epoch AI to produce 300 advanced math problems”, it said, and “OpenAI retains ownership of these questions and has access to the problems and solutions, with the exception of a holdout set”. Tamay Besiroglu and Jaime Sevilla described what was being carved out: “We are finalizing a 50-problem set for which OpenAI will only receive the problem statements and not the solutions. This allows us to independently test OpenAI and other AI models on solutions no AI developer has access to.”

On 19 January 2025 TechCrunch had reported Epoch's lead mathematician, Elliot Glazer, saying Epoch “can't vouch for” OpenAI's results until its own evaluation was complete.

Notice what the top layer becomes once it is written down as numbers. 300 problems visible to the developer being scored. 50 statements-only. Named parties who know which is which. That is a boundary a reader can reason about. “Access-controlled”, published without those figures, is a statement of intent rather than a control. And the remaining layers — contrast pairs, slices, the whole workflow, production logs — exist because even a correctly protected set answers only the question it was built to ask.

Contamination, prompt overfitting, format exploitation and population mismatch can coexist, so fixing one is rarely enough. Two of the four have been measured directly. The measurements are what turn the mitigations into consequences rather than housekeeping.

Contamination. Hugh Zhang and colleagues at Scale AI stated the worry precisely: “However, there is growing concern that some of this performance actually reflects dataset contamination, where data closely resembling benchmark questions leaks into the training data, instead of true reasoning ability.” Then, in 2024, they bought a control. They commissioned GSM1k — “Grade School Math 1000”, in fact 1,250 new grade-school math problems — matched to GSM8k on human solve rates, number of steps in solution and answer magnitude. On the fresh set accuracy fell by up to 8%. And the size of a model's GSM8k–GSM1k gap tracked its probability of generating a GSM8k example, at Spearman r² = 0.36. String matching against a training corpus never produced that number. A matched replacement benchmark did.

Prompt overfitting. Melanie Sclar and three co-authors varied only formatting choices that preserve the meaning of a prompt, and reported this in 2024: “We find that several widely used open-source LLMs are extremely sensitive to subtle changes in prompt formatting in few-shot settings, with performance differences of up to 76 accuracy points when evaluated using LLaMA-2-13B.” Format performance was only weakly correlated between models, so the best format for one model is not the best format for another. Their method, FormatSpread, exists so that a paper can report a range instead of a single-format score. A leaderboard row is one draw from a spread that can be that wide. That is why the final protocol has to be locked and the prompt search effort disclosed.

Format exploitation and population mismatch do not yet have one canonical number each in this lesson. They have the same remedy shape: change one thing you believe is decorative, and see whether the score moves.

The exam was never re-run. Only the comparison was.

A 2024 re-analysis in the journal Artificial Intelligence and Law opens on the claim it is about to move: “Perhaps the most widely touted of GPT-4’s at-launch, zero-shot capabilities has been its reported 90th-percentile performance on the Uniform Bar Exam.” Its author is Eric Martínez.

The paper leaves the score alone and changes only the reference population. Nothing about the model differs across the figures below. Only the question “compared with whom?” does. Against a July administration the result lands well below the headline. Against first-time takers it moves again. Against the candidates who actually passed, the essay percentile collapses. Population mismatch is not a caution in this section. It is a four-row table derived from one script.

Then there is everything the exam never sampled at all. Real intake records arrive as scanned forms with OCR errors and handwritten corrections. Local rules change after a benchmark is written. Staff need supported issue spotting rather than one chosen option. The exam rewards choosing an option when evidence is incomplete, where the desk needs abstention. And retrieval, citations, privacy and escalation decide whether an output is usable.

Build the evaluation before you pick a model, or every result turns into advocacy. Write the claim. Protect a final set. Add contrast cases. Check contamination. Report uncertainty and slices. Recreate the workflow.

Step 3 has a published recipe with a number attached to it. Six researchers at Apple generated GSM-Symbolic from symbolic templates, so that a question could be reissued with only its numbers changed. Every model they tested declined on that change alone. Then they built GSM-NoOp: into GSM8k problems they inserted a single clause that looks relevant and does no work. Their 2024 paper reports what that cost. “Adding a single clause that seems relevant to the question causes significant performance drops (up to 65%) across all state-of-the-art models, even though the clause doesn't contribute to the reasoning chain needed for the final answer.”

That is the entire design of a contrast case, and it is the cheapest diagnostic on the list. Hold the surface form. Move one decisive fact. Read the difference. A model that keeps its answer when the numbers move, and loses it when a decorative clause arrives, has told you which of the two it was tracking.

The other steps set the conditions under which that reading is trustworthy. A set whose access boundary is written down as figures rather than adjectives. A locked protocol, reported alongside the prompt search behind it. Confidence intervals and a failure taxonomy rather than a single mean. And a final measurement of the assembled application, with its tools, retrieval, latency and fallback in place.

No single benchmark captures instruction following, factual support, robustness, multilingual behavior, tool use, safety and product outcomes. A strong evaluation program uses several instruments with explicit boundaries.

There is an enforceable version of that standard. DoNotPay was marketed as “the world's first robot lawyer”, and the FTC charged it under Section 5 of the FTC Act. The analysis of the proposed consent order, published in the Federal Register on 30 September 2024, records: “The proposed complaint alleges that the Service was not designed to operate like a human lawyer, and that the company's claims were false, misleading, or unsubstantiated.” The Commission also alleged that “DoNotPay employees had not even tested the quality and accuracy of the legal documents and advice generated by the service”.

The same document sets the price: “Provisions III requires DoNotPay to pay the Commission $193,000 in monetary relief”. It also requires notice to 2021–2023 subscribers, and bars representing that the service “operates like a human lawyer” without competent and reliable evidence. The FTC announced the finalized order on 11 February 2025.

Competent and reliable evidence is the same object this lesson has been assembling piece by piece. A stated population rather than a bare percentile. An error bar on the instrument itself. An access boundary published as numbers. Contrast cases that move one decisive fact. A measurement of the whole workflow instead of one exam.

The next module begins post-training. Those techniques can change benchmark behavior. They can also overfit to evaluator preferences and hide weaknesses outside the measured distribution.