The Pulse
whatmodel.sh Finds AI Labs’ Scores 6.4% Higher on Average
An analysis on whatmodel.sh compares 611 matched AI benchmark results across 180 models and 103 tests. It finds lab-reported scores average 6.4% above independent results, while warning that different test settings and selective reporting l
AI.info Team · Written with AI · fact-checked ·
AI labs’ benchmark scores average 6.4% higher than independent results for the same models on the same tests, an analysis published on whatmodel.sh finds. The comparison covers 611 matched pairs across 180 models and 103 benchmarks. Its headline number is a relative difference, not a gap of 6.4 percentage points on benchmark score scales.
What the 6.4% measures
For each pair, the analysis divides the difference between the lab’s score and the independent score by the independent score, then averages those percentages, giving every pair equal weight. The calculation includes ties and cases where the lab score is lower. Its median relative difference is 1.2%, while the mean’s 95% interval runs from 3.8% to 9.5%; the interval resamples whole models because results for one model are not independent.
The direction of the differences also tilts toward labs, though not in every comparison. Across all 611 pairs, including ties, the maker’s score is higher 61% of the time, and the median gap on each test’s own scale is 0.8 points. Excluding exact ties, lab scores are higher in 375 comparisons and independent scores in 216, or 63% of pairs with a nonzero difference.
Gaps vary sharply by test
The size and direction of the differences depend on what a benchmark asks models to do and how evaluators run it. On SWE-bench Verified and MBPP, each with 13 matched pairs, the median lab advantage is 9 points; lab scores are higher in 92% of SWE-bench comparisons and 77% of MBPP comparisons. By contrast, the median gap is zero on MMLU, which has 20 pairs, and HumanEval, which has 34.
Several setup-sensitive evaluations show a pronounced maker-side tilt in the analysis. Labs score higher in 86% of 14 AIME 2024 pairs and 84% of 19 GSM8K pairs; the median gaps are 3.8 and 3.3 points, respectively. The site says differences in sampling and reasoning budgets can affect math results, while coding and agent tests can depend on prompts, few-shot examples or the software harness used to run the model.
A gap is not proof of inflated reporting
The comparison does not establish that labs deliberately overstate performance. Its author flags differences in reasoning effort, agent setup, prompts, benchmark subsets and test versions, any of which can make two scores difficult to compare directly. The analysis also says labs choose which benchmarks to publish, so the matched pairs are not a random sample of all model evaluations.
Some maker-level figures underline why the overall average should not be read as a verdict on every company. In the site’s groups with at least 10 pairs, Alibaba Cloud has a median gap of 1.1 points across 148 comparisons, while Meta’s is 0.7 points across 89. EleutherAI’s median is slightly negative, at -0.1 across 48 pairs; the site cautions that small groups can swing substantially.
Benchmark users still need the test conditions
For teams choosing a model, the findings make the fine print behind a benchmark score consequential: which version of the test ran, what prompts and tools were used, and how much reasoning time the model received. A score copied from a model card may not match a result from another evaluator, even when both claim to measure the same benchmark. The analysis gives buyers a broad comparison of published results, but its own caveats leave the main practical question unresolved: how much of each gap reflects changed test conditions, and how much reflects a difference in the model’s measured performance?