Benchmarking is Broken -- Don't Let AI be its Own Judge

AI summary

Overview

Research area: AI evaluation and benchmarking methodology — a position paper on the science and governance of how AI models are measured, spanning benchmark design, data quality control, mechanism design, and community governance.

Technical level: Intermediate. The argument is largely conceptual and accessible, but the paper includes formal scoring equations, a commitment/slashing scheme, and references to a broad benchmark literature.

Scope: The paper diagnoses structural failures in current AI benchmarking and proposes a community-governed, proctored evaluation paradigm called PeerBench, instantiated as a live prototype.

What This Paper Is About

AI benchmarking today relies largely on static, public test sets maintained by single teams, combined with self-reported results — a setup the authors argue is vulnerable to data contamination, cherry-picking, biased test construction, and unverifiable claims. The paper's goal is to argue that this laissez-faire arrangement is untenable and to sketch a concrete alternative: evaluation reframed as a standardized, proctored examination with secret test items, sealed execution, rolling renewal, and transparent community governance.

Key Contributions

1. Structural critique. A formal critique of the structural flaws undermining today's benchmarks — contamination, fragmentation, biased test data, and monopolization of evaluation authority.
2. Position statement. An argument that AI evaluation should be reframed as a secure, standardized examination, together with seven design principles that balance openness and rigor.
3. Prototype architecture. The PeerBench design, described as a minimum viable version of the desiderata: a ten-step workflow, cryptographically signed artifacts, a lightweight reputation scheme, and score-normalization methods intended to turn heterogeneous community inputs into a longitudinal, contamination-resistant leaderboard.
4. Comparison and counterargument analysis. A comparison table of existing evaluation platforms (static evals, Scale SEAL, LiveBench, ARC-AGI, Chatbot Arena) against a desired paradigm across dimensions such as dynamic updating, data source diversity, transparency, contamination resistance, and data quality control, plus responses to four anticipated objections.

Main Findings

• Contamination is measured, not hypothetical. The paper cites retrieval-based audits reporting over 45% overlap on QA benchmarks, and that GPT-4 infers masked MMLU answers in 57% of cases — well above chance. It also cites allegations that Llama 4 gained significant improvements via seeded paraphrases, and notes that N-gram audits such as those used on Qwen-1_8B can help detect leakage but rely on partial knowledge of training data.
• Test construction itself can be biased. The paper's example is Humanity's Last Exam, where organizers select five specific models and curate tests consisting solely of items that all five chosen models fail — which the authors argue unfairly penalizes those five models and artificially advantages any new model.
• Static benchmarks saturate and go stale. SuperGLUE was rapidly saturated, with LLMs hitting performance ceilings shortly after release. The authors argue the lack of liveness — continuous inclusion of fresh, unpublished items — renders today's metrics a stale snapshot.
• The measurement ecosystem is fragmented. Each benchmark repository often introduces custom tokenizers, scoring rules (BLEU, ROUGE, EM, proprietary AI-scores), and ad-hoc scripts, making results hard to reproduce and compare, and enabling hype-driven "state-of-the-art" claims.
• Private benchmarks trade one problem for another. Proprietary or paywalled evaluations can reduce contamination, but they shift epistemic authority to the curator, who alone controls evaluation access, task updates, and scoring — making scientific progress contingent on opaque processes, discretionary labor, and sustained funding.
• Evaluation lacks proctoring. Unlike high-stakes human exams such as the SAT, GRE, or bar examinations, AI evaluations lack proctors, identity checks, and appeals processes, so teams may fine-tune on test sets, exploit unlimited submissions, or report selectively within current norms.
• Temporal fairness is an unsolved core challenge. The paper argues that if evaluation is not simultaneous across models, information can leak across models and periods, and if test data are created after a model appears, contributors can cherry-pick items to favor or handicap it. It states two necessary conditions for the fairest score: tests must be created before a model is released and remain fully secret until evaluation, and all eligible models must be evaluated at the same time on the same undisclosed items, which are then discarded.
• The PeerBench prototype operationalizes part of the vision. It features three leaderboards (data contributor, reviewer, model) driven by three scoring rules: contributor score as the sum of test quality plus verification bonuses; reviewer score as the Pearson correlation between a reviewer's ratings and consensus quality; and model score as a quality-weighted average of test scores.
• The prototype explicitly prioritizes immediacy over ultimate fairness. PeerBench adopts a hybrid of immediate scoring (Design Choice A) and synchronized windows (Design Choice B, with registration windows on a geometric progression of 2^0, 2^1, 2^2, …, 2^k days), but in the prototype it prioritizes immediate scoring, relying on reputation and slashing to mitigate contamination risk.
• No empirical results are reported. The paper presents a critique, design principles, and a prototype; the content provided does not report experimental evaluations, accuracy figures, or measured outcomes from PeerBench itself beyond the live prototype link.

Methodology in Plain English

This is a position paper, so the "method" is argument and design rather than experiment. The authors first survey existing benchmarking practice — static suites such as MMLU, GSM8K, and SuperGLUE; dynamic efforts such as LiveBench and Dynabench; and crowdsourced platforms such as Chatbot Arena and the HuggingFace Open LLM Leaderboard — and catalogue what each gets right and where each breaks down. They then draw an analogy to human standardized testing, which is proctored, regularly updated, and governed by rigorous procedures, and argue AI evaluation should meet at least that bar.

From the critique they distill seven principles: secret test sets, proctored execution, community governance, continuous renewal and liveness, auditability and integrity, equitable access, and multi-metric reporting. They then turn those principles into a concrete prototype design — defining the actors (data contributors, reviewers, model creators, a coordination server, and end users), a ten-step workflow (test submission with hash commitment, model evaluation, peer review, weight calculation, reservoir management, reputation updates, new model integration), three leaderboards, and security mechanisms including partial revelation of live tests in read-only, non-copyable form such as images, full publication after retirement, and collateral slashing for misconduct. Finally, they anticipate four objections — loss of open benchmarks, trust in secret tests, cost, and slowed innovation — and answer each.

Why This Matters

Impact on research. The paper argues that contaminated, gameable benchmarks blur scientific signal: once contamination is plausible, generalization claims become suspect, resources get misallocated toward leaderboard positions rather than capability, and community trust erodes. A certificate-grade evaluation layer would let researchers distinguish genuine capability gains from favorable curation.

Real-world applications:

• Model procurement and regulation. Regulators, journalists, and practitioners consulting leaderboards could apply trust thresholds that reflect their tolerance for uncertainty, using auditable scores rather than self-reported numbers.
• Competitive fairness for smaller labs. The paper's equitable-access principle allows any bona-fide team — academic, corporate, or independent — to submit a model subject only to compute reimbursement fees, so a small laboratory competes on the same footing as a large vendor.
• Educational and aptitude testing design. The framework borrows directly from SAT/GRE-style practice: domain-specific subscores such as maths, coding, and reasoning, percentile ranks, and uniformly computed fairness metrics on bias and robustness rather than a single headline number.
• Open-source development workflow. The proposed two-tiered system keeps "practice" sets — retired questions or legacy benchmarks — openly accessible for debugging and method development, while a "final" set of fresh, unseen questions determines certified scores, similar to the public/hidden split in Kaggle competitions.

Industry relevance. The paper notes that companies reportedly invest hundreds of thousands of dollars in compute resources to achieve top scores on evaluations such as ARC-AGI, which makes the credibility of those scores commercially consequential. It also proposes an economic model in which staked collateral is slashed for malicious tests or systematic deviation from consensus, and used to reward honest contributors, potentially combined with fees if benchmarking on PeerBench becomes a paid service to model creators — an attempt at a self-sustaining loop rather than a goodwill-dependent one.

Future Directions

• Mechanism design and game-theoretic security analysis. The authors explicitly call for work strengthening the framework's economic and adversarial robustness, noting that PeerBench prioritizes instant evaluation over ultimate fairness and relies on reputation and slashing to mitigate contamination risk.
• Resolving the temporal fairness dilemma. The tension between immediate scoring and synchronized cohort evaluation remains open; the paper notes the ideal is extremely demanding for test creators and reduces direct comparability between models that do not appear in the same cohort.
• Institutional hosting and funding models. The paper suggests existing neutral organizations such as NIST or MLCommons, or a newly established not-for-profit foundation, could host the evaluation service, with costs managed through modest submission fees and public funding to support academic participation, and containerized inference submissions to protect proprietary model weights.
• Governance, transparency, and cost questions. Open proposals include a multi-institutional exam board with rotating membership, publication of statistical summaries on topic distribution, difficulty calibration, and demographic coverage with each test release, and post-retirement scrutiny of all items — alongside the acknowledged trade-off of a slower feedback loop.
• Unspecified design details. The paper notes that the weight calculation formula is an instance subject to further discussion and change, leaving room for refinement of scoring, normalization, and decay methodology (it mentions logistic time decay as a possibility for discounting stale results).

Target Audience

This paper benefits most from readers who build, consume, or govern AI evaluations: benchmark designers and dataset curators, evaluation researchers and metrists, model developers who publish leaderboard claims, policymakers and regulators who rely on AI performance claims, and mechanism-design or game-theory researchers interested in decentralized reputation and slashing systems. It is also useful for journalists, procurement teams, and practitioners who need to judge how much weight a headline benchmark score deserves.

Authors’ abstract

The meteoric rise of AI, with its rapidly expanding market capitalization, presents both transformative opportunities and critical challenges. Chief among these is the urgent need for a new, unified paradigm for trustworthy evaluation, as current benchmarks increasingly reveal critical vulnerabilities. Issues like data contamination and selective reporting by model developers fuel hype, while inadequate data quality control can lead to biased evaluations that, even if unintentionally, may favor specific approaches. As a flood of participants enters the AI space, this "Wild West" of assessment makes distinguishing genuine progress from exaggerated claims exceptionally difficult. Such ambiguity blurs scientific signals and erodes public confidence, much as unchecked claims would destabilize financial markets reliant on credible oversight from agencies like Moody's. In high-stakes human examinations (e.g., SAT, GRE), substantial effort is devoted to ensuring fairness and credibility; why settle for less in evaluating AI, especially given its profound societal impact? This position paper argues that the current laissez-faire approach is unsustainable. We contend that true, sustainable AI advancement demands a paradigm shift: a unified, live, and quality-controlled benchmarking framework robust by construction, not by mere courtesy and goodwill. To this end, we dissect the systemic flaws undermining today's AI evaluation, distill the essential requirements for a new generation of assessments, and introduce PeerBench (with its prototype implementation at https://www.peerbench.ai/), a community-governed, proctored evaluation blueprint that embodies this paradigm through sealed execution, item banking with rolling renewal, and delayed transparency. Our goal is to pave the way for evaluations that can restore integrity and deliver genuinely trustworthy measures of AI progress.