Measuring what Matters: Construct Validity in Large Language Model Benchmarks

AI summary

Overview

Research area: Evaluation methodology for large language models (NLP / machine learning), specifically construct validity — whether benchmark measurements actually represent the abstract phenomena they claim to measure.

Technical level: Intermediate. The paper is written for an academic audience but its arguments and its checklist are accessible to anyone who reads or uses LLM benchmark results.

Scope: A systematic review of 445 LLM benchmark papers, conducted by 29 expert reviewers, that diagnoses recurring weaknesses in how benchmarks define phenomena, choose tasks, score models and justify claims, and converts those findings into eight recommendations plus an actionable checklist.

What This Paper Is About

LLM evaluation is used both to assess capabilities and to catch safety or robustness problems before deployment, but reliably measuring abstract things like "safety" or "robustness" requires measures that genuinely represent those phenomena. This paper audits a large body of benchmark papers to see how often that requirement is met. It finds recurring patterns across phenomena, tasks and metrics that undermine the validity of the claims benchmarks make, and it offers guidance to fix them.

Key Contributions

1. A large systematic review of LLM benchmarks. A corpus of 46,114 articles from ICML, ICLR and NeurIPS (2018–2024) and ACL, NAACL and EMNLP (2020–2024) was filtered down to 445 benchmark articles, each coded by expert reviewers against a shared codebook.
2. An empirical diagnosis of construct validity gaps. The paper quantifies how benchmarks define phenomena, sample task items, choose metrics, and justify validity — showing where best practice is uncommon.
3. Eight recommendations with checklists. Concrete guidance grouped into: define the phenomenon; measure the phenomenon and only the phenomenon; construct a representative dataset for the task; acknowledge limitations of reusing datasets; prepare for contamination; use statistical methods to compare models; conduct an error analysis; and justify construct validity.
4. A worked example on GSM8K. The widely used grade-school maths benchmark is walked through the recommendations to show how they apply in practice, including where GSM8K already satisfies them and where it falls short.

Main Findings

• Phenomena are often defined, but definitions frequently conflict. 78.2% of reviewed benchmarks defined their target phenomenon, yet of those, 52.2% of definitions are widely agreed upon and 47.8% are contested. Benchmarks cover reasoning (18.5%), alignment (8.1%) and code generation (5.7%) among many other phenomena.
• Phenomena are often treated as composites. 61.2% of benchmarks defined their phenomenon as a composite, versus 36.5% describing a single unified whole. Of those with sub-elements, 261 measured them separately, 144 were not applicable and 46 did not.
• Real-world tasks are rare. Less than 10% of benchmarks used complete real-world tasks. 40.7% used constructed tasks (28.5% exclusively), while partially real-world tasks appeared in 32.3% and representative tasks in 36.9%.
• Task items are heavily reused and often conveniently sampled. Only 33.6% relied on a single task source. Authors most commonly handcrafted new items (43.3%), reused data from existing benchmarks (42.6%), generated data with LLMs (31.2%), or drew on human exams and other pre-existing sources (38.2%). 12.3% used convenience sampling exclusively and another 27.0% used it as part of their strategy. Targeted sampling was used at least partially by 55.2%, criterion sampling by 46.2% and random sampling by 17.1%.
• Exact matching dominates scoring, and statistical testing is rare. Exact matching was used at least partially by 81.3% of benchmarks (exclusively by 40.7%). Soft match appeared at least partially in 20.9%, LLM-as-a-judge in 17.1% and human ratings in 13.0%. Only 16.0% used uncertainty estimates or statistical tests to compare results.
• Only about half justify their validity. 53.4% of articles presented evidence for the construct validity of their benchmark; 35.2% compared to other benchmarks of similar phenomena, 32.4% to a human baseline and 31.2% to a more realistic setting.
• Response-format requirements can confound results. Free response was the most common format (46.4% partially, 17.8% exclusively), followed by multiple choice (40.0% partially, 18.5% exclusively) and short free response (38.5% partially, 13.2% exclusively). Structured formats such as JSON were required at least partially by 21.1% and exclusively by 8.6%, and the paper flags such format constraints as a potential confounder.
• Synthetic expansion is common. 47.5% of reviewed benchmarks generated synthetic examples to increase size, which the authors warn can compound non-representative sampling.
• Contamination and benchmark overfitting are live risks. The paper notes that widely used benchmarks attract methods specifically suited to solving them, and that contamination may undermine results even when model developers act in good faith.
• Almost every paper had at least one weakness. About half discussed validity, but nearly every reviewed paper fell short in at least one area.

Methodology in Plain English

The authors ran a systematic literature review rather than an experiment.

They assembled 46,114 conference papers from six venues over multi-year windows. To narrow this down, they searched titles and abstracts for the keyword "benchmark" alongside either "LLM" or "language model", producing 2,189 candidate articles — most from recent years, with only 14 from 2018 and 2019.

They then applied four inclusion criteria: the paper had to concern LLM capabilities (not purely technical matters like inference speed or energy consumption); it had to introduce an empirical benchmark and report LLM performance (not opinions, reviews or policy frameworks); the benchmark had to be compatible with text and vision models (excluding audio or video requirements); and it had to be a novel benchmark or a substantial modification of an existing one, not a lightly repackaged combination.

The first three criteria were screened automatically using GPT-4o mini (reference 104), a step validated against human labels on a sample of 50 articles and achieving an F1 score of 84%. This reduced the set to 522 articles for manual filtering. Those 522 were assigned to 29 reviewers matched by expertise, who applied all four criteria manually, yielding 445 included articles.

Reviewers used a codebook built a priori around phenomena, tasks, metrics and claims, drawing on prior literature for indicators of face, predictive, content, ecological, convergent and discriminant validity. The framing treats a benchmark as a task and a metric used together to represent a phenomenon — for instance, GSM8K's phenomenon is "multi-step mathematical reasoning", measured by short free-response grade-school word problems, scored by exact match. Each article was coded by a primary reviewer; a second reviewer mapped responses onto a simplified option list, verified by the primary reviewer. For reliability, 46 papers were reviewed twice, giving a mean Brennan–Prediger Kappa of .524 across all 30 categorical questions.

The first author then read a subset of 50 articles and reviewed all 445 annotations, synthesising findings into draft recommendations through inductive open coding. The recommendations were refined collaboratively across five meetings. The final dataset contains responses to 21 question items across the 445 benchmark articles.

Why This Matters

Impact on research. The paper argues that weak construct validity produces results that look precise but measure the wrong thing — for example, benchmark scores that reflect formatting proficiency or prior knowledge rather than the reasoning ability they claim to test. It also warns that repeated optimisation against a widely used benchmark can effectively contaminate it over time, similar to reusing a validation set. Its checklist is offered both as a design tool for new benchmarks and as an evaluation framework for existing ones.

Real-world applications. The benchmarks reviewed span many applied domains, which is where their validity claims land:

• Medicine and health — benchmarks that ask models to answer medical licensing exam questions.
• Software engineering — benchmarks that detect errors in computer code, or require writing a correct SQL query from a natural-language query and a database structure.
• Information integrity — benchmarks that reconcile conflicting information on Wikipedia, or compile news sources with differing opinions and test how well models synthesise them.
• Commerce and agentic use — benchmarks that complete e-commerce tasks collected from real people on a mock website, and "agentic capability" benchmarks that bundle sub-abilities like intent recognition, alignment and structured output generation.

Industry relevance. The paper notes that its venue-based sampling misses benchmarks developed and released by industry labs without formal peer review, even though such benchmarks can be influential. It also notes that improving on a benchmark requires attention and resources, so authors should explain why their benchmark is relevant — a justification only about half of reviewed benchmarks provided.

Future Directions

• Dynamic and procedurally generated benchmarks. The paper proposes dynamic benchmarks as a response to contamination and notes that procedurally generated tasks in particular can keep benchmarks up to date.
• Held-out task items and ongoing contamination checks. Contamination tests are recommended at creation time and, by embedding checks within the benchmark itself, as ongoing verification of validity.
• Factor analysis to disentangle capabilities. The paper points to emerging work using factor analysis to extract latent capability dimensions with less interference from auxiliary tasks.
• Extending the checklist beyond new benchmarks. The authors suggest the checklist can serve as an evaluation framework for existing benchmarks or for adapting them to new domains or capabilities, and recommend reporting it as an appendix with explanations for any skipped items.
• Open limitations the authors raise. Their focus on leading conference proceedings may exclude impactful benchmarks from industry or specialised venues; automated GPT-4o mini screening may have introduced undetected false negatives; distributional shift in language usage may have contributed to lower inclusion of older papers; and limiting the number of reviewers per paper reduced review robustness.

Target Audience

Benchmark creators and evaluation researchers in NLP and machine learning will get the most direct use from this paper, since the recommendations and checklist are aimed at design decisions. It is also valuable for AI practitioners and product teams who consume benchmark leaderboards and need to judge whether a reported score actually supports a deployment decision. Journal and conference reviewers, policy and standards bodies concerned with evaluation methodology, and students learning how to build evaluations will find the diagnosis of common failure patterns instructive.

Authors’ abstract

Evaluating large language models (LLMs) is crucial for both assessing their capabilities and identifying safety or robustness issues prior to deployment. Reliably measuring abstract and complex phenomena such as 'safety' and 'robustness' requires strong construct validity, that is, having measures that represent what matters to the phenomenon. With a team of 29 expert reviewers, we conduct a systematic review of 445 LLM benchmarks from leading conferences in natural language processing and machine learning. Across the reviewed articles, we find patterns related to the measured phenomena, tasks, and scoring metrics which undermine the validity of the resulting claims. To address these shortcomings, we provide eight key recommendations and detailed actionable guidance to researchers and practitioners in developing LLM benchmarks.