Are We Testing Too Much? A Survey of Benchmark Compression for Efficient Evaluation of Large Language Models

AI summary

Overview

Research area: Natural Language Processing — specifically the efficiency of large language model (LLM) evaluation and the statistical problem of estimating full-benchmark results from partial evaluations.

Technical level: Intermediate. The survey is readable without deep statistics background, but it assumes familiarity with concepts such as sampling, item response theory, matrix completion, bandits, and inverse-probability weighting, which it reviews across method families.

Scope: A systematic survey of 119 papers that organizes the field of LLM benchmark compression into a single taxonomy, compares method families by their information needs, costs, reliability checks, and limitations, and offers guidance on method selection and validation.

What This Paper Is About

Benchmarks are how the field measures and improves LLMs, but many widely used benchmarks are redundant — prior evidence from LongBench, GSM8K, and MMLU shows fewer than 20% of test samples can preserve results close to the full set. Benchmark compression aims to estimate full-benchmark scores and rankings from a partial evaluation, cutting cost while staying close to the full result. Research in this area has grown rapidly since 2024, but the authors state that no survey had systematically organized its methods and findings — this paper does that.

Key Contributions

1. First systematic survey and taxonomy. To the authors' knowledge, this is the first systematic survey of LLM benchmark compression, synthesizing 119 papers under a common definition and introducing a principled taxonomy and decision path organized by evaluation setting and available information.
2. A formal common formulation. The paper defines a shared evaluation contract: a full evaluation produces outcomes over units U aggregated by a rule G into θ̂_full, while a compressed evaluation uses a lower-cost subset S ⊆ U plus auxiliary information Z and a recovery rule R to produce θ̂_comp = R(Y_S, Z), aiming for L(θ̂_comp, θ̂_full) ≤ ε while C_comp < C_full.
3. Cross-family synthesis and guidance. It compares method families in depth to clarify their requirements, reliability, and limitations, providing guidance for method selection and validation plus a roadmap for future research.
4. A categorized paper catalogue. Appendix tables map individual papers to the taxonomy and record secondary attributes including domain, venue, year, GitHub availability, saved unit, and the returned result.

Main Findings

• Redundancy is the enabling condition. Compressibility arises because benchmark items measure overlapping capabilities and provide limited additional discrimination. In a model-by-item response matrix this shows up as correlated model responses, low item discrimination, and low effective dimensionality.
• Compression is now a large, fast-growing literature. The survey covers 119 papers, with publication counts growing rapidly since 2024 (Figure 1 tracks quarterly and cumulative counts from 2023 to August 2026).
• The taxonomy has two top-level branches and six compression families. Papers are classified by contribution type, then by whether the subset is fixed before evaluation or selected dynamically, then by scope and information. The six compression families are Content-Based, Historical-Response, Target-Conditioned, Cross-Benchmark, Adaptive Item Selection, and Sequential Decision and Stopping.
• Information source determines what a method costs and risks. Content-Based Compression needs only benchmark-side information and has low construction cost, but small random subsets have high variance and text-diverse items may still test similar abilities. Historical-Response Compression needs a model-by-item response matrix, which requires many full evaluations unless public results exist, and its advantage can shrink under large model shifts. Target-Conditioned Compression pays for a probe, output scan, or cheap judge on every target.
• Cross-benchmark sharing is more efficient but riskier. Joint compression removes redundancy both within and across benchmarks, and some methods skip complete benchmarks or costly tasks — but they rely on cross-benchmark relations that may not hold for new models, especially when learned from past suite results.
• Adaptive methods trade cost for flexibility and latency. Adaptive Item Selection can direct later tests to the most informative items, but sequential rounds increase wall-clock time and give models different test paths, complicating comparison and reproduction. Sequential Decision and Stopping can stop once a score is precise enough, but close models may require almost the full benchmark, and fixed-budget methods redistribute rather than reduce the budget.
• Random sampling remains a strong baseline. The paper reports that random sampling can remain competitive with more complex selectors, and under large model shifts random sampling can stay competitive with historical-response methods.
• Compression and repair are different goals. Some methods remove questions that are guessable from option wording or that conflict with the intended capability ordering. Their goal is a cleaner benchmark even if scores or rankings change — that is benchmark repair, not score reconstruction, and any repaired benchmark must be validated as a new benchmark.
• No universal compression ratio exists. Required item count depends on the score difference of interest, paired variance, task-group coverage, and allowed error rate. When retained items cannot reliably separate two models, the comparison should remain unresolved.
• Protocol must be held fixed. Full and compressed evaluations should use the same judges, prompts, decoding settings, and agent harnesses; aggregate judge accuracy can hide local errors, and removing weaker judges can discard useful signals.

Methodology in Plain English

This is a survey rather than an experimental paper, so the "method" is a literature analysis. The authors first fix a shared mathematical definition of what a compressed evaluation is, then use three ordered questions to label each paper: (1) is the contribution an operational cost-reduction method or a foundational analysis, (2) is the evaluated subset fixed in advance or chosen dynamically, and (3) what scope and information does a static method use. This produces a hierarchical taxonomy where each paper receives one primary label plus secondary attributes. The authors then compare the six compression families along subset selection, score recovery, lifecycle cost, and limitations, and they catalogue individual papers in appendix tables with domain, venue, year, repository, saved unit, and the result type returned (score, ranking, decision, or reliability interval).

Why This Matters

Impact on research. The paper gives a shared vocabulary and comparison framework for a fast-growing but fragmented literature, making method requirements, saved units, and failure modes explicit. That supports more consistent reporting and future quantitative synthesis across studies that currently use widely different evaluation settings.

Real-world applications:

• Leaderboard and benchmark reporting, where full results remain necessary for comparison with prior work and leaderboards, while compression reduces cost during iteration, screening, and repeated validation.
• Human-judgment evaluation pipelines, including chat and arena-style comparisons, medical review, and safety evaluation, where methods such as judge sampling, proxy correction, and selective escalation can reduce the number of human or strong-judge reviews.
• Agent and code evaluation, where methods can skip costly runs, environments, or target tasks by substituting cheaper proxy tasks or selected interaction traces.
• Multilingual and multi-benchmark suites, where cross-benchmark and tensor-based methods share information across languages and benchmarks rather than compressing each one independently.

Industry relevance. Evaluation cost directly affects how often teams can re-test models during development. The paper argues that documenting item count, model calls, human review, judge calls, and wall-clock time separately lets practitioners state which costly unit is actually being saved, and that methods should fall back to broader random sampling, direct testing, continued evaluation, or abstention when their requirements are not met.

Future Directions

1. Generalization across models and over time. Subsets are often selected from records of existing models, and models from other families or newer, more capable models may show different response patterns that reduce coverage and discrimination. The authors call for compression methods that transfer more reliably and for periodic or continuous refresh of compressed benchmarks.
2. Updating established benchmarks. Combining compression with benchmark repair could yield a recalibrated release that removes flawed, redundant, overly easy, or weakly discriminating items — lowering cost while improving discrimination among stronger models, and providing a cheaper shared reference.
3. Compression before release. For new benchmarks, compression should be integrated during construction so the public version ships a compact but informative item set covering the intended capabilities, rather than publishing a large redundant benchmark and compressing it later.
4. More comparable reporting. Because studies use widely different settings, the authors position their common formulation as a step toward consistent reporting and future quantitative synthesis.

Target Audience

Researchers and practitioners who evaluate LLMs and need to reduce evaluation cost without breaking comparability: benchmark maintainers and leaderboard operators, evaluation and model-development teams choosing between sampling, historical-response, target-conditioned, and adaptive methods, and NLP researchers who need a single map of the benchmark compression literature. Reviewers and students looking for an entry point to the subfield will benefit from the taxonomy, the decision path, and the catalogued list of 119 papers, while readers needing full implementation details should consult the appendices the survey points to rather than the main text.

Authors’ abstract

Benchmarks are essential for evaluating and improving large language models (LLMs). Yet many widely used benchmarks are redundant, making evaluation unnecessarily expensive. Benchmark compression lowers this cost by estimating full-benchmark scores and rankings from a partial evaluation. Research in this area has grown rapidly since 2024, but to our knowledge no survey has systematically organized its methods and findings. We review 119 papers to explain how methods choose what to evaluate, recover full-benchmark results, and verify their reliability. From this analysis, we develop a taxonomy covering compression within individual benchmarks, across benchmark suites, and during adaptive evaluation, together with work on redundancy and reliability. We compare their information needs, total costs, reliability checks, and limitations. Methods reduce evaluation costs by selecting representative subsets, using cheaper or reusable information, and testing adaptively, but risk poor coverage, inaccurate estimates, and sequential delays. The resulting taxonomy and practical guide help researchers choose, validate, and deploy methods for more efficient and reliable LLM evaluation.