Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks

AI summary

Overview

Research area: Large language model evaluation, psychometrics, item response theory (IRT), and computerized adaptive testing (CAT).

Technical level: Intermediate. The motivation and headline results are accessible to anyone working on LLM benchmarking, but the method assumes familiarity with IRT machinery (discrimination, difficulty, guessing parameters, Fisher information, EAP/WLE estimation).

Scope: The paper proposes ATLAS, an IRT-based adaptive testing framework that estimates a latent ability score for each LLM instead of averaging accuracy over a fixed benchmark, and shows it can match whole-bank ability estimates while administering up to 90% fewer items across five benchmarks.

What This Paper Is About

Benchmarks with tens of thousands of items — and in some cases exceeding 100,000 items — are expensive and slow to run, often taking days or weeks, yet they are still scored as simple average accuracy over a fixed item set. This treats poorly discriminative and highly discriminative items as equally informative and masks real differences between models that happen to land on the same accuracy. ATLAS instead calibrates items with a three-parameter logistic IRT model and adaptively serves each model only the items that carry the most statistical information about its current estimated ability, stopping when a precision target is met.

Key Contributions

1. Diagnosing the limits of raw accuracy. The authors identify fundamental limitations of accuracy-based evaluation and show that IRT-based ability estimates produce more informative model comparisons, including substantial rank reordering and more stable rankings across benchmark subsets and related domains.
2. The ATLAS framework. A large-scale adaptive testing framework for LLMs that achieves up to 90% item reduction while maintaining target measurement precision through standard-error-controlled stopping, enabling precision-targeted rather than fixed-length evaluation.
3. Robustness beyond the calibration distribution. Strict architecture-family holdout and temporal generalization experiments show that calibrated item parameters transfer reasonably well to unseen model families (a held-out Mixtral family of 321 models) and to newer model populations.
4. Rigorous psychometric validation. The paper reports model-fit diagnostics (M2 with RMSEA) and demonstrates common-person linking to align item parameters and support scalable cross-model comparability — validation that TinyBenchmarks and MetaBench do not report.

Main Findings

• Large item reduction with preserved precision: ATLAS reduces the number of required items by up to 90% while maintaining measurement precision, and matches whole-bank ability estimates using only 41 items (0.157 MAE) on HellaSwag, which has 5,600 items.
• Best accuracy–efficiency tradeoff on every benchmark: An ATLAS variant achieves the lowest Information Efficiency Score (IES) across all five benchmarks. It attains the lowest MAE on TruthfulQA (0.064 with 48 items) and HellaSwag (0.157 with 41 items), and matches MetaBench-Primary on WinoGrande while using 2× fewer items (70 vs. 133).
• Beats other static subset methods: ATLAS outperforms TinyBenchmarks, which uses 97–100 items but yields higher error across all benchmarks. Overall ATLAS requires 30–89 items per benchmark versus hundreds in static subsets.
• Ability preserves global accuracy structure: Reconstructed accuracies (via the p-IRT estimator) closely match raw full-bank accuracies across all five benchmarks, with MAE consistently below 5%. On WinoGrande, ATLAS 0.1 attains the lowest reconstruction MAE (0.048) using only 70 items versus 133 for MetaBench-Primary.
• Ability separates models that accuracy cannot: Two models with identical accuracy (0.833) on WinoGrande receive different ability estimates (θ̂_A = 1.2 vs. θ̂_B = 0.6). Model A succeeds disproportionately on harder items (D6–D9) while Model B succeeds mainly on easier ones (D1–D4). Among more than 3,000 evaluated models, 23–31% shift by more than 10 rank positions, and models with identical accuracies receive meaningfully different ability estimates.
• Ability fixes ceiling and floor effects: Accuracy and ability are highly correlated (0.99 on GSM8K; 0.96 on HellaSwag), but at the low end accuracy collapses into a narrow band (0.10–0.15) while IRT spans roughly θ ≈ −3 to −1; at the high end IRT preserves discrimination across roughly θ ≈ 1.5 to 2.5.
• Ability rankings are more stable: Split-half ranking stability (Spearman correlation, averaged over 10 random partitions) improves from 0.943 to 0.981 on WinoGrande, 0.981 to 0.992 on TruthfulQA, 0.985 to 0.993 on GSM8K, 0.968 to 0.977 on ARC, and 0.994 to 0.996 on HellaSwag. Cross-benchmark consistency improves from 0.504 to 0.738 between GSM8K and MMLU Elementary Math, and from 0.439 to 0.904 between MMLU College Chemistry and MMLU High School Chemistry.
• Better model fit: Under the limited-information M2 statistic with RMSEA, TinyBenchmarks shows poor fit on all five benchmarks, MetaBench ranges from 0.04–0.14 (poor on TruthfulQA, marginal on ARC), and ATLAS achieves acceptable or good fit on every benchmark (0.0438–0.0690).
• Generalizes beyond calibration: On ARC, withholding all 321 Mixtral-family models (calibrating on the remaining 3,296) raises accuracy reconstruction MAE only modestly from 0.032–0.034 to 0.044–0.045. Temporal generalization — calibrating on 2,998 models released before 2024-05-01 and testing on 322 later models — raises ability MAE from 0.084–0.117 to 0.126–0.162 while accuracy reconstruction changes only from 0.032–0.034 to 0.044–0.045.
• Practically cheap: Test overlap rates remain between 11–23% and item exposure stays below 12% across all benchmarks. End-to-end runtimes range from 9.4 to 75.5 seconds per model and scale with bank size (shortest for TruthfulQA, longest for HellaSwag).
• 3PL versus 2PL is benchmark-dependent: The benefit of the 3PL model over a 2PL variant that omits the lower-asymptote parameter is not universal. On TruthfulQA, 3PL improves both ability estimation and accuracy reconstruction while using similar or fewer items; on WinoGrande, GSM8K, ARC, and HellaSwag, results are mixed, with 2PL sometimes achieving slightly lower ability MAE but often requiring substantially longer tests.

Methodology in Plain English

The authors reframe LLM evaluation as a measurement problem from psychology. Instead of asking "what percentage did this model get right," they ask "what latent ability level best explains this model's pattern of correct and incorrect answers."

The pipeline has four stages:

1. Assemble a response matrix. They pull model responses from the HuggingFace Open LLM Leaderboard across ARC, GSM8K, HellaSwag, TruthfulQA, and WinoGrande. Models are kept only if they answered every item. Models in the extreme low-ability tail (below the 0.1st percentile) are dropped because near-zero response patterns destabilize parameter estimation; the high-ability tail is small and non-degenerate, so it is kept. The remaining models are split 90% calibration / 10% testing using stratified random sampling over 10 bins.

2. Filter out useless items. Two filters run before calibration: items with response standard deviation below 1% or mean accuracy above 95% are discarded, and items whose point-biserial correlation with models' total scores falls below 0.1 are removed as non-diagnostic.

3. Calibrate items at scale. Each item gets a discrimination parameter, a difficulty parameter, and a lower-asymptote (guessing) parameter under the 3PL model. Fitting the full bank at once would be computationally prohibitive, so items are split into K non-overlapping subsets of at least 100 items each and calibrated independently. Because every model answered every item, the model population itself acts as a set of "common persons" that links the separate subsets onto a single scale using mean–sigma transformations. This lowers complexity from O(|I|³) to O(K · max_k |I_k|³). Reference abilities are estimated with the Weighted Likelihood Estimator, which includes a bias-correction term and stays finite at ability extremes.

4. Adaptive testing. For each model, the test starts at θ̂ = 0 (the conventional scale midpoint), with a minimum of 30 and maximum of 500 items. The first item is the one whose difficulty is closest to zero. Thereafter, each next item is drawn at random from the top-5 unadministered items ranked by Fisher information at the current ability estimate — a "randesque" strategy that avoids over-relying on one item type. After each response, ability is updated with Expected A Posteriori estimation, which is numerically stable with the sparse early response record, and the standard error is recomputed. Testing stops when either the maximum item limit is hit or the standard error falls at or below a threshold (τ = 0.1, 0.2, or 0.3 in the experiments), after the minimum number of items.

Final ability estimates are validated against whole-bank references from the full item set, and predicted percent-correct scores are reconstructed from θ using the performance-IRT estimator built on the test characteristic curve. Baseline comparisons are against random sampling of 100 items, TinyBenchmarks, and MetaBench-Primary and MetaBench-Secondary. The authors also run explicit psychometric fit checks rather than assuming the IRT model holds.

Why This Matters

Impact on research. The paper argues that the field's default metric — average accuracy over a fixed set — is a lossy summary that hides real capability differences, especially at the extremes of the ability distribution where accuracy saturates or collapses. By showing that ability-based rankings are more stable across item splits and more consistent across related benchmarks, it offers a psychometric justification for moving beyond accuracy as the primary reported number. It also sets a validation bar (reporting fit statistics, documenting calibration and linking) that prior IRT-based benchmarks like TinyBenchmarks and MetaBench did not meet.

Real-world applications:

• Model selection and procurement. Teams choosing between models with near-identical headline accuracy can use ability estimates and their standard errors to pick the one that is genuinely stronger, rather than trusting a tie in the leaderboard.
• Cheaper continuous evaluation. Because ATLAS finishes in 9.4 to 75.5 seconds per model and uses 30–89 items per benchmark instead of hundreds, providers can re-run evaluations far more frequently during training or release cycles.
• Private and domain-specific item banks. The calibration and linking procedure lets an organization spend its labeling budget on a smaller, well-characterized set of discriminative items rather than administering a full benchmark to every candidate model.
• Ranking in saturated regimes. For benchmarks where most strong models cluster at the top, ability estimates preserve discrimination where accuracy produces ceiling effects.

Industry relevance. The paper's cost argument is direct: benchmarks with tens of thousands of items, and some exceeding 100,000 items, take days or weeks to run, and static subsets chosen by clustering or by expensive iteration do not guarantee that the chosen items are informative for the specific model being scored. ATLAS replaces that with per-model item selection driven by an explicit information criterion and a stopping rule tied to a precision target, which is exactly the kind of cost control that matters when evaluation is a recurring operating expense rather than a one-off study. The code and calibrated item banks are released at https://github.com/Peiyu-Georgia-Li/ATLAS.

Future Directions

• Extending beyond the five calibrated benchmarks. The paper calibrates on ARC, GSM8K, HellaSwag, TruthfulQA, and WinoGrande. Whether the same partition-based calibration and linking approach transfers to generative, multi-turn, or code-execution evaluations is an open question.
• Comparing stopping rules head to head. The authors note ATLAS can also operate under a fixed-length stopping rule rather than the precision-based SE rule used in the experiments, but do not evaluate that configuration here; a systematic comparison would clarify when precision-based stopping is worth the variable test length.
• Choosing the IRT parameterization per benchmark. The 3PL-versus-2PL comparison is benchmark-dependent, improving TruthfulQA but mixed elsewhere, and the paper does not offer a criterion for deciding which parameterization to use on a new benchmark.
• Understanding the limits of transfer. Temporal generalization degrades ability MAE noticeably more than accuracy reconstruction (0.084–0.117 to 0.126–0.162 on ARC), so how often item banks need recalibration as new model populations are released remains unresolved.

Target Audience

Evaluation researchers and benchmark maintainers who need to reduce the cost of scoring LLMs without losing measurement fidelity; machine learning engineers and model-selection practitioners who must rank models whose accuracy scores are nearly identical; and psychometricians or quantitative methodologists interested in applying IRT and computerized adaptive testing outside of human test-taking, including those studying model-fit validation and common-person linking at scale.

Authors’ abstract

Evaluating large language models (LLMs) typically requires thousands of benchmark items, making the process expensive, slow, and increasingly impractical at scale. Existing evaluation protocols rely on average accuracy over fixed item sets, treating all items as equally informative despite substantial variation in difficulty and discrimination. We introduce ATLAS, an adaptive testing framework based on Item Response Theory (IRT) that estimates model ability using Fisher information-guided item selection. ATLAS reduces the number of required items by up to 90% while maintaining measurement precision. For instance, it matches whole-bank ability estimates using only 41 items (0.157 MAE) on HellaSwag (5,600 items). We further reconstruct accuracy from ATLAS's ability estimates and find that reconstructed accuracies closely match raw accuracies across all five benchmarks, indicating that ability $θ$ preserves the global performance structure. At the same time, $θ$ provides finer discrimination within accuracy-equivalent models: among more than 3,000 evaluated models, 23-31% shift by more than 10 rank positions, and models with identical accuracies receive meaningfully different ability estimates. Code and calibrated item banks are available at https://github.com/Peiyu-Georgia-Li/ATLAS.git.