When LLMs get significantly worse: A statistical approach to detect model degradations

AI summary

Overview

Research area: Statistical machine learning applied to large language model (LLM) evaluation — specifically hypothesis testing for detecting accuracy degradation in optimized or compressed models.

Technical level: Intermediate. The paper combines classical nonparametric statistics (McNemar's test, binomial tests, Fisher's method, Bonferroni correction) with practical LLM benchmarking infrastructure (LM Evaluation Harness, vLLM). Readers need some familiarity with p-values, null/alternative hypotheses, and standard LLM benchmark suites.

One-sentence scope: The paper builds a statistically rigorous, false-positive-controlled framework for deciding whether an observed accuracy drop between a baseline LLM and an optimized variant reflects real degradation or harmless evaluation noise, and validates it on quantized, sparsified, and pruned Llama and Mistral models.

What This Paper Is About

LLM inference optimizations — quantization, sparsity, speculative decoding, different serving stacks or hardware — are supposed to be either provably lossless or to sacrifice accuracy only negligibly, but in practice even theoretically lossless changes produce different generations because floating-point arithmetic is non-associative and results vary across hardware, framework, and batch. Because benchmark accuracy is estimated on a finite sample and the same benchmark items are used for both the baseline and the optimized model, the two accuracy estimates are correlated, and naively treating them as independent overestimates the uncertainty of the difference. The paper's goal is to provide tests that correctly account for this correlation so that small but real degradations can be flagged with a controlled false-positive rate, and so that noise from lossless changes is not mistaken for degradation.

Key Contributions

1. An exact one-sided McNemar test for model degradation. The authors define the degradation probability as the fraction of disagreeing examples where the baseline model scores 1 and the optimized model scores 0 (b/(b+c) from the 2×2 contingency table), prove that the optimized model is worse than the baseline if and only if this probability exceeds 1/2, and compute exact one-sided p-values from the binomial distribution. They show McNemar's original statistic (b−c)²/(b+c) is just a scaled affine transform of the squared degradation probability, and that their one-sided exact version gives higher test power.

2. An asymptotic test-power analysis with a concrete dataset recommendation. The test power is shown to depend only on the signal-to-noise ratio sqrt(N/p_updownarrow)·δ, where δ is the accuracy difference, N the sample size, and p_updownarrow the flip probability. The authors prove that discarding examples on which both models agree leaves this SNR unchanged, leading to Recommendation 1: examples unlikely to flip should be removed from datasets used for degradation analysis.

3. Three aggregation schemes for multi-benchmark evaluation. Pooled test (sum b and c across tasks and run one exact test), Max drop test (take the largest standardized degradation statistic across tasks and calibrate it by Monte Carlo binomial simulation), and Fisher's method (combine per-task p-values into a chi-squared statistic with 2T degrees of freedom). A combined decision rule flags degradation if any of the three rejects, which Bonferroni-style controls type-I error at 3α — with experiments indicating effective control is usually tighter than that.

4. An implementation and an empirical case study. The tests are released as a script on top of the LM Evaluation Harness (github.com/amazon-science/LLM-Accuracy-Stats), and are evaluated on Llama-3.1 8B Instruct, Llama-3.3 70B Instruct, Mistral-Small-3.1-24B-Instruct-2503, and a pruned-and-distilled Llama-3.1 8B (non-instruct) model.

Main Findings

• Correlation matters a lot. Since the two models are evaluated on the same examples, Var[δ̂] = p_updownarrow/N rather than the naive sum of variances. The paper notes that for typical optimized LLMs (γ around 50%, p_updownarrow around 10%), Var[δ̂] ≈ (1/5)(Var γ̂ + Var β̂), meaning a naive approach overestimates the uncertainty of the accuracy drop by about sqrt(5) ≈ 2.2 and wrongly dismisses real degradations as noise.

• Small degradations can be detected confidently. The abstract reports that empirical accuracy degradations of 0.3% can be confidently attributed to actual degradations rather than noise. In the 8B KV-FP8 case, an accuracy drop of 0.79% yields a pooled p-value of 1.69e-05, a max-drop p-value of 9.28e-04, and a Fisher p-value of 4.44e-04.

• Theoretically lossless changes still flip a noticeable fraction of answers. The four lossless 8B variants (identical rerun, transformers engine, tensor parallelism 1, and A100 hardware with TP8) produce flip rates of 1.31% to 2.76% (the text summarizes the range as 1.3% to 2.8%). None are flagged as degraded. On one task the A100 run is 0.53% better than baseline.

• All three INT4 (w4a16) checkpoints are strongly flagged. On Llama-3.1 8B Instruct, w4a16 gives a pooled accuracy of 40.70% versus a baseline of 42.43% (δ̂ = 1.73%, SE 0.22%, flip probability 12.63%) with p-values of 4.80e-15 (pooled), 0.00e+00 (max drop), and 2.84e-15 (Fisher). On Llama-3.3 70B Instruct the w4a16 pooling of 17.70% against a 57.15% baseline is so extreme that the authors say it indicates a bug in vLLM; a spot check in transformers showed a much less severe drop.

• KV-cache FP8 is flagged even though its aggregate drop is small. On the 8B model the KV-FP8 variant drops 0.79% overall (0.78% in one passage of the text), which would not be flagged by the paper's baseline rule of δ̂ > 2%; on the 70B model the drop is described as 0.3% (0.29% in Table 2) and is still detected, while both baseline criteria fail.

• Flip probability alone is not a reliable degradation signal. The 8B FP8 variant has a flip probability of 8.71%, close to the 9.03% of 8B KV-FP8, yet shows no accuracy concern (pooled p-value 6.01e-01). The paper argues the Dutta et al. (2024) hard flip threshold of ≥5% would produce a false positive here.

• A pruned and distilled model shows an unmistakable drop. The Llama-3.1 8B Base variant has a pooled accuracy drop of 2.6% with p-values that are effectively zero. Its 2:4 Sparse configuration reaches 20.99% flip probability, δ̂ of 2.59%, SE 0.29%, and p-values of 1.09e-19 (pooled), 0.00e+00 (max drop), and 1.89e-35 (Fisher) — the strongest flagged degradation in the study.

• Dataset trimming preserves the signal at half the cost. Using the generative MMLU-Pro variant (12,032 examples) with Llama-3.1 8B Instruct, ten iterations at temperature 0.3 produced roughly 20% flips per run; 5,604 examples never flipped. Removing them left 6,807 examples. Full-dataset flip rates were 2,472/12,032 (20.55%) for w4a16 and 1,864/12,032 (15.49%) for KV-FP8; on the trimmed set the rates rose to 2,116/6,807 (31.09%) and 1,662/6,807 (24.42%) respectively.

• Aggregation choice depends on the degradation pattern. Synthetic experiments reported in the appendix indicate the pooled test is most powerful when degradation occurs across tasks, the max drop test is best when a single task is affected, and Fisher's method balances the two.

Methodology in Plain English

The core idea is that comparing two LLM accuracy numbers on the same benchmark is statistically close to a before-after medical trial on the same patients: the two measurements are not independent. The authors therefore treat the evaluation as a joint experiment over each test example and count the four possible paired outcomes — both models fail, baseline passes while the optimized model fails, baseline fails while the optimized model passes, and both pass — forming a 2×2 contingency table.

Decisions hinge only on the two disagreement counts. Among examples where the two models disagree, the question becomes: what fraction of those disagreements are degradations (baseline succeeded, optimized model failed)? This quantity is the degradation probability, and the paper shows the optimized model is genuinely worse exactly when this probability exceeds one half. Because that count is a binomial random variable, exact one-sided p-values can be computed with standard libraries such as SciPy, avoiding any need for a normal approximation. The authors also derive an asymptotic normal version (essentially a z-test) purely to analyze statistical power and to show why removing non-flipping examples leaves the detection signal unaffected.

For multiple benchmarks, the paper provides three ways to combine results and a simple rule of flagging degradation if any of them rejects. The evaluation side uses the Leaderboard v2 suite — Big Bench Hard, GPQA, IFEval, MATH hard, MMLU-pro, and Musr — totaling 25,282 examples (20,282 for the 70B model, which omits MATH because baseline scores were near zero due to a parsing mismatch). Models run through vLLM v0.10.0 and lm-eval v0.4.8, with the baseline being the official BF16 checkpoint served with tensor parallelism 8 on 8 H100 GPUs. To identify examples worth keeping for faster degradation testing, the authors ran the base model ten times at temperature 0.3 (chosen to give roughly 20% flips per run) and counted how often each dataset item was solved correctly.

Why This Matters

Impact on research: The paper challenges the widespread practice in the model-compression literature of reporting only aggregate accuracy differences or means, which the authors note is done by Kurtic et al. (2025), Frantar et al. (2023), Frantar and Alistarh (2023), Sun et al. (2024), and Kübler et al. (2025). It also pushes back on the claim by Dutta et al. (2024) that accuracy is not a suitable degradation signal at all, showing that the 0-2% range they dismiss as negligible often contains statistically significant degradation once estimation error is handled correctly, and that flip probabilities can be misleading because lossless changes also produce flips (Yuan et al., 2025 report deviations up to 9% for reasoning models in lossless cases).

Real-world applications:

• Validating quantized model checkpoints (INT4, INT8, FP8, KV-cache FP8) before deployment, distinguishing genuine accuracy loss from evaluation noise.
• Regression testing of serving stacks: catching degradation introduced by a framework change, hardware change, or tensor-parallel configuration change.
• Compressing evaluation suites by pre-filtering examples that are unlikely to flip, cutting evaluation cost while retaining detection power.
• Comparing candidate optimization schemes during model development when the true accuracy difference is below what a fixed threshold like 2% would flag.

Industry relevance: The work comes from Amazon and is released as a usable script on top of the widely adopted LM Evaluation Harness, so practitioners serving models with vLLM or similar engines can adopt the test without re-engineering their evaluation pipeline. Because inference cost reduction is a primary commercial concern, an easy way to certify that a speedup did not silently degrade quality has direct value for deployment decisions.

Future Directions

• Extending beyond binary scores: the paper states that a generalization to non-binary scoring criteria is presented in Appendix D, but the main experiments all use binary accuracy criteria.
• Wider validation across model families and optimization types: the case study covers Llama-3.1 8B, Llama-3.3 70B, Mistral-Small-3.1-24B, and one pruned/distilled Llama variant; sparsity is represented by a single 2:4 Sparse configuration.
• Refining the flip-prone example selection: the trimming procedure used ten runs at temperature 0.3 on MMLU-Pro; whether the same selection transfers to other datasets, tasks, or models is left open.
• Tighter combination of the three aggregation tests: the authors note the three p-values are statistically dependent and therefore rely on a Bonferroni-style 3α rule rather than a tightly combined p-value, leaving room for a better-calibrated joint test.

Target Audience

This paper is most useful for ML practitioners and evaluation engineers who benchmark LLMs before and after inference optimizations; researchers working on quantization, sparsity, speculative decoding, or serving-system changes who need to report accuracy impacts defensibly; and statisticians or applied scientists interested in adapting classical paired-comparison tests (McNemar, Fisher's method) to modern model-evaluation pipelines. Familiarity with basic hypothesis testing and LLM benchmark suites (Big Bench Hard, MMLU-pro, GPQA) is helpful but the paper is written to be followed without deep statistical theory.

Authors’ abstract

Minimizing the inference cost and latency of foundation models has become a crucial area of research. Optimization approaches include theoretically lossless methods and others without accuracy guarantees like quantization. In all of these cases it is crucial to ensure that the model quality has not degraded. However, even at temperature zero, model generations are not necessarily robust even to theoretically lossless model optimizations due to numerical errors. We thus require statistical tools to decide whether a finite-sample accuracy deviation is an evidence of a model's degradation or whether it can be attributed to (harmless) noise in the evaluation. We propose a statistically sound hypothesis testing framework based on McNemar's test allowing to efficiently detect model degradations, while guaranteeing a controlled rate of false positives. The crucial insight is that we have to confront the model scores on each sample, rather than aggregated on the task level. Furthermore, we propose three approaches to aggregate accuracy estimates across multiple benchmarks into a single decision. We provide an implementation on top of the largely adopted open source LM Evaluation Harness and provide a case study illustrating that the method correctly flags degraded models, while not flagging model optimizations that are provably lossless. We find that with our tests even empirical accuracy degradations of 0.3% can be confidently attributed to actual degradations rather than noise.