How to Correctly Report LLM-as-a-Judge Evaluations

AI summary

Overview

Research area: Statistical evaluation methodology for large language models, specifically LLM-as-a-judge evaluation and bias correction.

Technical level: Intermediate. The paper uses binomial confidence intervals, the delta method, and the Rogan–Gladen prevalence correction, but all formulas are stated explicitly and the motivation is presented through a simple 2x2 error table.

Scope: The paper proposes a plug-in bias-correction estimator with confidence intervals for LLM-judged binary accuracy scores, characterizes when this beats human-only evaluation, and shows robustness to distribution shift between calibration and test data.

What This Paper Is About

When researchers use an LLM to judge whether model responses are "correct," they usually report the raw fraction of responses the LLM marked correct, denoted p̂. Because LLM judges make mistakes, this raw score is systematically biased: it overestimates true accuracy when true accuracy is low and underestimates it when true accuracy is high. This paper provides a simple corrected estimator, θ̂, together with confidence intervals that account for uncertainty in both the test set and a small human-labeled calibration set, and identifies the conditions under which this LLM-based procedure is statistically preferable to just paying humans to label the test set.

Key Contributions

1. A bias-corrected plug-in estimator with valid confidence intervals. The paper derives θ̂ = (p̂ + q̂₀ − 1) / (q̂₀ + q̂₁ − 1) (building on Rogan and Gladen, 1978, and Lang and Reiczigel, 2014), where q̂₁ is the judge's estimated sensitivity and q̂₀ its estimated specificity, plus a confidence interval that combines uncertainty from the test dataset size n and the calibration label counts m₀ and m₁.

2. An adaptive calibration-allocation strategy. Because the two label types contribute asymmetrically to interval width, the authors propose Algorithm 1, which uses a small pilot calibration set (e.g., m_pilot = 10 per label type) to estimate the error ratio κ = (1 − q̃₀)/(1 − q̃₁) and then allocate calibration samples optimally. Proposition 5.2 states that minimum interval length is achieved when m̃₀ ≈ (1/p̃ − 1)√κ · m̃₁.

3. Parameter regimes for preferring LLM-as-a-judge over human-only evaluation. Proposition 6.1 gives a condition on θ, q₀, and q₁ under which the variance of the corrected LLM estimator is no larger than that of a human-only estimator φ̂ = M₁/m built from the same annotation budget.

4. Robustness to distribution shift. The authors show that existing bias-correction approaches, notably prediction-powered inference (Angelopoulos et al., 2023a; 2023b), can become biased when the test distribution ℙ differs from the calibration distribution ℚ, whereas their Rogan–Gladen-based estimator θ̂ remains unbiased under such shifts.

Main Findings

• The naive score is provably biased whenever the judge is imperfect. For q₀ + q₁ < 2, E[p̂] = θ + (2 − q₀ − q₁)((1 − q₀)/(2 − q₀ − q₁) − θ). Bias is positive when θ is below the threshold (1 − q₀)/(2 − q₀ − q₁) and negative above it. The extreme case q₁ = 1, q₀ = 0 gives p̂ = 1 regardless of θ.

• Bias decreases as the calibration set grows. Proposition 5.1 shows that under m = 2m₀ = 2m₁ and q = q₀ = q₁ with 0.5 < q ≤ 1, once m ≳ 2q/(2q − 1)², the absolute bias of θ̂ is always smaller than that of p̂ for all θ ∈ [0, 1]. Fewer samples are needed when the judge is near-perfect; substantially more are needed as q approaches 0.5.

• The confidence interval achieves nominal coverage. In Monte Carlo simulations, the interval based on p̂ attains near-zero coverage except at a few values of θ, while the proposed interval's coverage stays close to the nominal 95% level across all tested values of θ.

• Adaptive allocation shortens intervals. Under a fixed calibration budget of m = 200, the allocation produced by Algorithm 1 yields shorter confidence intervals than the symmetric split m₀ = m₁ = 100.

• Calibration budget targets are quantifiable. With q̂₀ = 0.7, q̂₁ = 0.9, and n → ∞, and when p̂ = 0.3 is estimated from the test set, achieving an interval shorter than 0.1 requires roughly m ≈ 200 calibration examples.

• On Chatbot Arena, bias correction reduces bias for all six target models. Using GPT-4.1-mini as judge on win-rate estimation for Alpaca-13B, Claude-v1, FastChat-T5-3B, GPT-4, LLaMA-13B, and Vicuna-13B, averaged over 100 random 90% test / 10% calibration splits, θ̂ reduces bias relative to p̂ in every case. Table 1 reports empirical coverage of the 95% intervals and average interval lengths for these win-rate estimates (the table's contents are truncated in the provided text).

• None of the seven evaluated judges lands in the favorable variance regime on Chatbot Arena. Figure 6 maps estimated sensitivity and specificity of seven contemporary LLM judges for Alpaca-13B pairwise comparisons; none falls inside the region where calibrated LLM-as-a-judge evaluation has lower variance than human-only evaluation, so human-only evaluation retains a variance advantage on this benchmark. Claude-Opus-4.6 lies closer to the boundary of that region.

• Task difficulty shifts the regime. On the AlpacaEval benchmark, the paper reports (Appendix I) that the condition already holds when GPT-4.1-mini is used as the judge.

• Distribution shift breaks some existing methods but not this one. Section 8 shows estimators from existing bias-correction methods can still be biased when ℙ ≠ ℚ, while the proposed estimator remains unbiased.

Methodology in Plain English

The setup treats each response as having a true binary label Z (human judgment of correct or incorrect) and an LLM judgment Ẑ. Two error rates describe the judge: sensitivity q₁ = Pr(Ẑ = 1 | Z = 1) and specificity q₀ = Pr(Ẑ = 0 | Z = 0). A test set of size n produces the raw score p̂. A separate, smaller calibration set of size m (with m₀ truly incorrect and m₁ truly correct examples, m = m₀ + m₁) has both human labels and LLM judgments, and is used to estimate q̂₀ and q̂₁.

Substituting these estimates into the inverted law-of-total-probability relation yields the corrected estimator. For uncertainty, the authors apply the delta method and binomial variance formulas to get Var(θ̂) as a ratio whose numerator combines a test-set term p̂(1 − p̂)/n with two calibration terms weighted by (1 − θ̂)² and θ̂², all divided by (q̂₀ + q̂₁ − 1)². They then build the interval using an "add two successes and two failures" adjusted Wald approach, replacing p̂, q̂₀, q̂₁ with smoothed versions p̃, q̃₀, q̃₁ and truncating the interval to [0, 1].

Because the test set can be made arbitrarily large at low cost, the paper often takes n → ∞, leaving calibration size as the binding constraint. The allocation algorithm exploits this: a pilot sample estimates the error ratio, and then the budget is split unevenly between label types according to the derived optimum. Validation proceeds through Monte Carlo simulation with (q₀, q₁) = (0.7, 0.9), θ over 21 values in {0, 0.05, ..., 1}, n = 1000, m = 200, and 10,000 replications, followed by real-world evaluation on Chatbot Arena and AlpacaEval. A plug-in Python implementation is provided at https://github.com/UW-Madison-Lee-Lab/LLM-judge-reporting.

Why This Matters

Impact on research. Reported benchmark improvements can reflect judge bias rather than genuine model gains, and different evaluation procedures induce biases of different magnitude and direction. A corrected estimator with honest error bars makes comparisons across papers and models more trustworthy, and gives a principled way to decide whether scarcity of human annotation should be spent on direct labeling or on calibrating a cheap LLM judge.

Real-world applications.

• Benchmark leaderboards for chatbots and assistants, where win rates and accuracy scores are increasingly computed by LLM judges rather than crowd workers.
• Model development pipelines that use an LLM judge as an automatic reward or regression signal during training and would otherwise chase biased scores.
• Safety and alignment evaluations, where both over- and under-estimating a failure rate changes deployment decisions.
• Content moderation and grading systems where an automated classifier is validated against a small human-labeled audit set.

Industry relevance. LLM-as-a-judge is attractive precisely because it is cheap and scalable, but the paper shows that its cost advantage is only statistically justified when the judge's sensitivity and specificity are high enough. Firms can use the reported interval-length curves to budget a fixed number of human labels and hit a target precision, rather than labeling an arbitrary amount.

Future Directions

• Extending beyond binary judgments. The main analysis covers correct/incorrect; the paper discusses an extension to more than two categories, such as grading or rating-scale evaluation, in Appendix B, leaving that direction partially open.
• Modeling human disagreement. The paper explicitly assumes human disagreement is absent or resolved by a predefined protocol and treats human evaluation as ground truth; extensions that model disagreement directly are listed as future work in Appendix A.
• Closing the gap to the favorable regime. Since none of the seven judges evaluated on Chatbot Arena fell into the region where calibrated LLM judging beats human-only evaluation, the open question is whether better judges, easier tasks, or both are needed, and how to predict this in advance.
• Handling distribution shift in stronger correction methods. The paper shows prediction-powered inference can be biased under ℙ ≠ ℚ while the proposed estimator is not; whether lower-variance methods can be made shift-robust is left open.

Target Audience

Practitioners and researchers who run LLM-as-a-judge evaluations and report accuracy or win-rate numbers, including benchmark maintainers, evaluation engineers, and applied ML scientists. It is also relevant to statisticians interested in measurement-error correction under distribution shift, and to anyone allocating a limited human-annotation budget between direct labeling and judge calibration. Readers need comfort with binomial confidence intervals and the delta method to follow the derivations, but the core correction formula and allocation rule are usable without them.

Authors’ abstract

Large language models (LLMs) are widely used as scalable evaluators of model responses in lieu of human annotators. However, imperfect sensitivity and specificity of the LLM judges induce bias in naive evaluation scores. We propose a simple plug-in framework that corrects this bias and enables statistically principled uncertainty quantification. Our framework constructs confidence intervals that account for uncertainty from both the test dataset and a human-labeled calibration dataset. Additionally, it uses an adaptive strategy to allocate calibration samples for tighter intervals. Importantly, we characterize parameter regimes defined by the true evaluation score and the LLM judge's sensitivity and specificity in which our LLM-based evaluation yields more reliable estimates than human-only evaluation. Moreover, we show that our framework remains unbiased under distribution shift between the test and calibration datasets, in contrast to existing approaches.