Behind Aggregate Scores: What Systematically Varying Arena-Hard's LLM Judge Reveals

AI summary

Overview

Research area: LLM-as-a-judge evaluation methodology; pairwise preference benchmarking (Arena-Hard v2.0); measurement reliability and robustness of automated evaluation.

Technical level: Intermediate. The core argument is intuitive, but the analysis relies on test–retest reliability, bootstrap intervals, and linearly weighted Cohen's κ.

Scope: A controlled factorial study holding 500 Arena-Hard v2.0 prompts and two models' responses fixed while varying eight GPT-4.1 judge instruction setups and repeating each run, showing that stable aggregate scores can hide large changes in individual verdicts.

What This Paper Is About

Leaderboards built on LLM judges report a single aggregate win rate, and that number is often treated as if the underlying individual judgments were equally solid. This paper asks whether those verdicts are repeatable when nothing changes, and how much they move when the judge's instructions change. It shows that the headline score can stay nearly flat while a fifth or more of the individual prompts flip between candidate win, baseline win, and tie.

Key Contributions

1. Separates aggregate stability from prompt-level stability. The paper shows that an aggregate score can be reproduced while individual verdicts change, and argues these are distinct quantities that evaluations must report separately.
2. Measures instruction sensitivity against test–retest disagreement. Rather than reporting raw change rates, each instruction contrast is expressed as an excess: the boundary-change rate between two setups minus the average of those two setups' own unchanged rerun disagreement.
3. Runs a complete 2×2×2 factorial over judge instructions. Criteria (Arena-Hard's original vs. accuracy-prioritising), output format (reasoning before verdict vs. verdict label alone), and task framing (no scope statement vs. coding-assistance description) are crossed on 500 fixed prompts with fixed responses, each setup run twice and the default run a third time, all on a pinned model version at temperature zero.
4. Locates the instability. Changes concentrate near the default setup's decision boundary: prompts whose order-averaged score is close to zero are far more likely to flip, and roughly a quarter of prompts keep the same outcome across all 17 runs.

Main Findings

• Unchanged reruns move the aggregate by about one point but change 21.2% of verdicts. The default setup's three runs scored 52.82, 54.02, and 53.50, yet 21.2% of prompts (95% CI 18.4 to 24.1) changed their winner-or-tie outcome across the three pairwise comparisons, averaged.

• Only 131 of 496 prompts (26.4%) were stable across all 17 runs. Of the 496 prompts with complete outcomes in every run, 365 (73.6%) changed outcome in at least one run; six prompts were tied in every run.

• Label-only output is the largest perturbation. It lowered the paired aggregate by 4.09 points and changed 41.6% of prompt outcomes — 28.2 percentage points more than the two setups' unchanged reruns averaged. Measured against the default setup's test–retest disagreement alone the excess is +20.5 points (95% CI 16.4 to 24.5); measured against the label-only setup's own, +36.0; the reported 28.2 is the average of the two.

• Label-only output mostly produces ties, not reversals. 31.4% of prompts became ties and 6.2% left a tie, with 4.0% reversing; the tie rate rose from 18.5% to 43.5%.

• Accuracy-prioritising criteria shift many verdicts while barely moving the score. The criteria package changed 26.9% of prompts with a −0.49-point aggregate change, an excess of 6.7 percentage points over test–retest disagreement.

• Coding-assistance framing adds a small excess. The excess is 3.8 points (95% CI 1.8 to 6.0), not distinguishable from zero against the default setup's test–retest disagreement alone, and concentrated in the 247 mathematics prompts, where the description is inaccurate.

• Test–retest disagreement itself varies widely by setup. It runs from 17.7% to 21.2% for setups that reason first and from 2.0% to 7.0% for label-only setups, so one setup's repeat cannot stand in for another's.

• Changes concentrate at the decision boundary. Mean absolute margin was 0.69 among changed prompts and 1.05 among unchanged prompts (Mann–Whitney p = 3.6×10⁻⁹). On an unchanged rerun, 43.4% of prompts with a margin of zero changed, against 33.8% at 0.5 and 17.6%, 4.3%, and 5.5% at the three higher margins.

• Prompt sensitivity splits the benchmark into two groups. 162 prompts (32.7%) kept the default outcome under all seven other setups and 30 (6.0%) changed under all seven; the stable group had a mean absolute margin of 1.30 against 0.65 for the always-changing group (p = 5.7×10⁻⁶).

• Aggregate scores across the eight setups were tightly clustered. Raw win-equivalent scores ranged from 49.36 to 53.87 despite the prompt-level differences.

• Judge substitution can move verdicts without moving the score. In the May runs, Gemini 2.5 Flash changed the paired aggregate by 0.18 points while 38.3% of prompts changed outcome (19.0 points above the default's May–August disagreement); Claude Sonnet 4.5 lowered the aggregate by 3.97 points and changed 35.0%.

• Agreement between judges is lower than agreement between reruns of one judge. Linearly weighted Cohen's κ was 0.435 for GPT-4.1 vs. Gemini (95% CI 0.378 to 0.492), 0.549 for GPT-4.1 vs. Claude, and 0.478 for Claude vs. Gemini, against κ = 0.763 (95% CI 0.730 to 0.794) between the default setup's own runs.

• Movements cancel in the average. Of the 105 prompts that changed between the default setup's first two runs, 56 moved toward the candidate and 49 toward the baseline.

• Leaderboard position is more fragile than the two-model comparison. Scored with Arena-Hard's own method against its published judgments of 27 other models, the three unchanged October runs place GPT-4.1 mini 11th, 7th, and 11th of 28. An implementation check scored the May default run at 48.0% (90% interval 46.3% to 50.3%) against Arena-Hard's published 48.6%.

• A verifiable case went the wrong way under the default setup. A competitive-programming prompt with three worked examples: the candidate's submission fails all three, the baseline's passes all three, yet the May default setup recorded a decisive win for the failing submission; the accuracy-prioritising criteria, both judge substitutions and three multi-contrast conditions preferred the correct submission, and the August repeat reversed the verdict.

Methodology in Plain English

The researchers took a fixed test set — 500 prompts from Arena-Hard v2.0's hard split, with GPT-4.1 mini as the candidate and o3-mini as the baseline — and never changed the prompts or the two answers being compared. Only the judge's instructions varied.

They crossed three binary choices to make eight setups: original Arena-Hard judging criteria versus criteria that make factual and technical accuracy the primary requirement; reasoning before the verdict versus the verdict label alone; and no task description versus a coding-assistance description. GPT-4.1 judged all 500 prompts under all eight setups, each setup run twice and the default setup a third time, all on the same day with the pinned version gpt-4.1-2025-04-14 at temperature zero. That is 17 runs and 17,000 calls, of which 16,996 returned a parsed verdict.

Arena-Hard judges each pair twice with the answer order reversed, so each prompt gets an order-averaged score on a −2 to +2 scale. A prompt counts as changed if its outcome moves between candidate win, baseline win, and tie.

The key analytical move is the comparison baseline: for each instruction change, they computed how often verdicts differed between the two setups, then subtracted the average of how often each of those setups disagreed with itself on a rerun. Whatever remains — the "excess" — is attributed to the instruction change rather than to the judge's inherent noise. Intervals come from 20,000-draw percentile bootstraps that resample whole prompts. Claude Sonnet 4.5 and Gemini 2.5 Flash were substituted as judges in an earlier May experiment, run once each, and are reported separately because provider code and generation settings also differ.

Why This Matters

Impact on research: The paper undercuts a common inference — that a reproducible leaderboard score implies reproducible judgments. It offers a concrete reporting protocol: prespecify a small set of alternative judge setups, repeat each unchanged, and report both the paired aggregate change and the prompt-level change rate for each, alongside every setup's own test–retest disagreement. It also frames judge reliability in the language of measurement validity, where test–retest reliability is reported before a test is used to decide about an individual.

Real-world applications:

• Preference labels for training. When judge verdicts become preference data, a 21.2% flip rate on rerun means a fifth of labels could differ on a second pass, and label noise propagates into the trained model.
• Regression testing in model release pipelines. A judge that changes verdicts on a rerun can flag or hide regressions that are not real changes in the model.
• Safety review triage. If individual verdicts route responses to human review, unstable verdicts mean unstable triage queues.
• Leaderboard and model-selection decisions. Positions separated by about two points are shown to be fragile: the same setup placed a model 11th, 7th, and 11th of 28 across three unchanged runs.

Industry relevance: Judge pipelines typically make one call per item and report no agreement statistic, so per-call variation flows directly into published numbers. The paper's practical guidance is cheap to adopt — repeat the run, report agreement, and test a few defensible alternatives — and it shows the cost of not doing so is concentrated exactly where decisions are made: near the decision boundary.

Future Directions

• Test whether majority verdicts over repeated runs reduce instability. The paper measures the variation but does not test whether aggregating several runs stabilises individual verdicts.
• Test a second model pair with a wider capability gap and a second benchmark. The current pair sits near 50% by design, which makes boundary crossings easy to observe; the rates should not be assumed typical of pairs with larger gaps.
• Separate genuine ambiguity from judge noise. Repeats show prompts move but cannot distinguish prompts that are truly ambiguous from prompts on which the judge is simply unstable.
• Establish ground truth and test judge combinations. The study has no human labels apart from four checkable prompts, so it measures how much verdicts change, not which setup is more accurate; whether combining judges improves reliability or accuracy was not tested.

Target Audience

Researchers and engineers who build or consume LLM-judge evaluations: benchmark maintainers, alignment and preference-data teams, and anyone using pairwise verdicts as labels, regression signals, or safety evidence. It is also useful for methodologists interested in measurement reliability and for readers of leaderboards who want to know what an aggregate score does and does not certify. Readers need only moderate statistical background; the central result is readable from the headline numbers alone.

Authors’ abstract

LLM judges are used to determine which of two answers is better. Trusting these verdicts requires knowing how repeatable they are and how much they depend on the judge's instructions. We show that similar aggregate scores can conceal substantial changes in individual verdicts, both when the same setup is repeated and when the instructions change. Using 500 Arena-Hard v2.0 prompts with two models' answers held fixed, we run GPT-4.1 as the judge under all eight combinations of criteria, output format and task framing, and repeat every setup unchanged at temperature zero on a pinned model version. Rerunning the default setup moves the aggregate score by about one point but changes the win/tie/loss outcome on 21.2% of prompts. Instruction changes introduce further sensitivity: in one setup, which asks for the verdict without reasoning and also drops several other instructions, outcomes change on 28.2 percentage points more prompts than the two setups' unchanged reruns do on average, and the aggregate score falls by 4.09 points. In another, where the criteria prioritise accuracy, outcomes change on 6.7 percentage points more prompts by the same comparison, while the aggregate score barely moves. A stable aggregate score therefore does not establish that individual verdicts are repeatable. This distinction matters when those verdicts are used as preference labels or to assess individual responses. Evaluations that rely on individual verdicts should repeat each setup they use and report both aggregate scores and prompt-level agreement.