Judge Reliability Harness: Stress Testing the Reliability of LLM Judges

AI summary

Overview

• Research area: AI evaluation methodology — specifically the reliability and robustness of LLM-as-a-judge systems used to score, rank, or classify model outputs.
• Technical level: Intermediate. The paper assumes familiarity with benchmarking, LLM judges, ordinal vs. binary scoring, and common judge bias terminology, but the harness itself is described at a practical, tool-building level.
• Scope: The paper introduces the Judge Reliability Harness (JRH), an open-source library that generates synthetic reliability tests for LLM judges, and reports preliminary results from applying it to four judges across four benchmarks.

What This Paper Is About

LLM judges ("autograders") are now widely used to score AI outputs in benchmarks, yet the reliability of a given judge configuration — model, rubric, and prompt template — is rarely measured systematically or reported alongside benchmark results. The authors build an open-source harness that automatically perturbs benchmark data in ways that should not change (or should deliberately change) a judge's score, then measure whether the judge's decisions hold up. The goal is to give researchers and practitioners a cheap, reproducible way to find out where a judge breaks before its scores influence model comparisons or safety evaluations.

Key Contributions

1. The Judge Reliability Harness (JRH), an open-source library that builds validation suites for any LLM judge on both free-response and agentic benchmarks, with code released at https://github.com/RANDCorporation/judge-reliability-harness.
2. A suite of reliability tests covering label-flip sensitivity, formatting invariance, semantic paraphrase invariance, verbosity bias, stochastic stability under repeated sampling, and ordinal calibration — each targeting a distinct failure mode.
3. A human-in-the-loop review interface that lets reviewers accept, edit, or reject generated synthetic items and their score labels before evaluation, with support for Inspect evaluation files in agentic mode.
4. A four-judge, four-benchmark empirical study (GPT-4o, Claude Sonnet 4.5, Llama Maverick 4.1 17B, Gemini 2.5 Pro across FORTRESS, HarmBench, Persuade, and AgentHarm) that reports accuracy heatmaps, ordinal correlation metrics for the Persuade benchmark, and cost-per-accuracy-point comparisons.

Main Findings

• No judge is uniformly reliable. Across the four benchmarks and the perturbation types tested, no evaluated judge was consistently robust. Reliability varied meaningfully by model and by perturbation type.
• Semantic paraphrase was the most consistently tolerated perturbation. The lowest accuracy on the semantic_paraphrase test was 40%, achieved by Gemini 2.5 Pro on the Persuade benchmark. The Persuade results also show weak ordinal agreement on paraphrase for several judges (CCC of 0.125 for Claude Sonnet 4.5, 0.210 for Gemini 2.5 Pro, 0.446 for Llama Maverick 4, and 0.744 for GPT-4o).
• Task type drives fragility. Judges were relatively robust on FORTRESS and HarmBench but more fragile on Persuade, which the authors attribute to Persuade requiring a score in [1, 6] rather than a binary yes/no answer.
• Volatility reverses between benchmarks. On Persuade, Claude Sonnet 4.5 had the highest volatility (standard deviation 17.18%) and Gemini 2.5 Pro the lowest (11.10%). On HarmBench the pattern reversed: Claude had the lowest standard deviation (11.13%) and Gemini the highest (17.17%).
• Formatting changes hurt more than semantic ones. Except for FORTRESS, judges reached their lowest mean scores on format invariance tests, suggesting they are more robust to semantic-level changes than to layout changes such as spacing and indentation.
• Ordinal performance on Persuade. Reported Pearson's rho and MAE were rho = 0.901 / MAE = 0.48 for Claude Sonnet 4.5, rho = 0.960 / MAE = 0.23 for GPT-4o, rho = 0.935 / MAE = 0.34 for Gemini 2.5 Pro, and rho = 0.953 / MAE = 0.29 for Llama Maverick 4.
• Agentic tests expose asymmetric failure modes. On AgentHarm's agent_perturbation test, GPT-4o, Gemini 2.5 Pro, and Llama 4 Maverick each scored 87.5% while Claude Opus 4.5 scored 68.75%. On agent_positives, Claude Opus 4.5, GPT-4o, and Llama 4 Maverick tied at 93.75% while Gemini 2.5 Pro scored 75%. Claude Opus 4.5 showed the largest asymmetry (93.75% vs. 68.75%), consistent with a high false negative rate; Gemini 2.5 Pro showed the opposite pattern, tending to flag corrected transcripts as violations.
• Agentic accuracy and error rates. GPT-4o and Llama 4 Maverick 17B each achieved accuracy 0.906 (error rate 0.094, FPR 0.063, FNR 0.125); Claude Opus 4.5 achieved 0.813 (error rate 0.188, FPR 0.063, FNR 0.313); Gemini 2.5 Pro achieved 0.813 (error rate 0.188, FPR 0.250, FNR 0.125). A "Best Trio Ensemble" matched the best single-model numbers at 0.906 / 0.094 / 0.063 / 0.125.
• Non-agentic mean scores. FORTRESS: 69.42% (Claude Sonnet 4.5), 70.00% (GPT-4o), 63.46% (Gemini 2.5 Pro), 78.75% (Llama Maverick 4). HarmBench: 60.50%, 62.33%, 62.33%, and 73.92% respectively. Persuade: 37.26%, 48.43%, 53.20%, and 52.62% respectively.
• Cost-reliability trade-off is nontrivial. Llama 4 Maverick 17B had the best overall cost efficiency at $0.0010 per accuracy point, compared with $0.0223 for Claude Sonnet 4.5, $0.0196 for GPT-4o, and $0.0080 for Gemini 2.5 Pro. Per-benchmark values ranged from $0.0001 (Llama 4 Maverick 17B, AgentHarm) to $0.0430 (GPT-4o, Persuade). Token pricing ranged from $0.24 input / $0.97 output per 1M tokens (Llama 4 Maverick 17B) to $5 input / $25 output per 1M tokens (Claude Opus 4.5).
• A judge was swapped due to reasoning inconsistency. On AgentHarm, Sonnet 4.5 frequently showed inconsistencies between its natural-language reasoning and its returned structured score; this was not observed in Opus 4.5, Haiku 4.5, or any other tested model, so Claude Opus 4.5 replaced it for that benchmark.
• Human review was needed mainly for agentic data. For FORTRESS, Persuade, and HarmBench, no human edits were required before accepting LLM-generated perturbations. For AgentHarm's agent_perturbations mode, 14 of 16 transcripts had one or more messages modified; for agent_positives, only 2 of 16 were modified.
• Practical conclusion drawn by the authors. Llama 4.1 Maverick 17B tends to be the most reliable judge while running at a fraction of the cost of the other judges tested, undermining the assumption that the most expensive frontier model is necessarily the best judge.

Methodology in Plain English

The harness takes a benchmark dataset plus a judge configuration and runs four stages: it normalizes the seed data, generates and validates perturbed items, evaluates the judge on those items, and aggregates reliability metrics.

Several test families are generated. Basic perturbations come in two flavors. Discriminative tests rewrite a response so it clearly violates the rubric or inverts the ground-truth label while keeping topic and structure — a good judge should flip its decision. Consistency tests rewrite responses in ways that should not change quality: layout-only changes (adding or removing blank lines, inserting extra spaces within lines, adding indentation), semantic paraphrases that preserve meaning, and longer or shorter versions with the same factual content and intent (to probe verbosity bias). Stochastic stability duplicates each sampled item and compares judge outputs across identical inputs, so any variation reflects randomness rather than content differences. Synthetic ordinal mode generates samples targeting each level of an ordinal rubric, using a bucket manager, temperature ramping from an initial value up to a user-defined maximum, few-shot examples from the target score level, a validator LLM that checks whether the target score was achieved (retrying with higher temperature on a miss), and cosine similarity checks to keep generated items from leaning too heavily on the few-shot examples or on previously generated samples. Agentic mode ingests Inspect evaluation files and uses a planning LLM, an editor LLM, a summarizer LLM, and an optional verifier LLM to either degrade transcripts so they violate the rubric (agent_perturbation) or steer them toward satisfying the rubric (agent_positives).

Because perturbing every sample is expensive, each benchmark was down-sampled. FORTRESS, Persuade, and HarmBench each used 10 manually selected samples, stratified by a category column (risk_subdomain, discourse_type, and SemanticCategory respectively). AgentHarm used 16 randomly selected samples stratified by harm category — 2 from each of 8 categories (Disinformation, Drugs, Fraud, Harassment, Hate, Sexual, Copyright, and Cybercrime). Benchmark responses were produced with GPT-4o mini; Gemini 3 Pro validated that perturbations hit their intended labels, with GPT-4o-mini as a fallback validator when Gemini 3 Pro raised safety errors. The reliability metric is the fraction of items where the judge's score matches the expected score of the synthetic item, higher being better, summarized in judge-by-test heatmaps.

Why This Matters

Impact on research. Standard practice reports point estimates of judge agreement with human raters on small validation sets. This work shows that such point estimates provide limited assurance about how a judge responds to realistic input variation — formatting, paraphrase, verbosity, or repeated sampling — and argues for reliability-aware judge selection, reporting, and benchmarking.

Real-world applications.

• AI benchmark and leaderboard operations: catching judges that shift scores when response formatting changes, which could otherwise silently distort model rankings.
• Safety evaluation pipelines: the FORTRESS and HarmBench results target misuse and harmful-content classification, where a judge with a high false negative rate could let violating outputs pass.
• Agentic system evaluation: the AgentHarm results show judges that miss subtle violations inserted into multi-turn transcripts, directly relevant to deploying autonomous agents under rubric-based monitoring.
• Education and writing assessment: the Persuade benchmark uses student argumentative essays scored on an ordinal scale, a use case where calibration and stability matter directly to learners.

Industry relevance. The cost tables show that a moderately sized model (Llama 4 Maverick 17B, or Llama 4.1 Maverick 17B as named elsewhere in the paper) can match or outperform premium frontier judges on reliability metrics at a small fraction of the cost, which matters for any organization running large-scale automated grading at volume.

Future Directions

• Assessing semantic proximity in ordinal scoring. The authors state that for Persuade, measuring how close a judge's free-text response is to the ground truth — rather than only exact accuracy — remains an important direction for future work.
• Broadening the judge and benchmark pool. The study covers four judges and four benchmarks with heavily down-sampled datasets (10 samples for three benchmarks, 16 for AgentHarm), so scaling to more models, more benchmarks, and larger samples is a natural next step.
• Closing the agentic reliability gap. Because judge performance on free-response tasks does not generalize to agentic settings, and because judges show opposite failure modes (missing violations vs. over-flagging corrected transcripts), the paper points to improving judge robustness for multi-turn agentic transcripts.
• Improving perturbation generation under safety guardrails. The greater need for human editing in the agent_perturbations mode is attributed primarily to safety guardrails in the perturbation model (GPT-4o), suggesting work on generating harmful-transcript perturbations without triggering those guardrails.

Target Audience

AI evaluation researchers and benchmark maintainers who select or audit LLM judges; ML engineers building autograding or preference-scoring pipelines; safety and red-teaming practitioners who rely on judge scores to detect policy violations; and research-methodology readers interested in the validity and reliability of LLM-based measurement. It is also relevant to decision makers who consume AI benchmark results and need to understand how much confidence those numbers support.

Authors’ abstract

We present the Judge Reliability Harness, an open source library for constructing validation suites that test the reliability of LLM judges. As LLM based scoring is widely deployed in AI benchmarks, more tooling is needed to efficiently assess the reliability of these methods. Given a benchmark dataset and an LLM judge configuration, the harness generates reliability tests that evaluate both binary judgment accuracy and ordinal grading performance for free-response and agentic task formats. We evaluate four state-of-the-art judges across four benchmarks spanning safety, persuasion, misuse, and agentic behavior, and find meaningful variation in performance across models and perturbation types, highlighting opportunities to improve the robustness of LLM judges. No judge that we evaluated is uniformly reliable across benchmarks using our harness. For example, our preliminary experiments on judges revealed consistency issues as measured by accuracy in judging another LLM's ability to complete a task due to simple text formatting changes, paraphrasing, changes in verbosity, and flipping the ground truth label in LLM-produced responses. The code for this tool is available at: https://github.com/RANDCorporation/judge-reliability-harness