JEV-as-a-Judge: Accept When Confident, Escalate When Unsure

AI summary

Overview

• Research area: LLM-as-a-judge evaluation, judge reliability and calibration, and cost-aware escalation (cascades) between judges.
• Technical level: Intermediate.
• One-sentence scope: The paper empirically tests a proprietary decision-only judge (TypeSafe JEV) against sixteen generative and reward-model judges — seventeen configurations are actually compared in Table 1 — on preference, factuality, and answer-adjudication benchmarks, measuring accuracy, confidence quality, latency, and fees, and showing that a confidence-gated cascade can accept JEV's verdicts when it is confident and escalate to a stronger LLM when it is unsure.

What This Paper Is About

Evaluating model outputs with LLM judges is useful but expensive and hard to trust, because judges can be overconfident and reasoning-heavy judges cost more in tokens and latency when evaluation is repeated at scale. The paper asks whether a cheap, decision-only judge that returns a verdict plus label probabilities can serve as a first pass, and whether its confidence signal reliably identifies the specific judgments that need a stronger (and pricier) judge. The goal is an empirical "operating envelope" showing where cheap judging suffices, where escalation earns its cost, and when confidence can connect the two.

Key Contributions

1. A matched empirical comparison of a decision-only judge against a broad judge panel. JEV 1.13 is compared with thirteen hosted judges (including GPT-4.1 mini, GPT-4.1, GPT-5.2, GPT-5.4, GPT-5.6 Sol, GPT-6 Astra, GPT-OSS 120B, Qwen3.6/3.8 27B, Claude Sonnet 5, Gemini 3 Flash, Gemini 3.1 Pro) and four local baselines (Qwen3 32B, Qwen3.5 27B, PairRM-hf, Skywork-Reward-V2-Qwen3-8B) on 1,312 items across reward-benchmark, factuality, and adjudication tasks, with fees and latency measured on a separate frozen 120-decision panel.
2. A confidence/calibration characterization of a typed-probability judge. The paper reports Brier scores, clipped NLL, ten-bin ECE, error-detection AUROC, temperature-scaling transfer, and interface consistency (Choice vs Noul vs Score) for JEV and comparators.
3. A frozen, pre-specified escalation policy. Pairs are judged in both orders and the aligned probabilities averaged, with thresholds fitted on a 96-pair selection set (64 RewardBench, 32 JudgeBench) under a coverage-maximizing rule subject to a two-point accuracy tolerance; the policy is then evaluated on held-out and extension data.
4. A workload-level operating envelope plus human adjudication. Blinded adjudication of 183 judge-disagreement items and a sensitivity analysis with human-corrected labels are used to check whether gaps reflect labels or model error.

Main Findings

• Ordinary preference and evidence-grounded factuality are close. JEV scores 92.2% on RewardBench (400 pairs) versus GPT-6's 93.5%, a paired difference of −1.25 points (95% cluster interval [−3.8, 1.5]), and 87.5% on HaluEval (240 judgments) versus GPT-6's 86.7%, +0.83 points ([−1.25, 2.92]). The abstract reports JEV within three percentage points of the strongest comparator on these two workloads at 0.36% of the comparator's fee.
• Human adjudication favors the stronger judge more than labels do. On RewardBench the adjudication sides with GPT-6 on 17 of 29 disagreements and with JEV on 5 (7 indecisive), a human-adjudicated difference of −3.0 points ([−5.3, −0.8]); on HaluEval it sides with GPT-6 on 8 of 10 (−2.5, [−5.0, −0.4]). It also finds 24 of the 26 HaluEval items both judges "miss" carry labels the evidence does not support; with those corrected, JEV scores 95.8% and GPT-6 98.3%.
• Difficult correctness is where JEV loses. On JudgeBench (350 pairs) JEV scores 78.6% against 93.1% for GPT-5.6 and GPT-6, −14.6 points ([−18.9, −10.3]). The gap is widest in reasoning (68.4% versus 95.9%, n=98) and coding (76.2% versus 97.6%, n=42), narrowest in knowledge (84.4% versus 90.9%, n=154). Adjudication sides with GPT-6 on 57 of 69 disputed items and with JEV on one (11 indecisive), a difference of −16.0 points ([−20.0, −12.3]), with reasoning, coding, and math at 27–0, 9–0, and 7–0.
• Presentation and style change the picture. Reversal changes 3.25% of RewardBench decisions and 11.14% of JudgeBench decisions (both-orders-correct accuracy 91.0% and 74.0%); JEV picks the first position 48.9% and 48.4% of the time. On RM-Bench (80 prompts, nine style pairings, both orders = 480), JEV scores 84.0% on matched-style pairs but 74.8% when the rejected answer is the more elaborately written one, a drop of −9.2 points ([−14.0, −4.8]); GPT-6 moves from 93.3% to 94.6% (+1.3, [−1.9, 4.2]). On hard pairs the gap is −19.8 points ([−27.7, −12.7]).
• Answer format matters less than workload. Under a four-way contract with an "ambiguous" outcome on the 150 existing replies, JEV's agreement moves from 91.3% (direct adjudication) to 86.0% (gold-blind extraction), Δ=−5.3 points ([−14.0, 2.0]), while GPT-4.1 mini stays at 87.3%. Across six reply conditions on forty HaluEval questions, JEV's direct agreement is 100.0% on multiple choice and 92.5% on free response (−7.5 points, [−11.7, −3.3]).
• Reference-free prose is a boundary for all judges tested. JEV, GPT-4.1 mini, and GPT-5.4 score 71.2%, 62.5%, and 72.5% on eighty document-grounded summaries, and 52.5%, 53.8%, and 55.0% on eighty reference-free responses. Without a reference all three are near chance and remain confident (mean maximum probabilities 0.90, 0.95, 0.96; JEV Brier 0.815 with error-detection AUROC 0.518; GPT-5.4 Brier 0.813).
• Controls saturate. Fourteen of the fifteen applicable configurations score 108/108 on the GSM8K trajectory controls and 64/64 on the synthetic evidence controls; Qwen3.6's 105/108 and 61/64 are entirely invalid outputs, its valid judgments being all correct. JEV returns a valid verdict on 1,312/1,312 base items.
• Cost and latency differ by orders of magnitude. On the isolated 120-decision panel, JEV's median latency is 0.152 seconds and its fee $0.044 per 1,000 judgments, versus 0.548 seconds and $0.390 for GPT-4.1 mini and 1.885 seconds and $12.182 for GPT-6 — about 9 and 277 times cheaper. JEV 1.13.0 charged $0.042 per million input tokens and nothing for output at collection time.
• Confidence orders JEV's errors. Pooled over the three base tasks (990 base-order judgments), JEV is right on 47.7% of the 65 items with q<0.6, 76.5% of the 85 in [0.7, 0.8), 93.9% of the 147 in [0.95, 0.99), and 99.1% of the 322 at q=1. GPT-6 on the same items moves only from 78.5% to 99.1%, so its advantage sits where JEV is unsure. On items JEV would accept at q≥0.9 the two are nearly interchangeable (95.8% versus 96.5%; 97.2% versus 98.5% under human-corrected labels); on escalated items GPT-6 leads by

Authors’ abstract

LLM-as-a-judge enables evaluation across diverse tasks, but inference cost and confidence reliability become critical at scale. We study whether a decision-only judge can provide an economical first pass and identify when stronger evaluation is needed. Comparing jev-as-a-judge with sixteen generative and reward-model judges, with blinded human adjudication, we find it within three percentage points of a state-of-the-art LLM judge, our strongest comparator, on ordinary preference and evidence-grounded factuality at 0.36% of the comparator's fee. Larger gaps arise when judgments require checking a derivation or resisting an elaborately written wrong answer. On several benchmarks, JEV's gap to this comparator is concentrated in low-confidence decisions. A frozen cascade that accepts confident verdicts and escalates uncertain ones retains 99% of the comparator's accuracy at lower cost.