SCOPE: Selective Conformal Optimized Pairwise LLM Judging

AI summary

Overview

Research area: Natural Language Processing — specifically LLM-as-a-judge evaluation, uncertainty quantification, selective prediction, and conformal risk control.

Technical level: Advanced. The paper relies on finite-sample statistical validity arguments (linear expectation constraints, exchangeability, false-discovery-rate control) and assumes familiarity with conformal prediction and calibration theory.

Scope: The paper introduces SCOPE, a calibration layer that guarantees the error rate among accepted pairwise LLM judgments stays below a user-specified level, paired with a bidirectional entropy-based uncertainty score (BPE) designed to neutralize the judge's position bias.

What This Paper Is About

LLMs are widely used to compare two responses and pick a winner, but these judges are miscalibrated and biased — for example, they can be highly confident yet systematically swayed by which response appears first, or by stylistic artefacts like length. The paper's goal is to make such judging trustworthy by adding a principled abstention mechanism: the judge returns a verdict only when it is uncertain enough to be safe, and the error rate among the verdicts it does return is bounded by a number the user chooses (α).

Key Contributions

1. SCOPE (Selective Conformal Optimized Pairwise Evaluation): a conformal-based method for selective LLM pairwise evaluation with a finite-sample guarantee that the error rate among non-abstained judgments is at most α, under exchangeability between calibration and test data.
2. BPE (Bidirectional Preference Entropy): a bidirectional, permutation-invariant uncertainty estimator. It queries the judge under both response orderings, averages the two preference probabilities into a bias-neutral preference, and converts that into a binary entropy score, mitigating position effects with only two forward passes per pair.
3. A linearized, finite-sample calibration rule that selects the largest feasible uncertainty threshold, maximizing coverage subject to the risk constraint, with a proof (Theorem 2.1) that the marginal test-time FDR is at most α.
4. An empirical validation across three main benchmarks and four judge models showing that BPE improves calibration and discrimination over standard confidence proxies, and that SCOPE meets the risk bound while accepting substantially more judgments than empirical-thresholding alternatives.

Main Findings

• BPE improves calibration and discrimination: Across MT-Bench, RewardBench, and Chatbot Arena, BPE mostly achieves the highest AUROC and AUPRC and lower ECE than Predictive Probability and Verbalized Confidence.

• BPE strictly beats Swap-and-Aggregate: S&A shares BPE's bidirectional prediction rule but uses discrete agreement voting. BPE improves over S&A on ECE, AUROC, and AUPRC in every model–dataset cell. Example: on RewardBench with Qwen2.5-32B-Instruct, BPE reaches ECE 0.071, AUROC 0.822, AUPRC 0.975.

• SCOPE respects the target risk: At α = 0.10, empirical FDR is tightly concentrated around the target, typically 0.097–0.099 across datasets and models. Coverage at that setting includes 0.246 (MT-Bench, Qwen-7B), 0.983 (RewardBench, Qwen-32B), and 0.583 (Chatbot Arena, Llama-70B).

• Baselines violate the constraint: Vanilla prediction has full coverage (1.000) but MT-Bench risk of 0.217–0.269. Heuristic thresholding retains high coverage (0.809–0.958) but still violates the constraint in most settings (MT-Bench risk 0.184–0.251). Naïve calibration exceeds α on MT-Bench for Qwen-7B (0.116) and Qwen-14B (0.124), and on Chatbot Arena for Qwen-14B (0.114).

• SCOPE accepts more judgments than empirical thresholding: On MT-Bench with Qwen-7B, SCOPE more than doubles coverage relative to Naïve (0.246 vs. 0.102). The abstract states SCOPE accepts up to 2.4 times more judgments under the same risk constraint than vanilla baselines; the largest such ratio visible in Table 3 is against the Naïve baseline in that same MT-Bench/Qwen-7B cell.

• Coverage scales with the risk budget and model strength: For MT-Bench, coverage moves from (0.018, 0.246, 0.556, 0.802, 0.991) for Qwen-7B to (0.072, 0.543, 0.790, 0.950, 1.000) for Llama-70B as α ranges from 0.05 to 0.25. On Chatbot Arena, Qwen-7B moves from 0.052 coverage at α = 0.05 to 0.998 at α = 0.25.

• BPE is efficient relative to Simulated Annotators: Simulated Annotators require multiple generations per instance and were restricted to Qwen-7B and Qwen-14B. BPE matches or improves calibration and achieves substantially stronger discrimination, e.g., AUROC improving from 0.59 to 0.69 for Qwen-7B and from 0.60 to 0.78 for Qwen-14B on MT-Bench. One exception: on RewardBench with Qwen-7B, Simulated Annotators reports slightly lower ECE (0.103) than BPE (0.104).

• Split-to-split variance depends on judge strength: The ±1σ bands across 1000 random splits are widest for the weakest judge (Qwen-7B) and much tighter for Qwen-32B, with variability most pronounced at larger α; mean risk curves remain stable and track the target.

• BPE preserves exchangeability: Because BPE is a deterministic mapping built from two greedy forward passes at temperature T = 0, exchangeability of the labeled pairs transfers to the induced tuples used in calibration, which is the assumption behind the validity proof.

Methodology in Plain English

Each evaluation instance is a query plus two candidate responses, and the ground truth is which response a human preferred. Instead of always forcing a verdict, the system uses selective prediction: it outputs a judgment only when its uncertainty score is at or below a threshold, and abstains otherwise.

To build a trustworthy uncertainty score, the authors ask the judge twice — once with responses in the original order and once swapped. Because "response A wins" corresponds to label A in the forward prompt and label B in the reversed prompt, the two probabilities can be aligned and averaged into a single preference probability. The binary entropy of that averaged probability becomes the uncertainty score, measured in nats and maximized at 0.69 (ln 2) when the averaged probability is exactly 0.5. Position bias pushes the averaged probability toward 0.5, so biased instances naturally receive higher uncertainty.

Calibration uses a labeled set with human preferences. Directly controlling the ratio of errors to acceptances is unstable when few items are accepted, so the authors use a linearized loss: each accepted sample contributes −α if correct and +(1−α) if incorrect. They require the cumulative calibration loss to be at most −1, a finite-sample "budget" that absorbs the worst-case contribution of one unseen test point. The chosen threshold is the largest one satisfying this condition; if none exists, the system abstains on everything. At test time, a judgment is accepted only if its entropy score is at or below that threshold, and the theorem states the resulting marginal error rate among accepted judgments is at most α.

Why This Matters

Impact on research: The paper moves LLM-as-a-judge reliability from heuristic confidence scores to formal risk control. It states that SCOPE is the first framework to provide finite-sample false-discovery-rate control for single-model pairwise LLM judging, and BPE actively mitigates position bias, which supports fairer model comparisons. The framework requires no retraining and no access to model internals beyond preference-token probabilities.

Real-world applications:

• Model leaderboards and arena-style crowdsourced comparisons, where a position-biased judge can flip the ranking of closely matched models.
• Filtering training signals for RLHF and reward models, where systematically wrong pairwise decisions can bias a policy toward stylistic artefacts such as verbosity.
• Large-scale annotation pipelines, where distilled preferences can inherit a judge's self-preference and familiarity biases downstream.
• Automated benchmarking and generation-quality evaluation, where abstaining on genuinely ambiguous comparisons is preferable to emitting a confident but wrong verdict.

Industry relevance: The calibration deploys as a drop-in layer on top of any existing judge, including specialized fine-tuned critics, without retraining. It exposes a single user-tunable knob, α, that trades coverage against certified error rate — an operationally useful control for teams that need a known reliability budget rather than an opaque quality score. The cost is approximately two forward passes per instance for BPE.

Future Directions

• Robustness beyond exchangeability: The guarantee assumes calibration and future queries are exchangeable; distribution shifts across benchmarks, prompt variations, or strategic model behavior may weaken it.
• Black-box deployment: BPE is a white-box measure requiring model probabilities or logits, so adapting it to fully black-box or API-only evaluators without approximation remains open.
• Beyond pairwise judging: Extending selective guarantees to point-wise evaluation, multi-response ranking, rubric-based scoring, and interactive critique.
• Overhead and scale: Reducing the roughly two-forward-pass cost of bidirectional evaluation for very large evaluation campaigns.

Target Audience

Researchers and practitioners working on LLM evaluation and benchmarking, uncertainty quantification, selective prediction, and conformal risk control; ML engineers and platform teams who deploy LLM judges in leaderboards, annotation pipelines, or RLHF reward modeling; and statisticians interested in finite-sample validity guarantees applied to modern language-model evaluation.

Authors’ abstract

Large language models (LLMs) are increasingly used as scalable judges in pairwise evaluation, but they remain prone to miscalibration and biases. We propose \textsc{Scope} (Selective Conformal Optimized Pairwise Evaluation), a framework that calibrates an acceptance threshold so that, under exchangeability, the error rate among non-abstained judgments is at most a user-specified level $α$. To supply \textsc{Scope} with a bias-neutral uncertainty signal, we introduce Bidirectional Preference Entropy (BPE), which queries the judge under both response positions and converts the order-averaged preference probability into an entropy-based score. Across various pairwise judging benchmarks, BPE outperforms standard confidence proxies in calibration and discrimination, while \textsc{Scope} consistently satisfies the target risk bound (empirical FDR $\approx 0.097$--$0.099$ at $α=0.10$) and retains substantial coverage. Compared to vanilla baselines, \textsc{Scope} accepts up to $2.4\times$ more judgments under the same risk constraint, demonstrating that BPE enables reliable and high-coverage LLM-based evaluation.