PRECISE: Reducing the Bias of LLM Evaluations Using Prediction-Powered Ranking Estimation

AI summary

Overview

Research area: Machine learning evaluation methodology — specifically statistical estimation of information retrieval / ranking metrics without full human annotation, applied to e-commerce search.

Technical level: Intermediate. The paper assumes familiarity with ranking metrics (Precision@K) and basic statistical estimation, but it frames the underlying idea (combining cheap-but-biased machine labels with expensive-but-scarce human labels) in accessible terms.

One-sentence scope: The paper introduces PRECISE, a Prediction-Powered Inference extension that produces low-variance, bias-corrected Precision@K estimates for search ranking systems by combining a small human-annotated query set with LLM judgments on a much larger unlabeled query set, and validates it on the public ESCI dataset and a live Indian e-commerce deployment.

What This Paper Is About

Search teams need human-annotated relevance judgments to know whether a ranking change helps or hurts, but human annotation is slow and expensive, and LLM judges — while cheap and abundant — carry systematic biases. The authors build a statistical framework that combines a small "gold" human-labeled set with LLM annotations on a large unlabeled set so the final metric estimate is both low-variance and corrected for LLM bias. The complication they solve is granularity: humans and LLMs label at the query–document level, while ranking metrics are computed per query and then averaged across the dataset.

Key Contributions

1. PRECISE-PPI, an extension of Prediction-Powered Inference to sub-instance annotations. The standard PPI++ estimator assumes the annotator labels at the same granularity as the metric "instance." The authors reformulate the estimator so the integration space matches query–document-level annotator output, making it valid for ranking metrics such as Precision@K.

2. A tractable reformulation of the integration space. They show that iterating over all binary vectors of length |C| (corpus size, in the order of millions) can be reduced to iterating over binary vectors of length K, because all corpus vectors in which the K retrieved documents are zero collapse into a single all-zero K-length vector. This reduces computational complexity from O(2^|C|) to O(2^K).

3. Empirical validation on both a public dataset and a production system. Experiments on ESCI (KDD Cup 2022) with known ground truth quantify bias and standard error across estimators and annotator models; a deployment on an Indian e-commerce search application with 8.5k Body queries (100 human-annotated, 8.4k unlabeled) demonstrates a pre-deployment decision that was later confirmed by A/B testing.

4. A cost-performance analysis of annotator choices. The paper compares Claude 3 Sonnet, Claude 3 Haiku, and the jina-reranker-v1-turbo-en cross-encoder on bias, standard error, and dollar cost, and reports that LLM judges with prompted uncertainty calibration outperform off-the-shelf cross-encoder confidence scores for PPI.

Main Findings

• Large unlabeled sets are unnecessary. With n = 30 gold samples, using 100× unlabeled data (3,000 queries) gives nearly identical performance to 2000× unlabeled data (60,000 queries) while reducing costs by 95%. Table 1 reports for true Precision@4 = 89.73%: Sonnet at 3k unlabeled had bias 0.52 and standard error 3.67, versus bias 0.82 and standard error 4.45 at 60k; Haiku at 3k had bias 0.42 and standard error 4.10, versus bias 0.01 and standard error 4.80 at 60k.

• PRECISE-PPI reduces variance relative to gold-only estimation and corrects LLM bias. On ESCI Precision@4 (Figure 3), the estimator produces tighter 95% confidence intervals than the gold-only estimator, with greater reduction at higher λ values, and is substantially less biased than the LLM-only prob and bin approaches.

• Calibrated LLM judges beat cross-encoders. In the cost-performance comparison (N = 60,000 unlabeled, n = 30 gold, Table 3): Gold-only bias 1.04 / std. error 4.45; Claude 3 Sonnet bias 0.70 / std. error 3.50 at $945.6; Claude 3 Haiku bias 0.29 / std. error 3.86 at $79.3; Jina Turbo bias 0.51 / std. error 4.26 at under $5.0.

• Haiku is a strong low-cost substitute for Sonnet. The paper reports comparable performance (bias 0.29 vs 0.70, standard error 3.86 vs 3.50) at roughly 12× lower cost ($79 vs $946 for 60k queries).

• Calibration quality tracks PPI effectiveness. Claude 3 Sonnet showed most true positives scoring at or above 0.5 and true negatives at or below 0.4; Haiku was slightly weaker with some true positives in the 0.6–0.8 range; Jina Turbo was poorly calibrated, with many true positives receiving scores at or below 0.4, explaining why it barely improved variance over gold-only estimation.

• The framework picked the winning treatment before A/B testing. Under strict relevance criteria on the production application data (Table 2, with the note that numbers are anonymized by introducing a randomly-selected baseline value), T1 showed clear improvement over control across all K values, described as a +13.4% relative improvement in Precision@4, with T2 showing similar but slightly lower gains. PRECISE-PPI at λ = 0.95 estimated 55.10% / 54.60% / 55.30% (K = 1/2/4) for production search, 59.50% / 59.10% / 60.30% for V1, and 59.70% / 59.20% / 59.40% for V2.

• Production A/B testing confirmed the offline prediction. In an equal-allocation A/B test (Table 4), T1 delivered +90 bps average orders per customer, +6 bps add-to-cart per customer, +407 bps BAU daily sales, and +137 bps average sale price; T2 delivered +42 bps, +5 bps, +174 bps, and +11 bps respectively.

• The simpler treatment won. The more elaborate V2 prompt preserving Indian ethnic context underperformed the simpler V1 translation-and-correction prompt, a counterintuitive result the authors flag as a reason to validate PRECISE-guided decisions with A/B testing.

• Search quality improved, especially on Hinglish queries. Across all corrected queries (Table 5), T1 achieved +571 bps CTR, +304 bps clicks per query session, +404 bps clicks per customer, and +782 bps average browse depth. On Hinglish queries, T1 gained +77 bps CTR, +406 bps clicks per query session, +494 bps clicks per customer, and +579 bps browse depth, while T2 lost ground on several Hinglish metrics (-154 bps CTR, -259 bps clicks per query session, -233 bps clicks per customer).

• Offline evaluation was fast. PRECISE-based treatment comparison finished within 2 hours of domain expert annotation, versus what the authors describe as weeks for traditional approaches.

Methodology in Plain English

The team starts from Prediction-Powered Inference, in which you take an average of machine-predicted metric values over a large unlabeled set and then add a correction term computed on a small human-labeled set. That correction term is what removes the annotator's systematic bias, while the large unlabeled set is what drives down variance. The standard formulation breaks when the annotator labels pieces of an item rather than whole items — which is exactly the ranking situation, where a human labels one query–document pair but the metric of interest is per-query Precision@K.

The fix is to redefine what you sum over. Instead of enumerating every possible binary relevance vector across the entire document corpus (intractable, since the corpus is in the millions), the authors note that Precision@K only ever looks at the top K retrieved documents. Every corpus-sized vector that puts zeros on those K documents can be merged into one all-zero vector of length K. So the sum runs over the 2^K binary vectors of the top-K positions. Since K is typically small (they mention K ≤ 10 as the practical regime), this is computable. The per-vector probability is the product of the annotator's independent per-document probabilities, raised or inverted according to each vector entry.

For the annotators, they prompt Claude 3 Sonnet and Claude 3 Haiku to produce a relevance verdict plus a verbal confidence label from a fixed six-level scale ("About Even" through "Almost Certain"), which they map to numeric scores in [0.5, 1.0], subtracting from 1.0 for irrelevant predictions. They then apply isotonic regression calibration on the labeled set. They compare two naive baselines against PRECISE-PPI: prob, which averages annotator probability scores across the K ranks, and bin, which thresholds scores at 0.5 and computes Precision@K against the K-hot prediction vector.

Evaluation proceeds in two stages. First, on ESCI — where the ground truth for the unlabeled set is known — they sample 50 gold datasets of size n = 30 and n = 100 from a pool of N = 60,000 unlabeled queries and inspect the sampling distribution of each estimator against the true Precision@4. Second, in the production setting they split 8,500 Body queries into 100 human-annotated and 8,400 unlabeled queries (an 84× ratio), estimate Precision@K for control and two reformulation treatments, and later check the prediction against a live A/B test.

Why This Matters

Impact on research. The paper extends Prediction-Powered Inference to a class of problems — hierarchical metrics where annotations live at a finer granularity than the metric itself — that the original framework does not cover. The complexity reduction from O(2^|C|) to O(2^K) is the enabling step that makes PPI practical for standard IR metrics, and it generalizes to other metrics computed over a bounded top-K window.

Real-world applications.

• Search relevance launch decisions. Teams without a standing annotator pool can decide whether to ship a ranking change using a few dozen to a few hundred human judgments rather than an infeasible full annotation effort.
• Query reformulation and translation. The deployed use case translates Hinglish queries into English and corrects typos and grammar — a setting where a small reformulation error can degrade relevance, making accurate pre-launch estimation business-critical.
• LLM-judge pipeline design. The calibration findings give concrete guidance on which judge models to use for bias-corrected estimation, and which to avoid.
• Cost-constrained evaluation of anytime-changing models. Since ranking models change frequently, the ability to re-evaluate with 3,000 unlabeled queries and 30 gold samples — instead of 60,000 — matters for repeated evaluation cycles.

Industry relevance. The work comes from Amazon AI and is grounded in a production Indian e-commerce application serving resellers in Tier-2 and Tier-3 cities, where the catalog contains millions of products. The reported business impact — +407 bps BAU daily sales and a several-fold return on investment for Treatment 1 — is presented as validation that a PPI-guided deployment decision produced measurable revenue outcomes. The reported 2-hour turnaround from annotation to decision is a directly operational claim.

Future Directions

1. Replacing or supplementing the gold set. The authors identify the reliance on a human-labeled "gold" set as the major bottleneck of any estimation method and suggest LLM-generated synthetic "silver" labels as a possible substitute.

2. Handling dynamic corpora. Extending PRECISE to retrieval systems where new documents are continuously added would avoid full re-annotation when the corpus shifts.

3. Multi-turn and multi-modal evaluation. Conversational search and multi-modal retrieval require sub-example-level estimates and present new bias-correction challenges, since evaluating them often involves user simulation or multi-modal benchmarks.

4. Ensembling multiple LLM judges. Combining judges with different strengths and biases could yield more robust assessments than any single-model evaluator within the PRECISE framework.

5. Online / real-time estimation. Adapting the framework so relevance assessments can be generated in real time would broaden applicability, and the authors note theoretical overlap with doubly robust estimation for online ranking.

Target Audience

Applied machine learning engineers and applied scientists responsible for search or recommendation quality who need to make launch decisions without a large annotation budget; information retrieval researchers interested in statistically grounded evaluation; and practitioners building LLM-as-a-judge pipelines who want to understand where judge bias enters a metric and how to correct for it. Readers should be comfortable with ranking metrics, confidence intervals, and the general idea of bias-variance trade-offs, but no deep statistical theory background is required to follow the argument.

Authors’ abstract

Evaluating the quality of search, ranking and RAG systems traditionally requires a significant number of human relevance annotations. In recent times, several deployed systems have explored the usage of Large Language Models (LLMs) as automated judges for this task while their inherent biases prevent direct use for metric estimation. We present a statistical framework extending Prediction-Powered Inference (PPI) that combines minimal human annotations with LLM judgments to produce reliable estimates of metrics which require sub-instance annotations. Our method requires as few as 100 human-annotated queries and 10,000 unlabeled examples, reducing annotation requirements significantly compared to traditional approaches. We formulate our proposed framework (PRECISE) for inference of relevance uplift for an LLM-based query reformulation application, extending PPI to sub-instance annotations at the query-document level. By reformulating the metric-integration space, we reduced the computational complexity from O(2^|C|) to O(2^K), where |C| represents corpus size (in order of millions). Detailed experiments across prominent retrieval datasets demonstrate that our method reduces the variance of estimates for the business-critical Precision@K metric, while effectively correcting for LLM bias in low-resource settings.