Predicting Human-Gain Curves under Local Proxy Optimization from Repeated Ratings

AI summary

Overview

Research area: AI evaluation and alignment — specifically, how to predict whether optimizing a measurable proxy (a reward model, an LLM judge, or even a single human rater) will actually improve human-perceived quality, and by how much.

Technical level: Intermediate. The paper is written for readers comfortable with correlation, variance, KL divergence and regression, but the core idea can be grasped without following the cumulant algebra.

Scope: The paper proposes PACE (Proxy Alignment and Curvature Expansion), a pre-optimization diagnostic that uses a small matrix of repeated human ratings to forecast the human-gain curve induced by gently optimizing a fixed proxy, and validates that forecast prospectively on SummEval, WMT MQM, OpenMEVA-MAGS, a GPT-4o judge proxy, and a LoRA-DPO training bridge.

What This Paper Is About

Practitioners routinely optimize a proxy score — a learned reward model, a judge's rating, or a reranker — believing that higher proxy scores mean better human-perceived outputs. The paper argues that average proxy–human correlation cannot answer the practical question: as optimization strength increases, does human quality rise, flatten, or fall? PACE answers this before any large candidate pool is built, by estimating two coefficients from repeated ratings on a small calibration set and using them to forecast the held-out human-gain curve over a checked range of optimization strength.

Key Contributions

1. A formal definition of the human-gain curve. The authors model local proxy optimization as an exponential tilt of a reference item distribution, measure intervention strength by the induced KL radius d, and define the human-gain curve as the change in expected human gold quality along that path.

2. A scale-free two-coefficient local law. They derive that the gain follows A√d + Bd + O(d^{3/2}), where A = √2·Cov(J,r)/σ_r measures first-order proxy–human alignment and B captures leading curvature (a second-order proxy–gold moment minus a proxy-skewness correction). The law does not require the proxy and human scores to share a numerical scale.

3. A prospective calibration protocol from repeated ratings. Repeated human ratings are split by item into 30% calibration and 70% evaluation sets, and the two coefficients estimated on calibration items are tested against the exact held-out gain curve on disjoint items, across 50 resamplings per path.

4. Matched-information comparisons and boundary checks. PACE is compared with flexible proxy-conditional calibrators (binning, splines, isotonic regression), with discrete selection rules such as Best-of-N translated into induced KL radius and relative effective sample size, and with a three-seed LoRA-DPO training experiment as a bridge to actual optimization.

Main Findings

• PACE beats simple baselines on held-out prediction. At d = 0.10, endpoint MAE for A+B versus A-only versus zero-gain was 0.0174 / 0.0310 / 0.2147 on SummEval, 0.0118 / 0.0165 / 0.0680 on the GPT-4o judge proxy, 0.0846 / 0.1298 / 0.8524 on WMT MQM, and 0.0148 / 0.0162 / 0.3327 on OpenMEVA-MAGS. The reliability-only surrogate A_ν was far worse in three of four settings (0.0880, 0.1508, 0.6230, 0.0207).

• Curvature is a small but systematic correction. Adding the Bd term reduced pre-pool CurveMAE by 34.3% (SummEval), 21.7% (GPT-4o), 26.4% (WMT) and 6.4% (OpenMEVA), yet the median share of B̂d in the predicted gain magnitude was only 1–13%. The endpoint reduction grows with radius, reaching 8–44% at d = 0.10.

• The improvement is aggregate, not universal. A+B had lower CurveMAE than A-only on 55.6–67.0% of splits, and the OpenMEVA paired interval includes zero.

• Competitive with flexible calibration under matched information. Seeing only calibration items, A+B had lower CurveMAE than binned calibration in all four settings and than splines in three. Isotonic regression was close, with a significant reduction only on WMT (−0.00039, 95% interval [−0.00070, −0.00009]).

• More information helps calibrators, and the two-term summary still suffices. When target-pool proxy scores were available, isotonic and spline calibrators beat the pre-pool A+B reference in most settings (e.g., isotonic direct integration 0.00909 on SummEval and 0.05211 on WMT); binned calibration did not. Projection and direct integration differed by only 0.00002–0.00286 in aggregate CurveMAE, and replacing the two-term curve with cubic or spline fits changed CurveMAE by at most 2.1×10⁻⁵.

• Curvature sign is usually right, but magnitudes are fragile. B̂ agreed in sign with the held-out moment coefficient on 89.4–97.0% of splits and kept its sign across within-split bootstrap draws in 96.7–100%, yet 25–53% of split-level intervals included zero. On SummEval, a second expert rating cut the mean absolute error of B̂ from 0.0618 to 0.0428; injected noise removed the A+B advantage at 1.0 rating SD for experts and 0.5 SD for crowd workers.

• Proxy-side outliers destabilize curvature. After 1% positive 8-SD proxy contamination, B̂'s correlation with the clean OpenMEVA estimate was −0.11.

• Source coefficients are priors, not invariants. Source-only transfer beat zero gain on all three routes, but target calibration improved every route (SummEval → WMT: 0.0436 → 0.0259).

• Best-of-N mostly falls outside the checked window. At N = 2 the median induced radius was already 0.166–0.193, and only 4 of 2,101 Best-of-N points (across 191 paths and eleven values of N from 2 to 64) lay inside d ≤ 0.10 — all at N = 2 on SummEval expert paths where 85–89% of items share the top score. Prompt-conditional BoN at N = 2 left about 88% of prompts outside the window.

• The LoRA-DPO bridge realizes about half the idealized gain. With mean KL radius 0.03968, realized silver gain was 0.02081 versus an idealized tilt gain of 0.04197, a gap of 0.02116. Against realized gain, A+B had lower MAE than A-only in every seed and at all ten checkpoints (0.01721 vs 0.01905 averaged over seeds), but against the idealized frontier A-only was closer (0.00210 vs 0.00395), so part of A+B's realized-gain advantage is error cancellation. Fourteen of 46 comparator variants, including flexible calibrators, had lower realized MAE.

• Sanity checks support the theory. Theoretical and fitted coefficients correlated at least 0.90 for B and essentially one for A; random and shuffled proxies drove A and realized gain near zero while A_ν stayed large.

Methodology in Plain English

The researchers treat proxy optimization as a reweighting of a fixed pool of items: turning up optimization strength shifts probability mass toward items with high proxy scores. The strength of this shift is measured by KL divergence, which they call the radius d. They then prove that, near the reference distribution, the resulting change in average human quality follows a simple two-term curve in √d and d.

To estimate the two coefficients before any optimization happens, they use repeated human ratings. In human-proxy settings, one rater's scores act as the proxy and the remaining raters define a leave-one-rater-out human gold target; for external proxies such as reward models or LLM judges, the model supplies the proxy score and the repeated ratings supply gold. Items are split 30/70 into calibration and evaluation sets, the split is repeated 50 times per rating path, and the forecast from calibration is scored against the exact finite-pool gain curve on held-out items, always within d ≤ 0.10. They compare against an A-only predictor, a zero-gain predictor, a reliability-only surrogate, and flexible calibrators (binned means, penalized cubic splines, isotonic regression) fitted on the same calibration items. Evaluation uses endpoint MAE at d = 0.10 and CurveMAE averaged over five radii from 0.01 to 0.10, with paired differences tested by a 2,000-draw path-clustered bootstrap.

Why This Matters

Impact on research. The paper reframes proxy evaluation as a question about an intervention rather than an association: knowing that a reward model correlates with human judgments does not tell you whether optimizing it will help. It also supplies an analytic alternative to black-box reward-model benchmarks — a two-parameter summary that is interpretable, scale-free, and testable prospectively on disjoint items.

Real-world applications:

• Deciding whether to run Best-of-N or reranking at inference time, and at what N, before paying for a large candidate pool.
• Vetting a reward model or LLM judge for RLHF or DPO training, using a small repeated-rating calibration set rather than a full training run.
• Designing human evaluation studies, since the analysis shows how much repeated annotation is needed before curvature estimates stabilize.
• Choosing between releasing a stronger optimizer and abstaining, when the checked KL window is unreachable or curvature confidence intervals span zero.

Industry relevance. Teams that build preference-optimized models must decide how much optimization pressure to apply, and overoptimization is a known failure mode. A pre-optimization forecast that costs a few hundred repeated ratings, rather than a full training run, is directly actionable for model-release and data-collection decisions. The paper's warning that Best-of-N at N = 2 already exceeds the validated window is a concrete practical constraint.

Future Directions

• Extending beyond the local, one-sided regime. The law describes only small KL radii on the positive tilt branch; the terminal regime where the distribution concentrates on proxy maximizers, and the negative branch, are explicitly out of scope. A theory covering saturation and downturn would make the diagnostic usable at larger optimization strengths.

• Closing the realized-versus-idealized gap. The LoRA-DPO experiment showed realized gain at roughly half the idealized tilt gain, which means the forecast systematically overpredicts what an actual optimizer delivers. Understanding why optimizers realize only part of the frontier — and modeling it — is an open question.

• Robust curvature estimation. B̂ is destabilized by proxy outliers (correlation −0.11 after 1% contamination) and needs more human ratings per item for stability. Robust moment estimators, heavier-tailed corrections, or variance-reduction schemes for the repeated-rating design are natural next steps.

• Rules that are not raw-score tilts. Best-of-N and rank tilts require translation into an induced radius and path-specific calibration. A general theory of the gain curve for arbitrary selection rules, rather than a per-rule translation, remains open.

Target Audience

This paper is most useful to researchers and engineers working on RLHF, DPO, reward modeling, and LLM-as-a-judge evaluation; to evaluation scientists who design repeated-rating annotation studies; and to statisticians interested in the geometry of proxy optimization. Readers need comfort with KL divergence and moment expansions to follow the derivations, but the empirical protocol and the practical warnings about Best-of-N and curvature instability are accessible to practitioners who simply want to know how hard they can safely optimize a proxy.

Authors’ abstract

Optimizing a proxy objective can improve human-perceived quality, but the resulting gain can vary with optimization strength. Average proxy-human correlation alone does not reveal how human-perceived quality changes as optimization strength increases. We propose PACE (Proxy Alignment and Curvature Expansion), a pre-optimization diagnostic that uses repeated human ratings to predict the human-gain curve induced by a fixed proxy. The method models proxy optimization as a KL-regularized exponential tilt. At KL radius $d$, the local gain is approximated by $A\sqrt{d}+Bd$, where $A$ measures first-order proxy-human alignment and $B$ captures the leading second-order correction along the proxy-optimization path. We estimate $A$ and $B$ from repeated human ratings on calibration items and test the resulting gain-curve prediction on held-out evaluation items. Across SummEval, analysis-eligible WMT MQM clusters, an OpenMEVA-MAGS ROC/WP repeated-rating slice, and a GPT-4o judge proxy experiment, PACE consistently reduces held-out gain-prediction error relative to $A$-only, zero-gain, and reliability-only baselines. With the same pre-pool information, it is competitive with flexible proxy-conditional calibration, and a LoRA-DPO experiment provides a modest bridge to actual optimization. Transfer experiments across datasets and evaluation slices show that source coefficients are useful priors, but target calibration further improves prediction. The resulting diagnostic is local and one-sided: it characterizes gentle optimization along the positive tilt branch of a fixed item pool.