Maximizing the efficiency of human feedback in AI alignment: a comparative analysis

AI summary

Overview

• Research area: Human-Computer Interaction and AI alignment, specifically the sampling and aggregation stages of Reinforcement Learning from Human Feedback (RLHF) reward modeling. The paper sits at the intersection of preference learning, active sampling, game theory, statistics, and social choice theory.
• Technical level: Intermediate. Readers should be comfortable with the idea of pairwise preference comparisons, latent utility scores, and reward models, but the paper explains its algorithms step by step and includes full pseudocode.
• Scope: A simulation-based comparative study of eight preference-elicitation methods (random sampling, Copeland round-robin, Borda count, Elo, Swiss tournament variations, and the authors' Swiss InfoGain) measured by how well their estimated item values correlate with ground-truth latent values under constrained annotation budgets.

Code is released at https://github.com/achouliaras/aics2025_rlhf_sampling, and an extended version with appendix is at https://arxiv.org/abs/2511.12796. The work was supported in part by EU Horizon projects 101092912 (MLSysOps) and 101160671 (DIGITISE).

What This Paper Is About

RLHF normally collects human preference data by showing annotators randomly chosen pairs of model outputs, then fitting a Bradley-Terry model to those comparisons. The authors argue this default has gone essentially uncontested since Christiano et al. introduced it, and that it wastes human effort: random pairs often produce redundant or uninformative comparisons, especially when the annotation budget is small. The goal of the paper is to ask how human effort can be spent better, and to compare alternative pairing and aggregation strategies across different annotation budgets to find which ones yield accurate reward models most efficiently.

Key Contributions

1. A systematic head-to-head comparison of eight preference-elicitation methods under matched annotation budgets: the Bradley-Terry estimator, Borda count with random sampling (Borda-RNG), Borda count with Copeland's round-robin sampling (Borda-Copeland), Elo rating with random sampling (Elo-RNG), Elo with Copeland sampling (Elo-Copeland), a pure Swiss tournament using Elo ratings (Swiss Tournament), random sampling followed by a Swiss tournament (Elo-RNG+Swiss), and the proposed Swiss-InfoGain. A Quicksort-based Borda approach inspired by Maystre and Grossglauser was also implemented but omitted from the figures due to poor performance.
2. The Swiss InfoGain algorithm, a Swiss tournament system that replaces Elo-closeness pairing with a pairing rule based on an approximation of mutual information gain, (IG(x_i, x_j) = P(x_i \succ x_j) \cdot P(x_i \prec x_j)), derived from a second-order Taylor approximation of entropy following S. Shams. The approximation is maximized when the predicted win probability is 0.5, i.e., when the outcome is most uncertain.
3. Identification of the regime where the Bradley-Terry default becomes sub-optimal, including the crossover points at which other methods overtake it as the annotation budget grows.
4. A resource-aware framing of the RLHF pipeline, positioning these methods as improvements to the upstream data-assembly stage rather than as replacements for the Bradley-Terry probability model or for active-learning reward modeling.

Main Findings

• Swiss InfoGain wins in low-to-medium budgets. It outperformed all other methods across most of the comparison budget range tested, and it beat even the Copeland methods while using roughly 9 times less data in the fixed-budget experiment.
• Borda-Copeland is strongest with essentially unlimited budgets. It displayed the best performance in the fixed-budget experiment at around 0.96 correlation, but requires 4950 pairs — all symmetrical item combinations — versus roughly 550 pairs for the other methods, about 9 times more comparisons.
• The crossover points are large. The Borda-Copeland method needed more than 15,000 pairs to clearly outperform Swiss InfoGain (measured as approximately 83% greater data efficiency for Swiss InfoGain). For the Bradley-Terry model to match Borda-Copeland's performance, a staggering 20,000 pairs were required. To outperform Swiss InfoGain, Bradley-Terry needed more than 17,500 pairs, corresponding to roughly 86% greater data efficiency.
• Swiss InfoGain is cheap relative to its competitors. Across the 100 seeds it needed on average fewer than 2500 pairs; the pure Swiss tournament method operated on average around 500 pairs across the same seeds.
• Random sampling variants underperform. Borda-RNG, Elo-RNG, and Elo-RNG+Swiss displayed subpar performance compared to the Bradley-Terry estimator, and only the full Swiss tournament methods were able to systematically beat Bradley-Terry in low annotation budgets.
• A 50-50 split is a good default. When allocating budget between initial random pairing rounds and Swiss pairing rounds, the authors examined all combinations and found the 50-50 rule yielded better results on average. This is convenient in practice because the randomly sampled pairs can be pre-generated, reducing computation during the annotation session.
• There is an operational trade-off. The Bradley-Terry model with random pairs can generate all pairs before the annotation task begins, whereas the Swiss tournament methods need multiple rounds to analyze results and propose new pairs.
• Ties are modeled explicitly. The probability of a tie is modeled as (P(x_a \sim x_b) = \frac{1}{3}e^{-|v(x_a)-v(x_b)|/\sigma}), so that nearly equal items are equally likely to be preferred either way or tied (probability 1/3 each), with the remaining mass allocated by the Elo-style formulation. The authors note that for large value differences, the tie probability can exceed the probability of the weaker item being chosen.

Methodology in Plain English

The authors ran simulations rather than human studies. They created a set of (N = 100) items, each assigned a latent true value drawn from a normal distribution (\mathcal{N}(1000, 200)). Comparisons between items were generated probabilistically: when values are close, a tie or either win is equally likely at 1/3 each; as the gap grows, the better item wins more often according to an Elo-style formula with a scale factor of 400. The Elo system's stated behavior is that a 100-point difference gives the better item 64% chance of winning, 200 points gives 72%, and 400 points gives around 90%.

Each method then proposes which pairs to compare, sees the simulated outcomes, and produces an estimated value (\hat{v}) for every item. Success is measured by the correlation between estimated and true values, (r(\hat{v}, v)), which approaches 1 for a perfect estimator. Elo-based methods started from an initial rating of 1000, and non-Elo methods had their results rescaled back to the (\mu = 1000), (\sigma = 200) distribution.

Two experiments were run. The first compared all methods at a roughly matched budget of around 550 pairs, with the Copeland variants as the expensive ideal case. The second swept budgets from 500 all the way to 20,000 pairs for a subset of methods (Bradley-Terry, Borda-RNG, Elo-RNG, and Elo-RNG+Swiss), using 100 random seeds and reporting mean values with 95% confidence intervals. The Copeland methods could not be scaled across that full range because of their fixed 4950-pair granularity.

Why This Matters

• Research impact: The paper challenges an assumption that has largely gone unexamined in RLHF — that random pair sampling plus Bradley-Terry fitting is the natural default. It shows that the default's efficiency collapses at low budgets and that the choice of sampling strategy is a first-class design decision rather than an implementation detail. It also connects preference learning to social choice theory and game theory tooling that the RLHF literature rarely uses.
• Real-world applications:
  • Studying LLM alignment pipelines where annotator time is expensive and limited, the setting the paper explicitly targets with the phrase "constrained annotation budgets."
  • Budget planning for annotation campaigns: the crossover points reported here give teams a rough sense of when it is worth paying for the costly round-robin coverage and when a cheaper adaptive scheme suffices.
  • Reducing annotator burden, which the authors argue is a prerequisite for "scalable and sustainable deployment of preference-based learning systems."
  • Designing data-collection tools, since sequential methods like Swiss InfoGain require multiple annotation rounds in which results are analyzed before the next pairs are suggested, whereas random pairs can be pre-generated.
• Industry relevance: Any organization training reward models on human preference data faces the same trade-off between annotation cost and model quality. The paper's central claim — that adaptive, resource-aware strategies reduce redundancy, enhance robustness, and yield statistically significant improvements in preference learning — has direct implications for how preference data pipelines are built. The released code makes the methods directly reusable.

Future Directions

• Validation with real annotators and real model outputs. The reported experiments are simulations over synthetic latent values; the paper does not report results from human annotation sessions or from any named language model, so the degree to which these gains transfer to real, noisy annotator behavior remains untested here.
• End-to-end evaluation inside a full RLHF loop. The authors explicitly position their methods as improvements to the upstream data-assembly stage. Whether better item-level rankings translate into better downstream policy behavior — the reward model must amortize the signal across every state in a trajectory, or every token in the case of large language models — is left open.
• Combining with active learning and modern reward modeling. The paper states its methods do not aim to replace the Bradley-Terry probability model or newer active-learning approaches, so integrating the two families is a natural next step.
• Accounting for the sequential cost. Swiss InfoGain and related methods need multiple rounds and therefore incur latency and orchestration overhead that random pre-generated pairs avoid. A full cost model that includes this overhead alongside raw comparison counts is not reported.

Target Audience

Researchers and practitioners working on RLHF, reward modeling, preference learning, or AI alignment who make decisions about how human preference data is collected. It is also relevant to human-computer interaction researchers interested in reducing annotation workload, and to engineers building annotation or evaluation platforms for language models. Readers with a basic grasp of pairwise preference modeling will follow the paper comfortably; the algorithm listings make the implementation details accessible without requiring deep statistical background.

Authors’ abstract

Reinforcement Learning from Human Feedback (RLHF) relies on preference modeling to align machine learning systems with human values, yet the popular approach of random pair sampling with Bradley-Terry modeling is statistically limited and inefficient under constrained annotation budgets. In this work, we explore alternative sampling and evaluation strategies for preference inference in RLHF, drawing inspiration from areas such as game theory, statistics, and social choice theory. Our best-performing method, Swiss InfoGain, employs a Swiss tournament system with a proxy mutual-information-gain pairing rule, which significantly outperforms all other methods in constrained annotation budgets while also being more sample-efficient. Even in high-resource settings, we can identify superior alternatives to the Bradley-Terry baseline. Our experiments demonstrate that adaptive, resource-aware strategies reduce redundancy, enhance robustness, and yield statistically significant improvements in preference learning, highlighting the importance of balancing alignment quality with human workload in RLHF pipelines.