Learning From Preferences and Pairwise Comparisons

Asking “Which response is better?” can be more reliable than asking a reviewer to assign an exact score from one to ten. Pairwise data reveal relative preference under a stated criterion.

The criterion must still be explicit. Helpful, concise, safe, persuasive, and factually supported are different dimensions, and one comparison can mix them.

The arithmetic that turns a pile of “this one is better” into a single score is old. It is older than the systems that now depend on it. Bradley and Terry published it in Biometrika in 1952, in a paper about incomplete block designs. Their model says the probability that A is chosen over B depends only on the difference between two latent scores.

That is still the assumption a modern reward model is fitted under. The paper that opened modern preference learning says so in its own words, rather than leaving the lineage to be inferred: “This follows the Bradley-Terry model (Bradley and Terry, 1952) for estimating score functions from pairwise preferences, and is the specialization of the Luce-Shephard choice rule (Luce, 2005; Shepard, 1957) to preferences over trajectory segments.” That is Christiano and colleagues, writing in 2017. They note that it is the same idea as the Elo rating used for chess. A seventy-year-old paper on incomplete block designs is not background reading for reward modelling. It is the equation being solved.

Preference learning appears in products far beyond conversational AI. Wherever it appears, the same pile of pairwise votes can be assembled into very different scores. It depends on who is allowed into the comparison, and how often.

Chatbot Arena is the case where that became public. On 29 April 2025 a paper called The Leaderboard Illusion counted the models being tested there privately. Singh and twelve co-authors report: “At an extreme, we identify 27 private LLM variants tested by Meta in the lead-up to the Llama-4 release.” The same paper estimates that Google and OpenAI had received 19.2% and 20.4% of all arena data, against 29.7% for 83 open-weight models combined.

The platform did not deny the practice. LMArena's response of 9 May 2025 says: “In a future policy release, we will explicitly state that model providers are all allowed to test multiple variants of their models pre-release, subject to our system's constraints.” Nothing there is cheating. It is a documented policy. But a score assembled from twenty-seven private attempts, of which one is submitted, is not the same measurement as a score assembled from one attempt. The arena's arithmetic cannot tell the reader which it is holding. Before asking what a preference dataset says, ask what got to enter it.

A preference model converts many relative judgments into estimates that can guide ranking, selection, or optimization.

How little of that judgment it can take is worth a number. In 2017 Christiano and colleagues trained agents on Atari games and simulated robots with no reward function at all. The only signal was non-expert answers to which of two short video clips looked better. That was enough “while providing feedback on less than 1% of our agent’s interactions with the environment”. The sharpest case was a behaviour nobody had written a reward for. A Hopper robot doing a sequence of backflips “was trained using 900 queries in less than an hour.” Nine hundred binary choices. The agent learned to land upright and repeat.

The pipeline that produced that is five steps, and none of them is optional. Sample pairs that reveal a useful distinction rather than a trivial win. Collect the judgment with the context, the criterion and a way to say neither. Fit the model that estimates which features make an output more likely to be chosen. Validate on held-out pairs, annotator slices and criterion consistency. Then use the result with guardrails, ranking candidates or shaping optimization while factual and policy checks stay outside the learned score.

These feedback formats trade annotation effort, information density, and cognitive burden against each other. A scalar rating is compact to report, but every reviewer uses the scale differently and anchors move the judgment. A pairwise choice is often easier for subtle quality and supports preference modelling, at the cost of needing many comparisons. A full ranking carries the richest relative information and collapses fastest, because inconsistency rises with the length of the list.

What none of the three formats removes is disagreement. The size of it has been measured rather than estimated. OpenAI reported it in 2022 for the labelers who built InstructGPT: “Despite the complexity of the task, we find that inter-annotator agreement rates are quite high: training labelers agree with each-other 72.6 ± 1.5% of the time, while for held-out labelers this number is 77.3 ± 1.3%.” The authors are right that the number is high for a task this soft. Read from the other end, it says something else. On roughly one comparison in four, two trained people looking at the same pair under the same instructions gave opposite answers.

An independent measurement lands in the same region. Zheng and colleagues, at LMSYS and UC Berkeley, report in the MT-Bench study that “The agreement under setup S2 (w/o tie) between GPT-4 and humans reaches 85%, which is even higher than the agreement among humans (81%).” Three quarters to four fifths is the ceiling, not the floor. A pairwise label is not a reading taken off an instrument. It is one person's answer, and the next person disagrees about one time in five.

A useful comparison dataset needs more than a button labeled A or B. It needs a reason to believe that disagreement between reviewers has not destroyed the conclusion. That reason can be measured instead of assumed.

A government laboratory measured it first, on a collection that still exists. In 2000 Ellen Voorhees, at NIST, had different assessors judge the same TREC documents. Their sets of relevant documents overlapped only around 0.4. The rankings of retrieval systems computed from those disagreeing judgment sets nonetheless correlated at a Kendall's tau of 0.890. Twenty-five years later, eight researchers repeated the procedure on the TREC Deep Learning 2019 collection and measured tau = 0.897. They summarise the original in a line: “Voorhees (2000) report overlap among annotators as a measure of agreement, observing values around 0.4.”

That pair of numbers — overlap around 0.4, tau of 0.890 and 0.897 — is the whole reason a preference study can be interpreted at all. Individual judgments are contested. The comparative ordering built from enough of them can still be stable. Which of the two situations you are in is a property of how the study was run, not something visible in the final score. So run it in a way that lets you check.

Six moves make that possible. Separate the criteria, so accuracy, style, safety and relevance are not hidden inside one vague question. Sample informative pairs, including close calls and important failures. Allow uncertainty, with tie, both-poor and insufficient-evidence answers. Record reviewer context: expertise, instructions, language, conflicts of interest. Test stability by repeating selected pairs across time, groups and presentation order — the Voorhees design in miniature. And validate downstream, by measuring whether optimizing the preference improved the product outcome that motivated collecting it.

Preference data can train rankers, reward models, or selectors. They do something else as well. They can reveal dimensions that are hard to encode as one target.

What they cannot do is become the objective outright. That failure has been given a curve rather than a warning. Gao and colleagues fitted scaling laws for the gap between a learned proxy reward and the gold reward it stands in for. Their abstract names the mechanism: “Because the reward model is an imperfect proxy, optimizing its value too much can hinder ground truth performance, in accordance with Goodhart’s law.” Rafailov and colleagues found direct alignment algorithms deteriorating the same way in 2024: “we find that DAA methods deteriorate not only across a wide range of KL budgets but also often before even a single epoch of the dataset is completed”. The decay does not wait for a long run to reveal itself.

Later lessons distinguish demonstrations, weak labels, active queries, and reinforcement signals. Each method uses human information differently and creates different failure modes.