Skip to content
AI.info

Generative AI

Preference Data, Rubrics, and Reward Models

Design pairwise preference data and reward models while measuring disagreement, positional bias, and reward-model generalization.

By the end you can

“Better” is not a scalar until a policy makes it one

Two answers can trade factual accuracy, completeness, tone, caution, and speed. Asking a rater to choose one without a rubric hides those tradeoffs inside an unexplained label. Preference data becomes useful when the comparison context and decision rule are explicit. The label then records a judgment under a policy, not a universal ranking of human value.

How unstable that judgment is has been measured. Trained labellers on a 2020 summarization study agreed with the researchers who wrote their instructions 77% ± 2% of the time. Those researchers agreed with each other only 73% ± 4% of the time. Stiennon and colleagues report both figures in “Learning to summarize from human feedback”. The chance baseline for a two-way comparison is 50%. So the people who defined the standard could not reproduce it among themselves three times in four. This lesson treats that gap as its object of study, not as noise to be averaged away.

A preference label is a documented policy observation, not an objective property of the response.

Visual

From candidate responses to a learned reward signal

Each transition can introduce systematic bias, and each of the later ones has been measured. At the rubric step: swapping the display order of two near-identical answers flipped the verdict more often than not for two of the three LLM judges tested in the MT-Bench study. At the disagreement step: the summarization and InstructGPT papers both reported inter-annotator agreement in the seventies, not the nineties. At the reward-model step: a signal carrying no quality information at all reproduced most of the measured gain. At the out-of-sample step: reward models were being shipped without a benchmark, which is why RewardBench exists. The figures behind each of those sentences are in the sections below.

FigureProcess · 6 steps
  1. 1

    Sample prompts

    Choose domains, risks, languages, and difficulty levels.

  2. 2

    Generate candidates

    Vary policies, checkpoints, decoding, and response styles.

  3. 3

    Apply a rubric

    Raters compare outputs under written criteria and escalation rules.

  4. 4

    Resolve disagreement

    Record uncertainty, adjudication, and rater population.

  5. 5

    Train the reward model

    Fit a score whose differences predict observed preferences.

  6. 6

    Validate out of sample

    Test new prompts, policies, adversarial styles, and distribution shifts.

Comparison

One comparison can hide several objectives

Separating dimensions improves analysis even when the final policy combines them. The split is not tidy bookkeeping. RewardBench is organised the same way — prompt-chosen-rejected trios spanning chat, reasoning and safety — and that structure is what let its authors locate the difficulty. The subsets with low ceilings turn out to be mostly in Chat Hard and Reasoning, rather than spread evenly across the suite. A single undifferentiated “overall quality” label cannot tell you which dimension moved. So it cannot tell you whether a policy improved its answers or improved its manner.

FigureComparison · 3 columns

Task correctness

Does the answer satisfy the factual or procedural contract?

  • Needs evidence or expert judgment
  • May allow several valid outputs
  • Should dominate in consequential tasks
  • Cannot be inferred from eloquence

Interaction quality

Is the answer clear, relevant, appropriately concise, and responsive?

  • Depends on user context
  • Can conflict with completeness
  • Easy to reward through surface style
  • Needs accessibility and language slices

Risk behavior

Does the answer avoid prohibited help and escalate uncertainty appropriately?

  • Includes harmful compliance and false refusal
  • Policy can vary by domain
  • Requires adversarial cases
  • Should record consequence and reversibility

Example

Preference collection has its own attack surface

These effects can make a reward model favor how an answer looks over what it says. Three of the five below have been quantified on named datasets.

  • Position bias: Raters prefer the first or second response independent of quality, and the effect survives when the rater is itself a model. The MT-Bench study showed each LLM judge the same pair of near-identical answers twice, once in each order, and counted how often the verdict held. With the default prompt, Claude-v1 was consistent on only 23.8% of cases and favoured whichever answer came first 75.0% of the time. GPT-3.5 was consistent 46.2% of the time. GPT-4, the best of the three, managed 65.0%, rising to 77.5% with a few-shot prompt. Zheng and colleagues put it plainly — “The position bias can be very significant. Only GPT-4 outputs consistent results in more than 60% of cases.”
  • Length bias: Detailed answers receive higher ratings even when they contain unsupported claims, and the effect is large enough to account for most of what RLHF appears to buy. Singhal and colleagues threw the learned reward model away and ran PPO against a reward that scored nothing but proximity to a target length. It nearly kept up: “We find that PPO performance with this length-only reward is close to standard PPO (56% vs 58% win-rate of standard PPO on WebGPT and 64% vs 63% win-rate of standard PPO on RLCD).” Those are simulated-preference win rates against SFT. On WebGPT, only 2.0% of standard PPO's reward improvement was attributable to non-length features.
  • Authority bias: Formal language and citations are mistaken for correctness. A rubric that never asks a rater to check the evidence behind a claim will reward the appearance of evidence instead of the substance of it.
  • Annotator mismatch: General raters judge specialized legal, medical, or technical content, and a shipped dataset can be thinner than its headline agreement figure suggests. InstructGPT was built with roughly 40 screened contractors, and the paper reports: “Despite the complexity of the task, we find that inter-annotator agreement rates are quite high: training labelers agree with each-other 72.6 ± 1.5% of the time, while for held-out labelers this number is 77.3 ± 1.3%.” Its limitations section adds the design choice that matters more than the rate. Most comparisons were labelled by a single contractor, for cost reasons. On those items disagreement was not resolved, adjudicated or recorded. It was never observed at all.
  • Policy drift: Rubrics change while old and new labels stay mixed together, without version tags. An agreement statistic computed across two rubric versions describes neither of them.

Key idea

A reward model can be optimized beyond its competence

The reward model learns from a limited distribution of prompts and candidate responses. Policy optimization may then discover outputs that score highly because they exploit style or formatting shortcuts. This is distribution shift created by the optimization itself. Once the policy moves away from the reward model's training data, independent human evaluation and adversarial candidate generation are needed.

The failure was demonstrated against real human labels first. In the 2020 summarization work, policies were trained with a range of KL penalties against an earlier version of the reward model, rm3. Under light pressure they improved. Then they turned. The authors describe the figure: “Under light optimization, the models improve (according to labelers). However, as we optimize further, true preferences fall off compared to the prediction, and eventually the reward model becomes anti-correlated with human preferences.” Not merely less accurate, and not merely saturated: anti-correlated. Pushing the score higher made the labellers like the output less.

The same paper prices the obvious fix. An ablation over 7 reward models of 160M–13B parameters, trained on 8k–64k human comparisons, found that doubling the training data raised validation accuracy about 1.1%. Doubling model size raised it about 1.8%. Buying more preference labels does not buy much reward-model accuracy.

About two years later the same effect was mapped as a curve. “Scaling Laws for Reward Model Overoptimization” states its method in the abstract: “In this work, we use a synthetic setup in which a fixed "gold-standard" reward model plays the role of humans, providing labels used to train a proxy reward model.” The 6B-parameter reward model from Ouyang et al. (2022) played the part of the humans. It labelled 100,000 synthetic comparisons, with 10% held out as a test set, and those labels trained proxy reward models ranging from 3M to 3B parameters. Plotted against d = sqrt(KL), the gold score took two different shapes depending on how the pressure was applied: R_bon(d) = d(alpha_bon − beta_bon·d) for best-of-n, and R_RL(d) = d(alpha_RL − beta_RL·log d) for reinforcement learning. The alpha and beta coefficients vary smoothly with proxy reward-model size, along approximate logarithmic trends. So the overoptimization point is not an accident of one run. It is a property of the setup, and it has a location you can estimate before you cross it.

The stronger the policy optimizes a learned reward, the more important it becomes to test outside the reward model’s comfort zone.

Steps

Run a preference-data reliability study

Treat the way the annotators work as a measured system. Three of these six steps already have published worked examples, with numbers you can compare your own against.

Step 3, measure disagreement. The summarization labellers were onboarded, given written instructions and monitored throughout the project. The result, in the paper's own words: “As a result of our procedure, we obtained high labeler-researcher agreement: on a subset of comparison tasks, labelers agree with researchers 77% ± 2% of the time, while researchers agree with each other 73% ± 4% of the time.” Against a 50% chance baseline for two-way comparisons, that is agreement roughly three times in four, under close supervision. And the researcher-researcher rate sits below the labeller-researcher rate. That is not what a story about untrained annotators would predict.

Step 4, audit style shortcuts. The MT-Bench authors ran precisely this experiment. They took 23 model answers containing numbered lists and made each one longer by rephrasing the list and prepending it, adding no new information whatsoever. Claude-v1 and GPT-3.5 preferred the padded answer in 91.3% of cases; GPT-4 in 8.7%. As a calibration check on the setup, all three judges correctly returned a tie when shown two identical answers. So the 91.3% is a response to the padding, not to the harness. The paper names the failure: “Verbosity bias is when an LLM judge favors longer, verbose responses, even if they are not as clear, high-quality, or accurate as shorter alternatives.”

Step 5, validate the reward model. RewardBench was the first toolkit built for the job. Its abstract: “The RewardBench dataset is a collection of prompt-chosen-rejected trios spanning chat, reasoning, and safety, to benchmark how reward models perform on challenging, structured and out-of-distribution queries.” It holds 2,985 manually verified trios, and 75 reward models were evaluated on it. Re-running the whole suite costs about 1,000 A100-hours, which is the real budget line behind the phrase “validate out of sample”. Lambert and colleagues report that “the subsets with low ceilings” are “mostly in the Chat Hard and Reasoning sections”, and that the adversarial Chat Hard pairs are ones where “two answers are written in a similar style, but with slightly different subjects”. Style held fixed, substance varied. That is step 4's audit turned into a standing benchmark.

FigureProcess · 6 steps

  1. 1. Randomize presentation

    Shuffle candidate order and mask model identity where possible.

  2. 2. Calibrate raters

    Use shared examples, domain tests, and explicit uncertainty options.

  3. 3. Measure disagreement

    Report agreement by rubric dimension and prompt slice.

  4. 4. Audit style shortcuts

    Control length, tone, citations, and formatting while holding substance fixed.

  5. 5. Validate the reward model

    Test new policies, adversarial candidates, and expert-reviewed cases.

  6. 6. Version the policy

    Tie labels and models to the exact rubric and adjudication process.

Preference learning is only as clear as the judgment process

Pairwise comparisons can capture qualities that are difficult to express as one canonical answer. They also compress disagreement and context into a small label. It is a label that two researchers who wrote the rubric themselves assigned differently about a quarter of the time, that flips when the two answers change places, and that a length heuristic alone can largely reproduce.

The next lesson compares policy-optimization methods that use these preferences. Whatever the algorithm, the data and the rubric remain the source of the objective.

Key takeaways