Reward Models Inherit Value Biases from Pretraining

AI summary

Overview

Research area: AI alignment and machine learning — specifically reward modeling, RLHF (reinforcement learning from human feedback), and the origins of value bias in language models.

Technical level: Intermediate. Readers will get the most out of it with some familiarity with how language models are trained (pretraining, instruction tuning, preference finetuning), what a reward model does, and the basic idea of log probabilities. The core argument, however, is stated plainly enough for a general technical reader.

Scope: A systematic empirical study showing that reward models inherit systematic value biases from their pretrained base LLMs, tracing those biases from reward scores back through instruction-tuned and pretrained logits, and testing how durable they are under controlled preference training.

What This Paper Is About

Reward models (RMs) are the component in an AI alignment pipeline that scores how good a model response is, and they are almost always initialized from an existing language model before being finetuned on human preference data. The authors ask whether the choice of that base language model — Llama versus Gemma — silently shapes which human values the resulting reward model prefers, even when the preference data and finetuning recipe are identical. They find that it does, robustly, and that the bias originates in pretraining rather than in the reward-modeling stage.

Key Contributions

1. A new RM interpretability method. The authors combine "exhaustive token search" (evaluating every token in a reward model's vocabulary against a prompt) with validated psycholinguistic corpora — the Big Two (agency/communion) and the Moral Foundations Dictionary (MFD2) — to quantify reward models' value preferences in psychologically meaningful terms.

2. Evidence of systematic value differences in real RMs. Across 10 leading open-weight reward models on RewardBench, Llama-based RMs consistently favor agency-related concepts (freedom, success, capability) while Gemma-based RMs consistently favor communion-related concepts (love, friendship, care). The effect size is medium (Cohen's d ≈ 0.40–0.43).

3. Tracing the bias to pretraining. The same agency/communion split appears in the log probabilities of the instruction-tuned Gemma and Llama models, and also in their pretrained versions — before any preference finetuning occurs.

4. Formulating the base-model difference as an implicit reward model. The authors derive a usable implicit reward score (mixture-weighted log-ratio, MWLR) from the difference in log probabilities between two models, and show that its highest-ranked token for "What, in one word, is the greatest thing ever?" is "Freedom" for Llama and "Love" for Gemma — across all 21 Llama/Gemma model pairings tested.

5. Demonstrating the durability of inherited bias. Through controlled RM training with identical data and hyperparameters, the authors show the gap between Llama and Gemma RMs is widest at initialization and narrows but never closes, requiring roughly 100k+ preference pairs to meaningfully attenuate.

Main Findings

• Llama RMs prefer agency, Gemma RMs prefer communion. For positively framed prompts ("the greatest thing ever"), Llama-based RMs rank agency words like "success," "skills," and "capability" higher, while Gemma-based RMs rank communion words like "love," "friends," and "relationships" higher. The pattern reverses for negatively framed prompts ("the worst thing ever"), producing a significant three-way interaction (category × base model × prompt valence, p < .001).

• The bias shows up in the top-ranked tokens that actually matter. For Gemma RMs, on average 5 of the top 10 scoring tokens are communion words and none are agency words; for Llama RMs, 3.67 are communion and 2.33 are agency. These are the tokens most likely to shape a downstream finetuned model's behavior.

• The same split appears in the base models' log probabilities. Both instruction-tuned Gemma 2 2B / Llama 3.2 3B and their pretrained counterparts show the identical agency/communion interaction (F(1,208) = 58.3 and 43.2 respectively, both p < .001), locating the bias before preference training.

• The base-model difference is itself a reward model. Using the MWLR implicit reward score, the optimal token for the "Gemma → Llama" implicit RM is "Freedom" and the pessimal token is "Love." This holds across all 21 Llama 3 (1–70B) versus Gemma 2 (2–27B) pairings, and the gap increases with model size.

• Bias narrows but does not disappear during RM training. Llama and Gemma RMs start out furthest apart and converge over roughly the first third of training, then stabilize. Gemma RMs drift toward rewarding agency terms like "choice" while downgrading communion terms like "neighbors" and "volunteers"; Llama RMs show the mirror pattern.

• Data source barely matters; data quantity does. Training on Skywork (77k) versus Unified Feedback (850k) preferences makes little difference, but ~100k+ preference pairs are needed to meaningfully reduce the base-model gap in the authors' experiments.

• The bias extends beyond Llama and Gemma. Supplementary analysis of Qwen-based RMs shows a communion bias even stronger than Gemma's, and that gap does not close after 100k preferences.

• Some training methods preserve the bias more strongly. "Generalizable Reward Models" trained by Yang et al. (2024), which keep the base model's language head and apply a generative-preservation regularizer, still show a striking agency/communion gap after more than 630k preferences.

• Bias also appears on other value axes. On the five Moral Foundations dimensions, Llama RMs favor authority and fairness while Gemma RMs favor care, loyalty, and sanctity on positively framed prompts, though negative-prompt results are less consistent.

Methodology in Plain English

The researchers used a technique called exhaustive token search: instead of asking a language model to generate answers and inspecting them, they run a reward model over every single token in its vocabulary as a candidate one-word answer to a prompt. This avoids sampling artifacts and reveals the provably best- and worst-scoring responses. They repeated this with 54 value-laden prompts (half positively framed, half negatively framed).

To interpret the resulting word rankings, they borrowed two dictionaries from psycholinguistics — the Big Two corpus (agency vs. communion words) and the Moral Foundations Dictionary (authority, care, fairness, loyalty, sanctity) — both hand-coded and validated by human experts. Words from each value category were then compared across models using mixed-effects models and permutation tests.

To find the source of the bias, they repeated the analysis using log probabilities from the raw Gemma and Llama language models (both instruction-tuned and pretrained), rather than from reward models. They also derived a theoretical bridge: from the mathematics of KL-regularized RLHF, the log-probability ratio between any two models defines an implicit reward model. Because raw log-ratios are dominated by very low-probability "junk" tokens, they introduced a mixture-weighted log-ratio (MWLR) that weights each token's log-ratio by its probability under both models, so only tokens that actually affect behavior count. They validated MWLR by deliberately finetuning an "authoritarian" Gemma model and checking whether MWLR best recovered the injected words.

Finally, they trained their own reward models from scratch on both base models with identical hyperparameters (LoRA, Bradley-Terry loss, same learning rate, batch size, seeds), varying only the preference dataset (Skywork vs. Unified Feedback) and its size (13k, 27k, 53k, 106k), checkpointing every 1,000 steps to watch how the bias evolved.

Why This Matters

The paper argues that alignment is not something you can bolt on at the end. The massive pretraining corpus leaves a persistent "moral fingerprint" on a model that preference finetuning can shrink but not erase. This shifts responsibility upstream: whoever chooses the pretraining data and the base model has effectively chosen a set of values.

Real-world applications:

• Alignment and safety engineering. Teams building RLHF pipelines should audit not just their human preference data but the base model underneath it, since the RM is not a neutral judge.
• Base model selection for open-source developers. Choosing between Llama, Gemma, Qwen, and others is not purely a performance or licensing decision — it is a value decision with measurable downstream consequences.
• Pretraining data curation. The results argue for treating data filtering, reweighting, and augmentation as alignment interventions, not just quality-control measures.
• Reward model auditing and benchmarking. The MWLR and big-two-based methods give auditors a practical tool for quantifying a model's implicit value preferences before deployment.

Industry relevance: Any organization deploying RLHF-finetuned models — from frontier labs to enterprises finetuning open-weight models — is affected. The finding that the base-model gap grows with model size is particularly consequential for labs scaling up, and the observation that the GRM regularizer preserves bias even after 630k preferences shows that popular training methodology can unintentionally entrench it.

Future Directions

• Formal scaling laws. The authors observed that the agency/communion gap increases with model size but did not fit a formal scaling law relating model size and preference-data quantity to bias persistence. Deriving one is a stated priority.

• Broadening the base-model survey. The study centers on Llama and Gemma, with exploratory Qwen results that suggest a stronger communion bias. An exhaustive map of value biases across all open-weight base model families would be highly valuable.

• Mechanistic interpretability. The paper establishes that values are inherited but not how — which internal representations carry them, and at what point in pretraining they form, remains open.

• Mitigation strategies and RLHF-stage interactions. Targeted debiasing, data reweighting, and understanding how data composition at different stages of the RLHF pipeline interacts with pretraining bias are described as vital future work.

• Extending beyond short token-level responses. The exhaustive-token-search method is limited to short responses and requires care when comparing across tokenizers; preliminary multi-token results replicate the single-token findings, but broader generalization remains to be established.

Target Audience

This paper is most useful to alignment and safety researchers, reward-modeling practitioners, and machine learning engineers building or auditing RLHF pipelines. Open-source model developers choosing a base model will find it directly actionable, as will policy researchers interested in where value biases in AI systems originate. Psychologists and cognitive scientists working at the intersection of language and values will also find the psycholinguistic methodology familiar and the application novel. A general technical reader with basic knowledge of language model training can follow the argument, though the implicit-reward derivation in Section 3.2 assumes some comfort with RLHF mathematics.

Authors’ abstract

Reward models (RMs) are central to aligning large language models (LLMs) with human values but have received less attention than pretrained and post-trained LLMs themselves. Because RMs are initialized from LLMs, they inherit representations that shape their behavior, but the nature and extent of this influence remain understudied. In a comprehensive study of 10 leading open-weight RMs using validated psycholinguistic corpora, we show that RMs exhibit significant differences along multiple dimensions of human value as a function of their base model. Using the "Big Two" psychological axes, we show a robust preference of Llama RMs for "agency" and a corresponding robust preference of Gemma RMs for "communion." This phenomenon holds even when the preference data and finetuning process are identical, and we trace it back to the logits of the respective instruction-tuned and pretrained models. These log-probability differences themselves can be formulated as an implicit RM; we derive usable implicit reward scores and show that they exhibit the very same agency/communion difference. We run experiments training RMs with ablations for preference data source and quantity, which demonstrate that this effect is not only repeatable but surprisingly durable. Despite RMs being designed to represent human preferences, our evidence shows that their outputs are influenced by the pretrained LLMs on which they are based. This work underscores the importance of safety and alignment efforts at the pretraining stage, and makes clear that open-source developers' choice of base model is as much a consideration of values as of performance.