Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidance

AI summary

Overview

Research area: Reinforcement learning from human feedback (RLHF), reward modeling, and LLM alignment, with an information-theoretic approach to debiasing.

Technical level: Advanced. The paper assumes familiarity with RLHF/PPO, the Bradley-Terry ranking loss, mutual information, and variational bounds, though the core intuition is explainable without full mathematical background.

Scope: The paper proposes DIR (Debiasing via Information optimization for RMs), a single training framework that removes arbitrary inductive biases from reward models by maximizing mutual information with preference labels while minimizing mutual information with bias attributes.

What This Paper Is About

Reward models trained on human preference data tend to latch onto superficial cues rather than genuine response quality. Because annotators often favor longer, more detailed, more flattering, or more heavily formatted answers, reward models learn heuristics like "longer is better," which downstream RLHF then exploits, producing verbose, sycophantic, or gimmicky LLM behavior—a failure mode known as reward hacking. The paper's goal is a general-purpose, theoretically grounded debiasing method that works for many kinds of bias at once, rather than one bias type or one linear-correlation measure at a time.

Key Contributions

1. An information-theoretic debiasing objective for reward models. The authors frame debiasing as an information bottleneck problem: maximize the mutual information (MI) between the RM's preference prediction and the true response-content triplet, while minimizing the MI between the prediction and the bias attribute.

2. A tractable training procedure via variational bounds. The intractable MI terms are replaced by the Barber-Agakov (BA) lower bound for the maximization term—which turns out to be exactly the standard Bradley-Terry ranking loss—and the contrastive log-ratio upper bound (CLUB) for the minimization term, with an auxiliary estimator network that is updated iteratively to keep the bound tight.

3. Relative bias attributes instead of absolute ones. Rather than predicting a response's absolute length or exact style, DIR only models the relative bias between the chosen and rejected response (e.g., "was the chosen answer longer?"). This makes the bias estimator a lightweight binary classifier, extends naturally to non-scalar biases, and avoids distorting the reward landscape.

4. Broad empirical validation across three bias types. Experiments cover response length, sycophancy, and format bias, using Llama3.1-8B-Instruct as the RM backbone and Skywork-Preference-80K-v0.2 as the training data, evaluated on RM-Bench, RewardBench, and downstream RLHF benchmarks including GSM8K, MMLU, IFEval, ArenaHard, and MT-Bench. Code and recipes are released publicly.

Main Findings

• Length bias is substantially reduced. On RM-Bench, the standard Bradley-Terry reward model shows a Pearson correlation of 0.533 between response length and reward score; DIR reduces this to 0.468, the lowest among all compared methods, and produces a flatter binned mean-reward curve across length buckets.

• Downstream RLHF quality improves, not degrades. With Llama3.1-8B-Instruct as the initial policy, DIR achieves the best average score of 66.20 across benchmarks, outperforming the vanilla baseline, a length-penalty heuristic, PoE, ALBM, and InfoRM. The same trend holds when OpenRLHF-Llama3-8B-SFT is used as the backbone.

• Gains appear on instruction-following and reasoning tasks. On IFEval (Llama3.1 base) DIR reaches 78.00 versus 74.12 for InfoRM and 73.57 for ALBM. On ProcessBench with the OpenRLHF backbone, DIR scores 13.82 versus 10.85 for ALBM and 4.46 for the base policy.

• Better human preference with shorter outputs. On ArenaHard-v0.1 judged by Qwen3-235B-A22B-2507, the DIR-trained policy reaches a 54.3% win rate against the baseline and 41.9% against GPT-4o-0314, while generating 679 tokens on average compared with 722 for ALBM and 754 for the original baseline—resolving the usual win-rate-versus-verbosity trade-off.

• The method generalizes beyond length. The same framework applies to sycophancy and format bias, and DIR is shown to help with concurrent multi-bias settings.

• Existing approaches have structural limits. Pearson-correlation penalties capture only linear dependencies; the two-head length approach handles only scalar biases without theoretical guarantees; MMD-based invariance is over-restrictive and risks collapsing distinct response groups; and generic compression like InfoRM never explicitly constrains the bias attribute.

Methodology in Plain English

Think of a reward model as an information channel. A good channel transmits the signal you care about—the genuine quality difference between two responses—and blocks the noise you don't, such as length or sycophantic tone.

DIR formalizes this with mutual information. The preference term rewards the model for its scores being informative about which response humans actually chose. The debiasing term penalizes the model for its internal representations being informative about the bias attribute. The two terms are combined with a balancing weight λ.

Because mutual information cannot be computed exactly in high dimensions, the authors approximate it. The preference term's lower bound turns out to be identical to the familiar Bradley-Terry loss already used to train reward models, so no new machinery is needed there. For the debiasing side, they use the CLUB upper bound, which requires a small auxiliary network that tries to predict the bias attribute from the reward model's hidden states. Minimizing the CLUB bound effectively makes that auxiliary predictor fail—meaning the hidden states have stopped encoding the bias.

Two design choices make this practical. First, the bias is defined relatively: instead of asking the auxiliary network to guess an exact token count, it only guesses a binary label such as "was the chosen response longer?" Second, the auxiliary network reads the difference between the chosen and rejected hidden states, which focuses it on what actually distinguishes the pair. The auxiliary network and the reward model are trained alternately within each batch so the bound stays accurate throughout training.

Why This Matters

Impact on research. The paper provides a theoretically justified debiasing framework that subsumes and improves on prior heuristics. By reducing debiasing to a mutual-information trade-off, it connects reward modeling to the well-developed information bottleneck literature and offers a template that other bias definitions—social biases, verbosity, tone, formatting, tool-calling habits—can plug into without redesigning the objective.

Real-world applications.

• Customer-facing assistants that currently pad answers with unnecessary detail or agree with users regardless of correctness can be trained to be concise and honest.
• Scientific and technical writing assistants where reward models often over-reward markdown formatting and bullet-point structure over substantive content.
• Educational tutors where sycophancy—telling students what they want to hear—directly undermines learning outcomes.
• Enterprise document and code assistants where verbosity inflates inference cost and latency without improving usefulness.

Industry relevance. Reward hacking is one of the main reasons RLHF pipelines are unstable and expensive to tune, requiring repeated rounds of data curation and manual reward clipping. A plug-in debiasing loss that improves both RM robustness and downstream benchmark scores at no inference cost is directly applicable to production post-training stacks, and the released code lowers the barrier to adoption.

Future Directions

• Handling unspecified or latent biases. DIR requires the bias attribute to be labeled or at least computable for each training pair. Discovering biases automatically—without a human pre-specifying them—remains open.

• Interaction effects among multiple biases. The paper demonstrates concurrent multi-bias debiasing, but how biases correlate with each other (for example, format and length) and whether debiasing one inadvertently amplifies another deserves deeper analysis.

• Tuning the trade-off weight λ. The balance between preference fidelity and debiasing is controlled by a single hyperparameter with unclear guidance for new bias types; adaptive or principled selection would help practitioners.

• Scaling and backbone generality. Results are reported primarily for Llama3.1-8B-Instruct and one 8B SFT checkpoint. Whether the same behavior holds for much larger reward models, mixture-of-experts backbones, or process reward models is untested.

Target Audience

Researchers and engineers working on RLHF, LLM alignment, and reward model training will get the most value, particularly those who have encountered reward hacking or length bias in production pipelines. The paper is also relevant to readers interested in information-theoretic objectives for deep learning, since the BA/CLUB formulation is reusable beyond reward modeling. Readers without a background in RLHF or mutual information will find the motivation and results accessible, but the derivation of the bounds and the training algorithm will require some prerequisite knowledge.

Authors’ abstract

Reward models (RMs) are essential in reinforcement learning from human feedback (RLHF) to align large language models (LLMs) with human values. However, RM training data is commonly recognized as low-quality, containing inductive biases that can easily lead to overfitting and reward hacking. For example, more detailed and comprehensive responses are usually human-preferred but with more words, leading response length to become one of the inevitable inductive biases. A limited number of prior RM debiasing approaches either target a single specific type of bias or model the problem with only simple linear correlations, \textit{e.g.}, Pearson coefficients. To mitigate more complex and diverse inductive biases in reward modeling, we introduce a novel information-theoretic debiasing method called \textbf{D}ebiasing via \textbf{I}nformation optimization for \textbf{R}M (DIR). Inspired by the information bottleneck (IB), we maximize the mutual information (MI) between RM scores and human preference pairs, while minimizing the MI between RM outputs and biased attributes of preference inputs. With theoretical justification from information theory, DIR can handle more sophisticated types of biases with non-linear correlations, broadly extending the real-world application scenarios for RM debiasing methods. In experiments, we verify the effectiveness of DIR with three types of inductive biases: \textit{response length}, \textit{sycophancy}, and \textit{format}. We discover that DIR not only effectively mitigates target inductive biases but also enhances RLHF performance across diverse benchmarks, yielding better generalization abilities. The code and training recipes are available at https://github.com/Qwen-Applications/DIR.