BiasReducer: Adaptive Bias Mitigation for Reward Models

AI summary

Overview

Research area: Machine learning / AI alignment — specifically bias mitigation in reward models used for reinforcement learning from human feedback (RLHF).

Technical level: Intermediate. The paper assumes familiarity with reward models, preference pairs, and linear reward heads, but its core ideas are explained conceptually and can be followed without deep background in sparse autoencoders or concept erasure.

Scope: The paper introduces BiasReducer, a lightweight framework that edits only the linear reward head of a trained reward model to reduce its dependence on superficial response attributes, and adaptively chooses which attribute-specific edits to apply for each new dataset.

What This Paper Is About

Reward models score LLM responses and guide training toward human preferences, but they often favor superficial attributes such as length, formatting, or confidence — so a longer or more confident response can receive a higher reward even when it is less correct. Existing fixes either require retraining the reward model (expensive, needs new data) or apply a single fixed correction to a bias that must be specified in advance. BiasReducer instead learns reusable, attribute-specific corrections once, then selects which corrections to apply based on the reward behavior observed in each new dataset.

Key Contributions

1. A lightweight reward-model editing framework. BiasReducer learns representations of predefined response attributes using semantic supervision, and edits only the linear reward head while all other reward-model parameters remain fixed.

2. Separation of "how to edit" from "what to edit." The method learns how to reduce dependence on each predefined attribute independently of deciding which attributes to address for a given dataset. This lets the same edits be reused without retraining or specifying the target bias in advance.

3. Dataset-specific edit selection via rank aggregation. For a new dataset, BiasReducer ranks candidate corrections using two signals — how strongly the original reward varies with an attribute, and how much the corresponding edit changes reward scores — combined by Borda-style rank sum, using only responses and reward scores (not which response is preferred). Two variants are offered: BiasReducer-S (single correction) and BiasReducer-M (combines up to k_max corrections).

4. Broad empirical validation. Experiments across five reward models and three benchmarks show consistent gains over the original model and two training-based baselines, with gains also transferring downstream to response selection and policy training.

Main Findings

• Consistent improvement across all model–benchmark combinations. Both BiasReducer-S and BiasReducer-M outperform the original reward model in every model–benchmark combination tested.

• BiasReducer-M gives the strongest average gains. Across five reward models, BiasReducer-M improves RM-Bench-Hard, Arena-StyleConflict, and JudgeBiasBench by 8.3, 18.0, and 6.9 percentage points on average. BiasReducer-S improves the same benchmarks by 6.2, 15.7, and 6.0 points.

• Gains exceed training-based baselines. Compared with RRM, the stronger baseline on average, BiasReducer-M provides an additional 4.1, 13.2, and 2.3 percentage points on the three benchmarks. Mean accuracy on RM-Bench-Hard rises from 52.53 (original) and 56.75 (RRM) to 60.84 (BiasReducer-M); on Arena-StyleConflict from 32.84 and 37.62 to 50.79; on JudgeBiasBench from 67.02 and 71.58 to 73.89.

• Semantic supervision improves attribute alignment. BiasReducer raises the mean held-out attribute correlation from 0.26 (vanilla SAE) to 0.55, and yields substantially larger editing gains. Removing behavioral supervision lowers attribute-axis correlation from 0.55 to 0.17; removing intervention supervision reduces sign accuracy from 0.99 to 0.75. Artifact supervision has a smaller intrinsic effect but still improves editing across all three benchmarks.

• Edit direction and strength both matter. Using SAE decoder vectors as edit directions produces average gains of only 0.5, 0.4, and 0.6 percentage points, versus 6.2, 15.7, and 6.0 points with the covariance-based direction. Separate per-attribute strengths also beat a single shared strength: on JudgeBiasBench with Mistral-7B, the gain increases from 8.4 to 11.8 points.

• Dataset-specific selection beats a fixed edit. BiasReducer-M outperforms both a fixed sycophancy edit and the strongest source-selected fixed edit across all three benchmarks.

• Rank aggregation of two signals works best. On JudgeBiasBench, the rank-sum rule includes the best eligible correction in 86% of cases, compared with 71–74% for either signal alone.

• Selection is stable with far fewer examples. Using 100 examples per subset matches the selection decision from the full response pool in 89% of trials, rising to 97% with 200 examples.

• Robust to edit-complexity settings. Varying k_max across {2, 3, 4, 6} leads to only modest differences. Reducing dictionary size improves RM-Bench-Hard (+9.38 vs +8.31) but lowers gains on Arena (+17.30 vs +17.95) and JudgeBiasBench (+4.68 vs +6.87), while reducing the number of active coordinates lowers gains across all three benchmarks.

• Downstream benefits transfer. Using the edited Mistral-7B reward model with GRPO training reduced mean response length and sycophancy for both Qwen3-4B and Llama-3.2-3B-Instruct policies while maintaining comparable judged quality; the same pattern appears in best-of-N selection for N in {16, 64, 256}.

Methodology in Plain English

The approach runs in three stages, editing only the reward head weight while every other reward-model parameter is frozen.

1. Learn interpretable attribute representations. Given preference pairs of preferred and rejected responses, the method computes the difference in hidden representations ("preference contrast"). It trains a sparse-autoencoder-style encoder–decoder that compresses this contrast into a sparse latent vector. One coordinate is assigned to each of seven predefined attributes: length, verbosity, formatting, emoji/exclamation use, confidence, hedging, and sycophancy. Three supervision signals give these coordinates meaning: behavioral supervision links a coordinate to natural differences in the attribute; interventional supervision uses controlled rewrites (e.g., lengthening or shortening a response) to fix the direction; and an artifact term captures changes shared across rewrites so they are not misattributed to named attributes.

2. Learn how to reduce dependence on each attribute. For each attribute, the method computes a covariance direction capturing the hidden-state direction that co-varies with the attribute activation. Motivated by linear concept erasure, it then defines a candidate edit that shifts the reward head along that direction, with a strength parameter λ. For each attribute it picks the λ that induces the largest reward-score change while keeping the preference-accuracy drop on held-out source validation data within a fixed budget. The results are collected into an "edit bank" of (attribute, direction, strength) triples.

3. Decide what to edit for a new dataset. For a new dataset, the method ranks candidate corrections using two signals — how strongly the original reward varies with the attribute, and how much the corresponding edit would change reward scores — then combines the two rankings by rank sum. BiasReducer-S applies the top-ranked edit; BiasReducer-M composes up to k_max edits. Crucially, this selection step uses only responses, hidden representations, and original reward scores, not which response in each pair is preferred.

Evaluation uses five publicly released reward models (Qwen3-1.7B, GRM-3B, Mistral-7B, InternLM2-7B, GRM-8B), with Skywork-Reward-Preference-80K-v0.2 as source data (77,004 raw usable examples, 66,850 after decontamination, split into 53,678 training and 13,172 validation examples). Benchmarks are RM-Bench-Hard, JudgeBiasBench (seven Superficial Quality Bias subsets), and Arena-StyleConflict, constructed from Arena Human Preference 140K by retaining English, single-turn, non-code pairs where one response is at least 1.3× longer and uses at least as much Markdown formatting but is not preferred, yielding 1,000 pairs. The main metric is pairwise preference accuracy.

Why This Matters

Impact on research. The paper reframes reward-model bias mitigation as a selection problem rather than a retraining problem: learn reusable corrections once, then choose which to apply based on observed reward behavior. It also connects interpretable sparse representations of reward models to practical, deployable fixes — a link the authors note was missing from prior sparse-representation and concept-erasure work. The transductive selection protocol (benchmark responses available during selection, preference labels hidden) is a well-defined setting that future work can relax.

Real-world applications:

• RLHF pipeline tuning: Reducing verbosity and sycophancy in policies trained with GRPO while maintaining judged response quality, without retraining the reward model.
• Response selection / reranking: Best-of-N selection using an edited reward model produces shorter, less sycophantic outputs at comparable quality.
• Reward-model auditing and repair: Using the learned attribute coordinates to diagnose which surface attributes a deployed reward model is sensitive to, then applying targeted edits.
• LLM-as-judge calibration: The same attribute framework (length, formatting, confidence, sycophancy) applies to judge-style models that share these biases, though the paper does not test this directly.

Industry relevance. Because BiasReducer edits only the linear reward head and needs no retraining or extra preference data beyond a small pool of unlabeled examples at selection time, it is a low-cost maintenance operation for production reward models. The finding that 100–200 examples per subset largely reproduce full-pool selection decisions makes it practical for small target distributions.

Future Directions

1. Beyond linear reward heads. The method is defined for reward models with a linear reward head; extending to non-linear scoring heads or different reward-model architectures is not addressed.

2. From transductive to inductive selection. Selection currently uses the responses of the target benchmark itself. Whether the ranking signals can be estimated from a small held-out sample of the deployment distribution, before the target responses exist, remains open.

3. Broadening the attribute set. Only seven predefined attributes are considered. How well the approach scales to attributes not in this list, or to attributes that require new controlled rewrites, is untested.

4. Compositional interaction of edits. BiasReducer-M combines multiple attribute edits, but how the individual corrections interact when composed — and how k_max should be set in general — is only probed through sensitivity analysis rather than explained.

Target Audience

Researchers and engineers working on RLHF, reward modeling, and AI alignment; practitioners who deploy reward models for response selection or policy training and need bias mitigation without a retraining budget; and readers interested in interpretable representations of reward models via sparse autoencoders and concept-erasure-style editing.

Authors’ abstract

Reward models score responses from large language models (LLMs) and guide LLM training toward human preferences. However, reward models can favor superficial attributes such as length or confidence, leading LLMs to produce higher-scoring but not more correct responses. Existing mitigation methods either retrain the reward model or apply a fixed correction to one known bias, such as a preference for longer responses. Retraining requires additional data and computational resources, while existing editing methods require the target bias to be specified in advance and use a fixed edit for that bias. To this end, we propose BiasReducer, a lightweight framework that edits only the linear reward head and selects the relevant edits for each new dataset. First, BiasReducer uses a sparse autoencoder (SAE)-style encoder to learn which attributes (e.g., length and confidence) the reward model is sensitive to. Second, it learns how to reduce the reward model's dependence on each attribute by determining which direction to adjust the reward head and how much to adjust it. Third, for a new dataset, it ranks the attributes by their influence on reward scores, selects the relevant ones, and edits the reward model accordingly. BiasReducer consistently improves reward-model robustness to biases toward superficial response attributes. Across five reward models, BiasReducer-M improves the three benchmarks by 8.3, 18.0, and 6.9 percentage points on average, outperforming the two training-based baselines. The gains transfer downstream, reducing unnecessary verbosity and sycophancy while maintaining comparable judged quality.