Reward Modeling from Natural Language Human Feedback

AI summary

Overview

Research area: Natural Language Processing — reward modeling, reinforcement learning from human feedback (RLHF), and large language model alignment.

Technical level: Intermediate. The paper assumes familiarity with reward models, binary preference labels, and policy-gradient RL (GRPO) terminology, though the core idea — that a model can guess the right answer for the wrong reasons — is explained accessibly.

Scope: This paper introduces RM-NLHF, a training method that replaces outcome-only binary reward with a process reward derived from the similarity between human-written critiques and model-generated critiques, plus a MetaRM component to scale that supervision to data that lacks human critiques.

What This Paper Is About

Generative Reward Models (GRMs) are trained with reinforcement learning where the reward is simply whether the model picked the correct one of two responses. Because there are only two possible answers, a GRM can guess the correct preference label while producing critiques that are objectively wrong or nitpicky. The paper shows this mismatch is common and then proposes using natural language human critiques as an additional, process-level reward signal so the model is rewarded for reasoning well, not just for landing on the right label.

Key Contributions

1. Empirical demonstration of outcome–process inconsistency. Across multiple state-of-the-art models, 20-30% of correct preference predictions are accompanied by flawed critiques (Section 2.2), with specific measured rates such as 44.24% for RM-R1-DeepSeek-Distilled-Qwen-7B, 26.10% for gemini-2.5-pro, and 33.62% for claude-3.7-sonnet.

2. Identification of an accurate process-reward proxy. The paper compares LLM-as-a-Meta-Judge, similarity against all human critiques, and similarity against only the core human critiques, finding that F1-based similarity over core critiques is the best balance of accuracy, cost, and training stability (Section 2.3), and that using it improves GRM training over outcome-only reward (Section 3.1).

3. MetaRM and online MetaRM for scalability. Because human critiques are expensive, the authors train a scalar Meta Reward Model to predict process reward from the small labeled set and generalize to the larger unlabeled set, with an online variant that is continuously updated alongside the GRM to handle distribution shift (Section 3.2).

4. Theoretical framing. The paper explains four reward modeling mechanisms from a theoretical perspective, clarifying the evolution of reward modeling and the necessity of the proposed method, while identifying remaining theory-practice gaps (Appendix I).

Main Findings

• Outcome-only supervision is noisy for GRMs. Correct preference labels frequently co-occur with invalid critiques, whereas in mathematical reasoning the probability of this inconsistency is extremely low because the solution space is large. An incorrect outcome, by contrast, almost invariably indicates flawed critiques.

• Core-critique similarity beats LLM-as-a-Meta-Judge. Across the three judge models tested, Similarity w/ HC methods outperformed LLM-as-a-Meta-Judge by gaps ranging from 12% to 59%. The highest single score (0.9388) came from gemini-2.5-pro with Similarity w/ All HC using Recall, but the authors chose the more cost-effective gpt-5-mini with Similarity w/ Core HC and F1.

• Human critiques as process reward help. For both Deepseek-R1-Distill-Llama-8B and Deepseek-R1-Distill-Qwen-7B, training with human critiques outperformed outcome-only training, with stable improvements beginning after approximately 70 steps and persisting for the remainder of training.

• Naive mixing of reward types is catastrophic. Assigning human-critique reward to labeled data and outcome-only reward to unlabeled data performed worse than using outcome-only reward for both, which the authors attribute to conflicting reward signals undermining reward consistency.

• Online MetaRM works. Online MetaRM consistently outperformed offline MetaRM and achieved performance comparable to full human critique supervision while significantly reducing annotation requirements — though the authors note this may not hold out-of-distribution.

• RM-NLHF outperforms same-scale specialists. RM-NLHF-Qwen-7B reached an overall score of 0.6481 versus RM-R1-Qwen-7B (0.5759) and RRM-Qwen-7B (0.5895); RM-NLHF-Qwen-32B reached 0.7296 overall.

• Generative reward models scale better than scalar ones. Scalar reward models performed better at the 7/8B scale, but showed only marginal gains even up to 70B (INF-ORM-Llama3.1-70B overall 0.7182), while generative reward models improved substantially from 7B to 32B.

• Ablations confirm both design choices. Removing process reward lowered the average from 0.6481 to 0.6158; removing outcome regularization lowered it to 0.6282; and training MetaRM on additional data beyond HelpSteer3 lowered it to 0.6332, indicating HelpSteer3's critiques transfer well.

• Downstream RL gains. Using RM-NLHF-Qwen-7B versus Outcome-Only-Qwen-7B as the reward model for 100 GRPO steps on DeepSeek-Distilled-Qwen-7B and evaluating on Arena-Hard-V2.0 with four judge models (gemini-3-pro, gpt-5.1, qwen3-max-thinking, kimi-k2.5), RM-NLHF won 52.33% on average against 39.41%, with 8.26% ties — an average margin of 12.92%. Gains came with shorter, not longer, responses.

• Test-time scaling gains. In Best-of-N, RM-NLHF beat outcome-only at BoN@2 (64.90% vs 63.65% on MATH500; 76.95% vs 76.30% on HumanEval+), BoN@4 (66.80% vs 65.45%; 81.04% vs 75.77%), and BoN@8 (67.60% vs 65.99%; 85.98% vs 75.00%). In Feedback-Edit it reached 68.40% on MATH500 (vs 67.01%), 87.20% on HumanEval+ (vs 82.32%), and 7.03% on Arena-Hard-V2.0 (vs 6.55%).

• Modest compute overhead. Per training step, the synchronous variant raised time from 156s to 289s (7B) and 655s to 874s (32B). Asynchronous execution reduced this to 196s for 7B (a 26% increase) and 766s for 32B (a 17% increase).

• Critiques become more specific. RM-NLHF produced 4,436 unique words versus 1,633 for the baseline, with more critical/diagnostic language such as "unusable" (+0.033), "critical" (+0.030), "incorrect" (+0.022), and "unrelated" (+0.010), while the baseline over-relied on generic positive terms like "comprehensive" (-0.029), "clear" (-0.029), and "helpful" (-0.007).

• Reward hacking was observed and is documented. Recall-based similarity caused the GRM to generate excessive critiques; over-extended training with F1 similarity or MetaRM caused abnormally long responses, unreasonably high process reward, declining outcome reward, and repeated identical critiques — addressed by instructing the similarity prompt to check for repetition.

• MetaRM accuracy decays without online updates. Offline MetaRM started with comparable accuracy but gradually deteriorated through RL training, while online updates maintained high agreement with F1-based similarity scores from gemini-2.5-pro.

Methodology in Plain English

The authors start from an observation rather than a hypothesis: they take existing GRMs, ask them to judge pairs of responses, and separately check whether the written critiques actually make sense. They use gemini-2.5-pro as a referee on both mathematical reasoning tasks (MATH-500) and preference tasks (HelpSteer3), and find that in the preference setting, correct verdicts often come with bad reasoning.

Next they ask how to score the quality of a critique automatically. They annotate a small set of 49 HelpSteer3 samples that already contain human critiques, then compare three scoring strategies: asking an external LLM to judge the critique directly, measuring overlap with everything the human said, and measuring overlap with only the essential points the human raised. The last option, measured with F1 and computed by gpt-5-mini, turns out to be the best value for the money.

That score becomes a bonus reward. If the model gets the preference wrong, it gets zero regardless of its critique — a rule they call outcome regularization. If it gets the preference right, its reward is 1 plus a weighted bonus for critique quality.

Because human critiques are scarce, they train a small regression model (MetaRM) to predict that bonus from the query, the two candidate responses, and the model's critique. MetaRM is initialized on the labeled subset and then, during RL training, updated at every iteration before it is used to score the unlabeled data — this "MetaRM-first" ordering keeps the reward model aligned with the changing policy. They then train the actual GRM with GRPO on 164K pairwise preference samples and evaluate on six benchmarks released within one month before the base model's release to avoid contamination.

Why This Matters

Impact on research. The paper challenges a widely used assumption in RLVR: that verifiable outcome reward is sufficient. For tasks with a large solution space, like mathematics, a correct answer implies correct reasoning. For binary preference comparison, it does not. The paper quantifies that gap and offers a general recipe — similarity to human critiques as a process reward, amortized by a learned meta reward model — that applies to any domain where the answer space is small but the reasoning space is not.

Real-world applications (as implied or demonstrated by the paper):

• Aligning assistant models. RM-NLHF was used as the reward model in RL training of DeepSeek-Distilled-Qwen-7B, improving win rates on Arena-Hard-V2.0 by an average of 12.92% across four judge models.
• Best-of-N response selection. Using a pairwise GRM in a tournament to pick the best of N sampled responses improved MATH500 and HumanEval+ scores over an outcome-only reward model.
• Feedback-driven response refinement. In the Feedback-Edit setup, critiques from RM-NLHF guided gemini-2.5-pro to synthesize improved responses, lifting HumanEval+ to 87.20%.
• General evaluative judging. The paper notes GRMs cover tasks that rule-based verifiers cannot handle, including social intelligence, roleplay, and image generation.

Industry relevance. The method trains on 164K preference samples and adds only 26% (7B) or 17% (32B) per-step compute under asynchronous execution, making it practical at production scale. It is particularly relevant where pairwise preference data already exists with outcome labels only — the online MetaRM path means companies do not need to re-annotate critique text at scale.

Future Directions

• Extending to multiple-choice and true/false questions. The paper explicitly proposes that these tasks, which suffer from the same restricted-solution-space problem, could benefit from the process-level supervision this method provides.
• Open-ended tasks with verifiable correctness. The authors suggest applying the approach to open-ended tasks that lack explicit outcome reward but have checkable correctness criteria. The provided text is truncated at this point and does not state the full scope.
• Closing remaining theory-practice gaps. Appendix I identifies gaps between the theoretical account of four reward modeling mechanisms and observed practice; the paper does not report what those specific gaps are in the available content.
• Robustness beyond the training distribution. The authors explicitly note that the claim of matching full human-critique supervision with fewer resources "may not hold for out-of-distribution settings," leaving this as an open question to be investigated.
• Mitigating reward hacking. Appendix A documents multiple hacking modes (recall-driven critique inflation, repeated identical critiques, length inflation), which suggests continued work on reward robustness.

Target Audience

Researchers and engineers working on LLM alignment, reward modeling, and RL post-training will benefit most, particularly those training generative reward models or LLM-as-a-Judge systems on pairwise preference data. The paper is also useful for practitioners who already have large outcome-labeled preference datasets but limited budget for critique annotation, and for readers interested in the theory of reward modeling. Some familiarity with RL fine-tuning and preference-based evaluation is assumed; the motivating insight about guessing correctly for the wrong reasons is accessible without it.

Authors’ abstract

Reinforcement Learning with Verifiable reward (RLVR) on preference data has become the mainstream approach for training Generative Reward Models (GRMs). Typically in pairwise rewarding tasks, GRMs generate reasoning chains ending with critiques and preference labels, and RLVR then relies on the correctness of the preference labels as the training reward. However, in this paper, we demonstrate that such binary classification tasks make GRMs susceptible to guessing correct outcomes without sound critiques. Consequently, these spurious successes introduce substantial noise into the reward signal, thereby impairing the effectiveness of reinforcement learning. To address this issue, we propose Reward Modeling from Natural Language Human Feedback (RM-NLHF), which leverages natural language feedback to obtain process reward signals, thereby mitigating the problem of limited solution space inherent in binary tasks. Specifically, we compute the similarity between GRM-generated and human critiques as the training reward, which provides more accurate reward signals than outcome-only supervision. Additionally, considering that human critiques are difficult to scale up, we introduce Meta Reward Model (MetaRM) which learns to predict process reward from datasets with human critiques and then generalizes to data without human critiques. Experiments on multiple benchmarks demonstrate that our method consistently outperforms state-of-the-art GRMs trained with outcome-only reward, confirming the superiority of integrating natural language over binary human feedback as supervision.