R-Align: Enhancing Generative Reward Models through Rationale-Centric Meta-Judging

AI summary

Overview

Research area: Natural Language Processing — reward modeling and Reinforcement Learning from Human Feedback (RLHF) for large language model alignment.

Technical level: Intermediate. The core idea (judge the reasoning, not just the answer) is intuitive, but the paper assumes familiarity with reward models, RLVR/PPO training, and preference benchmarks.

Scope: This paper diagnoses "Spurious Correctness" in Generative Reward Models, builds a rationale-aware benchmark and metric to measure it, and proposes a training method (R-Align) that supervises reasoning traces rather than only final preference labels.

What This Paper Is About

Generative Reward Models (GenRMs) produce a written rationale before predicting which of two responses is better, but they are still trained and evaluated only on whether the final label is right — the reasoning itself is never checked. The authors show that a GenRM can pick the correct answer for the wrong reason (relying on superficial cues like bullet-point formatting instead of real content quality), and that this flaw predicts downstream RLHF failure even when benchmark accuracy looks strong. Their goal is to make GenRMs "right for the right reasons" by explicitly supervising the rationale.

Key Contributions

1. Rationale-aware GenRM benchmarking. The authors construct a benchmark enriched with golden rationales over HelpSteer3, RewardBench2, and PPE-Preference, and introduce Spurious Correctness (S-Corr) — the fraction of label-correct decisions whose rationale is misaligned with the golden judgment — alongside a stricter Fidelity Score (F-Score). They report significant rationale misalignment in open-source GenRMs and even top proprietary LLMs.
2. The R-Align training framework (Rationale-Centric Alignment). A Meta-Reward Model (MetaRM) performs three-stage Chain-of-Thought verification of the GenRM's rationale, and the reward is granted only when both the label is correct and the rationale is verified as aligned — explicitly penalizing spurious justifications.
3. Evidence that rationale alignment improves downstream RLHF. GenRMs trained with R-Align produce consistently better actor policies across STEM, coding, instruction following, and general tasks compared to an outcome-only RLVR baseline.
4. A demonstration that F-Score predicts RLHF outcomes where Label Accuracy does not, with the authors reporting Pearson correlations of 0.947, 0.924, and 0.963 for F-Score versus 0.366, 0.382, and 0.220 for L-Acc on HelpSteer3, RewardBench2, and PPE-Preference respectively.

Main Findings

• Benchmark accuracy does not predict RLHF quality. Qwen3-14B and RRM-32B score comparably on three RM benchmarks (HelpSteer3: 74.1 vs. 74.7; RewardBench2: 87.9 vs. 88.5; PPE-Preference: 65.2 vs. 65.1). Yet when each supervises RLHF training of a Qwen3-8B policy under identical settings, the RRM-32B-supervised policy shows pronounced collapse in the averaged score across general, STEM, code, and instruction-following domains, while the Qwen3-14B-supervised policy continues to improve.

• Spurious Correctness is widespread and tracks model capability. Using Gemini-3-Pro to verify rationale alignment, RRM-32B shows high spurious rates (59.0% on HelpSteer3, 36.7% on RewardBench2, 62.4% on PPE-Preference) versus Qwen3-14B (40.0%, 20.1%, 36.9%). Lower S-Corr is associated with larger or more advanced models, and thinking variants (e.g., Qwen3-4B-Thinking-2507, GPT-5-thinking, Claude-Sonnet-4.5-thinking) show marked reductions in S-Corr relative to their non-thinking counterparts. Gemini-3-Pro itself reaches very low S-Corr (5.6, 1.7, and 1.6 across the three benchmarks).

• Label accuracy saturates; F-Score discriminates. L-Acc scores cluster tightly — most models land in the 70%+ range on HelpSteer3 and around 90% on RewardBench2, with open-source scores on PPE-Preference stagnating in the 60%+ range — while F-Score separates models far more granularly.

• Standard RLVR can hurt reasoning quality. Applying RLVR to Qwen3-14B on HelpSteer3 raises L-Acc from 74.1% to 75.5% but spikes S-Corr from 40.0% to 46.9%, dropping F-Score from 44.5% to 40.1%. On PPE-Preference with Qwen3-8B, L-Acc gains of +1.5% come at the cost of reasoning quality.

• R-Align lowers S-Corr and raises F-Score. On RewardBench2, GenRM-R-Align-8B reaches an F-Score of 70.3% (L-Acc 89.8, S-Corr 21.7), which the paper reports as surpassing the larger GenRM-RLVR-14B baseline by 4.2% while reducing S-Corr by the same margin. On HelpSteer3, the method lifts Qwen3-14B's F-Score to 54.0% (+13.9% over RLVR).

• Rationale supervision improves the downstream policy. With GenRM-R-Align-8B as the reward model, the Qwen3-8B actor recovers from the RLVR baseline's regressions — LiveCodeBench 51.5 vs. 47.4 and AIME25 64.2 vs. 58.3 — and reaches 89.2 vs. 84.3 on WildBench. GenRM-R-Align-14B yields the highest average (61.7) among the reward models compared, including 92.6 on WildBench and 60.2 on Arena-Hard-v2.

• An alignment–capability trade-off is visible but mitigated. Even using the base model itself as the reward model improves the average (58.1 vs. 53.9 for Qwen3-8B), mostly in general-purpose behavior (WildBench, Arena-Hard-v2, MultiChallenge), while specialized STEM/code reasoning often degrades; R-Align reduces that tax.

• Meta-judging is validated and made scalable. Against 53 human-annotated HelpSteer3 instances where Qwen3-14B was label-correct, Gemini-3-Pro's meta-judging achieved an F1 of 0.9044. Because using Gemini-3-Pro for the high-frequency queries of RL training is prohibitively expensive, the authors compare open-weight alternatives and select GPT-OSS-120B as the training MetaRM, citing its highest alignment (90.68 F1 on RewardBench2, 83.53 on HelpSteer3, 84.33 on PPE-Preference).

Methodology in Plain English

The authors start from a simple observation: if you only grade a judge on whether it picked the right winner, you cannot tell whether it reasoned well. So they first define the failure mode — Spurious Correctness — and build a way to detect it.

Building the benchmark. They take three existing preference datasets (HelpSteer3, RewardBench2, PPE-Preference) and add a "golden rationale" to each sample using Gemini-3-Pro. For datasets that only have a final label, the model is prompted to explain why the labeled winner is better. For HelpSteer3, which includes multiple human judgments and critiques per pair, Gemini-3-Pro acts as a meta-reviewer that aggregates the human feedback into one coherent reference rationale. RewardBench2 entries (1 chosen, 3 rejected responses) are expanded into three pairwise samples each, and the "Ties" subset is excluded.

Measuring misalignment. A separate MetaRM reads the prompt, both responses, the golden rationale, and the GenRM's own rationale, and outputs a binary verdict on whether the GenRM captured the key discriminators. The verification proceeds in three stages: extract the key discriminators from the golden rationale, check whether the GenRM's rationale explicitly covers them (rejecting cases where it cites superficial style while the golden rationale points to a factual or calculation error), then emit the binary verdict. Three metrics follow: L-Acc (label correct), S-Corr (label correct but rationale rejected), and F-Score (label correct and rationale verified).

Training with rationales. The authors initialize GenRMs from Qwen3-8B and Qwen3-14B and train with PPO. The baseline (GenRM-RLVR) rewards the model whenever the predicted verdict matches the ground-truth label. R-Align replaces that with a stricter reward: 1 only when the label is correct and the MetaRM verifies the rationale, 0 otherwise. Training data combines a cleaned subset of Skywork Reward Preference 80K, 8K samples from Code-Preference-Pairs, the complete Math-DPO-10K dataset, and the HelpSteer3 training set — all augmented with golden judgments generated by Gemini-3-Pro.

Testing downstream. To check whether the reward models actually matter, the authors use each one to supervise RLHF training of a Qwen3-8B policy with PPO on prompts from Arena-Human-Preference. The policy's sampled responses are compared against reference responses generated by STEP3-VL-10B, with +1 reward if the GenRM prefers the policy output and −1 otherwise, plus a dynamic length penalty to mitigate length bias. The resulting actors are evaluated on AIME24, AIME25, GPQA-diamond, LiveCodeBench, MultiChallenge, Arena-Hard-v2, Wildbench, and IFBench.

Why This Matters

Impact on research. The paper argues that the standard practice of reporting reward-model label accuracy is an insufficient proxy for reward quality, and that a benchmark metric (F-Score) correlates far more strongly with actual RLHF outcomes than accuracy does. It also reframes reward hacking in RLHF as something detectable at the reward-model level, before optimization begins — the authors report that higher S-Corr is associated with policy degeneration.

Real-world applications:

• Chat assistants and subjective quality judgments, where preferences are non-verifiable and scalar scores easily latch onto length or formatting rather than substance.
• Coding assistants, where the paper's downstream results specifically cover LiveCodeBench-style evaluation and an R-Align-trained reward model preserved coding performance that the outcome-only baseline degraded.
• Instruction-following and conversational products, where GenRM-R-Align-8B improved WildBench, Arena-Hard-v2, and MultiChallenge scores for the trained policy.
• Automatic evaluation pipelines, where the MetaRM-plus-golden-rationale setup offers a way to audit whether a model judge is right for the right reasons rather than only scoring its verdict.

Industry relevance. The paper's MetaRM selection table directly addresses the cost of running a proprietary model for the high-frequency queries inherent to RL training, and identifies GPT-OSS-120B as a scalable open-weight substitute — a practical consideration for anyone deploying reward models at scale.

Future Directions

• Reducing dependence on golden rationales. The authors state as a limitation that R-Align needs high-quality reference rationales. Only HelpSteer3 provides dense human critiques with both labels and detailed comments; for most training data the rationales had to be synthesized by an advanced LLM conditioned on the ground-truth label. Efficacy is therefore bound by human annotation cost or by the reasoning capability of proprietary teacher models.
• Handling multiple valid reasoning paths. A strict-matching verification could in theory penalize a valid alternative rationale that reaches the same conclusion. The authors argue this is more a theoretical than practical concern, since advanced reasoners (e.g., GPT-5-thinking, Gemini-2.5-Pro) show very low spurious correctness, but the question of how to credit legitimately different rationales remains open.
• Extending rationale-centric supervision to domains where human labeling is scarce, which the conclusion explicitly names as future work on superalignment.
• Repairing the weak link between offline proxy metrics and online RLHF outcomes, given the paper's finding that standard benchmark metrics often fail to predict downstream policy performance.

Target Audience

Researchers and engineers working on reward modeling, RLHF/RLVR pipelines, and LLM-as-a-Judge evaluation; practitioners who need to choose or train reward models and care about whether benchmark scores translate to real policy quality; and evaluation-focused readers interested in metrics that go beyond outcome accuracy. Readers without background in reinforcement learning will follow the diagnosis and metrics easily, but the training sections assume familiarity with PPO and reward-model pipelines.

Authors’ abstract

Reinforcement Learning from Human Feedback (RLHF) remains indispensable for aligning large language models (LLMs) in subjective domains. To enhance robustness, recent work shifts toward Generative Reward Models (GenRMs) that generate rationales before predicting preferences. Yet in GenRM training and evaluation, practice remains outcome-label-only, leaving reasoning quality unchecked. We show that reasoning fidelity-the consistency between a GenRM's preference decision and reference decision rationales-is highly predictive of downstream RLHF outcomes, beyond standard label accuracy. Specifically, we repurpose existing reward-model benchmarks to compute Spurious Correctness (S-Corr)-the fraction of label-correct decisions with rationales misaligned with golden judgments. Our empirical evaluation reveals substantial S-Corr even for competitive GenRMs, and higher S-Corr is associated with policy degeneration under optimization. To improve fidelity, we propose Rationale-Centric Alignment, R-Align, which augments training with gold judgments and explicitly supervises rationale alignment. R-Align reduces S-Corr on RM benchmarks and yields consistent gains in actor performance across STEM, coding, instruction following, and general tasks.