Think Twice: Branch-and-Rethink Reasoning Reward Model

AI summary

Overview

Research area: Natural Language Processing, specifically reward modeling for LLM alignment (RLHF / RLAIF pipelines), sitting at the intersection of test-time reasoning and preference learning.

Technical level: Intermediate. The core idea is intuitive, but the training procedure assumes familiarity with reinforcement learning for LLMs (PPO, GRPO, reward shaping) and with reward-model evaluation benchmarks.

Scope in one sentence: The paper proposes a two-turn "branch-and-rethink" reward model (BR-RM) that first narrows evaluation to a few instance-critical dimensions and then re-reads the responses conditioned on those dimensions, trained with GRPO-style reinforcement learning, and reports state-of-the-art results on three reward modeling benchmarks.

What This Paper Is About

Most reward models compress every aspect of response quality — factuality, reasoning, safety, style, instruction following — into a single judgment made in one pass. The authors argue this causes judgment diffusion: attention spreads thin across many criteria, producing focus dilution (no dimension is examined deeply) and shallow analysis (subtle factual slips or logic bugs go unnoticed). The paper's goal is to transfer the "think twice" principle from reasoning solvers to reward judges by building a two-turn model that deliberately focuses on a small set of instance-critical dimensions and then performs a targeted second-pass re-evaluation.

Key Contributions

1. Diagnosis and empirical grounding of "judgment diffusion." The authors name and characterize the failure mode in which single-pass reward models spread their token budget thinly across many criteria, and support it with a preliminary study comparing token allocation between a recent reasoning reward model (RRM-32B) and their method on RM-Bench subsets.

2. The branch-and-rethink (BR-RM) framework. A two-turn reward model: Turn 1 performs adaptive branching, selecting a small set of instance-critical dimensions from nine candidates (e.g., factual accuracy, reasoning quality, safety) and producing an issue sketch; Turn 2 performs branch-conditioned rethinking, a targeted reread that tests those hypotheses and produces the final preference decision.

3. A GRPO-based training recipe for two-turn traces. The model is trained with GRPO-style reinforcement learning using a simple binary outcome reward plus strict format checks, with the same reward assigned separately to the first turn and the second turn — making it compatible with standard RLHF infrastructure without requiring explicit multi-turn RL support.

4. State-of-the-art results and extensive analysis. BR-RM-Qwen-14B reaches the highest average score (84.2) across RewardBench, RM-Bench, and RMB, with ablations on architecture components, reward design, reasoning-turn scaling, branch distribution, and inference latency. Code and checkpoints are released (github.com/yzjiao/BR-RM; nvidia/Qwen3-Nemotron-14B-BRRM and nvidia/Qwen3-Nemotron-8B-BRRM).

Main Findings

• BR-RM-Qwen-14B sets the best overall balance. It achieves an average of 84.2 across the three benchmarks, with 92.1 on RewardBench, 85.9 on RM-Bench, and 74.7 on RMB. The authors describe the 85.9 RM-Bench score as a new state of the art and the 74.7 RMB score as top-2.

• The 8B model beats much larger baselines. BR-RM-Qwen-8B scores 91.0 on RewardBench, 85.0 on RM-Bench, and 71.8 on RMB (average 82.6), outperforming baselines such as GPT-4o-0806 (average 77.7) and 70B scalar reward models.

• Scalar reward models plateau on reasoning tasks. INF-ORM-Llama3.1-70B dominates RewardBench (95.1) but scores only 70.9 on RM-Bench and 70.5 on RMB (average 78.8), which the authors read as diminishing returns from parameter scaling alone.

• Generative RMs trade off consistency. Skywork-Critic-Llama-3.1-70B is strong on RewardBench (93.3) but struggles on RMB (65.5), and GPT-4o-0806, despite a balanced profile, still trails leading scalar and reasoning RMs.

• Reasoning RMs require sufficient scale. The 32B RM-R1 models reach up to 83.9 on RM-Bench, whereas smaller reasoning models such as JudgeLRM-7B (64.3 average) underperform even weaker scalar RMs.

• Removing the second turn is the most damaging ablation. "Branching Only" caused a 3.7-point drop on RMB for the 14B model, because the model over-emphasizes granular flaws from the first pass without a comprehensive perspective.

• Branching without conditioning hurts. An "Unconditioned Rethinking" variant using a generic second-pass prompt dropped performance by up to 2.8 points, showing that the second pass needs an explicit directive from Turn 1.

• Merging both turns into one instruction degrades quality. The single-turn variant forces dimension selection, error sketching, and scoring simultaneously, producing shallower analysis.

• The binary reward with format checking is the best reward design. Removing format checks let the model skip reasoning; scoring on a scale from -3 to 3 made the model chase exact values instead of relative quality; adding an extra reward for branching nearly doubled computational cost and produced noisy, unstable signals.

• Two turns is the optimum. Training 14B variants with 1, 3, and 4 turns shows performance peaks at 2 turns and declines with additional iterations, which the authors attribute to "over-thinking," hallucinated errors during repeated verification, or drift from the original context.

• Branch selection is sparse and domain-specific. On RM-Bench, the model activates an average of 2.5 dimensions per instance, with Safety Awareness activated 91% of the time and Implementation Capability 94% of the time in their respective domains.

• The two-turn 8B model is faster than a larger single-turn baseline. On 2× H100 GPUs, BR-RM-Qwen-8B requires 9.5s per instance versus 10.3s for RM-R1-Qwen-Instruct-32B, while achieving higher average accuracy (82.6% vs. 81.2%). Scalar RMs remain far faster (InternLM2-7B-Reward takes 0.015s; INF-ORM-Llama3.1-70B takes 0.25s) but cap at 78.8% average accuracy in this comparison.

• The baseline allocates tokens diffusely in the preliminary study. In the Code task, RRM-32B devotes only about 25% of its tokens to Implementation Capability, whereas BR-RM concentrates over 70% on Information Accuracy and Logical Reasoning for Chat and over 75% on Implementation Capability and Computational Precision for Code.

Methodology in Plain English

Formulating judgment as a two-stage generation task. Instead of learning a scalar function R(x, y) that scores a response directly, the authors train a policy to generate a two-turn deliberation trace for each prompt-and-two-responses triplet, from which a final preference decision is extracted. The goal is for that extracted decision to match the human label.

Turn 1 — Adaptive Branching. The model must first pick a small subset of critical criteria from a predefined universal set (nine candidates, including things like factual accuracy, reasoning quality, and safety), and then write a short preliminary analysis for each candidate response. This forces the model to commit to hypotheses about which dimensions matter most for this specific comparison, rather than evaluating everything at once.

Turn 2 — Branch-Conditioned Rethinking. Given the Turn 1 output, the model re-reads both responses specifically through the lens of the flagged dimensions — verifying facts, checking fragile reasoning, or hunting for localized bugs. Task-specific evaluation hierarchies (for example, accuracy-driven tasks prioritizing correctness over clarity, creative tasks favoring intent alignment and novelty) provide a structured inductive bias for this second pass. The final verdict is extracted from this turn.

Training. The model is optimized with GRPO, a PPO variant chosen for stability and efficiency, using a clipped surrogate objective plus a KL-divergence penalty against a reference policy. Advantages are computed group-relative and critic-free: sample K traces per prompt, compute a terminal reward per trace, and center/whiten it within the prompt group. The reward has two parts: a format reward of 0 for well-formed traces and a penalty (with the penalty constant set to -100) for malformed ones, plus a binary outcome reward of 0 for correct decisions and -1 for incorrect ones, weighted by 10. The same reward is applied separately to the first turn and the second turn, so no special multi-turn RL machinery is needed.

Training data and setup. Training mixes HelpSteer3, Skywork Reward Preference-80K, Code-Preference-Pairs (8K), and Math-Step-DPO-10K. The implementation uses the NeMo-RL library with a custom two-turn environment. Learning rates were 5×10⁻⁷ for Qwen3-8B models and 1×10⁻⁶ for Qwen3-14B models, all trained for 400 optimization steps. Evaluation is by preference accuracy on RewardBench, RM-Bench, and RMB.

Why This Matters

Impact on research. The paper reframes reward modeling from regression to structured reasoning, and gives a concrete, named diagnosis (judgment diffusion) for a known weakness of LLM judges. It shows that allocating test-time compute to a targeted second pass can beat simply scaling parameters — competing 32B single-turn reasoning models are outperformed by an 8B two-turn model in both latency and accuracy in the reported benchmark. It also demonstrates a way to bolt two-turn behavior onto standard GRPO without multi-turn RL support.

Real-world applications:

• RLHF and RLAIF pipelines, where the reward signal is often the bottleneck; a more accurate judge produces more stable preferences for policy optimization.
• Rejection sampling and PPO in offline reinforcement learning, where the authors argue a noisy faster reward can destabilize training more than a slower accurate one.
• Best-of-N selection at deployment, since RMB explicitly supports Best-of-N evaluation where a judge must pick the best among multiple candidates.
• LLM-as-a-judge evaluation systems for grading model outputs in domains like code correctness, math reasoning, and safety compliance, where subtle errors matter more than style.

Industry relevance. The work comes from NVIDIA with academic collaboration (University of Illinois Urbana-Champaign), ships code and model checkpoints, and is built on the NeMo-RL post-training library. The latency analysis on 2× H100 GPUs — and the argument that BR-RM's cost is amortized in asynchronous RL setups where reward generation is decoupled from policy updates — is written for practitioners deciding whether a reasoning judge fits their training budget.

Future Directions

1. Integrating verification tools such as retrieval and code runners to check factual and programmatic claims raised during the rethinking turn.

2. Automatically generating dynamic evaluation rules to replace the static, predefined rubric of criteria used for adaptive branching.

3. Using model uncertainty to adaptively decide when and how deeply to re-evaluate an output, rather than always running a fixed number of turns.

4. Understanding the scale dependency of reasoning-based judging. The paper observes that reasoning mechanisms show clear gains only beyond 14B parameters in its baseline comparison; whether the two-turn recipe changes that threshold, and how far the approach scales beyond the 8B and 14B models tested, is not reported.

Target Audience

Researchers and engineers working on LLM alignment, reward modeling, and RLHF/RLAIF training pipelines; practitioners building LLM-as-a-judge or best-of-N selection systems who need to weigh judge accuracy against inference latency; and readers interested in how test-time reasoning techniques transfer from problem-solving models to evaluation models. A basic grounding in reinforcement learning for language models and familiarity with benchmarks like RewardBench is helpful but the architectural idea is accessible without it.

Authors’ abstract

Large language models (LLMs) increasingly rely on thinking models that externalize intermediate steps and allocate extra test-time compute, with think-twice strategies showing that a deliberate second pass can elicit stronger reasoning. In contrast, most reward models (RMs) still compress many quality dimensions into a single scalar in one shot, a design that induces judgment diffusion: attention spreads across evaluation criteria, yielding diluted focus and shallow analysis. We introduce branch-and-rethink (BR-RM), a two-turn RM that transfers the think-twice principle to reward modeling. Turn 1 performs adaptive branching, selecting a small set of instance-critical dimensions (such as factuality and safety) and sketching concise, evidence-seeking hypotheses. Turn 2 executes branch-conditioned rethinking, a targeted reread that tests those hypotheses and scrutinizes only what matters most. We train with GRPO-style reinforcement learning over structured two-turn traces using a simple binary outcome reward with strict format checks, making the approach compatible with standard RLHF pipelines. By converting all-at-once scoring into focused, second-look reasoning, BR-RM reduces judgment diffusion and improves sensitivity to subtle yet consequential errors while remaining practical and scalable. Experimental results demonstrate that our model achieves state-of-the-art performance on three challenging reward modeling benchmarks across diverse domains.