Exploring Reasoning Reward Model for Agents

AI summary

Overview

• Research area: Agentic reinforcement learning, reward modeling, and tool-using LLM agents.
• Technical level: Advanced. The paper assumes familiarity with RLVR, GRPO, advantage estimation, and policy optimization, though the core idea is graspable at an intermediate level.
• One-sentence scope: The paper builds a reasoning-based reward model that critiques agent trajectories and tests three ways of feeding that feedback into agentic RL, showing that combining scalar scores with natural-language critiques beats sparse outcome-only rewards.

What This Paper Is About

Most agentic reinforcement learning trains agents with a single binary signal: did the final answer come out right. That signal cannot distinguish a trajectory that reasoned well and failed only at the last step from one that was wrong throughout, which makes learning slow and coarse in long multi-step tasks. This paper introduces a reward model that reasons explicitly about a trajectory, describes its flaws in words, and assigns a quality score, then systematically tests how that richer feedback should be plugged into training.

Key Contributions

1. Agent-RRM, a multi-faceted reasoning reward model that produces structured output in three parts: a <think> reasoning trace analyzing logical consistency, a <critique> identifying specific reasoning or tool-use errors, and a <score> in [0, 1] for overall trajectory quality. It operates without ground-truth answers.
2. Three integration schemes with a shared framework: Reagent-C (text-augmented refinement at inference, no parameter updates), Reagent-R (scalar model reward blended with rule-based reward during RL), and Reagent-U (unified feedback, jointly optimizing initial and critique-refined trajectories in one RL loop).
3. Four released datasets: Reagent-RL-709K and Reagent-SFT-55.6K for agent training, plus Reagent-RRM-SFT-28K and Reagent-RRM-RL-90K for reward model training, with the reward-model annotations generated by GPT-OSS-120B across an ensemble of diverse policy models to cover many error patterns.
4. A broad empirical study across 12 benchmarks covering math, knowledge-intensive reasoning, web search, and multimodal agent tasks, establishing which feedback modality helps where.

Main Findings

• Textual critiques alone improve agents without any training. Reagent-C, which simply prompts the frozen agent to retry after seeing Agent-RRM's critique, gains on every benchmark. The widening gap between first and second attempts shows many failures stem from transient execution errors or logical oversights rather than missing capability.
• Model-based scores reduce reward sparsity. Reagent-R beats the rule-reward-only baseline everywhere, with gains such as 72.8% vs. 61.6% on Bamboogle and 41.0% vs. 32.0% on xbench.
• Unified feedback is the strongest configuration. Reagent-U reaches 43.7% on GAIA (text subset), 46.2% on WebWalkerQA, 76.8% on Bamboogle, and 60.0% on AIME24 — outperforming rule-only ablation, critique-only variant, and all listed open-source and proprietary baselines. Critically, critiques are used only during training; at inference Reagent-U runs as an ordinary agent with no extra refinement passes.
• Scores and critiques are complementary, not redundant. Reagent-R beats the sparse baseline but trails Reagent-U, indicating scalar scores differentiate trajectory quality while textual critiques supply the explicit structural guidance that scores lack.
• The scheme generalizes beyond text. On the full multimodal GAIA set (165 tasks requiring search, image description, Python, file reading, and audio), Reagent-U scores 38.8 pass@1 and 53.9 pass@3, beating MCP-R1 on both, indicating it does not overfit to web-search-style tasks.
• The reward weight has an optimum. Performance rises with λ, plateaus between 0.2 and 0.4, and declines slightly at 0.5 — overweighting intermediate-process rewards trades away final task completion.

Methodology in Plain English

The researchers start by collecting a large pool of question-answer pairs from public sources and filtering them for unambiguous ground truth, deduplication, and appropriate difficulty. They use a strong model to generate reasoning trajectories with a fixed set of six tools (search, web browse, Python interpreter, file reader, image describer, audio transcriber), keeping only trajectories that reach correct answers. That becomes cold-start supervised fine-tuning data.

Separately, they sample trajectories from several different models so the reward model sees many error styles, then have a large annotator model label each trajectory with the three-part judgment. The reward model itself is trained in two stages: supervised fine-tuning to learn the output format, then GRPO to sharpen its judgments and calibrate its scores.

For the agent, everything is built on Qwen3-8B using GRPO with group-normalized advantages. The three variants differ in where the reward model's output enters. In Reagent-C the critique is appended to the prompt and the agent regenerates, with no learning. In Reagent-R the reward model's scalar score is added to the rule-based correctness reward, weighted by λ = 0.3, giving dense supervision during RL. In Reagent-U both stages run: the agent produces an initial attempt, receives a critique, produces a refined attempt, and all attempts from both stages are pooled together. Advantages are normalized across the pooled set, so the agent is rewarded for producing good first attempts and for successfully acting on feedback. This teaches the agent to internalize the corrective reasoning so it no longer needs the critique at test time.

Training uses 8 A800 GPUs, LLaMA-Factory for SFT, rLLM and VeRL for RL, 300 RL steps, and batch size 32 for the agent (64 for RL rollouts) with 8 samples per prompt.

Why This Matters

Impact on research. The paper challenges the default assumption in agentic RL that outcome correctness is sufficient supervision. It shows that a reasoning reward model producing language critiques plus scores offers a practical middle ground between expensive step-level process reward models (which need heavy annotation and are prone to reward hacking) and pairwise preference models (which give no actionable guidance). The unified pooling scheme is a concrete recipe for turning inference-time critique into internalized policy improvement.

Real-world applications:

• Deep research and web-navigation assistants that must chain search, browsing, and synthesis across many steps.
• Multimodal document and file analysis agents handling mixed images, PDFs, and structured data.
• Customer-facing agents whose failures often occur on a single final tool call after otherwise correct planning.
• Any long-horizon tool-using workflow where binary success signals make training data inefficient.

Industry relevance. The approach is attractive to teams training agentic models because it reuses a reward model trained without ground truth, requires no human step-level labels, and adds no inference-time cost — the critique scaffolding disappears after training. That combination of cheaper supervision and no deployment overhead is directly usable in production agent pipelines.

Future Directions

• Scaling beyond 8B. All experiments use Qwen3-8B; whether structured reasoning feedback yields larger or diminishing returns at 32B, 70B, or larger remains untested.
• Broader tool sets and open-ended environments. The current setup uses six fixed tools and standardized benchmarks; extending to unpredictable real-world domains such as AI for science would test adaptability.
• Reward model robustness and hacking resistance. The paper argues scalar process rewards are vulnerable to reward hacking but does not deeply analyze whether its own critique-and-score model can be gamed over long training runs.
• Optimal feedback mixing. The λ analysis shows a narrow plateau; a principled method for scheduling or adaptively weighting process versus outcome rewards could remove that manual tuning.

Target Audience

Researchers and engineers working on agentic RL, reward modeling, or tool-using LLM systems who already understand policy optimization basics and want a concrete, replicable recipe for replacing or augmenting sparse outcome rewards. It is also useful for practitioners deciding whether to invest in process reward models versus reasoning-based critique models for long-horizon agent training. Readers without RL background will follow the motivation and results but should expect to skim the GRPO objective and advantage-normalization details.

Authors’ abstract

Agentic Reinforcement Learning (Agentic RL) has achieved notable success in enabling agents to perform complex reasoning and tool use. However, most methods still relies on sparse outcome-based reward for training. Such feedback fails to differentiate intermediate reasoning quality, leading to suboptimal training results. In this paper, we introduce Agent Reasoning Reward Model (Agent-RRM), a multi-faceted reward model that produces structured feedback for agentic trajectories, including (1) an explicit reasoning trace , (2) a focused critique that provides refinement guidance by highlighting reasoning flaws, and (3) an overall score that evaluates process performance. Leveraging these signals, we systematically investigate three integration strategies: Reagent-C (text-augmented refinement), Reagent-R (reward-augmented guidance), and Reagent-U (unified feedback integration). Extensive evaluations across 12 diverse benchmarks demonstrate that Reagent-U yields substantial performance leaps, achieving 43.7% on GAIA and 46.2% on WebWalkerQA, validating the effectiveness of our reasoning reward model and training schemes. Code, models, and datasets are all released to facilitate future research.