Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning

AI summary

Overview

Research area: Natural language processing, specifically LLM-as-a-judge evaluation and tool-integrated reasoning (TIR) trained with reinforcement learning.

Technical level: Advanced. The paper assumes familiarity with policy-gradient RL, reward modeling, and multi-turn tool-use pipelines, though the core idea is conceptually straightforward.

Scope: The paper introduces TIR-Judge, a family of 4B and 8B judge models trained end-to-end with multi-turn reinforcement learning that interleaves natural-language reasoning with Python code execution, and evaluates them across seven public judge benchmarks under pointwise, pairwise, and listwise settings.

What This Paper Is About

Most LLM judges produce verdicts through text-only reasoning, which is unreliable when a prompt demands exact counting, symbolic verification, or precise arithmetic — models tend to miscount or miscompute in their heads. This paper asks whether a judge can instead be trained to write and run code as part of its evaluation process, verifying constraints empirically rather than by intuition. The goal is a general-purpose judge whose reasoning and tool use are jointly optimized end-to-end with reinforcement learning, rather than having tools bolted on at inference time.

Key Contributions

1. TIR-Judge, an end-to-end multi-turn RL framework for judges with integrated code execution. The authors state this is the first approach that jointly optimizes reasoning and tool-use for training LLM-based judges via RL, using a Python execution environment inside the judgment trajectory.

2. Design strategies that make the RL work. These include task diversification across verifiable domains (competitive programming, mathematical reasoning) and non-verifiable domains (dialogue, safety, general coding), flexible judgment formats spanning pointwise, pairwise, and listwise evaluation, and a structured reward combining correctness, format, and tool-specific signals.

3. An iterative RL scheme that removes the need for distillation. TIR-Judge-Zero bootstraps purely from the initial checkpoint through alternating rejection sampling, supervised fine-tuning, and RL, while TIR-Judge-Distill offers an optional cold-start variant using roughly 10k tool-integrated trajectories from a Gemini-2.5-Flash teacher with code execution.

4. Broad empirical validation. Evaluation on seven public benchmarks showing gains of up to 6.4% (pointwise) and 7.7% (pairwise) over strong reasoning-based judges, with 8B listwise performance reaching 96% of Claude-Opus-4.

Main Findings

• Tool use only helps when it is trained, not prompted. Simply augmenting Qwen-3 models with code execution yields negligible gains (under 1%) or even negative gains. The authors attribute this to base checkpoints lacking robust code-generation ability, and conclude that RL is essential for unlocking tool use.

• Substantial gains on reasoning-centric judge benchmarks. On the PPE benchmark, TIR-Judge outperforms baselines of similar size by 4.8%–9.9% for pointwise judging and 4.5%–8.8% for pairwise judging.

• Parameter efficiency against much larger models. With only 8B parameters, TIR-Judge surpasses 32B reasoning reward models on the PPE dataset, and achieves similar accuracy on PPE and RewardBench to RRM-32B despite having roughly 1/4 to 1/8 of its parameters.

• Iterative RL reduces the need for distillation. TIR-Judge-Zero matches or beats TIR-Judge-Distill on 4 of 6 benchmarks in the pointwise setting and 3 of 6 in the pairwise setting. At 4B scale, TIR-Judge-Zero shows a 1.2% gain over its distilled counterpart.

• Listwise judging approaches frontier models. On RewardBench 2, TIR-Judge-Zero 8B averages 73.4, matching 96% of Claude-Opus-4's 76.5 despite being an 8B model. Other averages in that table include Gemini-2.5-flash-Preview at 75.9, TIR-Judge-Distill 8B at 71.6, GPT-4.1 at 69.7, Claude-Sonnet-4 at 69.5, TIR-Judge-Zero 4B at 68.3, TIR-Judge-Distill 4B at 67.3, GPT-4.1-mini at 65.7, and GPT-4o at 64.9.

• Diverse data mixture is essential. Training exclusively on chat or reasoning tasks transfers poorly across subtasks because tool-use prompts are scarce; unifying tool-use and non-tool-use tasks into one pipeline improves generalization.

• Tool-augmented judges beat text-only judges on reasoning and instruction following. In a controlled study with code execution disabled but training data held identical, tool-augmented models were consistently more accurate on reasoning and IF benchmarks, while text-only models performed slightly better on text-centric Chat and Safety tasks in RMBench.

• No inference-time overhead. Despite calling an external code executor, TIR-Judge is more efficient than baselines, which the authors attribute to the SFT data construction strategy that favors trajectories with shorter reasoning and fewer tool calls during rejection sampling.

• Iterative RL improves progressively. The paper reports substantial gains after the first round of RL, driven by rejection sampling producing more format-correct and efficient tool use; additional RL iterations further raise accuracy, whereas rejection-sampling fine-tuning alone yields only modest gains.

• Tool use improves downstream policy performance. In reward-guided best-of-N inference over AIME-2024, AIME-2025, BigCodeBench, and IFEval, TIR-Judge consistently surpasses both Majority Voting (Self-Consistency) and the RRM baseline, with absolute gains of 3.9%–6.7% over RRM on BigCodeBench and AIME.

• Execution errors are rare. In a breakdown of code execution errors, TIR-Judge-Zero shows 1.20% syntax errors, 0.00% runtime errors, and 0.17% formatting failures, versus 3.65%/0.00%/0.14% for TIR-Judge-Distill and 4.95%/0.00%/0.97% for Qwen-3-Tool.

• Case study illustrates the mechanism. On an IFEval prompt requiring a letter to appear at least 40 times, TIR-Judge-Zero 8B wrote code reporting counts of 58 and 60 with capitalization checks, while Qwen3-8B manually counted and produced an internally contradictory conclusion (both responses below threshold, yet claiming Response B had more occurrences while listing 20 versus 23).

Methodology in Plain English

The authors frame judging as a scoring or ranking problem under three input formats: pointwise (score one response), pairwise (pick the better of two), and listwise (pick the best of several). They extend the judge so that at each step it can emit a natural-language reasoning step, a block of Python code, and receive the execution output, repeating this cycle until it produces a final verdict. Crucially, the reference answer is unseen during evaluation, distinguishing this from verification settings.

Training data comes from about 26k preference pairs mixing human-annotated sources — HelpSteer 3 for general helpfulness, UltraInteract and S1 for reasoning, CodeRM for coding, Tulu 3 for instruction following, and Safe-RLHF for safety — with synthetic pairs generated by sampling responses from Qwen3-8B/14B, Gemma-2-9B, and Gemma-3-12B and automatically scoring them against verifiable functions or ground-truth solutions. Math data comes from MATH and DAPO-Math, and general reasoning data from WebInstruct and Loong. The authors apply strict 8-gram decontamination against evaluation benchmarks.

The RL algorithm is DAPO, an improved variant of GRPO. For each prompt, a group of G rollouts is sampled, each receiving a reward, and the policy is updated with a clipped policy-gradient objective with token-level advantages and a KL penalty.

Rewards are deliberately compositional. A correctness reward checks whether the judge's verdict matches the ground-truth preference. A format reward enforces tagged output for scores, preference labels, and code blocks, and includes a heuristic that denies positive format reward if the model calls tools on safety or general-helpfulness prompts where tools are unnecessary. A tool-specific reward grants credit only when code blocks run error-free within a budget of at most 3 tool calls per trajectory. The final reward is R_c × (0.1 + 0.9·I[R_t = 1 ∧ R_f = 1]), meaning full credit requires correctness, format compliance, and clean tool use together.

Two stabilizing tricks are applied: interpreter output is truncated to only the final error line to keep context short, and execution outputs are masked out of the loss so the model cannot memorize result strings.

For cold-starting, TIR-Judge-Distill uses Gemini-2.5-Flash with code execution to generate roughly 10k successful trajectories for supervised fine-tuning, keeping only trajectories that got the right answer. TIR-Judge-Zero skips this: it runs RL directly, then alternates rejection sampling (keeping only trajectories that are correct, well-formatted, and error-free, retaining the shortest or fewest-tool-call trajectory per prompt), SFT, and another round of RL, selecting the best checkpoint by held-out validation accuracy.

Backbones are Qwen3-8B and Qwen3-4B-Instruct-2507 without thinking mode, trained with Verl-Tool on 8 NVIDIA H100 80G GPUs. SFT uses batch size 64, learning rate 2e-6, context length 8192, for 1 epoch. RL uses micro batch size per GPU of 4, mini batch size 128, 8 rollouts, ε_low = 0.2, ε_high = 0.3, β = 0.01, max response length 8192, learning rate 1e-6, and 2 epochs. Five percent of prompts from each task were held out for validation.

Why This Matters

Judges sit at multiple points in the LLM development pipeline: providing preference signals during post-training alignment, verifying and selecting responses during best-of-N decoding at inference, and delivering automated assessments during evaluation. If judges misjudge deterministic constraints — counting characters, checking formatting rules, verifying arithmetic — the errors propagate into those downstream uses. This paper's finding that tool use must be trained rather than prompted suggests that the common practice of giving a judge a code interpreter at inference time is not enough, and that RL over the full reasoning-tool-verdict trajectory is what makes the capability reliable.

Real-world applications:

• Instruction-following evaluation pipelines, where constraints like exact word counts, formatting rules, or required keywords can be verified by code rather than approximated by the judge.
• Best-of-N response selection in deployed systems, where the paper reports 3.9%–6.7% absolute gains over the RRM baseline on competition math and complex code generation.
• Reinforcement learning from AI feedback, where judge accuracy directly determines the quality of preference signal used to align policy models.
• Automated code review and ranking, via the CodeJudgeBench evaluation that appears in the paper's benchmark suite.

Industry relevance: The parameter-efficiency result is the practically meaningful one for deployment. An 8B judge that approaches a 32B reasoning reward model, and reaches 96% of Claude-Opus-4 in the listwise setting, changes the cost calculus for teams that would otherwise pay for frontier-model judging or serve a much larger judge. The paper also reports no additional inference-time overhead from the tool calls, due to the preference for shorter trajectories with fewer tool calls.

Future Directions

The paper is truncated before its conclusion, so explicit future work is not reported. The following are open questions the work raises:

• How far iterative RL scaling can go without distillation. The paper demonstrates one or two RS-SFT-RL cycles; whether gains continue to compound over more rounds, and where they saturate, is not reported.
• Expanding beyond Python execution. Only a Python interpreter is used; whether the same framework transfers to search, retrieval, or other external tools in the judging loop is not tested.
• Handling text-centric tasks where tools hurt. Text-only judges performed slightly better on Chat and Safety subsets in RMBench, so learning a sharper policy for when not to invoke tools remains an open problem beyond the current format-reward heuristic.
• Generalization to longer or more complex constraints. The IFBench results suggest some transfer to non-verifiable constraints; the limits of that transfer, and behavior on constraints too complex to express in a few lines of code, are not characterized.
• Scaling the backbone. Results cover 4B and 8B Qwen3 models; whether the trends hold for larger backbones or non-Qwen families is not established.

Target Audience

This paper is most useful for researchers and engineers working on reward models and LLM-as-a-judge systems, particularly those already comfortable with RL fine-tuning pipelines such as GRPO or DAPO. It also suits practitioners building evaluation or preference-data infrastructure who need to decide between prompting a judge with tools and training one to use them. Readers looking for an accessible introduction to LLM judging may find the RL details and reward-design discussion dense, but the central argument — that tool-using judges must be trained end-to-end — is stated plainly enough to be useful background.

Authors’ abstract

Large Language Models (LLMs) are widely used as judges to evaluate response quality, providing a scalable alternative to human evaluation. However, most LLM judges operate solely on intrinsic text-based reasoning, limiting their ability to verify complex constraints or perform accurate computation. Motivated by the success of tool-integrated reasoning (TIR) in numerous tasks, we propose TIR-Judge, an end-to-end RL framework for training LLM judges that integrates a code executor for precise evaluation. TIR-Judge is built on three principles: (i) diverse training across verifiable and non-verifiable domains, (ii) flexible judgment formats (pointwise, pairwise, listwise), and (iii) iterative RL that bootstraps directly from the initial model without distillation. On seven public benchmarks, TIR-Judge surpasses strong reasoning-based judges by up to 6.4% (pointwise) and 7.7% (pairwise), and achieves listwise performance comparable to Claude-Opus-4 despite having only 8B parameters. Remarkably, TIR-Judge-Zero - trained entirely without distilled judge trajectories, matches the performance of distilled variants, demonstrating that tool-augmented judges can self-evolve through iterative reinforcement learning.