What Does Multi-Turn Reinforcement Learning Really Improve in LLM Agents?

AI summary

Overview

Research area: Multi-turn ("agentic") reinforcement learning for large language model (LLM) agents, with a focus on post-training analysis, evaluation methodology, and training strategy.

Technical level: Intermediate to Advanced. The paper is readable by a graduate-level ML audience, but it assumes familiarity with policy-gradient RL, log-likelihoods, and agent benchmarking.

Scope (1 sentence): The paper asks what multi-turn agentic RL actually improves by evaluating RL-trained agents along two axes — sampling breadth (K) and interaction depth (T) — and finds that the gains come mainly from error recovery and long-horizon execution rather than uniformly better single-step decisions, then proposes a budget-varying training method called BudgetMix RL.

What This Paper Is About

Multi-turn agentic RL (e.g., training an LLM agent with GRPO on code or tool-use tasks) reliably beats base models on final task success, but it is unclear which underlying capabilities actually improve. The authors show that conventional trajectory-level success rates hide two things: the gains shrink or reverse when the agent is given a tighter interaction-turn budget, and the model's quality on individual decisions does not consistently improve and can even degrade. The paper's goal is to characterize these effects mechanistically and to offer a practical training fix.

Key Contributions

1. A two-dimensional evaluation, Pass@(K,T). The authors combine sampling breadth (K independent trajectories) with interaction depth (T maximum turns) into a single metric, isolating each axis by fixing the other. Pass@T is defined as Pass@(K=C, T) with a constant sampling budget C.

2. Evidence that agentic RL gains are budget-dependent. RL models consistently outperform base models as K grows, extending the empirical Pass@K boundary — unlike prior single-turn RL observations — but under tight T budgets the advantage shrinks and can even reverse on certain tasks. The authors compute structural performance upper bounds to rule out the trivial explanation that small budgets make tasks unsolvable.

3. A step-level decomposition of where the gains come from. Using matched states sampled equally from base and RL trajectories, the authors show RL does not uniformly improve per-step action quality; its advantage concentrates at error states and at later stages of a trajectory.

4. A mechanistic account plus a mitigation. Log-likelihood analysis shows RL actions stay plausible under the base policy while whole RL trajectories are much more likely under the RL policy; training-dynamics analysis shows trajectory reward rising as step reward falls. The paper then introduces BudgetMix RL, which varies interaction budgets during training, and shows it improves performance across budgets while mitigating step-level quality decline.

Main Findings

• RL surpasses the base model's Pass@K boundary. As sampling breadth k increases, the RL model consistently outperforms the base model, in contrast to prior single-turn findings where RL was reported not to expand the capability boundary beyond the base model.

• The advantage collapses under tight turn budgets. With sampling breadth fixed at k = 8, increasing T makes the RL advantage more pronounced, while small T diminishes it and can reverse it on certain tasks. Even where substantial headroom remained below the structural upper bound, RL showed no clear advantage at small T.

• Single-step quality is not uniformly better. With G = 16 sampled actions per state and averaged quality scores, RL produced better step-level decisions on only 46.8% of DeepCoder-MT states and 42.8% of FrozenLake states, with average quality no better than — and slightly below — the base model. FrozenLake step quality was scored with finite-horizon dynamic programming at a discount factor of 0.95; DeepCoder-MT steps were scored by unit-test pass rates.

• RL excels immediately after errors. On an error-state pool built by injecting an incorrect action at the preceding step, RL chose the superior action in 65.7% of FrozenLake cases and 67.3% of DeepCoder-MT cases. Continuing from identical error states, RL beat the base model on FrozenLake (51.3% vs 38.6%) and DeepCoder-MT (38.5% vs 22.9%).

• The advantage emerges late in trajectories. Grouping states by normalized depth d_t = t / T_max, RL shows little or no advantage early and can underperform the base model; the gap narrows and reverses as interaction proceeds.

• Actions stay locally plausible; trajectories shift. Under the base policy, per-token action NLL on Terminal-Bench was 12.20 for RL actions versus 10.25 for base actions — heavily overlapping distributions. But cumulative log-likelihood ratio for successful RL trajectories reached roughly 228, versus 26 for successful base trajectories. The RL policy assigns much higher probability to whole trajectories even though individual actions look ordinary to the base policy.

• Trajectory reward rises while step reward falls. On DeepCoder-MT, standard agentic RL raised the trajectory-level reward R_multi by 0.331 while the step-level reward R_single dropped by 0.097, showing the effect emerges progressively during training rather than being a final-checkpoint artifact.

• Process supervision helps only when it is genuinely precise. On FrozenLake, oracle Q-value process rewards and GiGPO stabilized performance across budgets, especially tight ones. On BFCL, where no exact step-level oracle exists, GiGPO and PPO failed to improve performance under constrained budgets and performed similarly to GRPO.

• BudgetMix RL improves robustness across budgets. At 25%, 50%, and 75% of T_max, BudgetMix outperformed standard full-budget RL across all four main tasks, with average gains of 7.5, 5.3, and 4.0 percentage points respectively. At 100% of T_max it maintained parity, showing low-budget gains do not cost long-horizon capability. During BudgetMix training, R_multi rose steadily while R_single stayed stable or improved slightly, unlike full-budget RL where step quality degraded.

Methodology in Plain English

Evaluation design. The authors take a base LLM and an RL-trained version of it and score both with Pass@(K,T): sample K trajectories and count success if any one succeeds within T turns. Sweeping K probes how far the model's reach extends with more attempts; sweeping T probes how much it benefits from more interaction. Agents are told their total and remaining turn budget in the system prompt.

Where they tested. Four interactive benchmarks are used in the main evaluation — DeepCoder-MT (multi-turn code generation with unit-test feedback), BFCL (executable tool calling), InterCode-SQL (interactive SQL built from the Spider training set), and FrozenLake (planning) — plus Terminal-Bench for the likelihood analysis. FrozenLake training and test maps are generated independently and kept disjoint. Base models are Qwen3-4B and Qwen3-8B; algorithms are GRPO, GiGPO, and PPO, implemented on VeRL, all trained from the base model with a learning rate of 1e-6, weight decay 0.01, sampling temperature 1.0 and top-p 1.0, KL coefficient 0.001 with the low_var_kl estimator, and G = 16 trajectories per task instance.

Step-level probing. To compare decision quality fairly, the authors build a pool of intermediate states sampled equally from base and RL trajectories, so differences are not driven by which states each model happens to visit. At each matched state they sample 16 actions from each policy and average the quality scores. Quality is measured exactly where possible: dynamic programming gives the true Q* for FrozenLake, and unit-test pass rates score code in DeepCoder-MT. A separate error-state pool is built by deliberately injecting a wrong action one step earlier, so the two models face the same recovery problem.

Mechanistic probing. Two measurements are used. Action NLL under the base policy measures how "surprising" the RL model's individual actions are to the base model. The cumulative log-likelihood ratio measures whether a full recorded trajectory is more likely under the RL policy than the base policy. Training curves of trajectory reward and step reward are tracked together to see when the divergence appears.

The fix. BudgetMix RL changes one thing in standard training: instead of always using the full budget T_max, it builds candidate budgets at 10% intervals of T_max, takes the unique values, and uniformly samples one budget per task group. Budget, total and remaining, is stated in the prompt, and trajectories that exceed the budget without finishing get zero reward. The reward mechanism is otherwise identical, so BudgetMix drops into GRPO, GiGPO, or PPO unchanged.

Why This Matters

Impact on research. The paper argues that trajectory-level success rate — the dominant metric for agentic RL — is an incomplete measure. It can conceal both a dependence on generous turn budgets and a degradation in per-step decision quality. The Pass@(K,T) framing gives the field a way to report agent capability along two separable axes, and the likelihood and training-dynamics analyses offer a mechanistic story (composition of locally plausible actions into long-horizon behavior) that connects agentic RL to the single-turn RLVR literature on capability boundaries.

Real-world applications:

• Financial operations agents, where individual actions can be irreversible and a bad step cannot be retried.
• Database operations and SQL agents, where a wrong query may commit changes rather than fail harmlessly.
• Coding and terminal agents, where early mistakes compound and recovery ability matters more than first-try correctness.
• Tool-calling assistants under strict latency or API-call limits, where the deployment budget is smaller than the training budget.

Industry relevance. Production agents often run under tight step or cost limits that differ from training conditions. A model that only outperforms its base under a generous budget is a deployment risk. BudgetMix is a low-engineering-cost change — same reward, same algorithms, only the budget distribution differs — which makes it practical for teams already running GRPO/PPO/GiGPO pipelines. The paper's caveat is equally relevant: dense process rewards only help when they are genuinely accurate, and the authors found no such oracle for BFCL.

Future Directions

• Obtaining reliable process signals in agentic settings. The paper shows oracle Q-values stabilize training on FrozenLake but that no exact step-level oracle exists for tool-calling tasks such as BFCL. How to construct trustworthy step-level supervision without an oracle remains open.

• Better budget curricula than uniform sampling. BudgetMix samples budgets uniformly at 10% intervals. Whether adaptive, difficulty-aware, or curriculum-style budget schedules improve further on the reported gains is untested here.

• Extending the analysis to other algorithms and model families. The main experiments use Qwen3-4B and Qwen3-8B with GRPO, GiGPO, and PPO; the authors point to additional results in appendices across algorithms, architectures, tasks, and analysis settings, but the scope of generalization is a natural next question.

• Metrics beyond task success for deployment. The paper's ethics statement argues agents should be evaluated on robustness across execution constraints and reliability of individual decisions. Operationalizing that into standard benchmarks is not yet solved.

Target Audience

Researchers and engineers working on LLM agent training and post-training, especially those using outcome-based RL such as GRPO on multi-turn tool-use, coding, or planning tasks. It is also useful for evaluation and benchmarking teams who need to know why a headline success rate may not transfer to a budget-constrained deployment, and for practitioners deciding whether to invest in process reward models or in simpler training-time interventions like BudgetMix.

Authors’ abstract

While multi-turn agentic reinforcement learning (RL) excels on long-horizon tasks, its performance gains remain highly sensitive to interaction constraints. Evaluating across sampling breadth $K$ and interaction depth $T$ via $Pass@(K,T)$, we show that trajectory-level RL advantages depend on sufficient turn budgets, diminishing significantly when budgets tighten. Besides, agentic RL does not uniformly elevate single-step decision quality; rather, its strength concentrates in error recovery and long-horizon resilience. To mitigate the deployment risks caused by this budget dependence, we provide a preliminary discussion on the benefits and limitations of process supervision signals and introduce BudgetMix RL as an alternative that varies interaction budgets during training. Experiments demonstrate that BudgetMix RL achieves robust performance across diverse execution settings. Project Page: https://rethinking-agentic-rl.github.io/