More Methods, Better Agents? A Reality Check on Self-Distillation in Agentic Reinforcement Learning

AI summary

Overview

Research area: Agentic reinforcement learning (RL) and self-distillation, with a focus on benchmarking and evaluation methodology.

Technical level: Advanced. This assumes familiarity with reinforcement learning for language-model agents, policy distillation, and training-stability concerns.

Scope (one sentence): A large-scale, controlled empirical comparison of ten self-distillation methods for agentic RL against GRPO on four agent environments, across three model sizes, that checks not only final task performance but also training stability and measured compute cost.

What This Paper Is About

Self-distillation for agentic reinforcement learning has produced a growing pile of methods, but the abstract argues that the evaluation culture around them is shaky: papers often reuse published baseline scores that were collected under different experimental conditions, rarely compare contemporaneous methods directly, and give little attention to training stability or computational cost. As a result, it is hard to tell where reported gains actually come from, or which methods are genuinely practical. Because agent training is expensive, the authors build a shared, controlled testbed so that these methods can be compared on a common basis, and then report what that comparison reveals.

Key Contributions

1. A systematic head-to-head comparison. Ten representative self-distillation methods are evaluated against GRPO under one experimental protocol, on ALFWorld, ScienceWorld, Search-QA, and WebShop, across three model sizes.

2. A resource-intensive empirical record. The study comprises 132 training runs totaling 12,124.4 recorded H800 GPU-hours, with repeated evaluations over eight rollout seeds for the pretrained backbones and for all eleven trained methods.

3. An evaluation that goes beyond final scores. Alongside final task performance, the study examines training progress, training stability, and measured computational cost, treating these as first-class dimensions of comparison rather than afterthoughts.

4. A reusable benchmark and three headline conclusions. The authors frame the work as a benchmark for fair, comprehensive comparison, and distill the results into three findings intended to redirect future research toward reliable and cost-effective agent training.

Main Findings

• Gains over GRPO are narrow. The first finding is that improvements over the GRPO baseline remain limited in scope: no method consistently outperforms GRPO across the settings tested.

• Strong final performance can hide instability. The second finding is that a good end-of-training score is not evidence of a healthy training run; the authors report that unstable training can be masked by strong final performance.

• More compute does not reliably buy better results. The third finding is that additional training cost does not consistently translate into better performance.

• Baseline-vs-method comparisons are confounded in the existing literature. The abstract states that reusing published baseline scores under differing experimental conditions obscures the source of reported gains, which is the motivation for the shared protocol used here. The abstract does not give per-method or per-environment score breakdowns.

Methodology in Plain English

The authors set up one fixed experimental protocol and ran everything through it, instead of comparing against numbers copied from other papers. They picked GRPO as the reference point and put ten self-distillation methods up against it. The comparison spans four agent environments — ALFWorld, ScienceWorld, Search-QA, and WebShop — and three model sizes, so the conclusions are tested for robustness across both task type and model scale. Each configuration was trained multiple times rather than once, producing 132 training runs and 12,124.4 recorded H800 GPU-hours. Evaluation was repeated over eight rollout seeds, applied both to the pretrained backbones (to establish what the starting point can already do) and to all eleven trained methods (GRPO plus the ten self-distillation approaches). Crucially, the authors recorded not just the final score but how performance evolved during training, whether training was stable, and what the measured compute cost was. This design lets them ask three separate questions: does a method beat GRPO, does it do so reliably, and does the extra compute it demands pay for itself?

Why This Matters

Impact on research. The abstract positions the paper as a corrective to evaluation practice in agentic RL. If baseline numbers are reused across incompatible setups, and if stability and cost go unreported, the field cannot tell real progress from measurement artifact. A shared protocol with repeated seeds and published compute budget gives subsequent work a fairer reference point, and the three findings suggest the bar for claiming improvement should be higher than a single favorable score.

Real-world applications (each mapping to a setting the abstract names):

• Household and embodied task agents. ALFWorld-style environments stand in for agents that follow instructions and manipulate objects, relevant to robotics and home assistants.
• Scientific experimentation agents. ScienceWorld-style tasks represent agents that carry out multi-step procedures in simulated labs.
• Search-augmented question answering. Search-QA represents agents that issue queries, read results, and synthesize answers, relevant to retrieval-heavy assistants.
• Web commerce agents. WebShop represents agents that navigate sites and complete transactions, relevant to shopping and site-automation tools.

Industry relevance. The cost finding is the most directly commercial: 12,124.4 H800 GPU-hours concentrated in 132 runs illustrates how expensive agent training is, and the abstract's claim that extra cost does not consistently yield better performance argues against assuming that larger training budgets are automatically justified. For teams deciding which distillation method to adopt, an evaluation that reports stability and measured cost alongside accuracy is more useful for production planning than a leaderboard of peak scores.

Future Directions

• Explaining why GRPO is hard to beat. The abstract reports that no method reliably exceeds GRPO but does not identify the cause; isolating what self-distillation adds, or fails to add, over the base RL objective is an open question.

• Diagnosing and fixing instability. Since strong final scores can mask unstable training, follow-up work could characterize when instability appears and whether it can be detected or corrected during training rather than inferred afterward.

• Making training cost pay off. If extra compute does not consistently improve results, the natural next step is to identify which portions of the budget are wasted and design methods that are cost-effective by construction.

• Extending the benchmark. The comparison covers ten methods, four environments, and three model sizes; whether the same conclusions hold for other agent tasks, other model families, or longer training horizons is not addressed in the abstract.

Target Audience

Researchers and graduate students working on reinforcement learning for language-model agents, and especially anyone developing or evaluating self-distillation methods, will find this most useful, since it is aimed directly at how that subfield measures progress. Practitioners and infrastructure or ML-platform engineers choosing a training method for an agent product also benefit, because the study reports measured compute cost and stability rather than accuracy alone. Readers looking for a single new algorithm will not find one here; the contribution is the comparison itself, and the abstract does not provide the per-environment results or method-level rankings needed to pick a winner.

Authors’ abstract

Self-distillation in agentic reinforcement learning has produced a growing number of methods. However, existing evaluations often reuse published baseline scores under differing experimental conditions, provide few direct comparisons among contemporaneous methods, and give limited attention to training stability and computational cost. These limitations obscure the source of reported gains and the practical strengths and weaknesses of competing methods. Costly agent training further motivates a common basis for comparison. To address these gaps, we compare ten representative methods against GRPO on ALFWorld, ScienceWorld, Search-QA, and WebShop across three model sizes. Our study comprises 132 training runs totaling 12,124.4 recorded H800 GPU-hours, with repeated evaluations over eight rollout seeds for the pretrained backbones and all eleven trained methods. Beyond final task performance, we examine training progress, stability, and measured computational cost. Our evaluation yields three key findings. First, gains over GRPO remain limited in scope, with no method consistently outperforming it across settings. Second, strong final performance can mask unstable training. Third, additional training cost does not consistently translate into better performance. We establish this benchmark to support fair, comprehensive comparison and guide future research toward reliable and cost-effective improvements in agent training. Project page: https://aim-uofa.github.io/Self-Distillation-AgentRL/