RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning

AI summary

Overview

Research area: Large language model agents, reinforcement learning with verifiable rewards (RLVR), and on-policy knowledge distillation. Technical level: Advanced. Scope: The paper proposes a three-stage training recipe that builds a task-skill-conditioned teacher, distills it into a skill-free student alongside RL, and then automatically stops distilling when teacher supervision begins to hurt.

What This Paper Is About

Multi-turn LLM agents trained with reinforcement learning get only one scalar reward per whole trajectory, leaving the many intermediate decisions in a long interaction unsupervised. On-policy distillation (OPD) addresses this by having a teacher model—conditioned on privileged "skills" available only at training time—score the student's own tokens, with the hope that the student absorbs those skills into its weights. This paper shows that two common assumptions behind that recipe are false, and replaces them with an adaptive, signal-driven version.

Key Contributions

1. Identification of two failure modes of privileged-information distillation for agents. First, an unoptimized skill-conditioned teacher is not reliably better than the student it supervises—the shared policy is optimized mainly through the skill-free branch, so it never learns to act on the skill context, and larger models do not fix this. Second, the value of teacher supervision is stage-dependent: the teacher-student discrepancy first shrinks (useful transfer) and then widens (conflicting gradients), so continued matching caps the student near the teacher's ceiling.
2. The RetireOPD method. A decoupled, skill-conditioned teacher is first trained with environment rewards via GRPO, then a skill-free student is trained jointly with GRPO and OPD, and the teacher is retired online—no schedule fixed in advance—once behavior transfer stagnates and the student reaches a target fraction of the teacher's success rate.
3. A theoretical account of when to stop. A first-order analysis shows that the sign of the inner product between the reward gradient and the distillation gradient determines whether OPD helps or hurts, and that a non-decreasing teacher-student discrepancy is a usable proxy for that conflict.
4. Broad empirical validation. Across Qwen2.5-1.5B/3B/7B on ALFWorld and WebShop, RetireOPD beats RL, distillation, and hybrid baselines and surpasses its own skill-conditioned teacher in every setting; code is released.

Main Findings

• Teacher supervision only helps early, then conflicts. Under GRPO+OPD the teacher-student discrepancy narrows and then rebounds, while under pure OPD it keeps shrinking toward zero. Continuing to align the student to the teacher past the crossover produces a performance plateau or degradation, whereas removing OPD and continuing with GRPO alone raised success from 76.6% at the retirement point to 93.8%.
• Privileged context alone does not make a teacher good. Without explicit optimization, 3B and 7B skill-conditioned teachers reach only 28.9% and 23.4% on ALFWorld. Training the teacher with environment rewards lifts those to 79.7% and 90.6%, and the stronger teacher transfers to a stronger student.
• The right retirement point moves; fixed schedules are wrong. The step at which the discrepancy stops shrinking varies from roughly step 50 to 90 across models and tasks. Replacing adaptive retirement with ATOD-style linear annealing improves quickly early but plateaus, while the adaptive criterion keeps improving.
• Both retirement signals are needed. Using only alignment stagnation or only relative competence gives 89.0% on ALFWorld, while combining them plus retiring at the detected point reaches 92.2%; retaining OPD throughout gives only 82.8%, and GRPO alone 75.0%.
• Retirement reduces teacher dependence. With OPD retained, students inherit the quality gap between a 3B and 7B teacher (82.8% vs 89.8%). With adaptive retirement, the gap nearly vanishes (92.2% vs 91.4%).
• Headline gains. ALFWorld success improves over the GRPO baseline by 14.1 to 18.8 points (e.g., 93.8% vs 75.0% at 3B). WebShop accuracy improves by 11.8 to 19.0 points (e.g., 75.8% vs 56.8% at 1.5B). RetireOPD also beats GiGPO at every scale on both tasks.
• Robust thresholds. The detected retirement step stays within about ±5 steps for γ in [0.80, 0.96] and δ in [−0.10, 0.04], and success rates range only from 89.1% to 92.2% across a wide sweep—performance is not monotone in the switching step, so the method does not depend on one precise cutoff.
• Token-level case study. At a decision-critical ALFWorld step where the skill says to clean an object before placing it, the trained teacher assigns the token "clean" a log-probability of −0.826 versus −16.625 for the OPSD teacher—showing that privileged text only becomes useful supervision after the teacher is trained to exploit it. Later in training, the same teacher disfavors actions the student finds task-effective, illustrating the conflict.

Methodology in Plain English

The pipeline has three stages.

Stage 1 — Build a teacher that actually uses the skills. Start the teacher and student from the same base model. Give the teacher task-relevant skill text as extra context and train it with ordinary GRPO against environment rewards, so it learns to convert that text into good behavior. Then freeze it.

Stage 2 — Train the student on both signals at once. The student never sees the skill text. On trajectories the student itself generates, it is optimized with two losses: a GRPO term from the environment's success/failure reward, and an on-policy distillation term that pulls the student's per-token distribution toward the frozen teacher's distribution at the same prefix. The exact reverse KL is expensive to compute over the whole vocabulary, so the authors use a cheap single-sample estimate: the difference in log-probability of the actually sampled token under teacher and student. The distillation term is weighted by a small coefficient (λ = 0.01).

Stage 3 — Decide when to fire the teacher. Training is chopped into windows of W = 5 steps. Two quantities are tracked: how fast the average teacher-student log-probability gap is still shrinking, and how close the student's success rate is (averaged over two windows) to the teacher's. The teacher is dismissed at the first window where the gap has stopped shrinking (or reversed) and the student has reached at least 90% of the teacher's success rate. After that, training continues with GRPO alone, so no further teacher forward passes are needed.

The evaluation covers three Qwen2.5 scales on ALFWorld (text-based embodied household tasks) and WebShop (simulated shopping), using a shared skill bank and keyword-based skill retrieval, with baselines spanning prompting, four RL algorithms, two pure distillation methods, and two hybrid methods.

Why This Matters

The paper reframes "teacher supervision" from a fixed ingredient into a temporary scaffold whose usefulness can be read directly off the training signal. This is a shift in how the distillation/RL combination is controlled: instead of tuning an annealing schedule per model and per task, the method lets the training dynamics decide. It also undercuts a widely used assumption—that giving a model privileged context automatically makes it a better teacher—with concrete evidence that an untrained skill-conditioned branch can be far worse than the student it supervises.

Real-world applications:

• Web and e-commerce agents. WebShop is a stand-in for shopping assistants that search, navigate, and select products under constraints; the Alibaba Group co-authorship points directly at this setting.
• Embodied and household assistants. ALFWorld represents long-horizon, state-tracking instruction following, relevant to domestic robotics and text-based control.
• Tool-using enterprise agents. Any pipeline that uses retrieved domain knowledge or expert hints to improve a model can train with those hints and deploy without them, avoiding retrieval latency and context cost at inference.
• Cost-sensitive or on-device deployment. Retiring the teacher removes teacher forward passes entirely, and the final student needs no privileged context, which lowers serving cost.

Industry relevance is high because the method is a drop-in modification to an existing RL post-training loop: it requires no new data, no larger teacher model, and no inference-time scaffolding. The authors also emphasize that teacher quality matters more than teacher size, which matters for teams that cannot afford a bigger model.

Future Directions

• Do the retirement signals generalize? The two criteria are validated on two benchmarks and one family of privileged context (retrieved skills). Whether the same discrepancy-stagnation proxy works for hindsight rationales, tool traces, or long chain-of-thought supervision is untested.
• Beyond the local first-order argument. The theoretical justification is a local expansion assuming inactive clipping and ignoring finite-sample noise. A more global or noise-robust criterion for teacher usefulness could replace the threshold pair (γ, δ) and remove the hand-set constants.
• Retirement as the end of a one-way street. The teacher is discarded permanently. Re-engaging, cycling, or maintaining a population of teachers as the student's competence profile changes—rather than a single retire event—is an open design question.
• Scope and scale. Experiments stop at 7B and 150 training steps on two environments. Behavior on much larger models, longer horizons, multi-agent settings, or other RL algorithms beyond GRPO (PPO, RLOO, GiGPO are only baselines) remains to be characterized.

Target Audience

LLM post-training and RL researchers working on agentic systems; applied scientists and engineers building multi-turn tool-using or web agents who need dense supervision without an inference-time teacher; and practitioners interested in when and how to remove a distillation signal rather than annealing it on a fixed schedule. Readers should be comfortable with policy gradient methods, KL divergence, and the standard GRPO/OPD training loop.

Authors’ abstract

Multi-turn agents trained with reinforcement learning (RL) receive a single scalar reward per trajectory, which motivates self on-policy distillation (OPD) to supply dense token-level supervision from a self-teacher with privileged task skills, letting a skill-free student internalize them. This recipe, however, is undermined by two findings in agentic tasks: privileged information alone does not always make a teacher reliable, and the benefit of teacher supervision is stage-dependent. We therefore propose RetireOPD (Self-Retiring On-Policy Distillation), which first optimizes a decoupled, skill-conditioned teacher with environment rewards and then trains a skill-free student jointly with RL and OPD. Rather than following a predefined distillation schedule, RetireOPD adopts Adaptive Retirement: the student drops the teacher on its own once their discrepancy stops shrinking and it reaches a target fraction of the teacher's success rate, after which training proceeds with RL alone. Across Qwen2.5 models from 1.5B to 7B, RetireOPD improves ALFWorld success rate over RL baseline by 14.1% to 18.8% and WebShop accuracy by 11.8% to 19.0%, and surpasses its own skill-conditioned teacher in every setting.