ReSAIL: Mitigating Collapse in Iterative Agent Self-Distillation

AI summary

Overview

Research area: Machine learning for large language model (LLM) agents, specifically iterative self-distillation with privileged information (PI) and recursive self-improvement.

Technical level: Advanced. The paper assumes familiarity with KL divergence, Jensen-Shannon divergence, teacher-student distillation, and LLM agent training loops.

Scope: The paper identifies and mitigates a performance collapse that occurs when LLM agents repeatedly self-distill across multiple deployment cycles, introducing a plug-in method called ReSAIL that combines selective step-level distillation with retention of the teacher's PI-conditioned behavior.

What This Paper Is About

Iterative self-distillation lets an LLM agent collect experience during deployment, learn from it offline using training-only privileged information (such as a summary of a completed reference trajectory), and then redeploy the improved model as the teacher for the next cycle. In theory this is a path toward recursive self-improvement. In practice, the authors find that repeated cycles of two existing methods, SDPO and OEL, cause deployment success rates to collapse and the teacher's privileged-information-conditioned competence to decay, so the quality of supervision degrades with each cycle. ReSAIL is a plug-in augmentation that addresses both problems at once: it concentrates distillation on the interaction steps where privileged information most changes the teacher's predictions, and it prevents the student from losing the PI-conditioned behavior it will need when it becomes the next teacher.

Key Contributions

1. First evidence that collapse can be mitigated. The authors report the first evidence that performance collapse in iterative agent self-distillation over deployment trajectories can be effectively mitigated by a more robust learning mechanism, rather than by abandoning the self-distillation setup.

2. The ReSAIL method. They propose ReSAIL (Retentive and Selective Augmentation for Iterative Self-Distillation), a plug-in augmentation combining Trajectory-Balanced Selective Distillation (TBSD) with Privileged Retention (PR). TBSD itself consists of Sensitivity-Guided Selection (SGS) and Trajectory Loss Balancing (TLB).

3. Sustained gains across models and benchmarks. On ALFWorld and TextCraft, with Qwen3-4B and Qwen3-8B, ReSAIL is reported to improve final-cycle success rates by an average of 22.5 percentage points over the corresponding SDPO and OEL baselines, with gains sustained across three deployment cycles.

4. PI sensitivity as a general selection criterion. The authors show that Sensitivity-Guided Selection also works as an offline data filter for multimodal GUI agents, improving action prediction accuracy on AITZ.

Main Findings

• Iterative self-distillation collapses without intervention. Across the six model–benchmark settings examined (two model sizes x three benchmarks/splits), OEL finishes below its own first-cycle performance in all six settings, and SDPO does so in five.

• PI-conditioned competence also decays. With evaluation tasks and privileged information held fixed, both SDPO and OEL show declining mean PI-conditioned task success over successive cycles (Figure 1b).

• ReSAIL raises final-cycle performance. After three deployment cycles, SDPO+ReSAIL and OEL+ReSAIL outperform all unaugmented baselines on ALFWorld ID, ALFWorld OOD, and TextCraft at both Qwen3-4B and Qwen3-8B. Relative to their parent methods, ReSAIL improves final-cycle success by 7.0 to 26.1 percentage points for SDPO and by 20.3 to 37.0 points for OEL.

• Gains hold against the strongest alternative baseline. On ALFWorld OOD with Qwen3-8B, SDPO+ReSAIL and OEL+ReSAIL reach 75.3% and 78.4% success, compared with 63.8% for RFT, 62.2% for GRPO, and 46.6% for EPD.

• The advantage grows with each cycle. Averaged equally across the six model–benchmark settings, the margin over the corresponding parent method grows from 6.8 to 18.3 percentage points for SDPO and from 5.0 to 26.6 points for OEL between the first and third deployment cycles.

• Selection and retention act on different timescales. In component ablations with OEL and Qwen3-4B on ALFWorld, SGS alone gives the strongest first-cycle performance, improving over OEL by 11.5 and 14.4 percentage points on ID and OOD. Adding TLB improves third-cycle success over selection alone by 6.0 and 7.8 points, but that configuration still finishes below its own first-cycle performance. Adding PR reverses the cross-cycle trajectory: the full method exceeds TBSD by 17.7 and 21.6 points at Cycle 3, and outperforms PR alone by 9.9 and 13.3 points.

• What matters is which steps are selected, not how many. At the same selection ratio ρ = 0.05 with TLB and PR fixed, Top (SGS) selection improves first-cycle success over Random by 12.5 and 11.5 percentage points on ID and OOD respectively (62.2% vs 49.7% on ID; 57.3% vs 45.8% on OOD), while Bottom selection performs worse than Random on both splits (40.1% ID, 40.4% OOD).

• Privileged-view retention beats ordinary-view retention. At Cycle 3 with SGS and TLB fixed, no retention yields 56.5% ID / 45.6% OOD, ordinary-view retention yields 72.9% / 65.9%, and privileged-view retention yields the highest results at 74.2% / 67.2%.

• Retention preserves PI-conditioned competence, not just deployment success. Without retention, PI-conditioned success falls from 61.5% at Cycle 1 to 44.8% at Cycle 3. With PR it instead rises from 58.9% to 70.3%. Ordinary-view retention reaches 62.2% at Cycle 3, leaving an 8.1-percentage-point advantage for PR under PI versus only a 1.3-point advantage without PI.

• Moderate selection is better than using all steps. With TLB and PR enabled and λ = 0.5, selecting the top 1%, 5%, or 25% of steps outperforms using all steps (ρ = 1) on both ALFWorld splits; the default ρ = 0.05 improves over ρ = 1 by 5.4 and 7.0 percentage points on ID and OOD. For retention weight at ρ = 0.05, third-cycle success increases as λ rises from 0 to 0.5, while λ = 1 gives lower performance.

• Selection reduces training cost, the full method costs more. For Qwen3-4B on ALFWorld, SGS reduces student optimization cost by approximately 74% and total training-loop cost from 4.33 to 4.08 GPU-hours (about 6%), while improving first-cycle success. Adding TLB raises total cost slightly to 4.17 GPU-hours. The full ReSAIL configuration costs 5.32 GPU-hours, or 1.23 times that of OEL.

• PI sensitivity works as an offline filter for multimodal agents. On AITZ, retaining the fixed global top 80% of interaction steps gives OEL+SGS a final step-level action accuracy of 65.95% versus 64.89% for OEL, an improvement of 1.06 percentage points under the same update budget.

Methodology in Plain English

Each training cycle starts with a policy and a dataset of trajectories collected by that policy (or supplied offline), with no further environment interaction during training. Both a student and a frozen teacher are initialized from the same starting policy, and every policy has two input views that share parameters: an ordinary view that sees only the task and interaction history, and a privileged view that additionally sees a summary extracted from a completed trajectory. The base objective trains the ordinary-view student to match the privileged-view teacher's token-level output distribution via student-to-teacher KL divergence, averaged uniformly over all interaction steps.

ReSAIL adds two pieces on top of that base objective. First, Sensitivity-Guided Selection scores every interaction step by how much the frozen teacher's predictions change when privileged information is added, measured as the Jensen-Shannon divergence between the teacher's ordinary and privileged output distributions at that step. Only the top ρ fraction of steps across the whole update batch are used for distillation, and Trajectory Loss Balancing averages the selected losses within each trajectory before summing, so trajectories with many selected steps do not dominate those with few. Second, Privileged Retention adds a KL term that pulls the student's privileged-view output distributions toward the frozen teacher's privileged-view distributions at every step, including steps not selected for distillation, using the same trajectory-level averaging. The final loss is the selective distillation loss plus λ times the retention loss. Sensitivity scores and the selected set are recomputed each update and held fixed during backpropagation.

Why This Matters

Impact on research. The paper reframes iterative self-distillation as a two-sided problem: you must improve the deployed policy while also preserving the PI-conditioned competence that policy will need when it becomes the next teacher. The finding that both SDPO and OEL degrade in PI-conditioned success while task setups and privileged information are held fixed gives a concrete diagnostic for why recursive self-improvement loops stall. The comparison to prior collapse-mitigation work is also notable: Chen et al. (2026a) shift to off-policy context-distillation using successful trajectories generated by PI-conditioned teachers, which requires additional environment interaction and is inapplicable when the deployment environment cannot be revisited during training. ReSAIL targets the more restrictive setting where only already-collected deployment trajectories are available.

Real-world applications (as suggested by the benchmarks used):

• Embodied household task planning, where agents must sequence actions in environments like ALFWorld and where repeated deployment without collapse matters.
• Compositional crafting and multi-step recipe-style tasks such as TextCraft, where long action sequences benefit from selective supervision.
• Android GUI navigation, where offline filtering of pre-collected multimodal trajectories improved step-level action prediction accuracy on AITZ.
• General deployed agent pipelines where the training environment is not available for re-collection, so learning must come from logs already gathered during deployment.

Industry relevance. Deployed LLM agents accumulate interaction logs continuously, and reusing those logs for improvement without revisiting the environment is operationally attractive. The paper's cost accounting (4.33 to 4.08 GPU-hours with selection; 5.32 GPU-hours for the full method on eight H800 GPUs) gives practitioners a concrete sense of the tradeoff between cheaper selective training and the more expensive full retention objective. The described stack (slime, Megatron-LM, SGLang, PyTorch 2.11.0, CUDA 12.9, BF16 parameters with FP32 optimizer states) also makes the setup reproducible in standard training infrastructure.

Future Directions

• Understand why PI-conditioned competence decays without retention. The paper measures the decay under fixed tasks and fixed PI but does not report a mechanistic explanation, so the cause remains an open question.

• Extend beyond three cycles. All text results cover three deployment cycles of 30 updates each; whether ReSAIL's non-decreasing trajectory continues over longer deployment horizons is not reported.

• Remove the need for a frozen teacher or for extra environment interaction. The full method costs 1.23 times OEL in training-loop time because retention is applied at every step in the batch; cheaper ways to preserve PI-conditioned behavior could be explored.

• Broaden the evaluation of PI sensitivity as a data filter. AITZ results come from a single cycle of 100 updates with the top 80% of steps retained; whether the SGS criterion generalizes to other multimodal agents, larger retention ratios, or other offline datasets is not reported.

Target Audience

Researchers and engineers working on LLM agent training, self-distillation, and recursive self-improvement, particularly those who train agents offline from logged deployment trajectories and need repeated improvement across cycles without access to the original environment. The paper is also useful for practitioners applying teacher-student distillation with privileged information, since the ablation tables isolate which component helps at which timescale.

Authors’ abstract

Iterative self-distillation enables LLM agents to learn from successive deployments, offering a path toward recursive self-improvement (RSI). Yet our experiments with existing methods reveal a collapse in deployment performance across cycles, while task performance with privileged information (PI) also declines. We address this collapse by prioritizing informative interaction steps for distillation and preserving PI-conditioned behavior as the student becomes the next teacher. We introduce Retentive and Selective Augmentation for Iterative Self-Distillation (ReSAIL), a plug-in augmentation for iterative PI-based self-distillation. ReSAIL selects interaction steps where PI most strongly changes the teacher's predictions and balances the resulting distillation losses across trajectories. It also regularizes the student's PI-conditioned output distributions toward those of the frozen teacher at selected and unselected steps to preserve PI-conditioned behavior for supervision in the next cycle. On ALFWorld and TextCraft, ReSAIL sustains substantial gains across model scales over three cycles, with an average absolute gain of 22.5% in final-cycle success rates when added to self-distillation baselines. Sensitivity-guided selection of offline data also improves action prediction accuracy for multimodal GUI agents on AITZ. These findings provide the first evidence that a more robust learning mechanism can effectively mitigate performance collapse in iterative agent self-distillation over deployment trajectories.