On-Policy Self-Distillation with Teacher Evolution

AI summary

Overview

Research area: Large language model post-training, specifically on-policy self-distillation (OPSD) and reinforcement learning with verifiable rewards (RLVR).

Technical level: Advanced — assumes familiarity with policy-gradient methods (GRPO/PPO-style clipping), KL/Jensen–Shannon divergence distillation objectives, importance-sampling ratios, and EMA teacher updates.

Scope: The paper diagnoses why on-policy self-distillation stops improving during training and proposes Teacher-Evolution Self-Distillation (TESD), which trains the self-teacher with its own verifiable rewards and down-weights samples where teacher guidance is unreliable, evaluated on Qwen3-1.7B/4B/8B and Olmo3-7B-Instruct across six training tasks and eight holdout tasks.

What This Paper Is About

On-policy self-distillation gives a model dense, token-level supervision by using itself — prompted with a privileged context such as a correct solution — as its own teacher. The authors show this only works if the self-teacher is actually good at teaching, and in practice it is not: it assigns low confidence to some correct rollouts and high confidence to many incorrect ones, and its teaching ability declines rather than improves over training. TESD fixes this by letting the self-teacher generate its own rollouts, scoring them with an environmental reward, and using those scores both to train the teacher and to weight the student's distillation loss.

Key Contributions

1. A diagnosis of the OPSD bottleneck. Through correlation analysis on the Physics subset of SciKnowEval with Qwen3-8B, the authors show the self-teacher cannot reliably distinguish correct from incorrect student rollouts, is biased toward rollouts that resemble the privileged context (measured by normalized edit distance), and that its teaching ability trends downward with instability during training.

2. Teacher-oriented optimization. The self-teacher samples its own rollouts and is optimized with a GRPO-style clipped objective using a reward that combines binary task correctness with a normalized edit-distance (independence) term, so the teacher learns to produce correct solutions that are not mere copies of the privileged context.

3. Adaptive student-oriented optimization. The mean teacher reward per query is converted into a batch-normalized softmax weight (scaled by batch size, clipped to [0.05, 5.0]) that scales each query's distillation loss, suppressing gradient updates from queries where teacher guidance is unreliable.

4. Empirical validation across models and task types. TESD is compared against SDPO (a representative OPSD method) and a strengthened GRPO baseline on four science QA subsets, a tool-use task, and LiveCodeBench v6, plus a catastrophic-forgetting evaluation on eight holdout benchmarks.

Main Findings

• OPSD underperforms sequence-level RLVR at convergence. On SciKnowEval + ToolUse (mean over Chemistry, Physics, Biology, Material, Tooluse), SDPO reaches 69.66% on Qwen3-4B versus 73.40% for GRPO; on Qwen3-8B, 70.21% versus 74.94%; on Olmo3-7B, 66.84% versus 68.42%. Naively summing SDPO and GRPO losses lands between the two (70.86% on Qwen3-4B, 72.27% on Qwen3-8B, 67.00% on Olmo3-7B).

• TESD leads on all three base models in the no-rich-feedback setting. Mean scores: 74.32% (Qwen3-4B), 75.61% (Qwen3-8B), 69.75% (Olmo3-7B). On Qwen3-8B, TESD reaches Physics 80.55 and Tooluse 70.02 versus GRPO's 80.47 and 68.29.

• TESD also leads on LiveCodeBench v6 with rich feedback, except at the smallest scale. Means: TESD 56.45, SDPO+GRPO 56.07, SDPO 54.90, GRPO 53.98. SDPO+GRPO beats TESD only on Qwen3-1.7B (46.38 vs 45.98); TESD leads on Qwen3-4B (60.07 vs 59.88) and Qwen3-8B (63.31 vs 61.95).

• Less catastrophic forgetting. Averaged over eight holdout tasks plus the Physics training task with Qwen3-8B: base model 81.41, TESD 81.77, GRPO 80.66, SDPO 79.42. TESD is the only trained method that exceeds the base model on the average; it improves Physics from 58.44 (base) to 79.24 while raising HumanEval from 64.02 to 67.68 and TruthfulQA from 86.05 to 88.98.

• The edit-distance reward is essential to teacher training. Ablation on Qwen3-4B (Physics/Biology/Material mean): SDPO baseline 67.06, adding a sequence-level student loss 70.61, optimizing the teacher with only correctness via privileged context 69.35, and adding the edit-distance reward 73.96. Training the teacher on correctness alone lets it copy the privileged context, which hurts guidance.

• Weighting must preserve loss scale. On Qwen3-8B (Physics/Biology/Material mean): unweighted baseline 73.76, direct mean-reward weighting 69.53, softmax normalization 70.56, softmax plus scaling/clipping 75.49 — raw weighting degrades performance.

• TESD scales better with more rollouts. On the Biology subset with Qwen3-4B over 10 epochs, SDPO scores 54.62 / 55.25 / 55.12 at N = 16 / 32 / 48, while TESD scores 54.87 / 56.63 / 59.62.

• Behavioral differences. TESD keeps higher accuracy standard deviation than SDPO and produces shorter responses than GRPO, avoiding GRPO's lengthening trend and SDPO's early collapse in length. Its teaching ability rises and is more stable, and its distillation loss stays higher in mid/late training, meaning the teacher keeps providing informative guidance rather than the student overfitting to a weak early teacher. Token-level credit heatmaps show TESD concentrates on query-relevant keywords (e.g., "ionization", "gas", "E_{Ar}") instead of nearly every token as SDPO does.

• Adaptive weights behave as intended. Early in training weights drop unreliable samples to roughly 0.5 of the loss; as the teacher improves, weights shift toward 1.0, showing the two mechanisms are complementary.

• Not reported: variance or confidence intervals for the headline benchmark numbers; the paper reports the maximum mean@16 over training. A training-time analysis is referenced as being in the supplementary material, which is not included in the supplied content.

Methodology in Plain English

The setup starts from a baseline called SDPO: a model answers a question, one of its correct answers is pasted back in as a "hint," and the model-with-hint serves as a teacher whose per-token probability distribution the plain model is pushed to match. The problem is that the hinted model is no smarter than the student — it just has the answer — so its guidance is sometimes wrong.

TESD changes three things. First, the teacher gets to answer on its own, with the hint attached, producing its own rollouts. Second, those rollouts are graded by an automatic checker (does the answer match, does the code pass tests, is the tool call right) plus a term rewarding answers that are dissimilar to the hint — this is meant to stop the teacher from simply parroting the hint and to force it to reason across varied trajectories. Third, the teacher is trained with the same clipped policy-gradient recipe used for RL (group-relative advantages, importance-sampling ratios, asymmetric clipping), while the student's distillation loss on each question is multiplied by a weight derived from how well the teacher scored on that question — normalized across the batch with a softmax and scaled by batch size so the overall loss magnitude is unchanged. The student's loss and the teacher's loss are simply added per batch.

Computationally, each method uses 16 rollouts per prompt; TESD splits them 8 for the student and 8 for the teacher, whereas the baselines use all 16 for the student. Experiments use the verl framework with FSDP on a single 8×H20 node, 32 prompts per step, 30 epochs on science QA, 15 on ToolUse, and 40 steps on LCBv6, with λ_edit = 1.0 and an EMA teacher update rate of 0.01 (vs. 0.05 for SDPO).

Why This Matters

Impact on research: The paper reframes self-distillation failure as a supervision quality problem rather than a data or loss-shaping problem. Most prior fixes add more privileged context, degrade OPSD into a GRPO module, or add entropy-based routing. TESD instead treats the model's teacher role as a separately trainable role with its own reward signal, which is a structurally different and reusable idea for any dual-role training scheme.

Real-world applications:

• Post-training reasoning models for science and math question answering, where verifiable answers are cheap but token-level credit assignment is hard.
• Coding assistants, where unit-test outcomes and runtime errors provide dense privileged context and TESD showed gains on LiveCodeBench v6.
• Tool-calling and agentic systems, where success is checkable against an API specification (ToolAlpaca-style tasks).
• Continual or domain-specific fine-tuning where forgetting base capabilities is a concern — TESD was the only trained method to beat the base model on the nine-task holdout average.

Industry relevance: Both a lab-scale research group (State Key Lab. of AI Safety, ICT, Chinese Academy of Sciences; University of Chinese Academy of Sciences) and an industrial lab (Meituan) co-authored the work, and the method is built on standard open infrastructure (verl, FSDP, GRPO-style clipping), so it is directly pluggable into existing RL post-training pipelines at the cost of splitting rollout budget between two roles.

Future Directions

• Optimization geometry of TESD. The authors explicitly state they will study this next, which is open given that student and teacher losses are simply summed and the teacher is also EMA-regularized at 0.01.
• Extension to multimodal reasoning. Explicitly named as future work, raising the question of what "privileged context" and "edit distance" mean for images or audio.
• Reward design robustness. The edit-distance term is a proxy for independence with a fixed λ_edit = 1.0 across all experiments; whether it invites reward hacking (e.g., producing correct-but-verbose answers that avoid the hint) is untested, and the main text writes the reward as a sum while the appendix describes it as task correctness minus the independence term — a discrepancy worth resolving.
• Compute and rollout trade-offs. All methods use 16 rollouts, but TESD gives the student only 8. Whether reallocating budget toward the teacher remains favorable at larger rollout counts (the N = 48 result suggests yes on one subset) and how wall-clock cost compares is deferred to supplementary material.

Target Audience

Researchers and engineers working on LLM post-training, especially those building RLVR or distillation pipelines and encountering the plateau where token-level self-distillation stops paying off. It is also useful for readers interested in self-training and dual-role optimization, and for practitioners who need concrete hyperparameter details — the appendix provides full settings for GRPO, SDPO, and TESD, dataset splits (e.g., Chemistry 1890/210, LCBv6 131/131), and evaluation protocols. Readers without a background in policy-gradient methods will find the method section demanding.

Authors’ abstract

On-Policy Self-Distillation (OPSD), which aligns a student model's token-level distribution with that of its self-teacher prompted by privileged context, has emerged as a promising direction to mitigate reward sparsity in sequence-level post-training paradigms. However, OPSD often fails to achieve continuous performance gains during training. In this paper, we find that the model's limited self-teaching capability underlies this phenomenon by generating unreliable guidance on certain samples. To address this problem, we propose Teacher-Evolution Self-Distillation (TESD), which enables the self-teacher to generate rollouts and receive environmental feedback based on task correctness and context independence. First, using these rewards as training objectives, we perform teacher-oriented optimization to continuously enhance the model's self-teaching capability during distillation. Second, by estimating teacher guidance reliability from these rewards, we apply adaptive student-oriented optimization to down-weight samples with unreliable guidance. Extensive experiments across multiple datasets and model scales show that TESD consistently outperforms representative token-level and sequence-level methods, highlighting the necessity of improving supervision quality in OPSD.