Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement

AI summary

Overview

Research area: Large language model post-training, specifically on-policy reinforcement learning and knowledge distillation for reasoning models.

Technical level: Advanced. The paper assumes familiarity with KL divergence, policy gradient advantages, GRPO-style group-normalized rewards, and token-level logits.

Scope: A diagnostic study of what actually drives improvement in on-policy distillation, followed by a supervision-free training method (On-Policy Self-Adaptation, OPSA) derived from those findings and evaluated across five model checkpoints and five benchmarks.

What This Paper Is About

On-policy distillation (OPD) trains a student model on its own generated trajectories while a stronger teacher scores each token, supposedly transferring the teacher's knowledge through dense token-level advantages. The paper asks whether this supervision is actually reliable, given that the teacher is scoring trajectories it would never have generated itself, and then asks where the student's improvement really comes from if the teacher's signal turns out to be unreliable. The answer leads the authors to propose a training method that removes the teacher entirely and relies only on the student's own token entropy.

Key Contributions

1. A quantitative characterization of teacher noise in OPD. The authors define supervision as noisy when the sign of the teacher-provided advantage on the token(s) of a verifiable final answer disagrees with the verifier (negative advantage on a correct answer, or positive advantage on an incorrect one), and measure how often this occurs for teachers of different scales.

2. A demonstration that the student is insensitive to that noise. A controlled filtering experiment partitions trajectories into those containing noisy signals and those without; training on all trajectories, only noisy ones, or only clean ones all converge to comparable performance after a similar number of gradient steps.

3. An identification of what actually drives OPD gains. Learning concentrates on the small fraction of low log-probability tokens sampled by the student, and replacing all teacher advantages with a single fixed negative value reproduces OPD-level performance — implying the teacher is not the source of improvement.

4. On-Policy Self-Adaptation (OPSA), a supervision-free token-level RL method that updates only the lowest-log-probability 20% of sampled tokens and scales the magnitude of a negative advantage with token entropy. It requires no teacher, no verifiable rewards, and no hints.

Main Findings

• Teacher supervision is highly noisy and gets worse with scale. Using Qwen3-1.7B as the student (thinking mode disabled) and Qwen3-4B / 30B-A3B / 235B-A22B-Instruct as teachers, over 500 questions from DAPO-17k: for the 4B teacher, 20.4% of correct trajectories receive negative advantages on their answer tokens and 40.8% of incorrect trajectories receive positive advantages, an overall noise rate of 30.6%. This rises to 34.7% for the 30B-A3B teacher and 50.6% for the 235B-A22B teacher.

• The largest teacher is almost uniformly negative. With the 235B-A22B teacher, 97.8% of answer tokens enclosed in boxed{} receive negative advantages even when the answer is correct, and 96.6% receive negative advantages when the answer is incorrect — an overall noise rate of approximately 50%. The authors attribute this to growing distributional mismatch between student and teacher policies, making student trajectories increasingly off-policy from the teacher's perspective.

• Students improve just as much on noisy data as on clean data. Restricting training exclusively to trajectories containing noisy advantages, or exclusively to those without them, produces convergence comparable to standard OPD on all trajectories after a similar number of gradient steps, as measured by Avg@4 on AIME24.

• Most token advantages are essentially zero. During OPD with a Qwen3-4B-Instruct teacher, 29.2% of tokens have exactly zero advantage and 51.7% have advantage magnitude below 10⁻⁴. These near-zero advantages are concentrated among tokens the student assigns high log probability, where the teacher conditioned on the same prefix also assigns similarly high probability.

• High-log-probability tokens carry no useful learning signal. Training on varying proportions of the student's top-log-probability tokens yields no noticeable improvement in AIME24 performance, and the performance is essentially unchanged even when original advantages are replaced with random values sampled evenly from [−1, 1].

• A single fixed negative advantage matches teacher supervision. Restricting training to the 20% of tokens with the lowest student log probability and using teacher-free fixed advantages of −0.5 (negative) or +0.2 (positive): the fixed negative advantage produces steady Avg@4 improvement on AIME24 and response length that grows to approximately 12K tokens. The fixed positive advantage causes policy collapse — within the first 40 training steps response length falls to nearly zero while the gradient norm explodes, after which outputs are largely garbled.

• Entropy determines how much negative signal a token should get. With a dynamic advantage, a positive entropy correlation (δ = 1) reaches 50.0% Avg@4 on AIME24 versus standard OPD at 35.13%; the reversed correlation (δ = −1) becomes unstable between steps 350 and 450 and performs slightly worse than the fixed-negative baseline.

• OPSA improves reasoning substantially on Qwen3-1.7B. Avg@32 rises from 13.44 to 48.85 on AIME24 (+35.41 points, 263.5% relative), 9.69 to 35.31 on AIME25 (+25.62, 264.4%), and 5.73 to 23.33 on HMMT25 (+17.60, 307.2%). Pass@32 rises from 40.00 to 80.00 on AIME24, 30.00 to 66.67 on AIME25, and 23.33 to 50.00 on HMMT25.

• OPSA beats all supervision-free and supervised baselines on Qwen3-1.7B. Averaged across the three math benchmarks, OPSA reaches 35.83 Avg@32 and 65.56 Pass@32, surpassing the best baseline (GRPO at 24.79 Avg@32, OPSD at 56.67 Pass@32) by 11.04 and 8.89 points respectively. On AIME24 it exceeds OPD by 16.77 points in Avg@32 (48.85 vs 32.08).

• OPSA generalizes across model families and scales. Gains hold for Qwen3-4B (+38.75 Avg@32 on AIME24), Qwen3.5-9B (+11.46 on AIME24, +22.92 on HMMT25), and both Olmo3-7B Instruct (+7.40 on AIME24) and Olmo3-7B Think (+5.41 on AIME24), with Pass@32 relative gains of 47% to 122% across the Qwen3 series, including +10.00 points on HMMT25 for Olmo3-7B Think.

• Out-of-domain gains are small but consistent. MBPP+ (code) Avg@32 improves by +1.20 for Qwen3-1.7B, +1.42 for Qwen3-4B, +1.94 for Qwen3.5-9B, +1.82 for Olmo3-7B Instruct, and +1.78 for Olmo3-7B Think. GPQA-Diamond improves by +4.48, +2.83, +3.17, +1.52, and +2.07 respectively.

• TTRL degrades Pass@k. Although TTRL needs no external supervision, its self-consistency-based training sharpens the policy around a local optimum; on Qwen3-1.7B it reaches 11.81 Avg@32 / 27.78 Pass@32 averaged across benchmarks, below the untrained base model's Pass@32 of 31.11.

• OPSD needs a mode mismatch to work. The authors find OPSD yields meaningful gains only when thinking mode is disabled for the student but enabled for the teacher, creating a substantial distributional mismatch at the first token position.

• OPSA preserves diversity. Jaccard distance measured on AIME24 with 32 sampled responses per problem shows the gap between base and OPSA-trained models narrowing and approaching zero as response length grows.

• Fork tokens are the mechanism. Masking fork positions whose head-token sets contain reflective words largely eliminates the increases in both response length and accuracy, and response length collapses at approximately 300 training steps. The OPSA-trained model produces substantially more reflective expressions and shows a positive correlation between response length and accuracy.

• The token-selection ratio is not very sensitive. Training on the lowest 10% of tokens by log probability performs substantially worse (those tokens are almost entirely tail tokens outside the top-1 prediction, and over-sharpening causes a pronounced entropy decrease), while 20%, 30%, and 40% settings all raise Avg@4 above 45.

Methodology in Plain English

The authors first audit the teacher. They take Qwen3-1.7B as a student with thinking disabled, sample one correct and one incorrect response per question from 500 DAPO-17k questions, and have teachers of three sizes score the tokens of the final answer inside boxed{}. Because a verifier can check whether the answer is right or wrong, they can flag every case where the teacher's advantage sign contradicts the answer's correctness.

Next they run a filtering experiment: train on all trajectories, on only trajectory groups containing noise, and on only those without noise, and compare learning curves.

Then they dissect why OPD works at all. They derive the logit-level gradient of the OPD loss and note it vanishes when the advantage is near zero or when the student's probability on the sampled token approaches one. They visualize the advantage distribution, check which log-probability ranges the near-zero advantages fall in, and run controlled experiments training only on top-log-probability tokens with either real or random advantages. They then replace all teacher advantages with fixed constants (−0.5 or +0.2), always restricting updates to the lowest-log-probability 20% of tokens.

Finally they ask how large the negative signal should be per token, using a reparameterized advantage whose magnitude correlates with a token's entropy relative to the min/max entropy over the lowest-20%-logp positions in that rollout, controlled by a parameter δ. Setting δ = 1 (higher entropy gets stronger negative signal) wins, and combining it with a fixed base advantage of −3/4 yields OPSA: the update touches only the lowest-log-probability 20% of sampled tokens, with advantage A_i^dyn = −1/2 − (H_i − H_min) / (2(H_max − H_min)). No teacher, no ground-truth reward, no hint is used. Models are trained on DAPO-17k question text only, implemented in the slime framework on 8 NVIDIA H100 or H200 GPUs, and evaluated on AIME24, AIME25, HMMT25, MBPP+, and GPQA-Diamond.

Why This Matters

The paper reframes a widely used training paradigm: if OPD's benefit comes mostly from suppressing the student's own low-probability tail tokens rather than from matching the teacher, then a large class of teacher-dependent pipelines is paying a real cost — teacher forward passes, shared vocabularies, white-box logit access — for something a much simpler procedure reproduces. The result also cuts against the intuitive assumption that a bigger teacher gives cleaner supervision: the noise rate roughly doubles from the 4B teacher to the 235B-A22B teacher.

Real-world applications:

• Efficient post-training of reasoning models. Removing the teacher eliminates the extra forward passes needed to compute advantages, which the paper reports introduces negligible computational overhead relative to OPD and substantially reduces training time.
• Post-training where labels or verifiers are unavailable. OPSA needs no ground-truth answers, no reward model, and no reference solutions, only the model's own logits and entropy.
• Environments with restrictive model licensing or deployment. Because no white-box teacher logits or shared vocabulary are required, the method can be applied when a suitable teacher cannot be accessed.
• Code and general Q&A domains. Improvements carry over to MBPP+ and GPQA-Diamond without training on those tasks.

Industry relevance: Practitioners running RL post-training at scale can drop a large teacher model from the loop, avoid the engineering constraint of vocabulary alignment, and get larger reasoning gains — the reported Avg@32 on AIME24 for Qwen3-1.7B more than triples relative to the base model.

Future Directions

• Why do off-policy teachers become so uniformly negative at scale? The paper attributes the trend to growing distributional mismatch but does not resolve whether a better-matched or fine-tuned teacher could recover useful dense supervision.
• Can the entropy-adaptive signal be extended beyond the lowest-log-probability 20% of tokens? The ablation shows 20%, 30%, and 40% all exceed 45 Avg@4, leaving the upper boundary and the interaction with response length unexplored.
• **How does OPSA interact with verifiable

Authors’ abstract

On-policy distillation (OPD) offers dense token-level supervision as an alternative to the sparse outcome-level advantages of reinforcement learning with verifiable rewards (RLVR). However, the teacher scores student-generated trajectories that are inherently off-policy for it, so the reliability of its supervision, and hence the source of the student's improvement, remains unclear. We quantitatively analyze teacher supervision during OPD training and find substantial noise whose prevalence increases with teacher scale. Surprisingly, the student policy is insensitive to such noise, converging to comparable performance regardless of whether noisy supervision is retained or removed. Does OPD distill at all? By analyzing what drives its gains, we find that learning concentrates on low log-probability tokens, and using a single fixed negative advantage matches the performance of teacher-provided ones. This suggests that OPD works largely by suppressing low log-probability tokens, which requires no teacher. These findings motivate On-Policy Self-Adaptation (OPSA), a supervision-free method using entropy-adaptive negative advantages. It assigns stronger learning signals to high-entropy positions, suppressing tail tokens, and evenly redistributing probability mass among head tokens. Compared with the base \texttt{Qwen3-1.7B}, OPSA improves Avg@32 by 35.41 points on AIME24, corresponding to a 263\% relative gain, and more than doubles Pass@32 across all three benchmarks. It also outperforms OPD by 16.77 points in Avg@32 on AIME24. Extensive experiments and analyses across model families and tasks further demonstrate its effectiveness and generalizability.