One Symptom, Three Levers: A Critical Review of On-Policy Self-Distillation

AI summary

Overview

Research area: Machine learning — post-training of large language models, specifically on-policy self-distillation (OPSD) for mathematical reasoning.

Technical level: Intermediate to Advanced. The paper assumes familiarity with reinforcement learning from verifiable rewards (RLVR), GRPO, KL divergences, and token-level distillation, though it explains its mechanism from notation upward.

Scope: A critical review — reporting no new experiments — that maps the OPSD literature up to August 2026, treats diversity collapse as a shared symptom, and organizes the field around three controlling levers. It restricts itself to mathematical reasoning, where the method was introduced and its failure modes are best documented, and deliberately excludes the field's two largest branches (multimodal learning and tool-using agents) as well as external-teacher distillation.

What This Paper Is About

On-Policy Self-Distillation trains a language model on its own generations while the model itself — conditioned on privileged information such as a reference solution, a plan, or environment feedback — acts as the teacher. This removes the need for a second, larger teacher model, but the same asymmetry that generates the learning signal also biases it, and the field's dominant failure mode is now collapse: the progressive narrowing of the set of reasoning paths the model can produce. The paper's goal is not to propose a new method but to build a shared vocabulary across papers that name the same phenomena differently, and to draw a clear line between what is settled and what remains disputed.

Key Contributions

1. A unifying diagnosis of collapse as a symptom, not a method-specific bug. The review separates two families of causes: those that predate OPSD (entropy erosion in the RL gradient, and model collapse in any self-training loop) and one specific to OPSD (the pointwise mutual information signal produced when a teacher is conditioned on the solution).

2. A three-lever framework for controlling the dense signal. Lever A is signal geometry (which divergence, how dense), Lever B is the nature of the privileged information, and Lever C is loop stability (teacher dynamics and the decay of guidance). The paper states explicitly that the levers are not independent, and that privileged information bears on all three.

3. A shared vocabulary for phenomena named differently across papers. Examples include suppression of epistemic verbalization, rich-get-richer dynamics, and fork rate — three descriptions of the same narrowing mechanism observed from different angles.

4. An evaluation grid for reading claims in this literature. The paper specifies minimum standards: averages over several samples and seeds (avg@k) with confidence intervals, pass@k and G-Pass@k to expose lost diversity, and four protocol controls (null or random privileged information, equalized compute, contamination tests, and replication outside the Qwen family).

Main Findings

• OPSD matches or exceeds GRPO on mathematical reasoning at far fewer generated tokens. Where GRPO samples 8 rollouts of up to 16k tokens per problem, OPSD uses a single generation capped at 1,024 tokens. The authors report a rise from 36.7 to 43.9 on AIME25 by step 50 with the forward KL, and best-over-checkpoint scores for Qwen3-1.7B of 57.2% on AIME24, 43.9% on AIME25, and 29.2% on HMMT25 — with the end-of-training AIME25 value at 41.1%.

• The token savings are not a compute saving. An OPSD optimization step requires two forward passes and one backward pass, against a single backward pass for GRPO. At equalized budget, one OPSD step costs roughly twice a GRPO step: 20.6 s against 11.2 s on Qwen3-8B with 8×H100. A run on Qwen3-1.7B completes in about fifteen minutes on 4 H100 GPUs. The advantage is faster convergence in number of steps, not a lower unit cost.

• Density is fragile, not simply beneficial. "Denser ≠ Better" finds that distilling the full chain of thought helps on short-trace tasks such as tool use but degrades mathematics and science, whose long traces surface artefacts. In continual learning, SDPO specializes quickly and then collapses, whereas GRPO-style sparse-reward RL retains more.

• Distillation helps mainly on erroneous trajectories. By comparing the distillation gradient to an ideal per-token gradient, "Unmasking OPD" measures an alignment score that is positive when the teacher pushes toward success, null when the signal is spent on style, and negative when it pushes toward failure. The best teacher also depends on student capacity: on a 0.6B model, self-distillation is two to three times better than an external teacher, but the advantage does not carry over to a 1.7B model.

• Entropy is not a valid proxy for diversity. On Qwen3-8B, self-distillation raises pass@1 from 71.9 to 73.4 while pass@16 falls from 83.6 to 78.5. The same model displays token entropy higher than a GRPO-trained model even though its functional diversity is lower.

• The OPSD-specific mechanism is a pointwise mutual information signal. Conditioning the teacher on the solution turns it into an oracle: it strongly rewards tokens the solution already entails (connectives, verifiable content) and penalizes deliberation tokens such as "wait", "let", and "maybe" — the very tokens that enable multi-step search at inference time. Three works corroborate this: privileged context lowers the fork rate; the rich-get-richer dynamic lets rare but correct strategies die out; and out-of-domain performance collapses through suppression of epistemic verbalization.

• Self-distillation measures degradations up to −17% (avg@16) on thinking models, with comparable effects out of domain.

• Existing remedies act downstream, never on the teacher's input. One family targets the RL gradient (Clip-Higher in DAPO, or targeting tokens with high covariance between probability and logit update), one replaces a mode-seeking objective with a mass-covering divergence (DPH-RL, which partitions problems into mastered and unmastered and anchors only the mastered ones), and one reverses the sign of the update (Anti-SD, which replaces gradient descent toward the teacher with divergence ascent to encourage exploration). Only Anti-SD targets the PMI mechanism directly; none touches the information given to the teacher.

• Several benchmark figures in this literature are fragile. AIME comprises only thirty questions, so a single question flipping shifts the score by more than three points, and the spread between two decoding seeds can reach fifteen. AIME 2024 problem statements are partly present in pre-training data, to the point that some models complete half of them from memory while failing on benchmarks released after their training cutoff. On Qwen models, even a random training signal can raise the score — an effect absent on Llama and OLMo and traceable to pre-training rather than to the method under evaluation.

• The distortion rules stated in the review include the entropy-performance law R = −a·e^H + b (performance is bounded by an exhausted entropy budget that declines monotonically), and the model-collapse signature (tails disappear first, convergence to a single mode follows). Gerstgrasser et al. show collapse presupposes that synthetic data replace real data; OPSD does re-inject the reference solution at every step, though through the teacher's conditioning rather than the student's training distribution — so model collapse describes the shape of the phenomenon but not its cause.

• Privileged information has an older literature with a clear safety criterion. Vapnik and Vashist (2009) held that privileged information serves to learn better, not to be copied; Lopez-Paz et al. showed distillation is a special case of that framework. From robotics: "Learning by Cheating" works because the student can recover from the image what the teacher knew; Weihs et al. show information unavailable to the student is marginalized during imitation, producing an imitation gap; and RMA never copies the privileged information but has the student regenerate it. The review notes this lineage is largely absent from the OPSD literature, which has rediscovered its vocabulary without inheriting its results.

Methodology in Plain English

This is a review paper; the authors report no new experiments. Their approach is structural and comparative.

They begin by reconstructing the genealogy of the method: supervised fine-tuning gives a dense but off-policy signal, reinforcement learning gives an on-policy but sparse one, on-policy distillation gives both at the cost of a larger teacher, and OPSD removes that cost by replacing teacher capability with teacher information. They then walk through the full OPSD loop in notation — the student generates a rollout, the teacher scores it token by token while conditioned on the prompt plus reference solution plus instructions, the forward KL between the two distributions is computed at each position, clipped dimension-wise at τ, averaged over the rollout and then over the batch, and backpropagated into the student only while the teacher stays frozen. They do the same for SDPO, which swaps the reference solution for textual environment feedback and lets the teacher evolve under regularization.

With that shared baseline established, they survey post-founding work organized by lever rather than by method family. For each lever they follow a fixed pattern: where the field stood at the founding paper, what has appeared since, and what remains open — explicitly separating settled results from disputed ones. Throughout, they apply a fixed reading grid to every number they cite, checking the number of seeds, whether pass@k is reported, and which model family was used, and flagging checkpoint-selected best-score figures as inflated.

Why This Matters

Impact on research. The paper's claim is that collapse is one symptom wearing many names, and that the literature has been solving it piecemeal. By mapping the causes into two families — those inherited from RL and self-training, and the PMI mechanism specific to conditioned teachers — it gives future work a way to state which cause a proposed fix actually addresses. The authors note that two of the three remedy families target the general cause and only one targets the OPSD-specific mechanism, and that none intervenes upstream on the variable that matters most: what the teacher is shown.

Real-world applications:

• Training small mathematical reasoning models on a reduced budget without depending on a larger teacher model, which matters where a second large model cannot be run alongside training.
• Code and agentic tasks, where SDPO's execution feedback (runtime traces, verifier output, judge assessments) is naturally available and the family transfers with efficiency gains of the same order as in mathematics.
• Domains with a verifiable environment, where privileged information can be generated mechanically rather than requiring a dataset of annotated solutions.
• Test-time adaptation to a single hard question, a regime SDPO explores by iteratively distilling feedback into the policy — one that OPSD does not explore.

Industry relevance. The cost profile is the practical crux: OPSD claims far fewer generated tokens per problem but roughly twice the per-step cost at equalized budget (20.6 s versus 11.2 s on Qwen3-8B, 8×H100). Teams choosing between OPSD and GRPO are therefore trading step count against unit cost, and the review is explicit that the advantage is faster convergence in steps, not a lower unit cost. The evaluation grid — pass@k, seeds, contamination tests, non-Qwen replication — is directly actionable for anyone reading or producing benchmark claims on small reasoning models.

Future Directions

1. A weighting criterion that is both justified and computable. The two available criteria have symmetric defects: alignment with the ideal gradient is better founded but only measurable after the fact, since the trajectory outcome must be known before a token can be judged to have pushed toward success; teacher entropy is available online at every step, but nothing guarantees it approximates alignment. A criterion that is both available during training and correlated with a token's actual usefulness remains to be built.

2. The right granularity for selectivity. Weighting is applied today either token by token (Entropy-Aware OPD) or problem by problem (DPH-RL). Nothing indicates these are the optimal scales, nor that the same granularity suits short traces and long reasoning chains.

3. Converting an announced oracle gain into a realized one. Distilling only on positively aligned tokens — roughly half the total — is estimated to improve the signal by a factor of ten to fifteen. This is an oracle measurement obtained outside training, and no method has yet turned it into an effective gain.

4. Testing PMI as a predictor rather than an explanation. The review proposes that the more directly privileged information entails the tokens of the solution, the more the signal should inflate shortcuts and crush deliberation — a grid for predicting which privileged information will collapse the student. No study has yet tested it as a predictor.

5. The lever the remedies leave untouched. All current fixes intervene downstream, once the teacher has already been conditioned. The open question is which privileged information to give the teacher so that it guides without crushing deliberation — including teacher dynamics and the decay of guidance, which the truncated text introduces as Lever C but does not develop beyond noting that OPSD keeps its teacher frozen while SDPO regularizes it via an exponential moving average or interpolation with the initial teacher.

Target Audience

Researchers and practitioners working on post-training of language models — particularly those using or evaluating on-policy distillation, RLVR, or GRPO-style training for reasoning. It suits readers who need to interpret benchmark claims on small models critically, and those designing their own self-distillation setup who need to decide how to weight the signal, what privileged information to give the teacher, and how to keep the loop stable. It is less suited to readers seeking an exhaustive survey of on-policy distillation, for which the paper points to the surveys of Song and Zheng and of Zhang, or an organized map of method families, for which it points to an existing brief overview of OPSD that addresses neither failure modes nor open questions.

Authors’ abstract

On-policy distillation trains a language model on its own generations while a teacher scores them token by token. It combines the dense supervision of imitation learning with the on-policy sampling of reinforcement learning. But it requires a second, larger model to act as teacher. On-Policy Self-Distillation (OPSD) removes that cost. The teacher is the model itself, conditioned on privileged information the student will not have at test time, such as a reference solution, a plan, or environment feedback. The teacher is no stronger than the student, only better informed. Early results were promising, with accuracy comparable to reinforcement learning at a fraction of the generated tokens. But the same asymmetry that produces the signal also biases it. One failure mode now dominates the field: collapse, the progressive narrowing of the set of reasoning paths the model can produce. Collapse is not specific to OPSD, though privileged information aggravates it. This review treats collapse as a symptom governed by three levers: (i) where the signal is applied, that is, how tokens are weighted; (ii) what the teacher is shown, that is, the nature of the privileged information; and (iii) when the signal changes, that is, the teacher's dynamics and the decay of guidance. We restrict our scope to mathematical reasoning, where the method originated and where its failure modes are best documented. We report no new experiments. The contribution is structural: a shared vocabulary for phenomena named differently across papers, and a clear line between what is settled and what is still disputed.