When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better

AI summary

Overview

Research area: Natural Language Processing — specifically post-training and knowledge distillation for large language models, with a focus on reasoning models trained on mathematical problem solving.

Technical level: Intermediate. The paper uses standard distillation notation (reverse KL, per-token advantages) but the core idea is conceptually simple and the results are presented through accuracy curves and cost tables.

Scope: A systematic empirical comparison of on-policy distillation (OPD) against a proposed "Semi-OPD" variant across 17 teacher–student pairs (1.5B to 235B parameters), plus an analysis of when each is preferable.

What This Paper Is About

On-policy distillation trains a student model on text it generates itself, with a teacher model scoring each token — this has become a standard step for transferring reasoning ability. The paper asks whether regenerating those student rollouts throughout training is actually necessary. The authors propose Semi-OPD, which samples rollouts once from the initial student before training and reuses them, and they test when this simpler approach matches or beats the on-policy version.

Key Contributions

1. A simple alternative to OPD. Semi-OPD freezes a rollout cache sampled from the initial student, reuses it for the entire run, and still recomputes the student's log-probabilities (and therefore the per-token advantage) with the current parameters at every step.
2. A large-scale comparison. The authors evaluate 17 teacher–student pairs spanning 1.5B to 235B parameters across the Qwen3, DeepSeek-R1-Distill, and Nemotron-Cascade-2 families, in both thinking and non-thinking modes, on AIME 2024/2025/2026 and HMMT February 2026.
3. A predictive diagnostic. They show that the gain of Semi-OPD over OPD correlates strongly (Pearson r = −0.73) with the initial teacher–student top-64 output-token overlap ratio, turning that ratio into a practical decision criterion.
4. An explanation for OPD's failures. They argue effective distillation requires on-policyness with respect to both the student and the teacher, and show OPD sacrifices the latter for misaligned pairs.

Main Findings

• Semi-OPD wins on most pairs: Across the 17 pairs, Semi-OPD outperforms OPD in 14 cases, with up to +13.6% accuracy and an 11.4× training speedup. On the remaining three pairs it stays within a 0.1%–2.6% gap.
• Overlap predicts the winner: The accuracy gain from Semi-OPD is strongly negatively correlated with initial top-64 overlap (r = −0.73). OPD is beneficial only when teacher and student are highly aligned.
• Large gains at low overlap: The biggest improvements come from low-overlap pairs such as Qwen3-1.7B ← Qwen3-30B-A3B-Thinking (58.9% overlap, +9.4 last-step, +7.57 AUC) and Qwen3-1.7B ← Qwen3-4B-Instruct-2507 (62.5% overlap, +9.5 last-step, +7.09 AUC). On a high-overlap pair like Nemotron-Cascade-2-30B-A3B ← Cascade-2-Math-Expert (89.3% overlap), Semi-OPD falls only 0.1 points behind OPD.
• Thinking-mode pairs show collapse under OPD: For Qwen3 thinking-mode pairs, OPD undergoes substantial performance collapse during training while Semi-OPD keeps improving.
• Beats other OPD variants: On the Qwen3-1.7B non-thinking pair, Semi-OPD reaches 39.8 avg@32 versus Relay-OPD at 32.7, FastOPD at 30.7, Lightning OPD at 30.0, Speculative KD at 28.2, and Standard OPD at 26.2, with 34.7× less training time than Relay-OPD. On the Qwen3-4B non-thinking pair it scores 57.4 versus FastOPD at 52.5 and Lightning OPD at 52.2, with 19.5× less time than Relay-OPD.
• Cost: For 100 training steps on the Qwen3-1.7B pair, Semi-OPD costs 5.0 H100 GPU-hours versus 56.8 for OPD, 11.7 for FastOPD, and 173.3 for Relay-OPD. On the Qwen3-14B thinking pair, Semi-OPD costs 63.0 versus OPD's 480.0.
• Rollouts inflate under OPD: Over the last 10 shared steps, OPD's rollouts are 7–113% longer than Semi-OPD's in 15 of 16 pairs, with the largest growth in low-overlap non-thinking Qwen3 pairs. The exception is R1-Distill-1.5B ← JustRL-1.5B.
• More teacher-preferred tokens: Semi-OPD keeps a larger share of positively supervised tokens (those with p_T > p_S) than OPD in 15 of 16 pairs, again excepting R1-Distill-1.5B ← JustRL-1.5B.
• Full-trajectory coverage matters: Truncating cached traces to the first 1/4, 1/2, or 3/4 of their length (while scaling rollouts per prompt to keep the token budget fixed) hurts; training on all stages performs best.
• Student rollouts beat teacher rollouts: Semi-OPD outperforms distillation on teacher-generated rollouts, so the benefit is not merely from freezing a cache — keeping student-generated trajectories matters.
• Teacher-rollout SFT is unnecessary: Adding an epoch of SFT on teacher rollouts slightly raises overlap and still leaves Semi-OPD ahead of OPD, but Semi-OPD without that stage outperforms Semi-OPD with it.
• OPD reaches a rollout-determined equilibrium: Warming up OPD from a Semi-OPD or teacher-rollout checkpoint on low-overlap pairs still converges to the level reached by OPD from the initial student under the same rollout cap. On a high-overlap pair (86.8%), switching from Semi-OPD to OPD does improve performance and closes most of the gap.
• Gains transfer to code: Models distilled only on math data also improve on HumanEval and HumanEval+, with Semi-OPD still ahead of OPD.
• A strong initial student is required: With a base student, initial rollouts are too weak and Semi-OPD underperforms OPD; with an instruction-tuned student, the advantage returns.
• Not objective-specific: Semi-OPD also outperforms OPD under the top-k reverse KL objective, not just the sampled-token objective.

Methodology in Plain English

The setup is standard distillation. A prompt is fed to a student model, and for each token the student generates, the teacher's probability is compared against the student's. If the teacher assigns a higher probability, the student is pushed toward that token; if lower, away from it. In OPD, the student regenerates its responses at every training step, so the training data shifts as the student changes.

For Semi-OPD, the authors simply collect one response per prompt from the student before training starts and keep that dataset fixed. The scoring step still uses the live student parameters each step, so the supervision signal still adapts — only the text being trained on is frozen. This lets them run generation, teacher scoring, and student training as three separate phases, each using the full GPU cluster instead of sharing hardware among concurrent jobs.

To decide which method suits a given pair, they measure the "top-64 overlap": at each response position, they compare the student's 64 most likely tokens with the teacher's 64 most likely tokens and compute the fraction that coincide, averaged over all response tokens.

Experiments use DAPO-Math (14,116 English prompts after deduplication), 128 prompts per step with one rollout per prompt (roughly 110 steps per epoch), a constant learning rate of 1×10⁻⁶ with AdamW, bfloat16 precision, and training rollout caps of 4,096 tokens for non-thinking pairs and 16,384 for thinking pairs. Evaluation uses avg@32 at temperature 1.0 with a 32k response cap (40k for the Nemotron pair). Infrastructure is NeMo-RL with the DTensor backend, vLLM for generation, and 1–8 nodes of 8× H100 GPUs.

Why This Matters

Research impact. The paper challenges a default assumption in LLM post-training — that on-policy sampling is always better. It reframes the question as a trade-off between on-policyness with respect to the student (which reduces exposure bias) and with respect to the teacher (which determines whether the supervision signal is meaningful). It also provides a cheap, measurable diagnostic (top-k overlap) that other researchers can use to choose baselines fairly and to avoid accidentally using a teacher–student pair on which OPD is doomed.

Real-world applications:

• Cost-effective model compression: Teams distilling a large teacher into a small deployable model can cut training cost by up to 11.4× while improving accuracy on misaligned pairs.
• Math and reasoning tutoring systems: The evaluation targets competition math (AIME, HMMT), relevant to systems that need reliable step-by-step reasoning.
• Code assistant training: Gains from math-only distillation transfer to HumanEval and HumanEval+, suggesting broader reasoning transfer.
• Teacher selection and comparison: A frozen rollout cache can be reused across teachers, so evaluating a new teacher only requires scoring its log-probabilities on cached rollouts.

Industry relevance. Distillation is a routine post-training stage for shipping smaller models. A method that decouples rollout generation from training also simplifies infrastructure: no concurrent student generation, teacher serving, and training on the same cluster, and the phases can each use full hardware.

Future Directions

• Characterizing the OPD equilibrium. The authors observe that OPD converges to a performance level determined by the online rollout data distribution rather than by the initialization, and explicitly flag analyzing this equilibrium as open work.
• Stronger OPD variants under meaningful settings. The paper argues that future OPD methods should be evaluated on teacher–student pairs selected to make the setting informative, rather than on pairs where Semi-OPD is already a stronger baseline.
• Automatic method selection. Turning the top-k overlap ratio into a practical, pre-training decision rule for choosing between OPD and Semi-OPD for a given teacher, student, and target domain.
• When a student is too weak. Semi-OPD fails when the initial student cannot produce meaningful trajectories (e.g., a base model). How to bootstrap in that regime — short of the teacher-rollout SFT stage which the paper found unhelpful — remains open.

Target Audience

Machine learning researchers and engineers working on LLM post-training, distillation, and reasoning-model training. It is most useful for practitioners who are choosing a distillation recipe for a specific teacher–student pair and want a principled basis for deciding between on-policy and offline rollouts, and for researchers who need to design fair OPD baselines. Readers should be comfortable with reverse KL objectives and the basics of autoregressive language model training.

Authors’ abstract

On-policy distillation (OPD) has become increasingly popular for transferring teacher capabilities to student models. In this work, we ask a critical research question: Is on-policy sampling always beneficial for distilling arbitrary teacher-student pairs? We show that a simple alternative, Semi-OPD, which distills from offline rollouts generated by the initial student, can often outperform OPD in both accuracy and training efficiency. Across 17 teacher-student pairs ranging from 1.5B to 235B parameters, Semi-OPD outperforms OPD in 14 cases, with up to +13.6% accuracy and 11.4x training speedup. We further find that the choice between OPD and Semi-OPD depends on the alignment between the initial teacher and student, quantified by an output-token overlap ratio: OPD is beneficial only when the two are highly aligned with high overlap ratios. Our deeper investigation suggests that effective distillation requires on-policyness w.r.t. both the student and the teacher. For misaligned pairs, student rollouts can become increasingly off-policy w.r.t. the teacher as context length grows, weakening the distillation signal. In contrast, Semi-OPD is often more stable, as it distills on shorter contexts while covering full trajectories and exposing the student to more teacher-preferred tokens. Beyond proposing Semi-OPD as an efficient alternative, our work motivates the community to rethink when to use OPD and to study stronger OPD variants with meaningful teacher-student pairs.