On-Policy Distillation with Negative-Policy Rollouts

AI summary

Overview

Research area: Machine learning — large language model post-training, specifically on-policy distillation (OPD) and knowledge distillation for reasoning tasks.

Technical level: Intermediate. The paper assumes familiarity with RL-style post-training, token-level distillation rewards, and preference optimization objectives, though the core idea is explained conceptually.

Scope: The paper introduces Negative-Policy OPD (NP-OPD), a method that adds a lower-capability "negative" rollout policy to on-policy distillation, and evaluates it across three student scales, two generation modes, four model families' worth of configurations (Qwen3 and Gemma), 13 benchmarks, and multiple OPD reward variants.

What This Paper Is About

On-policy distillation trains a student model on its own generated outputs using token-level guidance from a stronger teacher, but this guidance only ever points the student toward the teacher — there is no explicit reference telling it what to move away from. When the teacher's distribution differs substantially from the student's, that positive-only signal can be too weak to learn from. This paper asks whether adding rollouts from a deliberately weaker "negative policy" — a model the student should move away from — can supply the missing negative signal without changing the distillation reward itself.

Key Contributions

1. NP-OPD method. A modification to on-policy distillation in which each training batch mixes student-generated trajectories and trajectories sampled from a negative policy in proportions 1−α and α, while token-level teacher supervision stays exactly the same as in standard OPD.

2. A rollout-stage, not reward-stage, intervention. Because the negative policy is introduced through the rollout distribution rather than the reward function, NP-OPD is compatible with recent OPD reward variants (ExOPD and OPD²) without modifying their formulations.

3. A mathematical account of why it works. The paper shows that under the assumption that the KL divergence between the teacher and the negative policy exceeds that between the student and the negative policy, tokens generated by the negative policy receive negative expected OPD reward, so increased sampling of those trajectories pushes the student away from the negative policy.

4. Two diagnostic metrics and their analysis. It defines negative-policy suppression precision (NSP) and negative-policy overlap reduction (NOR) and uses them to show that the intended mechanism — concentrated suppression of negative-policy-preferred tokens — actually occurs.

Main Findings

NP-OPD beats OPD across scales and domains (non-thinking mode). For Qwen3-1.7B with α = 1.0, Math average rises from 48.3 (OPD) to 56.3, Code from 27.2 to 31.1, and Science from 37.2 to 41.5. For Qwen3-4B, Math goes from 62.2 to 70.5, Code from 40.2 to 51.1, and Science from 47.8 to 51.2.

Largest single-benchmark gains are concentrated on hard reasoning benchmarks. For Qwen3-1.7B, NP-OPD raises AIME24 from 31.7 to 43.1, AIME25 from 21.7 to 33.3, AMC23 from 65.3 to 78.4, and HMMT25 from 13.3 to 20.0, relative to OPD. For Qwen3-4B the same benchmarks move from 53.8 to 67.9, 42.3 to 58.8, 90.0 to 95.3, and 26.2 to 43.8.

Gains hold in thinking mode and across OPD reward variants. On the 13-benchmark average for Qwen3-1.7B in thinking mode, NP-OPD improves OPD by 1.98 points (49.7 to 51.6) and OPD² by 1.76 points (53.7 to 55.5), while remaining comparable with ExOPD (+0.30, 52.8 to 53.1). For Qwen3-4B, gains are 1.11 points for OPD, 1.86 for ExOPD, and 2.10 for OPD².

Effectiveness transfers to the Gemma family. With Gemma-4-E4B-it as student and Gemma-4-12B-it as teacher, NP-OPD with α = 1.0 lifts the Math average from 63.0 to 66.4 and Science from 49.1 to 49.8; on Code, both OPD and NP-OPD fall below the base model's 56.5, but NP-OPD still improves over OPD (54.9 to 55.2).

A lower-capability rollout policy is what matters. Analyses using Qwen3-1.7B as student and Qwen3-8B as teacher in thinking mode show that rollouts from Qwen3-1.7B, Qwen3-4B, and Qwen3-8B (larger or equal capability) yield lower NSP and lower accuracy; only the lower-capability Qwen3-0.6B policy improves over on-policy OPD.

NP-OPD concentrates suppression on the intended tokens and shifts away from the negative policy. NP-OPD with Qwen3-0.6B as negative policy achieves higher NSP than OPD, and NSP is reported as highly correlated with average accuracy. On NOR, most trajectory points lie below the diagonal, with a mean of 0.019 under NP-OPD versus 0.005 under OPD; teacher rollouts give a mean of −0.030.

Rollout pre-generation cuts training time. For Qwen3-1.7B on a single 8×H100 node, NP-OPD with α = 1 reduces training time from 9h19m to 3h34m in thinking mode, excluding rollout pre-generation.

Simple degradation of the student is less effective than a separate negative policy. With Qwen3-1.7B as student and Qwen3-8B as teacher in thinking mode, temperature scaling (1.5) improves the average by 0.9 points and a persona prompt by 1.0 point, while a separate negative policy improves it by 3.4 points (42.9 to 46.3). Layer dropping (three layers) scores 42.1, below the 42.9 on-policy baseline.

A reinterpretation of recent OPD variants. ExOPD and OPD² reach higher NSP than standard OPD even with on-policy rollouts, which the authors suggest means these methods may already benefit from implicit negative signals, because ExOPD uses a reference model and OPD² uses the teacher-to-base delta.

Methodology in Plain English

The setup starts from standard on-policy distillation: a student generates its own answer to a question, and at every token the model compares the teacher's probability for that token with its own, producing a reward that nudges the student toward the teacher.

NP-OPD changes only who generates the text. Every training batch draws a coin flip with probability α; with that probability the trajectory comes from a separate negative policy instead of the student. That negative policy is chosen from the same model family but with lower capacity and lower reasoning performance — for a Qwen3-1.7B student, Qwen3-0.6B; for Qwen3-4B, Qwen3-1.7B; for Qwen3-8B, Qwen3-4B; for Gemma-4-E4B-it, Gemma-4-E2B-it. The same teacher reward is then applied to those tokens unchanged.

The paper shows algebraically that this mixing changes the sampling probability of each trajectory's gradient contribution by a term proportional to α times the difference between the negative policy's and the student's prefix probabilities. Because weaker models tend to be further from the teacher than the student is, tokens the negative policy prefers over the teacher carry negative expected reward, so amplifying their sampling pushes the student away from them. Crucially, the reward direction itself is never inverted: individual tokens the teacher prefers are still reinforced even if the negative policy also likes them.

Experiments use 30k questions mixed at a 1:1:1 ratio from Math, Science, and Code, a fixed batch size of 256, a learning rate of 5×10⁻⁶, a maximum rollout length of 16k tokens, 100 training steps, and rollout temperature of 1.0. Evaluation uses temperature 0.7 and a maximum generation length of 32k tokens, across 13 benchmarks: seven math, three code, and three science. The mixing ratio is swept over α ∈ {0.0, 0.25, 0.5, 0.75, 1.0}, where α = 0.0 is plain OPD and α = 1.0 uses only negative-policy rollouts.

To check the mechanism, the authors define two measurements. NSP measures what fraction of the tokens whose log-probability dropped the most are tokens the negative policy prefers over the teacher. NOR measures, at positions where the negative policy's and teacher's top predictions disagree, how much more the student's probability moved toward the teacher's top token than toward the negative policy's.

Why This Matters

Impact on research. The paper reframes the rollout policy in on-policy distillation — normally treated as a fixed given — as a design choice that can supply negative supervision. It also offers an explanation for why recent OPD reward variants work, suggesting they carry implicit negative signals, and shows that an explicit negative-policy signal stacks on top of them. Because the intervention sits in the rollout stage, it is orthogonal to reward-design research lines.

Real-world applications.

• Post-training smaller reasoning models in a family (for example a 1.7B model distilled from an 8B teacher) where the capability gap is large and teacher guidance alone is weak.
• Reasoning-heavy domains where the reported gains are largest: competition mathematics (AIME24, AIME25, AMC23, HMMT25), competitive programming (LCBv5, RGAlgo), and graduate-level science questions (GPQA Super, GPQA Sci, SciBench).
• Cost-constrained training pipelines: pre-generated negative-policy rollouts can be reused, reported as cutting training from 9h19m to 3h34m on a single 8×H100 node in one setting.
• Distillation on non-Qwen model families, demonstrated on Gemma-4-E4B-it with Gemma-4-12B-it as teacher.

Industry relevance. The method requires only an additional lower-capability checkpoint from the same family and no change to the loss, which makes it a low-friction addition to existing distillation pipelines. The rollout reuse and reduced training time are directly relevant to practitioners paying for GPU hours.

Future Directions

• How should the negative policy be selected? The paper fixes it as a lower-capacity model from the same family, but the accuracy-versus-rollout-capability analysis implies there is a preferred region of capability gap that is not fully characterized here.
• Can negative rollouts be synthesized from the student itself? The paper tests temperature scaling, persona prompting, and layer dropping, and finds gains (0.9 and 1.0 points on average) that fall well short of a separate negative policy (3.4 points), leaving open whether better synthesis methods exist.
• Does the NSP–accuracy correlation hold more broadly? The correlation is measured across OPD, NP-OPD, and alternative rollout settings using Qwen3-1.7B, Qwen3-4B, and Qwen3-8B; whether it generalizes to other families, scales, and domains is not established.
• A full account of implicit negative signals in existing OPD variants. The discussion notes that ExOPD and OPD² already raise NSP without explicit negative rollouts, but the paper does not quantify or formalize that implicit signal.

Target Audience

Researchers and engineers working on LLM post-training, distillation, and reinforcement-learning-style fine-tuning — particularly those who already have a working on-policy distillation pipeline and want to improve it without redesigning the reward. It is also relevant to readers interested in the role of data or rollout sources rather than loss functions, and to practitioners training small reasoning models against much larger teachers. Readers looking for an introduction to distillation or preference optimization from scratch would find the paper assumes that background.

Authors’ abstract

On-policy distillation (OPD) has been widely studied as a post-training method in which a student model obtains token-level supervision from a stronger teacher on its own rollouts. Recent studies have improved OPD through alternative distillation reward formulations and teacher configurations, while the objective of distillation remains centered on mimicking the teacher. However, when a stronger teacher has limited distributional overlap with the student, such positive guidance can provide insufficient learning signals. In this work, we introduce Negative-Policy OPD (NP-OPD), which complements teacher supervision with rollouts from a lower-performing, lower-capability negative policy that serves as a negative reference for the student. Rather than modifying the distillation reward formulation, NP-OPD introduces the negative policy at the rollout stage, continuously supplying tokens preferred by the negative policy over the teacher so that they remain exposed to teacher supervision throughout training. This provides an explicit negative signal through negative-policy rollouts while preserving the positive teacher supervision used in OPD. Through extensive experiments, we show that NP-OPD improves OPD across model scales, generation modes, reasoning domains, and different OPD variants. Furthermore, our analyses show that NP-OPD effectively suppresses tokens preferred by the negative policy over the teacher and moves the student away from the negative policy. These results support our design of introducing negative signals through negative-policy rollouts and provide new insight into the role of the rollout policy in OPD. Code will be available at https://github.com/naver-ai/np-opd.