On-Policy Delta Distillation

AI summary

Overview

Research area: Post-training of large language models (LLMs), specifically on-policy distillation (OPD) as a reinforcement-learning alternative for transferring reasoning ability from a large teacher model to a smaller student model.

Technical level: Advanced. The paper assumes familiarity with knowledge distillation, KL divergence objectives, policy-gradient formulations (GRPO-style rewards and advantages), and LLM post-training pipelines.

Scope: The paper proposes a new token-level distillation reward called the delta signal, derived from the difference between a reasoning-tuned teacher and its own pre-instruction-tuning base model, and shows across mathematics, science, and code benchmarks that it outperforms conventional on-policy distillation.

What This Paper Is About

Conventional on-policy distillation trains a student model to imitate a strong teacher's token-level output distribution, but this copies everything the teacher knows, including stylistic and natural-language preferences that have nothing to do with reasoning. The authors argue that what should be transferred is only the change in behaviour the teacher acquired during reasoning tuning. They define that change as the delta signal — the log-probability difference between the reasoning-tuned teacher and its base model before post-training — and use it as the primary reward for on-policy distillation, calling the resulting method On-Policy Delta Distillation (OPD²).

Key Contributions

1. A new distillation reward (the delta signal). Instead of the conventional OPD reward R_t^OPD = log π*(y_t|x,y_<t) − log π_θ(y_t|x,y_<t), the paper defines R_t^Δ = log π*(y_t|x,y_<t) − log π*_base(y_t|x,y_<t), isolating the learning trace the teacher acquired through reasoning tuning rather than its pre-existing preferences.

2. Two reward design mechanisms to make the delta signal trainable. Centering subtracts the expected reward under the student's on-policy distribution (computed over the top-k tokens, k = 1024, to save GPU memory), and joint conditioning defines the advantage A_t^{D²} = A_t^Δ when A_t^Δ · A_t^OPD > 0, and 0 otherwise, so updates are restricted to sign-consistent directions shared with the original distillation signal and the delta signal governs gradient magnitude.

3. Three diagnostic analyses of the delta signal. Word clouds, token-level signal visualizations (clipped at ±6), and word-level statistics across three domains, showing how the delta signal differs from the conventional OPD reward.

4. An extensive multi-domain empirical verification framework. A 1:1:1 mixed training set drawn from OpenMathReasoning, OpenScienceReasoning-2, and OpenCodeReasoning (100k questions), with evaluation on 7 Math, 3 Science, and 4 Code benchmarks, across Qwen3-1.7B, Qwen3-4B, Qwen3-8B in both non-thinking and thinking modes, plus Gemma4-E4B-it.

Main Findings

• Delta signal emphasizes reasoning connectives and suppresses exploratory narration. In word-cloud analysis with Qwen3-1.7B as student, Qwen3-4B-Thinking-2507 as teacher, and Qwen3-4B-Base as base on 10k math questions, the delta signal enhanced words such as hence, note, instead, however, and yet, while suppressing see, try, verify, and confirm — the latter also appearing in the base model's word cloud.

• Delta signals correlate better with reasoning correctness at the token level. On three simple questions with intentionally incorrect reasoning (math, science, code), OPD kept positive signals on incorrect reasoning spans because student negative signals were stronger in magnitude than the teacher's, while the delta signal suppressed those tokens more consistently. The paper states the signal is clipped to ±6 for visualization.

• Statistical analysis over large models shows consistent word-level shifts. Using Qwen3-8B as student, Qwen3-30B-A3B-Thinking-2507 as teacher, and Qwen3-30B-A3B-Base as base, with 10k questions per domain and 72.4M (Math), 53.3M (Code), and 36.7M (Science) generated tokens, tokens such as hence (57% enhanced in Math), however (55% in Math, 61% in Code), imagine (74% in Code), and thus (60% in Code) shifted toward enhancement, while perhaps (49% suppressed in Math, 38% in Science), tackle (73% in Code), and consider (31% in Code) shifted toward suppression. The ±1 thresholds correspond approximately to the top and bottom 4% of OPD signal values and 5% of delta signal values.

• OPD² leads on non-thinking Qwen3 mathematics. Average Math score: Qwen3-1.7B rises from 34.8 to 54.6 with OPD², versus 51.0 (OPD) and 51.4 (ExOPD); Qwen3-4B reaches 70.3 versus 64.0 and 66.4; Qwen3-8B reaches 71.6 versus 65.9 and 67.8.

• OPD² leads on non-thinking Qwen3 code and science. Code averages: 29.4 / 40.1 / 39.9 for 1.7B / 4B / 8B with OPD², versus 21.0 / 31.4 / 35.0 for OPD and 24.6 / 37.2 / 38.1 for ExOPD. Science averages with OPD²: 38.8 / 50.5 / 51.6, the highest for all three sizes.

• OPD² improves thinking-mode models that OPD often degrades. On thinking Math, OPD underperformed the original models at all three sizes (57.1 vs 59.2 for 1.7B; 70.9 vs 73.3 for 4B; 72.2 vs 73.7 for 8B), while OPD² improved them to 62.7, 74.8, and 75.9. On thinking Code, OPD² reached 40.4 / 50.8 / 57.8 versus baseline 29.3 / 48.7 / 50.8. On thinking Science, OPD² reached 43.4 / 51.5 / 54.6 versus baseline 40.1 / 51.3 / 53.0.

• Competition-level benchmarks show the largest gaps. On HMMT25 in thinking mode, OPD² raised Qwen3-8B from 44.3 to 52.3, compared with 43.3 (OPD) and 46.3 (ExOPD).

• Gains transfer to a different model family. Gemma4-E4B-it Math average improved from 60.6 to 67.8 with OPD², versus 58.9 (OPD) and 65.3 (ExOPD); on AIME24, OPD² raised 51.7 to 69.2 versus 59.6 for ExOPD.

• Code remains the hardest case for Gemma4. Gemma4-E4B-it already scored 55.2 on Code average; OPD dropped to 36.9, ExOPD to 45.1, and OPD² to 49.5 — no distillation method exceeded the original model in average Code score, though OPD² retained the most capability.

• ExOPD was configured with λ = 1.25, following the setting reported in its original paper.

Methodology in Plain English

The authors keep the standard on-policy distillation setup: the student model generates its own answers to questions, and each generated token receives a reward that says whether that token should be made more or less likely. In ordinary OPD, the reward compares the teacher's probability for that token with the student's — so tokens the teacher likes more than the student get boosted, and vice versa.

The authors change what is compared. They take a teacher model that has been tuned for reasoning and also its base version, the same model before reasoning tuning. The reward for a token becomes the difference between those two: how much more (or less) the reasoning-tuned teacher likes this token compared with the untouched base model. That difference is the "delta signal," and it isolates what reasoning tuning actually changed.

Because the delta signal does not reference the student at all, training on it alone would keep pushing toward the single highest-reward token. The authors fix this in two ways. First, they "center" the reward by subtracting its expected value under the student's own probability distribution (computed over 1024 tokens for memory reasons). Second, they gate the delta advantage: it is used only when it agrees in sign with the standard OPD advantage, and set to zero otherwise. When the student matches the teacher, no gradient is applied.

The empirical setup is deliberately broad. Training data mixes mathematics, science, and code questions equally (1:1:1) into a 100k-question pool, of which fewer than 30k are actually seen — less than one epoch, with each question used at most once. Training runs for 100 steps using TRL's GRPOTrainer with a single completion per question, temperature 0.7, a maximum completion length of 8k tokens, AdamW with a learning rate of 5×10⁻⁶, cosine decay, gradient clipping at 1.0, and all rewards uniformly scaled by 0.1 to reduce clipping frequency. Evaluation uses pass@1 across 14 benchmarks.

Why This Matters

Impact on research. The paper reframes what on-policy distillation should transfer. Rather than imitating a teacher wholesale, it proposes transferring only the teacher's learning trace. This is a fundamental redesign of the distillation reward, and the reported results suggest the redesign matters more than scaling: the paper notes that Qwen3-4B trained with OPD² surpasses Qwen3-8B trained with OPD or ExOPD on non-thinking Math. The release of results across two model families (Qwen3, Gemma4), three domains, two reasoning modes, and multiple model sizes makes the claim unusually well-tested for a reward-design paper.

Real-world applications:

• Deploying smaller, cheaper reasoning models that inherit capability from larger teachers for on-device or latency-sensitive assistants.
• Domain adaptation for mathematical, scientific, and programming assistants without hand-crafting reward models for each domain.
• Post-training code assistants where long reasoning traces need to become more correct and less exploratory.
• Reducing training cost for teams that cannot run full reinforcement-learning pipelines with reward models.

Industry relevance. The method needs no domain-specific reward design, trains for only 100 steps, and reuses an existing RL training framework (TRL's GRPOTrainer with vLLM rollouts). The one extra requirement is access to the teacher's base checkpoint, which is publicly available for both model families tested. This makes it practical for organizations that already have a strong teacher model and want a short, inexpensive post-training run. Code is to be released at https://github.com/naver-ai/opd2.

Future Directions

• The paper states it does not address top-k OPD variants that extend learning signals beyond sampled tokens. Combining the delta signal with methods that handle unsampled tokens is a natural unanswered question.
• Code performance on Gemma4 remained below the original model for every distillation method tried, including OPD². Why code is harder to improve in this setting, and whether recipe adjustments close the gap, is unresolved.
• Training dynamics, ablation study, and computational analysis are referenced in the paper's experiment plan, but their results are not reported in the content available here.
• The delta signal requires a teacher base checkpoint. Whether the approach extends to teachers whose pre-instruction-tuning checkpoint is unavailable, or to settings where the base differs in architecture, is left open.

Target Audience

Researchers and engineers working on LLM post-training, knowledge distillation, and reinforcement-learning-based reasoning training. The paper is most useful to readers who already understand KL-divergence distillation objectives and policy-gradient reward formulations, and who have access to paired teacher and teacher-base checkpoints. Practitioners seeking an inexpensive alternative to reward-model RL for reasoning post-training will find the reported training recipe and multi-benchmark results directly applicable.

Authors’ abstract

On-policy distillation is an alternative post-training method in reinforcement learning that alleviates the constraints imposed by reward models by providing token-level supervision from a teacher model. Although on-policy distillation has been studied and applied across various settings, its fundamental design remains underexplored. In this paper, we introduce a new distillation reward, termed the delta signal, instead of directly imitating the teacher's output distribution. The delta signal is defined as the difference between the teacher model and its base model prior to instruction tuning for reasoning capability. It therefore captures the changes induced by reasoning tuning and provides a more direct signal for transferring reasoning capabilities. Using extensive empirical evidence, we show that the delta signal substantially improves on-policy distillation and refer to the new distillation method as On-Policy Delta Distillation (OPD$^2$). Experiments across mathematics, science, and code-reasoning benchmarks demonstrate that OPD$^2$ consistently outperforms conventional on-policy distillation, enabling reasoning LLMs to achieve strong performance with only a short post-training period. Code will be available at https://github.com/naver-ai/opd2