Rethinking Relative Policy Differences Transfer of On-Policy Distillation

AI summary

Overview

Research area: On-policy distillation (OPD) and LLM post-training — specifically, how to transfer the relative policy change between an expert and a reference model into a different target model, rather than copying the expert's absolute output distribution.

Technical level: Advanced. The paper is built around a formal derivation (KL-regularized policy optimization, implicit reward parameterization) and assumes familiarity with reverse KL, policy gradients, advantage centering, and RL/SFT post-training pipelines.

One-sentence scope: A theoretical reformulation of relative policy difference transfer as "any-to-any" OPD, combined with a systematic empirical study of which policy differences carry useful supervision and which target models can actually absorb them.

Note: the paper content supplied to me is truncated (it cuts off partway through Table 5 and omits all appendices), so appendix-level derivations and the remainder of the code-domain results are not covered here.

What This Paper Is About

Conventional on-policy distillation trains a student on its own rollouts using token-level guidance from a teacher, which only works well when the teacher is meaningfully stronger than the student. A newer line of work sidesteps that requirement by transferring the policy difference between an expert and a reference model — the log-ratio of their output probabilities — instead of the expert's absolute distribution. This paper asks two questions that prior work left open: which relative policy differences actually provide useful supervision, and which target models are able to realize a useful difference once it is available.

Key Contributions

1. A formal derivation of relative policy difference transfer from implicit rewards. The authors show that the log-probability ratio between a proxy expert and a proxy reference defines an implicit reward for KL-regularized policy optimization, and from this they derive an on-policy update rule. Conventional OPD falls out as a special case (setting the proxy expert/reference equal to the teacher/target reference and α = β).

2. An "any-to-any" OPD objective (A²-OPD). The formulation separates the source of the learning signal (the proxy pair) from the policy it is applied to (the target), permitting arbitrary source–target combinations. The paper positions prior methods (OPD, ExOPD, Relative Policy Transfer) as special cases of the same objective in Table 1, differing only in the proxy pair, target reference, and transfer strength α/β.

3. A systematic empirical study across three sources of policy difference. The authors compare differences induced by reinforcement learning (RL), supervised fine-tuning (SFT), and model scaling, evaluated on a fixed target (Qwen3-8B) and then across six different targets.

4. A characterization of when transfer succeeds. The results identify task-directed training, reference-based filtering, and target compatibility — not raw policy distance — as the governing factors, including a demonstration that choosing an appropriate reference can rescue an otherwise failed transfer.

Main Findings

• A stronger model is not automatically a better teacher. Under a fixed Qwen3-8B target, task-optimized differences transfer consistently: RL and SFT pairs yield target gains (G_T) from 9.78 to 32.23. Four scale-only pairs yield much smaller or negative gains (G_T of +3.42, −1.03, +0.02, and +8.23), despite positive proxy pair capability gaps.

• Transfer depends on the task-relevant direction, not the size of the update. Proxy gain G_P correlates strongly with target gain G_T (Pearson r = 0.96, Spearman ρ = 0.93). But with Qwen3-8B as target and Qwen3-1.7B as proxy reference, the task-optimized pair (Qwen3-4B-RL / Qwen3-1.7B) and the scale-only pair (Qwen3-4B / Qwen3-1.7B) induce comparable update magnitudes (K_Induced = 0.3531 vs. 0.4181) yet produce very different target gains (25.85 vs. 3.42).

• The target follows a task-trained direction but not a scale-only one. Token-level overlap between target and proxy expert rises from 0.75 to about 0.81 for the task-trained pair and overtakes the scale-only pair within 20 steps, while the scale-only overlap drifts down to 0.78.

• The expert supplies capability; the reference filters what is already shared. With the reference fixed at Qwen3-4B, moving the expert from early to regular to late RL checkpoints raises G_P from 22.09 to 26.89 to 32.28, and G_T tracks it (22.42 → 26.84 → 28.18), though the realization ratio R_perf falls from 1.015 to 0.873. Fixing the expert at Qwen3-4B-RL and swapping the reference cuts G_T from 26.84 to 9.62 when the early RL checkpoint is used as reference, because the difference then exposes only the early-to-regular increment.

• The reference need not be the expert's training origin. Using Qwen2.5-3B as reference for the Qwen3-4B-RL expert gives the largest target gain in that sweep (29.09), despite the largest K_Base = 1.23.

• Expert–reference overlap predicts the residual available for transfer. With Qwen3-4B-RL as expert, top-16 overlap O_E,R drops from 0.916 (early-RL reference) to 0.839 (Qwen3-4B) to 0.625 (Qwen2.5-3B), while target gains rise from 9.62 to 26.84 to 29.09. Two RL increments with similar proxy gains (G_P = 4.80 and 5.39) but different overlaps (0.916 and 0.869) produce G_T of 9.62 vs. 25.03.

• A useful difference does not help every target. Across four proxy pairs applied to six targets, Qwen3 targets gain between 9.78 and 29.09, whereas Qwen2.5 targets under a Qwen3 reference gain almost nothing (G_T in [−2.46, 2.72]).

• Target distance does not predict who benefits. Neither R_perf nor transferred accuracy is monotone in K_Base, and Qwen3 targets still gain at the largest K_Base (R_perf ≈ 0.65 at K_Base > 1.2). The outcome splits by model family, not by policy distance.

• Matching the reference's model family can recover a failed transfer. Replacing the Qwen3 reference with a Qwen2.5-3B reference turns Qwen2.5 targets from near-zero or negative gains into positive ones (G_T = 6.74 / 2.25 / 5.76), while Qwen3 targets stay successful.

• Benefiting means moving toward the expert and generalizing. A target with G_T = +16.1 moves toward the expert and away from its own base, while a failing target with G_T = −1.7 moves toward neither. For the incompatible Qwen2.5-Math-7B target, training accuracy rises and even spikes but validation accuracy stays near zero, so the failure is not undertraining.

• The patterns appear in code generation as well as math. For same-generation Qwen3 → Qwen3 transfer, the Qwen3-4B-RL / Qwen3-4B pair applied to Qwen3-8B gives math 45.27 (G_T +26.8) and code 56.44 (G_T +2.3); the Qwen3-4B-RL / Qwen3-1.7B pair gives math 44.28 (G_T +25.9) and code 56.03 (G_T +1.9). The remaining code-domain rows are not present in the supplied text.

Methodology in Plain English

The authors start from a mathematical observation: if you rewrite the conventional OPD objective, the teacher's log-probability appears as a reward term and the target's initial policy appears as a KL anchor. Because the same reference policy appears in both places, it cancels, and the method degenerates into matching the teacher's absolute distribution.

They then ask what happens if you don't let it cancel. Treating the expert and reference as a "proxy pair" that solves a one-step KL-regularized optimization with some task reward, they show the log-ratio between the two encodes that reward up to a positive scale (an implicit reward). That ratio is then simply applied as a per-action weight to a target policy, with a separate KL term keeping the target near its own reference. Because the proxy reference and the target reference are independent, any pair can in principle feed any target — hence "any-to-any."

For training, they differentiate this objective and get a score-function gradient with a per-action advantage. The advantage contains an intractable normalizer term that depends only on the state, so they center the advantage across actions to remove it — which also reduces variance — and stop gradients through the centered advantage in the final loss.

Empirically, all experiments use this objective with a fixed transfer strength α/β = 1, top-16 token distributions instead of the full vocabulary, models from the Qwen3 and Qwen2.5 families, and proxy pairs obtained via RL or SFT on DeepMath-103K (math) and Eurus-RL-Code (code) with a rule-verifiable reward. Sampling uses temperature 1.0 and top-p 1.0. Evaluation covers eight math benchmarks (AIME24/25/26, SMT25, CMIMC25, HMMT25-Feb/Nov, HMMT26-Feb) with 16 samples each, and three code benchmarks (HumanEval+, MBPP+, LCB) with 8 samples each. Besides raw task accuracy, they use KL-based diagnostics (K_Proxy, K_Base, K_Target, K_Induced) and a top-16 overlap ratio between target and proxy expert to see the mechanism behind the scores.

Why This Matters

Impact on research. The paper reframes relative policy transfer as a family of KL-regularized policy optimization problems, which gives prior methods (OPD, ExOPD, Relative Policy Transfer) a common formal home and makes their differences explicit. It also replaces the intuition "pick the strongest available model" with a concrete set of conditions — task-directed training, a reference that filters shared behavior, and target/reference compatibility — that future work can test and build on.

Real-world applications (as implications of the findings):

• Reusing expensive RL or SFT post-training runs as supervision for other models, without needing the original post-trained model to be stronger than the target.
• Cheaper post-training pipelines: the paper argues relative transfer broadens applicability of OPD across models of different scales and architectures.
• Cross-family model development, where the choice of reference model family is shown to determine whether transfer succeeds at all.
• Domain reasoning models in math and code, the two domains evaluated here, where a single expert difference can be applied to targets at multiple scales.

Industry relevance. The related work cited in the paper describes OPD as a common component of production post-training pipelines: Qwen3 uses off-policy distillation followed by OPD for logit alignment, and MiMo-V2-Flash scales the approach with multiple domain-specialized RL teachers, enabling a 309B/15B-active student to match models with 2–3× more total parameters. Findings about which differences transfer and which targets can absorb them speak directly to how such pipelines choose teachers and students.

Future Directions

1. Extending beyond the Qwen3/Qwen2.5 families and the math/code domains. The paper's own patterns are established on math and extended to code; broader model families and tasks are not reported.
2. Turning "compatibility" into a predictive criterion. The authors show that raw policy distance fails to predict transferability and that family matching works in their experiments, but the supplied text does not give a general recipe for choosing a reference without trial and error.
3. A principled way to select the proxy reference. The reference does the filtering, and the alternative Qwen2.5-3B reference produced the largest gains in one sweep, so what makes a reference "appropriate" beyond sharing fewer high-probability tokens is an open question.
4. Theory for when a realizable difference generalizes. The paper separates "useful difference" from "target can realize it" and shows training accuracy can spike while validation stays near zero; a formal account of the generalization condition is not provided in the supplied content.

Target Audience

Researchers and engineers working on LLM post-training, distillation, and reinforcement learning from verifiable rewards, particularly those who already know conventional on-policy distillation and want to understand relative policy transfer. The paper is most useful to people deciding which teacher or expert–reference pair to use in a training pipeline, and to those who want a formal framing of relative transfer as KL-regularized policy optimization rather than a collection of empirical tricks. Readers without a background in policy gradients and KL-regularized objectives will find the derivation sections dense.

Authors’ abstract

On-policy distillation (OPD) trains a target policy on its own rollouts using token-level supervision from a stronger teacher, but typically relies on the teacher being substantially more capable than the target. Recent work relaxes this requirement by transferring the relative policy difference between an expert and a reference policy rather than the expert's absolute distribution. Yet it remains unclear which differences provide useful supervision and which targets can benefit from them. We develop a unified view in which the log-probability ratio between a proxy expert and a proxy reference defines an implicit reward for KL-regularized policy optimization. This formulation induces an on-policy update for the target, recovers conventional OPD as a special case, and enables any-to-any transfer across models that differ in scale, training stage or generation. Our systematic experiments yield three findings. First, a stronger expert is not necessarily a better source of supervision: differences induced by task-specific training, including reinforcement learning and supervised fine-tuning, transfer consistently, whereas those arising from model scale alone transfer weakly or not at all. Second, the expert and reference serve complementary roles: the expert provides the capability to be transferred, while the reference filters out shared behavior and isolates the relevant residual. Third, even a useful difference does not transfer to every target: success requires the induced update to move the target toward the expert and generalize beyond sampled rollouts. Raw policy distance does not predict transferability, whereas an appropriate reference can rescue otherwise unsuccessful transfer. These results characterize when relative policy differences provide effective supervision for OPD.