Calibrating Teacher--Student Discrepancy for On-Policy Distillation

AI summary

Overview

Research area: Large language model post-training, specifically on-policy knowledge distillation (OPD) for mathematical reasoning, with a focus on the reliability of teacher supervision signals.

Technical level: Intermediate. The core intuition is easy to grasp, but the paper is written in dense mathematical notation (token-level log-likelihoods, advantage decomposition, intervention sets) that assumes familiarity with policy-gradient-style training and LLM distillation.

Scope: The paper identifies a previously unaccounted-for source of noise in on-policy distillation — which it calls Teacher Self-Deviation (TSD) — and proposes a calibration method, Cal-OPD, that filters that noise out of the learning signal, evaluated across two Qwen3 teacher–student pairs on six mathematical reasoning benchmarks.

What This Paper Is About

On-policy distillation trains a weaker student model by having it generate its own reasoning trajectories and then comparing, token by token, how likely the teacher would have been to produce each token. The paper argues that this gap does not measure only the teacher's superior capability: part of it comes from the teacher wobbling on its own — being sensitive to context in ways that have nothing to do with task knowledge. The goal is to separate the teacher's genuine capability gap from this self-wobble, and to train the student only on the former. The problem gets worse under "privileged" OPD, where the teacher is given extra information like reference solutions or hints, which makes the teacher's self-wobble larger and lets the student learn even more of it.

Key Contributions

1. Identification and characterization of Teacher Self-Deviation (TSD). The authors define TSD as the change in teacher log-likelihood for a fixed token induced by a teacher-side contextual intervention, while the problem and student rollout are held constant. They empirically show TSD emerges without task-specific knowledge, is largely insensitive to intervention semantics and correctness, and concentrates on surface-form tokens.

2. The Cal-OPD method. Calibrated On-Policy Distillation uses a pair of contrasting positive and negative privileged interventions to estimate a per-token "self-deviation region" around the teacher's likelihood, then retains only the teacher–student discrepancy that falls outside that region. Privileged information is used to calibrate the teacher reference rather than to directly supervise the student.

3. Empirical demonstration across scales. Across two teacher–student configurations and six benchmarks, Cal-OPD achieves the best average in both configurations while retaining only about 52–65% of the original teacher–student discrepancy, and it also mitigates the response-length expansion seen in standard and privileged OPD.

4. A detailed diagnostic picture of where TSD lives. The paper provides token-form-level analysis (rankings of significant-TSD rates by token form), retention analysis across intervention groups, and consistency metrics (overlap, directional agreement, shared deviation ratio) across thresholds and model scales.

Main Findings

• Standard OPD optimizes noise, not just knowledge. The observed teacher–student discrepancy mixes the genuine capability gap with teacher-side deviations, and standard OPD learns both indiscriminately. In the 4B→1.7B setting, standard OPD improved the student only modestly (49.2 to 50.8 average), and in the 30B→4B setting it actually degraded the student (66.6 to 65.9), despite both teachers being substantially stronger than their students.

• Privileged OPD performs worst among the distillation methods. Privileged-OPD obtained the lowest average in both configurations, 49.3 and 64.0. In the 30B→4B setting it fell 2.6 points below the student baseline. The authors link this to privileged context substantially amplifying TSD, so stronger supervision transfers teacher-side deviation alongside task-relevant information.

• TSD emerges without task knowledge. At a threshold of τ = 0.01, task-agnostic instructions (which contain neither an answer nor a solution) induced significant TSD on 20.2% and 25.8% of tokens under the positive and negative variants, with their union covering 29.8% of tokens. This is comparable to evaluative feedback (29.1%) and answer-level privilege (29.4%), while solution-level privilege expanded the affected set to 39.6%.

• Richer privileged context broadens TSD rather than creating it. Under solution-level privilege, 98.3%, 98.6%, and 98.2% of tokens that showed significant TSD under task-agnostic instructions, evaluative feedback, and answer-level privilege, respectively, remained significant. In the reverse direction, only 74.2%, 72.6%, and 72.9% of solution-level significant-TSD tokens remained significant under those less informative interventions.

• TSD is largely insensitive to intervention semantics and correctness. Reversing answer correctness (from correct to incorrect) still yielded 88.7% directional agreement and a 74.7% shared deviation ratio. Solution-level interventions showed the highest positional overlap at 80.3% but the lowest shared deviation ratio at 66.1%, suggesting that richer context changes how TSD varies more than where it emerges. Interpretation: these shifts cannot be reliably read as task knowledge.

• TSD concentrates on surface-form tokens. Restricting to token forms occurring more than 20,000 times under the positive solution-level intervention at τ = 0.01, the 18 highest-ranked forms were natural-language markers — maybe (97.4%), however (97.4%), therefore (95.9%), consider (95.4%), altern. (94.3%), earlier (94.2%), seems (93.6%), another (92.9%), since (91.8%), try (91.8%), think (91.6%), says (90.4%), wait (90.0%), check (89.6%), let (89.5%), here (89.4%), because (89.4%), now (89.2%) — all above 89%. The 18 lowest-ranked forms were digits, symbols, and notation — }{ (2.8%), ### (3.2%), 0 (5.0%), 9 (6.8%), 8 (7.2%), √ (7.2%), ∘ (7.3%), _ (7.3%), 6 (7.4%), 7 (7.7%), 5 (7.8%), θ (7.9%), frac (8.0%), { (8.0%), 4 (8.8%), 2 (8.8%), 3 (9.2%), _i (9.2%) — all below 9.3%. That is nearly an order-of-magnitude separation.

• Cal-OPD wins on average in both configurations. It reached 53.1 for Qwen3-4B-Thinking-2507 → Qwen3-1.7B and 69.0 for Qwen3-30B-A3B-Thinking-2507 → Qwen3-4B. That is +3.9 and +2.4 over the student baselines, and +2.3 and +3.1 over standard OPD. Cal-OPD achieved the best result on 7 of 12 benchmark–configuration pairs.

• The choice of probing intervention matters. With λ = 5, the evaluative-feedback set achieved the best average (53.1). Instruction interventions retained nearly 70% of the teacher–student discrepancy. Solution-level privilege caused the largest degradation, dropping to 49.0 while retaining only about 20% of the discrepancy — interpreted as over-filtering, because solution-induced TSD contains a larger task-relevant component.

• Calibration changes training dynamics. For the evaluative configuration, the retained-discrepancy ratio decreased from 65% to 52% during training, while the zero-advantage token ratio rose from 27% to 34%. Standard OPD expanded average response length from about 9.8K to 11.8K tokens, whereas Cal-OPD ended at only 9.3K after an initial decrease to 8.5K. Despite the extra teacher computation for estimating TSD, shorter trajectories made Cal-OPD approximately 1.26× faster to train.

• The relaxation factor λ has a sweet spot. Performance was optimal at λ = 5, retaining approximately 52% of the discrepancy. Increasing λ beyond 5 sharply degraded performance, eventually dropping 1.7 points below the λ = 1 baseline at λ = 80, by which point the estimated TSD region reduced retained discrepancy to roughly 20% and over-filtered task-relevant supervision.

• Gains are not just signal attenuation. The authors report a comparison against a TSD-threshold filtering baseline matched in retained teacher–student discrepancy, where Cal-OPD remained stronger.

Methodology in Plain English

The authors start diagnostically. For each token the student generates, they ask the teacher to score that same token several times — once with no extra context, and then again with different pieces of extra context added only to the teacher: generic instructions ("reason carefully" vs. "solve quickly"), a verifier's judgment (correct vs. flawed), a supplied answer (correct vs. incorrect), and a supplied solution (correct vs. unrelated). Because the problem, the student's partial answer, and the token being scored never change, any shift in the teacher's score must come from the extra context itself. That shift is what they call Teacher Self-Deviation.

They then measure three things about these shifts: how often they occur, whether positions affected by weak interventions stay affected under richer ones, and whether flipping the semantics (correct vs. incorrect) flips the direction of the shift. Finally, they rank token forms by how often they show significant TSD.

For the method, they take the estimated range of the teacher's own wobble — the lowest and highest shifts observed under the contrasting interventions — and expand it slightly by a relaxation factor λ to account for using only two probes. The calibrated training signal is then the teacher–student gap measured from the edge of that range rather than from the teacher's raw score. If the student's score already sits inside the estimated wobble range, the signal is set to zero; otherwise only the portion beyond the nearest boundary counts. This calibrated quantity replaces the raw teacher–student discrepancy in the standard OPD objective.

Experiments train two student models (Qwen3-1.7B and Qwen3-4B) from two stronger teachers (Qwen3-4B-Thinking-2507 and Qwen3-30B-A3B-Thinking-2507) on DAPO-17K filtered by Qwen3-235B-A22B-Instruct-2507, using verl on 8 NVIDIA H20 GPUs (4 for student, 4 for teacher) for 100 steps, then evaluate on AMC23, AIME24, AIME25, AIME26, HMMT26, and MATH500 using Avg@16 accuracy (16 sampled responses per problem, averaged binary correctness). Baselines are standard OPD, ExOPD, EOPD, Uni-OPD, and Privileged-OPD.

Why This Matters

The paper reframes a common assumption in distillation: that the teacher's per-token judgment is an equally trustworthy reference everywhere. It shows that a measurable fraction of the teacher–student gap is the teacher being unstable rather than the teacher being knowledgeable. This matters because privileged distillation — supplying solutions or hints to the teacher — is a widely used way of getting stronger supervision, and the paper's results indicate that this approach can actively hurt, with Privileged-OPD producing the lowest averages of all distillation methods tested.

The work also adds to a growing line of research questioning whether teacher supervision is uniformly reliable, complementing approaches that weight supervision by entropy, uncertainty, reasoning position, or outcome consistency.

Potential real-world applications:

• Training smaller, cheaper reasoning models for deployment in tutoring, homework help, or mathematical problem-solving assistants, where a larger teacher must transfer capability to a model that fits on modest hardware.
• Improving cost-efficiency of post-training pipelines: the reported 1.26× faster training, driven by shorter generated responses, matters where rollout length dominates compute cost.
• Diagnosing and debugging distillation failures in production — the TSD region gives an explicit per-token signal of "the teacher is unreliable here," which can serve as a monitoring or auditing tool.
• Any setting where a privileged or context-augmented teacher is used, such as training with reference solutions, retrieved evidence, or optimized system prompts, where the paper's findings indicate the privileged signal should calibrate rather than directly supervise.

Industry relevance: The method requires extra teacher forward passes to compute the intervention-based TSD estimate, but the authors report a net training speedup due to shorter trajectories, which makes it relevant to teams that already run separate teacher and student inference on fixed GPU budgets. The finding that solution-level privilege over-filters useful signal is directly actionable for anyone currently using reference solutions in distillation.

Future Directions

• Better probes for the TSD region. The method uses only two contrasting interventions as a finite probe, and λ is introduced explicitly to compensate. How the estimate behaves with more interventions, or with interventions chosen adaptively per token, is not resolved.
• Why solution-level privilege over-filters. The paper observes that solution-induced TSD contains a larger task-relevant component, so using it for calibration degrades performance to 49.0. Untangling which part of solution-induced variation is knowledge and which is wobble remains open.
• Extension beyond mathematical reasoning. All evaluation here is on AMC23, AIME24, AIME25, AIME26, HMMT26, and MATH500 with Qwen3 models. Whether TSD has the same surface-form concentration and the same insensitivity to semantics in code, open-ended writing, or agentic tasks is not reported.
• Interaction with other reliability-aware methods. The paper compares against reward extrapolation, entropy-aware, outcome-guided, and privileged baselines, but how calibration composes with position-weighted or uncertainty-based supervision schemes is not reported.

Target Audience

Researchers and engineers working on LLM distillation, reasoning-model post-training, and teacher–student compression will get the most from this paper. It is also relevant to practitioners using privileged or context-augmented distillation who need to know when extra teacher information helps versus hurts, and to anyone studying the reliability of supervision signals in large-scale training pipelines. Readers should be comfortable with token-level likelihood notation and policy-gradient-style objectives; those seeking only the practical takeaway can read the introduction, the main results table, and the analysis section.

Authors’ abstract

On-policy distillation (OPD) improves reasoning models by learning the token-level discrepancy between a stronger teacher and an on-policy student. However, this discrepancy does not purely reflect the capability gap between the teacher and the student: it also contains deviations arising from the teacher itself, which are consequently mixed into the observed teacher--student discrepancy and indiscriminately learned by standard OPD during training. This issue is further exacerbated by privileged OPD, where privileged information induces larger teacher-side likelihood shifts, thereby encouraging the student to learn more of the teacher's own deviation. We introduce \textbf{Calibrated On-Policy Distillation (Cal-OPD)}, which estimates the teacher's self-deviation region through positive and negative privileged interventions and calibrates the original teacher--student discrepancy by retaining only the component that lies beyond this region. Experiments on mathematical reasoning benchmarks show that, while retaining only about 52--65\% of the original teacher--student discrepancy as the optimization signal, Cal-OPD consistently outperforms standard OPD and its variants across model scales.