Every Token Needs Its Own Teacher: Multi-teacher Adaptive On-policy Self-distillation

AI summary

Overview

Research area: Natural Language Processing / large language model post-training, specifically on-policy self-distillation (OPSD) for reasoning.

Technical level: Advanced. The paper assumes familiarity with knowledge distillation, KL divergence, and LLM post-training pipelines, though the central idea (each token gets its own teacher) is stated in accessible terms.

Scope: The paper argues that teacher supervision in on-policy self-distillation should be selected per token rather than per response, and introduces MTA-OPSD, a method that builds a pool of seven candidate teachers and assigns each token its own preferred teacher.

What This Paper Is About

On-policy self-distillation uses the same language model as both student and teacher, with the teacher additionally shown a privileged reference such as a ground-truth solution. Existing adaptive variants decide how much of that reference to expose only once per response, so every token in a rollout is supervised by the same teacher. The authors argue that teacher preference actually varies token by token within a single response, and they build a method that lets each token select its own teacher from a pool.

Key Contributions

1. A teacher-intervention pilot study that quantifies teacher preference at the token level. The study holds a student-generated prefix fixed, lets a candidate teacher write a short continuation at a chosen position, returns control to the student, and uses the resulting accuracy gain over pure student generation as a proxy for how well that teacher suits that position.

2. Two empirical selection rules derived from that study: majority voting over the sign of teacher advantage signals yields a reliable subset of teachers, and the teacher with the largest student-teacher KL distance within that subset tends to be the preferred one.

3. MTA-OPSD, described as the first token-level adaptive OPSD method. It builds a multi-teacher pool by varying reference truncation strategy (prefix, suffix, random-span) and exposure ratio, then performs two-stage per-token teacher selection: majority voting followed by KL-based selection.

4. Experiments on mathematical reasoning with Qwen3-1.7B, Qwen3-4B, and Qwen3-8B against OPSD, SDPO, and Paint, plus ablations isolating the multi-teacher pool, the assignment strategy, and each selection stage.

Main Findings

• Teacher preference varies across tokens within one response. For Qwen3-1.7B with intervention length k=256, the overlap of preferred teachers between regular-spaced token positions in the same rollout was only 48.44%, 50.00%, and 51.56% across the three position pairs — roughly 50%.

• Majority voting reliably filters teachers. The preferred teacher fell inside the majority subset in 77.94% (regular) and 88.09% (extreme) of cases for Qwen3-1.7B, and 87.11% and 90.68% for Qwen3-4B. For Qwen3-4B with k=256, average gain per teacher was +3.34% for the majority subset versus +1.88% for the minority subset (overall +2.34%).

• Large KL distance marks the preferred teacher. Selecting the farthest-versus-nearest teacher within the majority subset gave +2.42% versus +0.59% average gain for Qwen3-1.7B (Δ +1.83%) and +2.78% versus +1.21% for Qwen3-4B (Δ +1.57%).

• Best average performance at every model scale. MTA-OPSD reached averages of 57.0 on Qwen3-1.7B (base 51.1; OPSD 54.0; Paint 54.0; SDPO 49.4), 77.2 on Qwen3-4B (base 75.7; OPSD 76.4; Paint 75.6; SDPO 73.6), and 79.2 on Qwen3-8B (base 76.4; Paint 78.1; OPSD 77.6; SDPO 76.1). Average gains over the base model were +5.9, +1.5, and +2.8 respectively.

• Gains are strongest on the hardest benchmark. On AIME25, MTA-OPSD scored 41.1 (1.7B), 68.1 (4B), and 70.3 (8B), versus base-model scores of 33.6, 65.3, and 66.9.

• The pool alone is not the source of the gain. Against the best single teacher in the pool (prefix-0.75, average 56.2 on Qwen3-1.7B), MTA-OPSD scored 57.0. With the same pool, random per-token picking averaged 53.6 and averaging all teacher losses averaged 53.7, both below MTA-OPSD's 57.0.

• Both selection stages matter. Removing majority voting and using the farthest teacher gave an average of 52.9; using majority voting with nearest-teacher selection gave 53.3; majority voting with random selection within the subset gave 56.6; the full two-stage method gave 57.0.

• Higher reference exposure does not reliably mean larger KL. For Qwen3-4B with suffix truncation, only 49.1% of cases had a larger student-teacher KL at exposure ratio 0.75 than at 0.5 (50.9% for the 0.5 side) — evidence that exposure-ratio tuning alone cannot determine teacher suitability.

• Teacher assignments stay spread out. The authors report that selection frequencies are distributed relatively evenly across the pool rather than collapsing to one dominant teacher.

Methodology in Plain English

The starting point is a diagnostic experiment rather than a new architecture. The researchers take a prompt and a response the student model produced, pick a token position inside that response, and hand control briefly to one candidate teacher — a version of the same model shown a partial or full reference solution. The teacher writes a short continuation (64, 128, or 256 tokens), the student then finishes the response, and the team checks whether the final answer is correct. If this relay beats plain student generation more often, that teacher is judged a better fit for that position.

The study crossed 7 token positions with 7 teachers, over 64 prompts from DeepMath-103k, with 2 teacher continuations and 2 student relays per configuration — 75,264 generations in total.

Two patterns from that study became the method's two decision rules. First, at each token, look at the sign of each teacher's advantage — whether the teacher raises or lowers the probability of the token the student actually chose — and keep only the teachers that agree with the majority direction. Second, among those survivors, pick the teacher whose next-token distribution is farthest from the student's in KL divergence. The intuition offered is that the farthest teacher either reinforces the student's choice with a genuinely different distribution, or, when the majority opposes that choice, supplies a stronger corrective push away from a likely error.

Training then reduces to standard OPSD with the per-token selected teacher substituted for the single fixed teacher. Experiments use the veRL PPO trainer, full-parameter training on 4 H200 GPUs, a learning rate of 1e-6, global batch size 32, 300 training steps, checkpoint evaluation every 50 steps, and best Avg@12 reported.

Why This Matters

Impact on research. The paper redirects a line of work on adaptive reference exposure from the response level to the token level, and offers a reusable diagnostic — the student-teacher-student relay — for measuring teacher suitability without training. It also reframes the "sharp teacher distribution" problem in OPSD: instead of only softening supervision globally, it varies the supervisor per position.

Real-world applications.

• Post-training reasoning models for mathematics and quantitative domains, where the reported gains concentrate on the hardest benchmark (AIME25).
• Any distillation setup with a verifiable reference — code generation with unit tests, or agent training with environment feedback — where partial references can be constructed in multiple ways.
• Model compression and self-improvement pipelines where no separate, stronger teacher model is available or affordable.
• Curriculum and data-quality tooling: the pilot-study protocol can be repurposed to audit which reference formats actually help which parts of a model's output.

Industry relevance. The method requires no external teacher model and no architecture change — only multiple prompt variants of the same model and a selection step — which makes it relatively cheap to graft onto existing OPSD infrastructure. The reported +5.9 average gain for Qwen3-1.7B is the most striking result for small-model settings.

Future Directions

• Self-evolving teacher pools. The conclusion explicitly names exploring self-evolving teacher pools as future work; the current pool is a fixed set of seven truncation/exposure configurations.
• More stable multi-teacher training. Also named in the conclusion as open, implying that juggling multiple teacher distributions during training introduces stability issues the paper does not fully resolve.
• Better suitability signals than KL. Majority voting plus KL is a two-stage heuristic derived from correlations in the pilot study; whether a learned or more principled per-token criterion outperforms it is untested.
• Generalization beyond math and Qwen3. All experiments use Qwen3 models on AMC23, AIME24, and AIME25 with DeepMath-103K training data; other domains, other model families, and the cost of computing seven teacher distributions per token are not evaluated in the reported content.

Target Audience

Researchers and engineers working on LLM post-training, distillation, and reasoning-model training — particularly those already using or evaluating OPSD-style methods and looking for a way to reduce student-teacher mismatch without a separate teacher model. Readers who want the conceptual argument without the math will find the pilot study and the intuition in the discussion section accessible; reproducing or extending the method requires comfort with KL-based distillation objectives and PPO-style training infrastructure.

Authors’ abstract

On-policy Self-distillation (OPSD) has proven effective for post-training large language models by constructing a teacher conditioned on privileged references. However, existing approaches keep the teacher fixed either throughout training by consistently exposing it to the full reference, or within each response by adaptively varying partial reference exposure only across responses. In this paper, we argue that neither of them is sufficient. Instead, every token needs its own teacher, as the teacher preference can vary token by token within the same response. Through a teacher-intervention pilot study, we validate that the teacher preference differs across tokens, and reveal that a preferred teacher can be identified for each token through majority voting and KL-based selection. Motivated by these findings, we propose Multi-teacher Adaptive On-policy Self-Distillation (MTA-OPSD), the first token-level adaptive OPSD method. Our method constructs a multi-teacher pool consisting of candidate teachers, and enables every token to adaptively select its preferred teacher during training. Experiments with Qwen3 models on mathematical reasoning tasks show that MTA-OPSD consistently outperforms existing OPSD approaches with an average gain of up to $+5.9$ over the base model, highlighting the importance of adapting teacher supervision at the token level. The code is available at https://github.com/xingzhejun/MTA-OPSD.