Bridged Self-Distillation for On-Policy Learning

AI summary

Overview

Research area: Natural Language Processing / large language model post-training — specifically on-policy distillation (OPD) from stronger reasoning models into smaller students.

Technical level: Intermediate. Readers should be comfortable with reinforcement learning from verifiable rewards (GRPO), token-level probability distributions, and the basics of knowledge distillation.

Scope: The paper introduces Bridged Self-Distillation (BSD), a single training framework that distills reasoning ability from any teacher which can supply text reasoning traces, and validates it across twelve teacher–student pairs spanning shared-tokenizer, cross-tokenizer, and closed-API teacher access.

What This Paper Is About

On-policy distillation lets a small student model learn from a stronger teacher by having the teacher score the student's own generated attempts, but this breaks down in two common situations: the teacher's solution may be too hard for a small student to imitate, and the teacher's token-level probabilities may be unusable (if it tokenizes text differently) or entirely unavailable (if it is only a closed API). The authors propose BSD, which sidesteps both problems by asking the external teacher for nothing more than a written reasoning trace, then using that trace as privileged context for a frozen copy of the student itself, which scores the student's own rollouts in the student's own token space.

Key Contributions

1. A unified distillation framework. BSD turns reasoning traces from any external teacher into token-level guidance via a privileged "self-teacher" — a frozen copy of the student conditioned on the teacher's trace — so no teacher logits or vocabulary alignment are required.
2. Selective, verifier-directed updates. The token-level guidance only reweights the penalties on unsuccessful rollouts, while a verifier (not the teacher) determines the sign of every update; successful rollouts are left exactly as in GRPO.
3. A twelve-pair empirical study. The method is evaluated with four students (Qwen3-0.6B, Qwen3-1.7B, Qwen3-4B, Qwen3-4B-Instruct) against three teachers (Qwen3-32B for shared tokenizer, Nemotron-3-30B-A3B for cross-tokenizer, GPT-5.6 Luna for closed API) on AIME 2024, AIME 2025, and HMMT 2025, beating scenario-specific baselines.
4. Analysis of the guidance signal. The authors show the guidance concentrates on high-entropy (uncertain) tokens, correlates with the size of the gains, and that richer privileged context produces larger improvements.

Main Findings

• Consistent improvement over the base model. BSD improves all twelve teacher–student pairs, with gains of 1.2 to 4.7 points, and achieves the best average score among distillation methods in eleven of the twelve pairs.

• Shared-tokenizer setting. BSD outperforms SFT for all four students by 3.3 to 8.6 points in average score, while SFT falls below the base model for three of them. BSD matches vanilla OPD for Qwen3-1.7B and scores up to 3.6 points higher for the other three — even though Qwen3-32B shares the students' tokenizer.

• Cross-tokenizer setting. Cross-tokenizer OPD (xOPD) gives only marginal gains of 0.5 and 0.6 points for Qwen3-4B and Qwen3-4B-Instruct, improves Qwen3-0.6B by 3.4 points, and decreases Qwen3-1.7B by 0.8 points. BSD improves all four by 1.2 to 4.4 points and beats xOPD for three of four students, with Qwen3-0.6B the only exception.

• Closed-API setting. SFT falls below the base model for three of four students, with Qwen3-0.6B dropping from 10.8 to 2.8. BSD beats SFT for all four students by 3.1 to 9.7 points and beats rubric-based OPD (ROPD) by 0.6 to 3.6 points.

• Concrete average scores (Table 1). Qwen3-4B goes from 59.2 base to 63.9 (shared), 63.6 (cross), and 61.9 (closed API). Qwen3-4B-Instruct goes from 44.4 to 45.8, 45.6, and 46.4. Qwen3-1.7B goes from 35.8 to 38.6, 39.2, and 40.0. Qwen3-0.6B goes from 10.8 to 13.6, 12.8, and 12.5.

• Stable training, unlike SFT. With the closed API teacher, whose responses average only 247 tokens versus about 4,000 for the other teachers, SFT progressively shortens student outputs; performance for Qwen3-4B-Instruct falls to nearly zero by the end of training. BSD and the OPD baselines keep response lengths relatively stable across all twelve pairs. SFT also shows catastrophic forgetting on two out-of-domain benchmarks (Appendix I), which BSD does not.

• Privileged context helps transfer, not just accuracy during training. In the ablation with Qwen3-4B and Qwen3-32B, GRPO, answer-conditioned RLSD, and full-trace BSD have nearly identical training-accuracy curves (average differences below 0.2), yet Avg@4 rises from 59.2 (base) to 60.6 (GRPO), 61.7 (RLSD), and 63.9 (BSD-trace).

• More complete reasoning in the context is better. BSD-hint scores 61.7, BSD-guide 62.5, and BSD-trace 63.9, despite similar training curves.

• Both design choices matter. Extending the weighting to successful rollouts (BSD w/ succ.) scores 61.4, below BSD-trace's 63.9. OPSD, which uses the self-teacher signal on every rollout without a verifier, scores 59.7 and declines from 63.3 at step 25 to 55.8 at step 150, while its benchmark responses shorten from 16.8k to 12.5k tokens.

• Guidance targets uncertain tokens. The Spearman correlation between per-token guidance magnitude and student token entropy on unsuccessful rollouts ranges from 0.83 to 0.90 across the four students. Mean absolute guidance is 0.072 (Qwen3-0.6B), 0.115 (Qwen3-1.7B), 0.114 (Qwen3-4B), and 0.037 (Qwen3-4B-Instruct), and stronger guidance lines up with larger gains (+2.2, +3.5, +3.9, +1.5 respectively).

• Qualitative self-correction. On an AIME 2024 problem where the base Qwen3-4B finds N = 5694 but misreads the division (answering 573 instead of 699), the BSD-trained student re-checks the step and reaches 699. Over eight continuations from the same erroneous prefix, the BSD student solves it six times versus four for the base model.

Methodology in Plain English

The setup. You have a big, strong teacher model and a small student you want to improve. Ordinarily, on-policy distillation asks the teacher to grade the student's own attempts token by token. That needs the teacher's probability for each token — impossible when the two models split text into different tokens, and often impossible through a paid API.

The trick. BSD asks the teacher for one thing only: a written solution to each training problem. These traces are generated once, offline, before training starts. Then, for each problem, the student generates its own attempt without seeing the teacher's solution. A frozen copy of the student reads both the problem and the teacher's trace and scores the student's exact same tokens.

Why that is legitimate. Because both the scoring copy and the student are the same model with the same tokenizer, the two sets of token probabilities are directly comparable. The difference between them, taken token by token, says: "after reading the teacher's reasoning, would I have been more or less likely to write this token?"

How that becomes a training signal. A verifier checks whether the student's attempt got the right answer, and GRPO turns the group of attempts into an advantage. If an attempt succeeded, training is unchanged from GRPO — the student is reinforced normally. If an attempt failed, the token-level difference is turned into a positive weight: tokens the self-teacher would have down-weighted get a stronger penalty, and tokens it would have up-weighted get a weaker one. The weight is clipped, and a coefficient controlling its strength decays linearly over training, so the guidance fades as the student improves. Because the weight is always positive, it only changes how hard a token is penalized, never whether it is rewarded or penalized — the verifier keeps that authority.

Practical details. Training uses LoRA adapters for 150 updates on 2,400 OpenThoughts math problems per teacher–student pair, with the self-teacher synchronized every 10 updates. The extra forward pass of the self-teacher takes 7 to 18 seconds per update, under 10% of update time. For the Qwen3-4B student, BSD costs 31.4 GPU-hours with the Qwen3-32B teacher, 48.7 with Nemotron, and 33.3 with GPT-5.6 Luna — compared with 29.4 for OPD, 70.8 for xOPD, and 83.0 for ROPD respectively.

Why This Matters

Impact on research. Prior work on on-policy distillation has splintered into separate machinery for each kind of teacher access: distribution matching for shared tokenizers, vocabulary mapping for cross-tokenizer teachers, and rubric or sampling-based feedback for black-box APIs. BSD shows that a single, much simpler interface — just text — is competitive with or better than all three specialized approaches, which reframes reasoning traces as the common currency for distillation.

Real-world applications.

• Cost-efficient deployment: compressing the reasoning ability of a large model into sub-4B models that are cheaper and faster to serve, which is exactly what the paper tests.
• Distilling from proprietary models: improving open-weight students using only API text, without needing logit access or licensing a model's internals.
• Cross-vendor model mixing: pairing a student with a teacher from a different family and tokenizer, a situation that arises whenever the best available teacher is not the same architecture as the deployment target.
• Legal and domain-specific reasoning assistants: the authors' funding comes from a German judicial language-model project, where verifiable-answer tasks and small deployable models are both relevant.

Industry relevance. The method requires no teacher logits at inference or training time, adds under 10% to update time, and is implemented on standard tooling (verl, vLLM). It also avoids the failure mode most visible in practice: SFT on a terse API teacher's outputs collapses response length and benchmark performance, which BSD does not do.

Future Directions

• Scaling the study. The authors note their experiments are limited to students of up to 4B parameters from one model family and to English competition mathematics; whether the gains hold for larger students, other model families, and other domains is untested.
• Removing the verifier dependency. BSD needs automatically checkable answers to set the direction of each update. Whether agreement between student and external teacher could serve as the verifier for open-ended tasks is left open.
• Mechanistic attribution. The analysis links guidance magnitude to gains only through correlations across students; it does not establish which tokens the guidance actually helps, or why.
• Adaptive use of the trace. Since richer privileged context (hint → guide → full trace) monotonically improved results, a natural question is how to select or synthesize the right level of detail per problem rather than always using the full trace.

Target Audience

Researchers and engineers working on LLM post-training, reinforcement learning with verifiable rewards, or model compression who need to transfer reasoning ability across model boundaries — especially those constrained to API-only teachers or mismatched tokenizers. It is also useful for practitioners deciding between SFT and on-policy methods, since the paper documents concrete cases where SFT degrades below the base model while BSD does not.

Authors’ abstract

On-policy distillation can transfer the reasoning capabilities of a strong teacher to a smaller student. However, distillation from a stronger teacher model faces two mismatches: its solutions may be difficult for a smaller student to imitate, and its token-level probabilities may be incompatible with, or even unavailable to, the student. Existing methods handle these settings through different forms of supervision. To this end, we propose Bridged Self-Distillation (BSD), a unified framework for distilling from any teacher that provides reasoning traces. The model to be distilled from, termed the external teacher, first provides a reasoning trace as privileged context, which can be generated in an offline fashion. The student generates rollouts without this context, while a copy of the student conditioned on it, serving as the self-teacher, scores the same rollouts in the student's own token space. BSD uses the resulting token-level guidance to adjust updates on unsuccessful rollouts, with a verifier providing the outcome signal. As the external teacher is queried only to produce text, the same training procedure applies to all teacher scenarios: shared-tokenizer, cross-tokenizer, and closed API teachers. Across these scenarios, BSD improves over the base model and outperforms baselines tailored to each scenario, offering a novel and unified pathway for distillation. Code is available: https://github.com/TUMLegalTech/Bridged-Self-Distillation