Self-Distillation with Verbal Regularization

AI summary

Authors: Jiazheng Li, Yawei Li, Junru Lu, Zhenyi Shen, Di Yin, Lin Gui, Xing Sun, David Rügamer, Xinpeng Wang, Yulan He (King's College London; LMU Munich; Munich Center for Machine Learning; Tencent YouTu Lab) arXiv: 2610.16354v1 [cs.CL], 12 Oct 2026 · Code: https://github.com/lijiazheng99/VRSD

Overview

Research area: Natural Language Processing — large language model post-training, specifically on-policy self-distillation (OPSD) and knowledge distillation.

Technical level: Advanced. The core intervention is simple (a system prompt), but the paper builds a formal affine-attenuation model of privileged-information log-probability shifts and proves entropy conditions.

Scope: The paper proposes VRSD, a method that uses teacher-only system prompts to regulate how a self-teacher exploits privileged information during on-policy self-distillation, and validates it across three models and four tasks on accuracy, output entropy, and reasoning faithfulness.

What This Paper Is About

On-policy self-distillation uses the same model as both student and teacher, with the teacher additionally conditioned on privileged information (PI) such as a verified solution. The student never sees the PI, but is trained to match the teacher's next-token distributions along its own sampled responses. The problem is that PI can push the teacher toward shortcut commitments to the supplied answer, concentrating the teacher distribution and sharply lowering its entropy, which the student then inherits. The paper's goal is to keep the benefit of solution guidance while preventing this premature-commitment effect, using only changes to the teacher's conditioning rather than the architecture, loss, or the PI itself.

Key Contributions

1. Teacher-only verbal regularization for OPSD. The paper introduces system prompts applied only to the self-teacher that instruct it to verify intermediate steps, consider alternatives, and reconsider earlier reasoning before committing — without modifying model architecture, distillation objective, or the privileged information.
2. An affine attenuation model of PI-induced shifts. The authors formalize prompting as attenuating the influence of PI on the teacher distribution, (u_t^{d}(v) = \gamma_t u_t(v) + \kappa_t) with (\gamma_t < 1), and derive conditions under which prompting counteracts teacher-side entropy collapse.
3. Entropy analysis with an exact expression and a local sufficient condition. Theorem 4.1 gives an exact entropy change as an integral of a covariance along an attenuation path; Corollary 4.2 shows that if (\mathrm{Cov}_{q_t}(u_t, \log q_t) > 0), then teacher entropy increases for attenuation factors sufficiently close to 1.
4. Empirical validation across models and tasks. VRSD improves Pass@1 over OPSD in 11 of 12 model–task settings (3.9 percentage points on average), generally maintains higher token entropy, and yields evidence of more faithful reasoning in causal-intervention tests.

Main Findings

• Pass@1 improvements. VRSD (multi) improves average Pass@1 over OPSD by 7.2, 2.8, and 1.6 percentage points on Qwen3-4B, OLMo-3-7B-Instruct, and Qwen3-8B, respectively — an average gain of approximately 3.9 percentage points. It achieves higher Pass@1 in 11 of 12 model–task settings.
• Consistency gains beyond mean accuracy. VRSD (multi) matches or exceeds OPSD in Maj@16 across all settings and improves average Worst@16 on every model.
• Works even when OPSD hurts. On Qwen3-4B SQL, OPSD's Pass@1 (69.8%) falls below the starting model's (71.3%), while VRSD (multi) reaches 81.0%.
• Higher student entropy. VRSD produces higher next-token entropy than OPSD in 19 of 24 comparisons across the two prompt variants. On Qwen3-4B SQL, OPSD yields (H(p) = 0.03), versus 0.49 for VRSD (single) and 0.54 for VRSD (multi), alongside the Pass@1 rise from 69.8% to 81.0%.
• Multi-prompt beats single-prompt and concatenation. VRSD (multi) exceeds concatenation and every individual prompt across all four tasks, improving over the strongest individual prompt ((d_k)) by 0.5 to 4.7 percentage points. Concatenation falls below OPSD on three tasks.
• Students adopt the encouraged reasoning patterns. Measured as changes over OPSD averaged across three models: on grading, "checks rubrics" rises +56.2 (single) and +41.5 (multi), and "quotes evidence" +68.2 and +53.2; on chemistry, "discusses all options" +23.5 and +43.7, "rules out options" +13.8 and +22.6, and "checks conclusion" +73.6 and +5.7; on SQL, "names schema tables" +36.4 and +30.5, "names join keys" +38.0 and +29.4, and "plans query clauses" +53.4 and +45.1; on Python, "discusses boundary cases" +19.0 and +7.6, and "checks conclusion" +15.2 and +5.9.
• More faithful reasoning. In truncation-based causal interventions on Qwen3-4B, reasoning gain (RG) averages 5.8 percentage points for VRSD (single) and 5.0 for VRSD (multi), versus 3.7 for OPSD. The larger RG is accompanied by a lower misleading-reasoning rate (MR) than OPSD in seven of eight comparisons across the two variants and four tasks. On grading, both variants more than double OPSD's RG (+5.2 for OPSD versus +11.9 and +14.4).
• Measured PMI attenuation. Estimated (\hat{\gamma}) values with 95% bootstrap confidence intervals lie below the no-attenuation baseline, indicating statistically significant attenuation of PI-induced log-probability shifts. In Appendix C, mean covariance under (q_t) at prefixes with (|u_t(y_t)| > 1) ranges from 0.43 to 0.58, with the covariance positive at 79%–91% of those prefixes.

Methodology in Plain English

The setup keeps the standard on-policy self-distillation pipeline and changes only one thing: what the teacher is told. During training, the student model generates its own response to a question without any privileged information. The same model, acting as teacher, then scores each position of that response while being shown a verified solution. The student is trained to match those teacher distributions using a generalized Jensen-Shannon divergence objective.

The paper's intervention adds a system prompt that the teacher — and only the teacher — receives. The prompt instructs it to check intermediate steps, weigh alternative answers, and revisit earlier reasoning before committing. The student never sees this prompt at training or inference time. Two variants are tested: a single composite prompt shared by all rollouts, and a multi-prompt variant that splits each question's rollouts across several aspect-specific prompts.

To explain why this helps, the authors model the prompt's effect as shrinking the log-probability difference between teacher and student that PI induces (a PMI-style shift), then analytically track how that shrinkage changes the Shannon entropy of the teacher distribution. They verify the predicted attenuation empirically and check the entropy condition on prefixes where PI strongly changes the sampled continuation. Experiments fine-tune Qwen3-4B, Qwen3-8B, and OLMo-3-7B-Instruct across four tasks with automatic verifiers (ASAP-SAS grading, SciKnowEval chemistry, Spider SQL, LiveCodeBench Python), using 8 rollouts per prompt, 32 prompts per batch, a 2048-token prompt length, no KL regularization, and early stopping after four consecutive non-improving validations. Evaluation samples 16 responses per problem at temperature 0.6 and top-p 0.95.

Why This Matters

Impact on research. The work reframes privileged-information supervision as a conditioning problem rather than a data or loss problem. Instead of filtering shortcuts out of training targets (as Purified OPSD does) or discarding the privileged teacher signal (as AntiSD does), VRSD keeps the full reference solution and regulates its influence through the model's own instruction-following ability. The affine attenuation model and the entropy condition give a formal handle on when prompting can preserve exploration, which is directly reusable by other self-distillation methods.

Real-world applications (mapped to the paper's evaluated tasks):

• Automated short-answer grading, where students benefit from rubric-checking and evidence-quoting behavior rather than answer memorization.
• Chemistry question answering, where the model reasons from mechanism and explicitly rules options in or out.
• SQL query authoring, where the model names schema elements and join keys and plans clauses explicitly.
• Python code generation, where the model considers boundary cases and verifies its own conclusions.

Industry relevance. Because VRSD adds no learned module, no extra loss term, and no inference-time overhead, it drops into existing OPSD training pipelines. That is attractive for teams training reasoning models on verifiable tasks, where privileged solutions are readily available during training but absent at deployment.

Future Directions

• Automating prompt selection. The current method relies on manually curated, task-specific prompts. Future work could automate prompt selection and adaptation across problems and training stages, especially as student capability evolves.
• Direct control of PI-induced shifts. Verbal instructions control attenuation indirectly through wording. The authors call for methods that estimate and adjust PI-induced log-probability shifts directly, using the entropy analysis to decide where and how strongly to attenuate privileged solutions.
• Adaptive regularization strength. The theory provides a condition for entropy increase but the attenuation factor (\gamma_t) is not set per-prefix; designing adaptive schemes guided by the covariance condition is a natural extension.
• Generalization beyond verifiable tasks. The paper's four tasks all have automatic verifiers and correct model rollouts as PI. Whether the approach transfers to tasks without reliable verifiers or verified solutions remains open.

Target Audience

Researchers and engineers working on LLM post-training, knowledge distillation, and on-policy reinforcement-style methods, especially those building reasoning models on verifiable tasks. It also suits readers interested in the theoretical link between supervision signals and output-distribution entropy, and practitioners looking for a low-overhead modification to an existing self-distillation pipeline. Some familiarity with KL/Jensen-Shannon divergence and entropy is helpful for the methodology and theory sections.

Authors’ abstract

On-policy self-distillation (OPSD) trains a large language model using supervision from a teacher instantiated from the same model as the student. The teacher is conditioned on privileged information (PI), such as verified solutions. The student, which does not observe PI, learns from the teacher's next-token distributions along its own sampled responses. However, PI can bias supervision toward shortcut reasoning with limited verification or reconsideration, accompanied by a sharp drop in the entropy of the output distribution. To address this challenge, we introduce self-distillation with verbal regularization (VRSD), which uses system prompts to regulate how the teacher uses PI, thereby encouraging rethinking and exploration to counteract premature commitment. Theoretically, we model prompting as attenuating the influence of PI on the teacher distribution and derive conditions for entropy preservation. Across three models and four tasks, VRSD improves Pass@1 over OPSD in 11 of 12 model-task settings, by 3.9 percentage points on average, while generally maintaining higher token entropy.