Weaken to Strengthen: Weakening-Induced On-Policy Self-Distillation for Large Reasoning Models

AI summary

Overview

Research area: Post-training and self-distillation for large language models, specifically mathematical and code reasoning.

Technical level: Advanced — the method builds on on-policy distillation, reverse KL objectives and log-probability "policy shift" contrasts, though the core idea can be grasped without the formalism.

Scope: The paper introduces Weakening-Induced On-Policy Self-Distillation (WISD), a method that improves a reasoning model by distilling it against a deliberately degraded copy of itself, using no teacher model, reference solutions, or environment feedback.

What This Paper Is About

Existing self-distillation methods for reasoning models make the model "stronger" during training by feeding it privileged information such as reference solutions, expert demonstrations or environment feedback, then training it to behave that way without the extra context. That dependence is a limitation when only unlabeled prompts are available.

WISD flips the construction: instead of building a stronger teacher, it takes the original model as the strong member and creates a weaker copy by quantizing or pruning its weights. The difference in next-token probabilities between the original and the weakened copy becomes a dense training signal on the model's own generated solutions.

Key Contributions

1. A reversed self-distillation formulation. The paper reframes prior on-policy self-distillation as a model pair where the original policy is the weaker member and the context-conditioned model is the stronger one, then reverses the roles: the original model is the stronger member and a weight-degraded copy is the weaker one, giving a token-level log-probability contrast that needs no privileged information.

2. Two training-free weakening operators. It instantiates the weaker self with low-bit quantization (symmetric round-to-nearest, group size 128, default bit width b = 3) and magnitude pruning (group-relative threshold τ = 1/6 with per-row ℓ2 norm restoration). Neither requires calibration data, and both preserve architecture and tokenizer so predictions stay comparable.

3. Broad empirical validation. It evaluates WISD on five reasoning models — Qwen3-4B, Qwen3.5-4B, OLMo-3-7B-Think, Qwen3.6-27B and Qwen3.6-35B-A3B — spanning dense and mixture-of-experts designs, on AIME 2024, AIME 2025, HMMT 2025, LiveCodeBench v5 and v6, using only 3,200 unlabeled prompts per domain.

4. Analysis of where the signal goes. Using the ThinkARM taxonomy on 64 Qwen3-4B mathematics trajectories, the paper shows the contrast allocates disproportionate gradient mass to "exploration" and "verification" roles, and that simple confidence-based baselines do not reproduce the gains.

Main Findings

• Average gains across all settings: quantization and pruning improve average accuracy over the original models by 4.0 and 5.5 percentage points, respectively, with positive gains in 21 and 23 of the 25 model–benchmark settings.

• Largest improvement on Qwen3.5-4B: pruning raises HMMT 2025 accuracy from 57.7% to 70.5% and LiveCodeBench v6 accuracy from 39.1% to 56.7%; on the five-benchmark average, quantization adds 11.3 points and pruning adds 13.4 points (55.95 to 67.23 and 69.34).

• Gains scale to larger dense and MoE models: pruning improves Qwen3.6-27B's five-benchmark average by 5.2 points (76.58 to 81.76), and quantization improves Qwen3.6-35B-A3B's by 5.9 points (71.82 to 77.69), where HMMT 2025 rises from 68.5% to 74.6% and LCB v6 from 61.8% to 69.5%.

• Not every configuration improves: on Qwen3.6-27B, quantization slightly lowers the average from 76.58 to 76.29, and on Qwen3-4B both variants (57.93 and 58.60) beat the reported five-benchmark averages of OPSD (52.70) and SDPO (49.88).

• Weakening strength is non-monotonic: on Qwen3.5-4B / HMMT25, quantization accuracy is 56.04% at 8 bits, rises to 68.33% at 3 bits, then falls to 49.90% at 2 bits; pruning accuracy goes 60.94% (τ = 1/16), 69.38% (τ = 1/8), 70.52% (τ = 1/6), 54.79% (τ = 1/4), 49.58% (τ = 1/2). Both sweeps show intermediate damage works best.

• Other weakening methods also work, but not all: on Qwen3-4B, Smooth W4A4 (75.3 AIME24 / 55.2 LCB v5), Wanda (74.4 / 55.6) and SparseGPT (73.5 / 55.7) yield gains, while GPTQ at 3-bit (72.0 / 53.4) and plain W4A4 (70.0 / 53.0) do not give consistent gains over the 73.4 / 53.0 original.

• Confidence reinforcement alone is insufficient: "self-enhancing" (72.8 / 53.6) and "temperature weak" with T = 2 (73.1 / 53.0) stay close to the original model, while RTN 3-bit (75.4 / 56.4) and magnitude pruning (75.8 / 55.7) move further.

• The signal concentrates on reasoning-critical roles: quantization and pruning allocate about 22% of the logit-gradient signal to exploration and 11% to verification, versus token shares of 14.8% and 8.1% for those roles.

• Qualitative effect: case studies show the contrast reinforcing a coverage-calculation digit and the negation "not," and an intermediate 0 used to exclude an invalid root.

Methodology in Plain English

Start with an already capable reasoning model and a set of unlabeled prompts. Let the original model generate four responses per prompt, producing 12,800 cached trajectories per domain (3,200 prompts × 4, from Skywork-OR1-RL-Data for math and KlearReasoner-CodeSub-15K for code).

Next, make a damaged copy of that same model — either by rounding its projection weights to very low precision (3 bits by default, in groups of 128) or by deleting the smallest weights within each group (threshold 1/6 of the group maximum) and rescaling rows to restore their original norm. Embeddings, output heads, normalization and routing parameters are left untouched, and MoE experts are handled independently.

Both frozen models then score every token of the cached trajectories under the identical prompt and prefix. At each token position, the difference between the original model's log-probability and the weakened model's log-probability says which token the healthy model prefers over its degraded self. That difference is used to tilt the original model's next-token distribution into a target, and a student, initialized from the original weights, is trained by reverse KL to match that target.

Because trajectories and both scorers are fixed, the procedure is an offline version of the Lightning Weave DOPD recipe: learning rate 10⁻⁶, global batch size 64, distillation coefficient α = 2, 200 optimizer updates, BF16 student. The weakened model and the original are discarded at inference; only the trained student is used. Evaluation uses 32 samples per math problem and four per coding problem, at temperature 0.6 and top-p 0.95.

Why This Matters

Impact on research: The paper removes a practical precondition of on-policy self-distillation — the need for reference solutions, demonstrations or reliable environment feedback — and shows that a strictly worse model can still be a useful teacher when the contrast is structured. It connects decoding-time ideas like contrastive decoding (DoLa, DCD, PruneCD) and diffusion autoguidance to training-time distillation, and it demonstrates the effect across dense and MoE architectures up to Qwen3.6-35B-A3B.

Real-world applications:

• Math and science tutoring assistants that must improve on competition-style problems where gold solutions are expensive to obtain.
• Code-generation copilots tuned on private or proprietary repositories where no reference implementations or test harnesses exist for training.
• On-premise or regulated deployments where an external teacher model cannot be called and only the customer's own prompts are usable.
• Low-resource domains lacking verified answer keys or reward models, where unlabeled prompts are the only available training data.

Industry relevance: The method needs only a checkpoint and unlabeled prompts, with no calibration data and no reward model, and the weakening step is a few lines of weight manipulation. That makes it cheap to add to an existing post-training pipeline, in contrast to methods like GRPO that require verifiable outcome rewards.

Future Directions

• What makes a good weakening operator is unresolved. GPTQ and W4A4 fail where RTN and Smooth W4A4 succeed at similar bit widths, so bit width alone does not predict training value; the paper does not explain why.
• Choosing weakening strength automatically. Accuracy peaks at intermediate strength (b = 3, τ = 1/6) and collapses beyond it, so a per-model selection rule rather than a fixed default would matter for new architectures.
• Extending beyond quantization and pruning. The paper's framing is agnostic to how the weaker member is made; other training-free degradations, and combinations of operators, remain untested here.
• Combining with verifiable rewards. The comparison table positions WISD against GRPO, which uses outcome rewards that WISD does not; whether the dense contrast signal and sparse verifiable rewards are complementary is not explored in the reported content.

Target Audience

Researchers and engineers working on post-training and reasoning-model distillation, particularly those who already understand on-policy distillation and are deciding what supervision signal to use. It is also useful for practitioners with only unlabeled in-domain prompts and no teacher model, reference solutions or reward model, who want a concrete, low-overhead recipe with reported hyperparameters and benchmark numbers. Readers new to distillation will find the core intuition accessible, but the method section assumes comfort with reverse KL objectives and token-level log-probability ratios.

Authors’ abstract

Self-distillation offers a way to improve reasoning models when a suitable external teacher is unavailable. However, common approaches rely on privileged information to construct a stronger self-teacher, limiting their use when only unlabeled prompts are available. In this paper, we introduce Weakening-Induced On-Policy Self-Distillation (WISD), which instead constructs a weaker counterpart of the original model and uses the contrast between the two as a training signal without requiring privileged information. We explore two ways to construct the weaker model: low-bit quantization and magnitude pruning. We evaluate WISD across diverse architectures and scales on mathematical and coding reasoning tasks. Across these benchmarks, quantization and pruning improve average accuracy over the original models by 4.0 and 5.5 percentage points, respectively, achieving state-of-the-art performance even without privileged information. Notably, pruning raises Qwen3.5-4B's accuracy from 57.7% to 70.5% on HMMT 2025 and from 39.1% to 56.7% on LiveCodeBench v6. Quantization raises Qwen3.6-35B-A3B's accuracy from 68.5% to 74.6% and from 61.8% to 69.5% on the same benchmarks, respectively. These results establish WISD as a novel and effective self-distillation framework for large reasoning models.