DASH: Divergence-Adaptive Supervision Horizons for On-Policy Self-Distillation of Reasoning Models

AI summary

Overview

• Research area: Post-training of large language models for mathematical reasoning; specifically dense token-level supervision in on-policy self-distillation (OPSD) and its interaction with reinforcement learning with verifiable rewards (RLVR).
• Technical level: Advanced. The paper assumes familiarity with KL divergence, on-policy rollouts, advantage-style backward recursions, and policy-gradient gradient decompositions.
• Scope: The paper identifies that vanilla OPSD assigns a uniform coefficient to every token-level distillation loss and proposes DASH, a divergence-adaptive multi-step aggregation rule, validated on three mathematical reasoning benchmarks across three Qwen3 model scales.

What This Paper Is About

In on-policy self-distillation, a model generates its own reasoning rollout while a privileged version of the same model, which has seen a reference solution, supplies dense token-level distributional supervision at each visited prefix. The authors observe that vanilla OPSD weights every one of those local divergences with the same coefficient, so the objective cannot distinguish two rollouts whose local divergences have the same values but appear in a different temporal order. DASH's goal is to make those token-level weights depend on how the local divergences actually evolve along the generated sequence, without requiring any extra teacher or student forward pass.

Key Contributions

1. Identification of a temporal coefficient allocation gap. The paper shows that assigning the same weight to every local divergence makes the vanilla OPSD objective invariant to different temporal arrangements of the same divergence values, so supervision weights cannot adapt to temporal structure (Section 4.1).
2. The DASH method. DASH maps the gap between each local divergence and the sequence-level mean into an adaptive propagation gate, then uses a backward multi-step recursion to produce temporally conditioned token-level coefficients and an adapted effective supervision horizon (Section 4.2).
3. A structural reference via Proposition 1. A fixed-horizon gradient decomposition shows that order dependence appears in the trajectory score-function term through future-divergence coefficients G^D_{u+1} = sum_{t=u+1}^{H} d_t, whereas the direct distillation term retained by vanilla OPSD uses constant explicit coefficients of 1/T. The paper states explicitly that DASH does not estimate G^D, does not introduce score-function gradients, and does not perform future-to-past credit assignment.
4. Empirical validation with ablations. DASH obtains the highest score among the compared results in all nine benchmark–model settings and the highest macro-average at each model scale, supported by component ablations (Sections 5.3 and 5.4). The paper reports the gains require no additional teacher or student forward pass.

Main Findings

• Best in all nine settings: DASH obtains the highest score among the compared results in all nine benchmark–model settings and the highest macro-average at every model scale (Qwen3-1.7B, Qwen3-4B, Qwen3-8B).
• Matched OPSD improvement: Under the matched rerun protocol, DASH raises the four-seed OPSD average from 41.87 to 45.07 on Qwen3-1.7B, from 63.60 to 65.00 on Qwen3-4B, and from 64.80 to 66.40 on Qwen3-8B, corresponding to gains of 3.20, 1.40, and 1.60 points.
• Per-benchmark improvement: DASH improves over the matched OPSD rerun on every benchmark at every model scale. On Qwen3-1.7B the scores are AIME 2024 58.30, AIME 2025 45.80, HMMT 2025 31.10; on Qwen3-4B, 77.20 / 71.10 / 46.70; on Qwen3-8B, 78.90 / 71.40 / 48.90.
• Comparison to the strongest competing methods: PW-OPSD provides the highest competing macro-average at the 1.7B scale (43.43) and 8B scale (65.23), while OPSD provides the highest at the 4B scale (63.60). DASH exceeds the compared PW-OPSD result by 1.64 points on Qwen3-1.7B and by 1.17 points on Qwen3-8B, and exceeds the matched OPSD result by 1.40 points on Qwen3-4B. The paper notes this does not imply a statistical significance test against the single-run baselines.
• Fixed propagation is only part of the gain: Replacing the token-wise gates with a fixed coefficient λ ∈ {0.1, 0.3, 0.5, 0.7, 0.9} improves over vanilla OPSD in every tested setting, and the best fixed setting λ = 0.1 raises the macro-average from 41.87 to 43.63. DASH's 45.07 outperforms that best fixed setting by 1.44 points.
• Gate direction matters: An Inverse-gap variant with the same gap, sensitivity, and aggregation recursion but the opposite sign inside the gate function reaches a macro-average of 42.10 — only 0.23 points above vanilla OPSD, 1.53 points below the best fixed coefficient, and 2.97 points below DASH.
• Allocation, not just scale: A 2×2 factorial comparison separates relative coefficient allocation from average coefficient scale. Dynamic allocation improves performance by 2.40 points at the OPSD scale and 2.50 points at the DASH scale, while increasing the average scale contributes only 0.70 points under uniform allocation and 0.80 points under dynamic allocation.
• Sensitivity choice: Across κ ∈ {1, 2, 5, 10, 20}, every value outperforms the standard OPSD macro-average of 41.87, and κ = 5 achieves the highest macro-average of 45.07.
• Divergence choice matters substantially: Forward KL achieves the highest macro-average of 45.07, outperforming symmetric JSD and reverse KL by 6.84 and 3.60 points respectively, and performs best on all three benchmarks.
• Moderate vocabulary compression is tolerated, extreme compression is not: Retaining the teacher's top-100 tokens plus a merged tail yields 44.37 (0.70 points below full vocabulary and 2.50 points above standard OPSD), whereas retaining only the top teacher token plus tail drops the average to 34.83, a fall of 10.24 points from the main configuration.
• No extra forward passes: DASH reuses the teacher and student distributions that OPSD already computes.

Methodology in Plain English

The setup is standard OPSD. The student model sees only the problem and generates a rollout; the same model, additionally conditioned on a reference solution, acts as a frozen privileged teacher and provides distributions at every prefix the student visited. The local signal at each position is the forward KL divergence from teacher to student, d_t = D_KL(π_t^T || π_t^S).

Vanilla OPSD simply averages these per-token losses with a coefficient of 1/T.

DASH changes three things:

1. Clip the local signal. Each vocabulary-level contribution is capped at a threshold τ = 0.05, and the per-position signal r_t is the sum of those capped contributions.
2. Build a gate from the sequence-relative gap. For each position, DASH computes the gap g_t = r_t − r̄ against the sequence-level mean r̄, and converts it into a propagation gate λ_t = sg[σ(−κ g_t)] with sensitivity κ = 5, where the stop-gradient blocks gradients through the gate. Because of the negative sign, a below-average local signal produces a larger gate and an above-average one produces a smaller gate.
3. Aggregate backward. A recursion A_T = r_T, A_t = r_t + λ_t A_{t+1} runs from the end of the sequence to the start, and the objective minimizes (1/T) Σ A_t. This is equivalent to a weighted sum (1/T) Σ c_k r_k with c_1 = 1 and c_k = 1 + λ_{k−1} c_{k−1}.

The practical effect is that after a relatively small discrepancy the supervision horizon stays open longer, letting later signals propagate backward, and after an unusually large discrepancy the horizon closes sooner. Gradients flow only through the local signals.

Experiments use Qwen3-1.7B, Qwen3-4B, and Qwen3-8B Instruct, trained on OpenThoughts-Math-30K (29,434 mathematical reasoning examples with problem, reference solution, and final answer), evaluated on AIME 2024, AIME 2025, and HMMT February 2025, each with 30 problems, using Avg@12 with official answer keys. All rerun methods train for a 200-step budget with checkpoints saved every 20 steps and use "best-within-200-step reporting" rather than held-out validation selection. Runs use LoRA with rank 64 and scaling factor 128, learning rate 5×10⁻⁶, and global batch size 64; rollouts are sampled with temperature 1.1, top-p 0.95, and top-k 20. The main configuration uses a maximum student completion length of 1,024 tokens; 2,048 and 4,096 were also evaluated and yielded comparable performance, so 1,024 was used for computational efficiency. OPSD, DASH, and every DASH ablation use training seeds {0, 1, 2, 3} and report four-seed means. For Avg@12 evaluation, 12 responses per problem are sampled in thinking mode with temperature 1.0, top-p 1.0, and a maximum of 38,912 newly generated tokens.

Seven baselines are compared: Base, SFT, GRPO, OPSD, EOPD, AVSD, and PW-OPSD. Base, SFT, and GRPO are taken from Zhao et al.; OPSD, EOPD, AVSD, PW-OPSD, and DASH are rerun by the authors.

Why This Matters

Impact on research. The paper reframes dense token-level distillation supervision as an allocation problem rather than only a sparsity problem. It shows that the uniform coefficient profile of OPSD leaves temporal structure of the rollout unused, and that a sequence-relative gate combined with backward aggregation captures part of that structure at negligible additional training overhead. The motivation analysis in Figure 1 visualizes 2,500 of the 30k analyzed tokens, and the authors report a weak association between the current local divergence and the mean future divergence within student-generated trajectories.

Real-world applications:

• Mathematical reasoning assistants that must produce long, verifiable step-by-step derivations.
• Code generation and repair, where the paper notes program execution and unit tests serve as automatically verifiable signals in the broader RLVR paradigm.
• Post-training pipelines that must avoid expensive annotation, since the approach relies on automatic verification rather than human preference labels.
• Inference-cost-sensitive training, because DASH reuses distributions already computed by OPSD and adds no forward passes.

Industry relevance. The method is designed as a drop-in modification to an existing OPSD backbone — the authors note that OPSD, EOPD, PW-OPSD, and DASH use the same privileged-OPSD backbone while retaining their method-specific objectives or weighting rules. That makes it practical for teams already running on-policy distillation, and the 1,024-token default length keeps training within a reasonable compute budget. The three model scales tested (1.7B, 4B, 8B) span small-to-midtier deployment regimes, though the paper does not report results on larger scales in the provided content.

Future Directions

• Compatibility with outcome-level optimization. The provided content indicates Appendix E examines DASH's compatibility with outcome-level optimization through a complete GRPO grid and a cross-scale extension, but the results of those experiments are not included in the truncated text and are therefore not reported here.
• Estimating the trajectory term directly. Proposition 1 shows the exact gradient contains a score-function term weighted by future divergence-to-go, which DASH deliberately does not estimate. An open question is whether estimating or approximating that term would add further gains.
• Gate design beyond the tested sign mapping. Two adaptive mappings were tested (DASH and Inverse-gap); the space of gap-to-gate functions, and gate forms conditioned on richer context than the sequence mean, is unexplored here.
• Generalization beyond mathematics. All reported benchmarks are mathematical (AIME 2024, AIME 2025, HMMT February 2025); the paper mentions code generation as part of the RLVR landscape but does not report DASH results on code or other domains.

Target Audience

Researchers and engineers working on LLM post-training, particularly those already familiar with RLVR, GRPO, and on-policy distillation. It will be most useful to people implementing dense token-level supervision who want a low-overhead change to how per-token losses are weighted, and to readers interested in temporal credit assignment and advantage-style backward recursions in autoregressive generation. Readers without prior exposure to KL-based distillation objectives or policy-gradient derivations will find Section 4 and Appendix A demanding.

Authors’ abstract

Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models using automatically verifiable outcome signals, but these signals are typically sparse and at the sequence-level. On-policy self-distillation (OPSD) mitigates this sparsity by querying a privileged teacher at student-visited prefixes and providing dense token-level distributional supervision. Although this dense supervision alleviates signal sparsity, we find that standard OPSD still underexploits the temporal structure of the rollout. It assigns every local divergence the same coefficient, regardless of its position or the divergence sequence in which it occurs. In on-policy autoregressive generation, the same divergence magnitude can follow different discrepancy histories, reflecting different evolutions of the mismatch between the teacher and student. Since the local scalar alone cannot distinguish these temporal contexts, standard OPSD cannot adapt its token-level weights to the realized discrepancy sequence. To address this limitation, we propose Divergence-Adaptive Supervision Horizons (DASH). DASH maps the gap between each local distillation signal and the sequence-level mean to an adaptive propagation gate and then uses these gates to control backward multi-step aggregation. By doing so, DASH adjusts token-level supervision weights according to how local divergences evolve during generation. Experiments on three mathematical reasoning benchmarks across three model scales show that DASH improves over our matched vanilla OPSD reruns on every benchmark at all three scales. DASH reuses the teacher and student distributions that OPSD already computes, so the gains require no additional teacher or student forward pass. Code: https://github.com/DBtxy/DASH-OPSD