Distribution-Aligned Sequence Distillation for Superior Long-CoT Reasoning

AI summary

Overview

Research area: Machine learning / large language model post-training, specifically knowledge distillation for long chain-of-thought (CoT) reasoning.

Technical level: Intermediate (requires familiarity with supervised fine-tuning, KL divergence, temperature sampling, and exposure bias).

Scope: The paper proposes an enhanced sequence-level distillation pipeline — temperature-scheduled learning, divergence-aware sampling, and mixed-policy distillation — and uses it to train the open-source reasoning model DASD-4B-Thinking (student: Qwen3-4B-Instruct-2507; teacher: gpt-oss-120b) on 448K training samples.

Note: the supplied paper content is truncated. It covers the Abstract, Introduction, Preliminaries, and Sections 3–5 (temperature-scheduled learning, divergence-aware sampling, and the opening of mixed-policy distillation). Sections 6 (full training pipeline implementation) and 7 (comprehensive evaluation) are referenced but their content is not present in the provided text.

What This Paper Is About

Most open-source long-CoT distillation work treats the problem as ordinary supervised fine-tuning: collect teacher-generated responses, filter them with heuristics, and train the student on them. The authors argue this framing ignores the core principle of distillation — letting the student learn the teacher's full output distribution so it inherits the teacher's generalization ability — and that this omission creates three concrete failures: incomplete coverage of the teacher's sequence-level distribution, a mismatch between the teacher's distribution and the student's learning capacity, and exposure bias from teacher-forced training versus autoregressive inference. The paper's goal is to fix all three within the simple, tokenizer-agnostic sequence-level distillation setting, and to demonstrate the result with a 4B-parameter reasoning model.

Key Contributions

1. Temperature-scheduled learning. A two-stage curriculum that cold-starts the student on low-temperature (T=0.6), high-confidence teacher samples and then continues training on higher-temperature (T=1.0) samples that cover more of the teacher's modes — described as a "reverse" temperature annealing or easy-to-hard curriculum.

2. Divergence-aware sampling (DAS). A sentence-level distribution decomposition framework that classifies generated sentences into four types (Teacher Sentence, Student Sentence, Shared Sentence, Boosted Sentence) and shows that Teacher Sentences — where the teacher assigns high probability and the student low probability — correlate with correct answers. DAS then prioritizes training examples rich in such sentences.

3. Mixed-policy distillation. A lightweight final stage that re-generates responses with the trained student, truncates the student's prefixes, has the teacher complete the sequence, and fine-tunes on the teacher continuations that pass quality filters — combining off-policy and on-policy signals to mitigate exposure bias.

4. Open release. DASD-4B-Thinking plus a Mixture-of-Experts version, DASD-30B-A3B-Thinking-Preview, both derivative models of the Qwen family, released together with the training dataset on Hugging Face and ModelScope, using only 448K training samples.

Main Findings

• Final model performance: DASD-4B-Thinking scores 88.5 on AIME24 and 83.3 on AIME25 (highly challenging mathematical competition benchmarks), 69.3 on LiveCodeBench v5 (code generation), and 68.4 on GPQA-Diamond (doctoral-level scientific reasoning), reported as state-of-the-art among open-source models of comparable scale, and outperforming some larger models (e.g., 32B-scale).

• High temperature beats low temperature, but is harder to learn: With gpt-oss-120b as teacher and Qwen3-4B-Instruct-2507 as student, 50K math responses sampled at T=0.6 give AIME24 81.7 / AIME25 71.9, while 50K at T=1.0 give 83.1 / 76.1 — a +1.4 gain on AIME24 and +4.2 on AIME25. Figure 3 shows low-temperature data converges to lower loss faster, while high-temperature data yields a flatter, broader distribution covering more modes but higher loss.

• More high-temperature data has diminishing returns: Doubling the T=1.0 dataset from 50K to 100K samples leaves AIME24 unchanged at 83.1 and improves AIME25 only from 76.1 to 78.9, suggesting the student's capacity to absorb diverse teacher behavior is the bottleneck.

• Cold-start plus high-temperature training is best: 50K at T=1.0 with a cold start at T=0.6 reaches AIME24 85.2 / AIME25 81.3, beating all static-temperature baselines. With Qwen3-Next-80B-A3B-Thinking as teacher on 25K samples, cold start (T=0.6) plus T=1.0 gives 83.1 / 73.1 versus 79.0 / 71.3 for T=0.6 alone and 82.9 / 70.2 for T=1.0 alone (+4.1 on AIME24, +1.8 on AIME25 over the T=1.0-only setting).

• Multi-domain results also improve: With a 25K math + 10K code + 10K science mixture, T=0.6 alone gives AIME25 74.6, LCB v6 44.1, GPQA-D 65.5; T=1.0 alone gives 75.2 / 47.3 / 65.4; a doubled T=1.0 set gives 75.8 / 51.3 / 65.4; and cold-start (T=0.6) plus T=1.0 gives 77.5 / 51.0 / 66.4. The authors note the comparable LiveCodeBench result may reflect the small proportion of code data.

• DAS beats random sampling at equal budget: 50K Math + DAS at T=0.6 gives 83.3 / 74.2 versus 81.7 / 71.9 for random sampling; 50K Math + DAS at T=1.0 gives 85.0 / 79.2 versus 83.1 / 76.1 for random sampling, and surpasses even the 100K random-sampling run (83.1 / 78.9). With the Qwen3-Next teacher, 25K Math + DAS gives 82.5 / 71.9 versus 79.0 / 71.3 for random sampling.

• DAS transfers across domains and teachers: On the 25K math + 10K code + 10K science mixture, DAS gives AIME25 75.6, LCB v6 47.3, GPQA-D 65.7 versus 74.6 / 44.1 / 65.5 for random sampling. DAS data curated for the Qwen3-4B-Instruct-2507 student also generalizes to the Qwen3-30B-A3B-Instruct-2507 student, so re-sampling per student is unnecessary.

• Sentence-type analysis: Teacher Sentences receive higher probabilities in correct answers across models, including a replication on the open-source DeepSeek-Distill-Qwen3-8B; Shared and Student Sentences occur with low probability and have minor influence; Boosted Sentences show a potential negative correlation with test accuracy. Boosted Sentences cannot be identified before training, so the method focuses on Teacher Sentences.

• Exposure bias is measurable: Regenerating training data with the student at a maximum generation length of 1.5 times the teacher solution length shows that the student's cut-off rate rises with training-response length, confirming that longer teacher prefixes exacerbate exposure bias.

• Mixed-policy distillation helps slightly, masking hurts: From a baseline of 50K DAS at T=0.6 (83.3 / 74.2), 7.7K mixed-policy samples trained for one epoch without masking yield 83.3 / 74.8; masking student-generated segments and keeping only teacher completions drops to 80.8 / 72.3. The authors add 20K additional off-policy samples to balance the ratio of mixed-policy to off-policy data.

Methodology in Plain English

The authors start from a mathematical restatement of what SFT on teacher outputs actually does: sequence-level distillation minimizes the KL divergence between the teacher's distribution over whole responses and the student's, but because the space of possible responses is exponentially large, the approximation replaces the teacher's distribution with a single sampled response. That approximation reduces exactly to the standard SFT loss. From this, they conclude that the success of the paradigm depends on how well the sampled data represents the teacher's distribution — which motivates treating data selection as the central design decision rather than a heuristic afterthought.

Stage one addresses coverage. Sampling at a low temperature produces a sharp, peaked distribution of easy, consistent responses; high temperature flattens the distribution and covers more teacher modes but produces harder, noisier data. Rather than choosing one, they train the student first on low-temperature data to learn consistent patterns, then continue on high-temperature data. They validate this across two teacher models (gpt-oss-120b and Qwen3-Next-80B-A3B-Thinking) and across math, code, and science mixtures, and they track both response probability distributions (using the geometric mean of token-level probabilities as a per-response likelihood) and SFT loss curves.

Stage two addresses the mismatch between teacher distribution and student capacity. They generate responses with the distilled (already-trained) student on test prompts, split each response into sentences, and compute each sentence's probability under three models — the teacher, the pre-distillation student, and the post-distillation student — again as a geometric mean of per-token probabilities. Comparing these probabilities yields four sentence categories. Sentence-level rather than token-level comparison is deliberate: it works across model families with different tokenizers and vocabularies, unlike on-policy distillation which needs shared tokenization. They then correlate each sentence type's presence at a given sentence position with whether the final answer was correct, focusing on earlier positions where enough data exists. Teacher Sentences win, and crucially they can be identified before training since they only require the teacher's and the pre-distillation student's probabilities on teacher-generated text. DAS simply favors training examples rich in these sentences. This requires only per-token probabilities from the teacher (often exposed even by closed-source APIs) and the local student — not full-vocabulary logits, and not teacher probabilities on the student's own outputs, which proprietary models typically do not provide.

Stage three addresses exposure bias. They confirm it empirically by measuring how often student regenerations get cut off relative to teacher solutions, and finding the problem worsens with response length. The fix: take a small subset of training queries, let the trained student generate full responses, randomly truncate those prefixes, and ask the teacher to complete from the truncation point; keep only continuations passing quality filters, then fine-tune for a few steps. This is applied after the off-policy SFT phase, with the goal of both improving accuracy and encouraging more concise outputs.

Throughout, the pipeline inherits standard quality control from prior work, such as filtering truncated outputs and repetitive content.

Why This Matters

The paper reframes sequence-level distillation as a distribution-matching problem rather than a data-filtering problem, and shows that this reframing yields measurable gains with an order of magnitude less data than comparable open-source efforts (448K samples). It also demonstrates that a 4B model can match or beat models at the 32B scale on AIME24, AIME25, LiveCodeBench, and GPQA-Diamond, which matters for deployment cost. Practically, DAS needs only per-token teacher probabilities — often available through APIs — instead of full teacher logits, and works across mismatched architectures, vocabularies, tokenizers, and pretraining corpora, which removes two of the main barriers to logit-based distillation.

Real-world applications:

• Code generation assistants, where the reported 69.3 on LiveCodeBench v5 reflects the ability to solve competitive-programming-style problems on small, cheap-to-serve models.
• Mathematics tutoring and competition preparation, given the 88.5 / 83.3 on AIME24 / AIME25 (the latter being described as the more representative and challenging benchmark).
• Scientific question answering and expert assistance, given 68.4 on GPQA-Diamond, a doctoral-level benchmark.
• On-device or cost-constrained reasoning deployment, since the model is 4B parameters and the paper also previews a MoE variant (DASD-30B-A3B-Thinking-Preview) with sparse activation.

Industry relevance: the method lowers the labeled-data and teacher-access requirements for producing capable small reasoning models, which directly reduces serving cost, and the released model weights and dataset lower the barrier for organizations that cannot train from scratch.

Future Directions

• Scaling and evaluating the MoE variant. DASD-30B-A3B-Thinking-Preview is released but the provided content does not report its benchmark results, leaving open how the pipeline behaves at larger and sparsely activated scales.
• Systematic temperature selection. The authors state that optimal temperature combinations can be chosen based on evaluation performance and observed response probability distributions, implying a more principled scheduling rule remains to be derived.
• Bottleneck of student absorption capacity. The near-flat 50K-to-100K scaling at T=1.0 suggests the student's capacity limits gains from more diverse teacher data; how to absorb broader teacher distributions without simply adding data is unresolved.
• Tuning the mixed-policy stage. The ablation leaves open the correct ratio of mixed-policy to off-policy data (the authors added 20K off-policy samples to balance against 7.7K mixed-policy samples), and why masking degrades performance is diagnosed only briefly.

Target Audience

Machine learning researchers and engineers working on LLM post-training, distillation, or reasoning-model development, especially those with limited compute who need small models to inherit strong reasoning behavior from larger teachers. It is also relevant to practitioners who rely on closed-source teacher APIs, since DAS deliberately avoids requiring full teacher logits or teacher supervision on student-generated tokens. Readers without a background in knowledge distillation, KL divergence, and sampling temperature will find the middle sections dense.

Authors’ abstract

In this report, we introduce DASD-4B-Thinking, a lightweight yet highly capable, fully open-source reasoning model. It achieves SOTA performance among open-source models of comparable scale across challenging benchmarks in mathematics, scientific reasoning, and code generation -- even outperforming several larger models. We begin by critically reexamining a widely adopted distillation paradigm in the community: SFT on teacher-generated responses, also known as sequence-level distillation. Although a series of recent works following this scheme have demonstrated remarkable efficiency and strong empirical performance, they are primarily grounded in the SFT perspective. Consequently, these approaches focus predominantly on designing heuristic rules for SFT data filtering, while largely overlooking the core principle of distillation itself -- enabling the student model to learn the teacher's full output distribution so as to inherit its generalization capability. Specifically, we identify three critical limitations in current practice: i) Inadequate representation of the teacher's sequence-level distribution; ii) Misalignment between the teacher's output distribution and the student's learning capacity; and iii) Exposure bias arising from teacher-forced training versus autoregressive inference. In summary, these shortcomings reflect a systemic absence of explicit teacher-student interaction throughout the distillation process, leaving the essence of distillation underexploited. To address these issues, we propose several methodological innovations that collectively form an enhanced sequence-level distillation training pipeline. Remarkably, DASD-4B-Thinking obtains competitive results using only 448K training samples -- an order of magnitude fewer than those employed by most existing open-source efforts. To support community research, we publicly release our models and the training dataset.