Facts Don't Speak Louder than Words: The Behavioral Essence of Long-CoT Distillation

AI summary

Overview

Research area: Natural Language Processing / large language model post-training, specifically long chain-of-thought (Long-CoT) supervised fine-tuning and knowledge distillation.

Technical level: Intermediate (assumes familiarity with supervised fine-tuning, cross-entropy loss, chain-of-thought reasoning, and standard LLM benchmarking).

Scope: A controlled empirical study that tests the standard assumptions behind Long-CoT distillation data curation and shows that the choice of training loss must be matched to reasoning complexity.

What This Paper Is About

Long-CoT distillation is widely used to give language models strong reasoning abilities, but nobody has clearly established why it works. Current pipelines assume the distilled reasoning traces must be factually correct and must come from a broad, diverse set of prompts, so they use rejection sampling to gather correct traces at scale. This paper tests those assumptions directly and finds they are largely unnecessary — instead, it argues that what transfers is the behavior of structured, step-by-step reasoning.

Key Contributions

1. Challenging prevailing distillation assumptions: The authors empirically show that neither factual correctness of the reasoning traces nor broad prompt coverage is the dominant bottleneck for Long-CoT distillation in their controlled settings.

2. Proposing the Behavioral Hypothesis: They argue, and empirically support, that the essence of reasoning distillation is behavioral alignment — internalizing step-by-step methodology — rather than memorizing correct, diverse facts.

3. Isolating behavior from knowledge injection: By fine-tuning exclusively on problems the base model can already solve (Easy-444 and Easy-643 subsets), they obtain gains comparable to a broader Random-3k baseline, reducing the role of novel problem-specific knowledge.

4. Uncovering complexity-dependent optimization: They show the recently popularized probability-based loss (-p) helps on simple CoTs but degrades sharply on complex ones, while standard cross-entropy (-log p) scales robustly with complexity, and they explain this via gradient dynamics.

Main Findings

• Incorrect trajectories teach nearly as well as correct ones. Using 3,000 prompts from DAPO-Math-17K with 16 trajectories each (48,000 question-answer pairs), math-verify labeled 22,222 correct and 25,778 incorrect samples. Models trained only on incorrect trajectories reached 97.7% of the absolute accuracy of positive-trained models and retained 94.7% of the relative performance gain over base. This held across Qwen3, Qwen2.5, and Llama 3.1 families and both training objectives.

• The weak dependence on correctness extends beyond math. On MMLU-Pro (general reasoning), GPQA-Diamond (science), and LiveCodeBench-v2 (coding), positive and negative CoT distillation differed by roughly a 1% average margin. For example, Qwen3-4B-Base averaged 38.9 with negative trajectories under -log p versus 37.3 with positive; Qwen3-1.7B-Base averaged 23.9 (negative) versus 23.5 (positive) under -log p.

• Incorrect data helps when correct data is scarce. Starting from 512 correct trajectories for Qwen3-1.7B-Base and raising the positive-to-negative ratio from 1:0 to 1:5 raised average accuracy across five math benchmarks from 20.4% to 26.4%. The benefit shrank as the volume of correct data grew.

• Prompt diversity is not the bottleneck under a fixed trajectory budget. Holding total trajectories roughly constant while varying prompt counts from 500 (128 samples each) to 16K (4 samples each) produced nearly flat accuracy curves across four base models and five math benchmarks.

• Models do not rote-memorize flawed facts. Fine-tuning on negative samples improved accuracy even on the training set itself. On the training set, Qwen3-1.7B-Base reached 13.5 (+6.6) with negative data versus 14.3 (+7.4) with positive under -log p, and 15.7 (+8.8) versus 17.3 (+10.4) under -p.

• Gains come even from already-solvable problems. Fine-tuning on Easy-444 (Qwen3-8B-Base) and Easy-643 (Qwen3-14B-Base) — prompts the base models answered correctly at least 7 of 8 times — produced improvements closely matching the Random-3k baseline across math benchmarks.

• Loss choice interacts strongly with CoT complexity. Simple, Medium, and Complex datasets averaged 426, 1,337, and 9,062 tokens respectively. The -p loss beat -log p on simple CoTs but suffered pronounced degradation on the Complex dataset, where -log p scaled robustly. Word clouds shifted from task-oriented terms ("solution", "value") in Simple to meta-cognitive markers ("wait", "maybe", "alternatively") in Complex.

• The mechanism is gradient amplification. Because ∇θ(-log p) = (1/p)∇θ(-p), cross-entropy heavily penalizes low-confidence "hard tokens" while -p applies a uniform penalty and selectively under-learns them. Simple CoTs have few hard tokens (mostly noise), so -log p overfits; complex CoTs are dense with hard tokens that carry sequential structure, so -p fails to supervise them and coherence collapses. Consistent with this, after training on ~9K-token trajectories, -p-trained Qwen2.5-7B and Llama-3.1-8B-Instruct generated over 20K tokens on average, versus roughly 14K and 13K under -log p.

• A concrete example (Figure 6). For a telescoping-product problem, the negative sample hallucinated the starting index (k=2 instead of k=1), producing the wrong answer 503/2 instead of 502 — yet it still recognized the telescoping structure, unfolded the product step by step, and described the cancellation mechanism exactly as the positive sample did.

Methodology in Plain English

The authors set up tightly controlled fine-tuning experiments rather than proposing a new model. They take open-source base models (Qwen3-1.7B/4B/8B/14B-Base, Qwen2.5-7B, Llama-3.1-8B-Instruct) and generate reasoning traces from Qwen3-4B on the DAPO-Math-17K prompt set, then split the traces into correct and incorrect piles using an automatic math verifier. They fine-tune separately on each pile and compare. To test prompt diversity, they hold the total number of training trajectories roughly constant while varying how many distinct prompts those trajectories come from. To isolate behavior from new knowledge, they train only on problems the base model already solves. Finally, they build three datasets of increasing reasoning depth and compare two losses — standard cross-entropy and the plain probability-based objective — explaining the resulting performance crossing through the mathematical relationship between their gradients. Training used full-parameter fine-tuning with verl for 3 epochs (1 epoch for Simple-CoT, 15 epochs for the mixed-ratio experiments), batch size 128, learning rate 10^-5, on 8 NVIDIA A100-80G GPUs. Evaluation used averaged pass@1 (avg@k) with 16 samples at temperature 1.0 on AIME 2024, AMC23, Minerva Math, Olympiad Bench, and a 2,000-example subset of MATH, plus Evalchemy-based evaluation at temperature 0.6 on GPQA-Diamond, MMLU-Pro, and LiveCodeBench-v2.

Why This Matters

Impact on research: The paper undermines two assumptions that underpin most Long-CoT distillation data pipelines — that traces must be correct and prompts must be diverse — and reframes distillation as behavioral alignment. It also directly contradicts the emerging consensus that the -p loss is universally better for reasoning, showing the conclusion depends on trajectory complexity. This reframes data curation and loss selection as complexity-aware design choices rather than fixed best practices.

Real-world applications:

• Building reasoning models more cheaply, since scarce, high-quality correct data can be padded with incorrect traces.
• Reducing the cost of prompt collection by concentrating sampling on a small prompt set without losing performance.
• Choosing loss functions per training stage when models are trained on data of mixed reasoning depth.
• Diagnosing degenerate long-form generation (for example, runaway response lengths) as a symptom of under-supervised hard tokens.

Industry relevance: Rejection sampling and large-scale diverse data collection are expensive parts of post-training pipelines. The finding that flawed traces and narrow prompt sets work nearly as well suggests direct cost savings, and the complexity-aware loss guidance gives practitioners a principled rule for selecting objectives instead of defaulting to the newest loss.

Future Directions

• Determining whether the Behavioral Hypothesis holds at much larger model scales and trajectory budgets than those tested, since the paper notes prompt diversity was not the bottleneck "at the trajectory budgets considered."
• Extending the correctness and diversity findings to domains beyond those tested (the appendix covers science QA and alternative teachers, but the main study centers on math).
• Developing an adaptive or interpolated objective that captures the simplicity-level benefits of -p while retaining the hard-token supervision of -log p; the paper notes related work interpolates between the two losses based on data noise, and identifies trajectory complexity as another relevant factor.
• Clarifying how "hard tokens" should be identified or weighted in very long reasoning chains, including the stopping behavior that appeared to break down under -p.

Target Audience

Researchers and engineers working on LLM post-training, chain-of-thought distillation, and reasoning-model development who need to make practical decisions about data curation (correctness filters, prompt diversity) and loss selection. It is also useful for those studying supervised fine-tuning generalization, since the paper offers controlled evidence on what SFT actually transfers.

Authors’ abstract

While long chain-of-thought (CoT) distillation significantly improves the reasoning abilities of language models, its underlying mechanism remains poorly understood. Current pipelines generally rely on rejection sampling to gather reasoning traces across diverse prompts, assuming that factual correctness and broad problem coverage are indispensable. Surprisingly, we find that training exclusively on flawed reasoning trajectories or a restricted prompt set yields nearly comparable performance. Driven by this counterintuitive finding, we propose the Behavioral Hypothesis: the essence of Long-CoT distillation is behavioral alignment with structured reasoning patterns, rather than factual knowledge memorization. We then validate this by demonstrating substantial reasoning gains even when models are fine-tuned exclusively on previously mastered problems-effectively isolating behavioral alignment from novel knowledge injection. Having established that reasoning patterns are the true bottleneck, we investigate how to learn them across varying levels of complexity optimally. We reveal a critical interplay: while the recent probability-based loss excels at learning simple patterns, the standard cross-entropy loss is essential for scaling to complex CoTs. We believe these insights demystify CoT distillation and provide a principled foundation for the future development of reasoning models. Code is available on https://github.com/yuyongcan/Long-CoT-Distillation.