Supervised Fine-Tuning for Reasoning Large Language Models: A Survey

AI summary

Overview

Research area: Natural Language Processing — post-training of large language models, specifically supervised fine-tuning (SFT) on long chain-of-thought (CoT) data for reasoning models.

Technical level: Intermediate. The survey assumes familiarity with standard language-model training objectives (negative log-likelihood) and with the SFT-then-RL post-training pipeline, but it explains each method family in conceptual terms before naming specific systems.

Scope (one sentence): A systematic review of recent work on reasoning-oriented SFT, organized through two lenses — methodological design (data-centric and algorithm-centric) and analytical investigation (data characteristics, optimization dynamics, and mechanistic insights) — plus an outline of open challenges.

What This Paper Is About

Reasoning LLMs are typically trained by fine-tuning on long chains of thought, but the literature on how to do this well has grown quickly and fragmented across data-construction papers, loss-modification papers, and empirical analysis papers. The authors argue that existing reviews either emphasize reinforcement learning (RL) or treat SFT only as a preliminary to RL, leaving no holistic account of recent SFT-specific innovations. This survey fills that gap by organizing the field into a single taxonomy, synthesizing scattered empirical findings, and charting future directions.

Key Contributions

1. A two-lens organizing framework. The survey reviews reasoning SFT through methodological design and analytical investigation, rather than treating methods and empirical findings as separate literatures.

2. A method taxonomy split into data-centric and algorithm-centric approaches. Data-centric work is covered under data augmentation and data refinement; algorithm-centric work is organized by optimization granularity — token-level, sample-level, distribution-level regularization, and integration with RL — unified under a generalized objective that adds token-level weights wₜ and sample-level weights αᵢ to the standard NLL loss.

3. A reorganization of analytical studies into three dimensions: data characteristics (CoT length, distribution, difficulty, diversity, reasoning pattern, data quantity), optimization dynamics (training duration, checkpoint selection, learning rate, model capability), and mechanistic analysis (what SFT actually teaches, memorization versus generalization, and interaction between SFT and RL).

4. An agenda of open problems, covering the tension between reasoning gains and knowledge acquisition, inference efficiency, RL-compatible SFT objectives, and the need for causal rather than correlational analysis.

The paper also includes a supplementary 2D map (Table 1) crossing optimization granularity with data heuristics, and an appendix reorganizing data-centric methods along a pipeline: problem sourcing → CoT trajectory synthesis → trajectory verification → data filtering and curation.

Main Findings

• Standard SFT has documented weaknesses. Vanilla behavior cloning on CoT data suffers from exposure bias, severe overfitting, and limited generalization, which is the motivation for the algorithmic innovations the survey catalogues.

• Token-level methods split into reweighting and hard selection. Continuous reweighting scales token updates by gradient consistency with the global batch, predicted probabilities, or output entropy; discrete selection keeps tokens with the highest training loss or largest parameter update norms, or uses cross-model loss discrepancies, semantic-aware response-to-prompt attention, counterfactual perturbations, or explainability metrics. SFTKey goes furthest by masking the intermediate CoT entirely and optimizing only answer tokens.

• Reweighting goals shift when SFT is an RL initialization. Sharp, loss-driven imitation suits standalone SFT, but risks distributional collapse that destroys exploration; methods aimed at RL cold-start instead use sequence-relative weighting, finite-temperature Gibbs distributions, or multi-level importance sampling (PEAR) to preserve alternative-token probability mass.

• Sample-level methods disagree on which examples to favor. FLOW upweights "easy" low-pre-training-loss samples to mitigate catastrophic forgetting, while BFT prioritizes "hard" sequences with continuous low-probability spans. Other work frames sample weighting as a way to simulate RL's distribution shift offline, or learns data mixtures via minimax bilevel optimization.

• Longer CoTs are widely reported as better, but not uniformly. CoT supervision is described as essential — models fine-tuned without CoT degrade or generalize poorly — and CoT length is reported to matter more than problem difficulty. However, one study argues for an inverted-U relationship with accuracy: both overly short and overly long CoTs are suboptimal, with the optimal length rising for harder tasks and falling as model capability improves.

• Distributional alignment can outweigh surface correctness. LLM-paraphrased CoTs, even those containing errors, are reported to outperform manually constructed data, and teacher–student alignment in long-CoT distillation is repeatedly emphasized.

• Complex reasoning patterns are not automatically better. Disrupting CoT structure harms performance more than corrupting surface content, yet stronger teachers do not necessarily distill better, teachers producing convergent deductive trajectories can beat divergent branch-heavy ones, and SFT on CoTs with "Aha moments" may degrade performance. Retaining erroneous steps may help models learn verification, backtracking, and error correction, while removing repetitive steps can cut token cost and improve performance.

• Data quantity matters less than data quality. Fewer than 1K examples are often sufficient, and as few as 10 examples work in extreme settings; increasing SFT data may even hurt performance in some cases, and VLM post-training has been found relatively insensitive to data quantity.

• Training duration has opposite optima depending on the goal. For standalone SFT, multi-epoch training is reported as important, and one study finds OOD performance first decreases then recovers during training. When SFT initializes RL, excessive optimization is undesirable: early-stage checkpoints can be better starting points, and smaller learning rates better balance in-distribution and OOD performance.

• Reasoning SFT is capability-dependent. Weak models struggle to learn patterns encoded in long CoTs, though in VLM reasoning short-CoT SFT disproportionately benefits weaker models; remedies include mixing short-CoT data, using smaller teachers, or training a medium-sized intermediate teacher.

• What SFT teaches is contested. Evidence for format imitation includes gains from 10 content-corrupted examples and the finding that models trained on golden shortest paths fail to acquire backtracking (unlike RLVR-trained models). Counter-evidence includes Pass@k improvements and composition of learned simple skills to solve unseen problems.

• "SFT memorizes, RL generalizes" is a contested claim. One study reports continuous OOD decline on Points24 and navigation tasks; others report that long CoTs, sufficient optimization, high-quality data, and strong base models enable generalization, that training on incorrect CoTs can improve OOD generalization, and that CoT supervision with diverse prompts can surpass RL in generalization.

• SFT-before-RL is conditional, and some reported gains were bugs. SFT can impair subsequent RL in VLM settings, hybrid strategies can underperform RL-only training, and SFT benefits RL mainly when the base model is weak. One study found that reported gains over SFT-then-RL stemmed from training-framework bugs; after correction, a simple SFT-then-RL pipeline performed best. Stronger standalone SFT performance does not necessarily imply better RL initialization, and validation loss or Pass@k with a large k is proposed as a more reliable checkpoint indicator.

Methodology in Plain English

This is a survey, so the method is literature organization rather than experimentation. The authors define the problem formally — SFT minimizes negative log-likelihood over targets that decompose into a rationale r followed by an answer a — then sort the literature two ways.

For methods, they choose optimization granularity as the primary split criterion because, in their words, it gives a unified mathematical lens: token-level weights, sample-level weights, global distribution constraints, and joint SFT-plus-RL objectives all fit into one generalized loss. They acknowledge that other cuts (by data source or by heuristic such as model confidence, sample difficulty, or divergence from a reference policy) are possible, and supply a cross-reference table mapping granularity against heuristics.

For analysis, they collect empirical studies and regroup their conclusions into three buckets — data properties, optimization dynamics, and mechanism — rather than summarizing each paper in isolation. The appendix re-sorts data-centric work a second time along a data pipeline (sourcing, trajectory synthesis, verification, filtering) and assigns each paper to the stage of its primary contribution when it spans several.

The survey explicitly restricts its scope to recent work, especially post-2025, and states that earlier data-construction literature is not systematically reviewed.

Why This Matters

Impact on research. The survey gives a shared vocabulary and a single map to a literature that has been split between data papers and optimization papers. Its most actionable research claim is evaluative: most SFT methods are judged on post-SFT accuracy alone, which the authors argue is an incomplete criterion given that better standalone SFT does not reliably yield better RL initialization. It also flags that many widely repeated conclusions (for example, about CoT length) remain correlational and may be confounded by reasoning structure, token budget, problem difficulty, or teacher distribution.

Real-world applications.

• Mathematical problem solving, which the introduction lists alongside code generation, agents, question answering, and robotics as core reasoning domains.
• Code generation, including the tool-integrated and execution-verified trajectories discussed in the appendix (ToRA, MathCoder, OpenCodeReasoning).
• Agentic and tool-use systems that depend on multi-step decomposition and intermediate verification.
• Question answering and robotics, both named in the introduction as application areas of reasoning capability.

Industry relevance. Modern post-training pipelines rarely use SFT in isolation; with the success of DeepSeek-R1, SFT-then-RL has become a common recipe. Findings about which checkpoints to hand to RL, how to avoid distributional collapse during SFT, and how to control inference cost from verbose CoT cloning bear directly on training budgets and serving latency.

Future Directions

• Harmonizing reasoning with knowledge learning. Confidence-based SFT strategies boost reasoning but reinforce pre-existing beliefs, suppressing learning signals for novel facts and under-represented domains; capability-aware optimization that balances reasoning refinement with factual acquisition is proposed as the fix.

• Inference-efficient reasoning trajectories. Cloning verbose expert demonstrations conditions models to generate excessively long CoTs, incurring computational overhead and latency; the field needs methods that compress long logical steps into dense trajectories without losing deductive soundness.

• RL-compatible SFT. Beyond re-evaluating existing methods under a full SFT-then-RL pipeline, the authors call for SFT objectives that explicitly preserve exploration potential, policy diversity, and reward-improvable behavior.

• Causal analysis of reasoning SFT. Controlled studies that isolate the causal effect of individual data properties, model capability, and training dynamics, disentangling factors such as CoT length from reasoning structure and teacher distribution.

Target Audience

Researchers and graduate students working on post-training for reasoning models who need a structured entry point into the SFT literature and a map of where the method families sit relative to one another. It is also useful for practitioners building SFT-then-RL pipelines who want to know which design choices (data difficulty, checkpoint timing, learning rate, distribution regularization) have empirical support and which remain contested. Readers looking for a historical account of early CoT data-construction work will not find it here, as the authors state that scope limitation explicitly.

Authors’ abstract

Large reasoning models have recently demonstrated remarkable capabilities in solving complex tasks, where supervised fine-tuning (SFT) on long chain-of-thought data serves as a crucial foundation for eliciting and enhancing their reasoning abilities. Despite rapid progress in both improving and analyzing reasoning-oriented SFT, the field still lacks a systematic survey that consolidates its fast-growing literature. To fill this gap, we present a comprehensive review of recent advancements in reasoning SFT, examining the literature through the dual lenses of methodological design and analytical investigation. First, we review methodological improvements across the SFT pipeline and categorize them into data-centric approaches and algorithm-centric innovations. Second, we reorganize analytical studies along three dimensions: data characteristics, optimization dynamics, and mechanistic insights. Finally, by synthesizing current research focuses and remaining bottlenecks, we outline promising future directions for reasoning SFT. We hope this survey deepens the understanding of reasoning SFT and paves the way for advanced reasoning models.