Rethinking Expert Trajectory Utilization in LLM Post-training for Mathematical Reasoning

AI summary

Overview

Research area: Post-training of large language models for mathematical reasoning — specifically how supervised fine-tuning (SFT) and reinforcement learning (RL) should consume expert trajectories.

Technical level: Intermediate. The paper assumes familiarity with SFT, RL fine-tuning, and standard math benchmarks, but the core framework and arguments are explained with equations that are not overly dense.

Scope in one sentence: This paper introduces the Plasticity-Ceiling Framework to decompose post-training performance into an SFT foundation plus subsequent RL headroom, and uses it to determine which training paradigm, transition timing, and data properties maximize the final reasoning ceiling.

What This Paper Is About

Two post-training approaches dominate math reasoning: SFT, which imitates expert prompt–solution trajectories, and RL methods like GRPO/DAPO, which use prompt–answer pairs and let the model self-explore. Recent "Synchronized SFT-RL" methods weave imitation loss directly into the RL loop and claim efficiency advantages, but they rely on limited SFT data (only about 46K), while industrial practitioners favor a sequential SFT-then-RL pipeline without formal guidance on when to switch or what data to use. This paper builds an analytical framework to answer those open questions systematically: which paradigm is best, when to transition from SFT to RL, and how data scale and difficulty shape the ultimate performance ceiling.

Key Contributions

1. The Plasticity-Ceiling Framework — a theoretical mechanism that decomposes the post-training performance ceiling (A_post) into the realized SFT performance (P_sft) and the subsequent RL plasticity (PL_rl = A_post − P_sft), enabling quantitative comparison of paradigms and scaling choices.
2. A systematic benchmark of training strategies — Pure-SFT, Pure-RL (GRPO, DAPO_d), four Syn-SFT-RL methods (LUFFY, SRFT, UPT, Prefix-RFT), and the Sequential SFT-then-RL pipeline — establishing the sequential pipeline as the rigorous standard for stability and performance.
3. Actionable scaling guidelines — linking SFT transition timing, data scale, and trajectory difficulty to the final reasoning ceiling, and identifying the Minimum Validation Loss of SFT as a reliable predictor of the ultimate ceiling.
4. Refutation of the "Less is More" hypothesis in the SFT-then-RL setting, showing that data scale determines the primary post-training potential while trajectory difficulty acts as a performance multiplier.

Main Findings

• Sequential SFT-then-RL is the superior paradigm. Pure-RL and stable Syn-SFT-RL methods (GRPO, DAPO_d, LUFFY, Prefix-RFT) rapidly reach around 71.5 points, surpassing Pure-SFT (69.8) within 25 exaFLOPs, but then plateau. Even with infinite RL compute, DAPO_d improves only by an additional 2.8 points to reach a ceiling of 74.3, while LUFFY gains only 1.5 points to reach 72.7. Pure-SFT reaches a peak of 76.9 points, and SFT-then-DAPO_d achieves the best result at 78.1 points.

• Syn-SFT-RL is unstable and prior-sensitive. SRFT shows performance fluctuations with a standard deviation 2.6× higher than the stable DAPO_d baseline and fails to converge stably from a saturated SFT checkpoint. UPT converges well on Qwen2.5-Math-7B but loses that advantage on general-purpose Qwen2.5-7B, where it even underperforms vanilla GRPO.

• Transition to RL at SFT saturation. Switching to RL prematurely during the Adaptive Regime is consistently suboptimal. On SFT889K, switching early at 69.8 exaFLOPs yields a ceiling of only 80.7 points, whereas extending training to the Stable Regime (698.4 exaFLOPs) boosts the ceiling to its peak of 84.0 points. For Easy102K, the peak ceiling instead shifts into the Mild Overfitting Regime, and training to 335.9 exaFLOPs causes a rapid ceiling decline from collapsed RL plasticity.

• SFT saturation also maximizes out-of-domain generalization. ARC-C improves by 10 points to surpass 90.0 at 104.8 exaFLOPs (1,080 steps), and GPQA-D increases by 30 points to peak at 57.0 after 1047.6 exaFLOPs (10,800 steps). Transitioning from the Stable Regime (SFT-10800-RL) reaches 94.0 on ARC-C and 57.0 on GPQA-D, versus 92.0 and 51.0 when transitioning early (SFT-360-RL).

• Data scale is the primary determinant of the ceiling. S1K reaches an SFT performance of 74.1 with only 2.3 exaFLOPs, matching similar levels that require 69.3 exaFLOPs on Uniform102K (P_sft = 74.5) and 174.6 exaFLOPs on SFT889K (P_sft = 74.1) — but S1K saturates prematurely. Uniform102K reaches 74.8 at 104.0 exaFLOPs, while SFT889K reaches 77.3 at 698.4 exaFLOPs. SFT889K also maintains an average PL_rl of 7.3, exceeding S1K and Uniform102K by 2.4 and 3.7 points.

• Trajectory difficulty acts as a multiplier. At a controlled size of 102K, Hard102K achieves a higher average SFT performance of 75.4 points, outperforming Easy102K (73.5) and Uniform102K (74.6) by 1.9 and 0.8 percentage points, while maintaining a comparable RL plasticity of 4.1 points.

• Minimum SFT validation loss predicts the final ceiling. Across diverse SFT data configurations, minimum validation loss shows a strong negative correlation with A_post (Pearson r = −0.92), and this holds on Llama3.2-3B (Pearson r = −0.96).

• Cross-model validation on Llama3.2-3B confirms the pattern. SFT-then-RL achieves 68.7 points, far above Pure-RL or Syn-SFT-RL baselines. Within the SFT-then-RL runs, the Stable Regime achieves a peak max P_post of 70.4 at x_sft = 532.5 exaFLOPs, outperforming the Adaptive Regime's best (65.5 points at x_sft = 177.5) by 5.9 points. Data scale remains primary (max P_post ranges from 25.0 on S1K to 70.4 on SFT889K), and Hard102K leads at 56.4 among the 102K variants.

Methodology in Plain English

The researchers treat SFT-then-RL as one continuous pipeline rather than two isolated stages. They define the final post-training performance ceiling as the performance the model would approach given unlimited RL compute, then split that ceiling into two measurable parts: how far SFT itself got (P_sft), and how much extra headroom RL can still capture (PL_rl). This gives them a vocabulary for reasoning about trade-offs — for example, pushing SFT further may raise P_sft but shrink PL_rl.

To populate the framework, they built SFT datasets from distilled DeepSeek-R1 outputs at different scales (S1K, Uniform/Easy/Hard102K, and SFT889K with around 889K unique samples) and difficulty levels, along with a 199-example validation set (Val-199). They fine-tuned Qwen2.5-7B on these datasets, tracked validation loss over training, and at various checkpoints launched RL using GRPO, DAPO variants, or Syn-SFT-RL algorithms, recording how performance evolved.

They divide the SFT trajectory into four regimes based on validation loss relative to its global minimum L_min, using tolerance thresholds of 0.02 and 0.1: Adaptive (still underfitting), Stable (loss within 2% of minimum), Mild Overfitting (loss rises but stays under 10%), and Severe Overfitting (loss rises by 10% or more). They then compare which regime produces the best final ceiling.

Evaluation uses a leakage-controlled benchmark set. They filter out benchmark prompts with over 0.8 cosine similarity against training data using Qwen3-8B-Embedding, leaving 2,157 unique problems from GSM8K (1319 to 1317), OlympiadBench (675 to 291), Minerva (272 to 262), MATH (500 to 237), and AIME24/25 (30 to 25). They also test out-of-domain generalization on GPQA-Diamond (198 prompts) and ARC-C (1,172 prompts). All results use pass@1 at temperature 0.7 and top-p 1.0, with Avg@16 for AIME24/25 and a maximum generation length of 8,192 tokens. Findings are replicated on Llama3.2-3B. Compute ran on 16 GPUs (SFT in §6.1), 8 GPUs (RL and Syn-SFT-RL in §6.1), and 128 Ascend 910B NPUs (RL in §6.2).

Why This Matters

Impact on research. Prior work typically evaluated SFT in isolation, comparing accuracy right after fine-tuning. This paper argues that the right lens is the whole post-training trajectory — data choices that look efficient for SFT can silently reduce the model's later capacity to improve under RL. Framing performance as a ceiling plus plasticity gives the field a way to reason about scaling decisions that previously rested on empirical intuition.

Real-world applications:

• Training specialized math and science reasoning models where teams must decide how much SFT data to collect before running expensive RL.
• Budget allocation for compute-constrained teams: knowing that premature RL wastes the headroom that a saturated SFT phase would unlock.
• Data curation pipelines that prioritize volume before difficulty, rather than chasing small "high-quality" sets.
• Using SFT validation loss as a cheap early-stopping signal to pick the RL launch checkpoint without running full RL experiments.

Industry relevance. The paper is directly relevant to organizations building large reasoning models, since it benchmarks the sequential pipeline that major labs already favor and quantifies when the synchronized alternatives actually fail. The finding that Minimum Validation Loss predicts the final ceiling gives engineering teams a low-cost proxy for deciding when to move from one expensive training stage to the next.

Future Directions

• Testing whether the Plasticity-Ceiling Framework transfers to other verifiable tasks such as logic and coding, where the paper notes the findings are currently untested.
• Extending the analysis to non-verifiable tasks like creative writing, where no automatic reward signal exists.
• Investigating whether the conclusions hold for alternative trajectory styles, such as Gemini-style or human-annotated reasoning traces, rather than the R1-style distilled trajectories used here.
• Refining the framework's estimator for the asymptotic ceiling and the alignment offset (epsilon) in cases where the SFT endpoint misaligns with the global RL scaling trend.

Target Audience

Researchers and engineers working on LLM post-training, particularly those building reasoning models for mathematics and related domains. It is also valuable for graduate students studying SFT/RL trade-offs, practitioners deciding how to allocate training compute and curate fine-tuning data, and anyone evaluating the claimed efficiency advantages of synchronized SFT-RL algorithms. Readers should already understand the basics of supervised fine-tuning and reinforcement learning from human or rule-based feedback; the framework itself is accessible without deep theoretical background.

Authors’ abstract

Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) dominate the post-training landscape for mathematical reasoning, yet differ fundamentally in their reliance on expert trajectories. To understand the optimal way to harness these trajectories for maximizing performance, we propose the Plasticity-Ceiling Framework. This framework empirically grounds the post-training landscape by decomposing the final performance ceiling into the foundational SFT performance and the subsequent RL plasticity (i.e., the maximum improvement via RL). Through extensive benchmarking, we establish the Sequential SFT-then-RL pipeline as the superior standard, overcoming the stability and premature convergence deficits inherent in synchronized approaches. Furthermore, we derive precise scaling guidelines: (1) Transitioning to RL at the Stable or Mild Overfitting Regime of SFT maximizes the final ceiling by securing a robust SFT foundation with substantial RL plasticity; (2) Refuting the ``Less is More'' hypothesis in SFT-then-RL scaling, we demonstrate that Data Scale determines the primary post-training potential, while Trajectory Difficulty acts as a performance multiplier; and (3) The Minimum Validation Loss of SFT serves as a reliable indicator for selecting the expert trajectories that maximize the ultimate performance ceiling. Our findings provide actionable guidelines for extracting maximum value from expert trajectories.