Fusion Training for Mathematical Generalization in Large Language Models

AI summary

Overview

Research area: Natural Language Processing — post-training and fine-tuning of large language models, specifically "Thinking Mode Fusion" (TMF), the technique used in the Qwen3 model family to put a long-reasoning mode and a concise-answer mode inside one model.

Technical level: Intermediate. The paper uses standard supervised fine-tuning (SFT) with LoRA on a small open-weight model, so no unusual machinery is required to follow the method, but it assumes familiarity with chain-of-thought reasoning, chat templates, and SFT data mixing.

One-sentence scope: The paper systematically varies two knobs in TMF training — the ratio of thinking to non-thinking data and the order in which the two data types are shown — and measures how each mode's accuracy and response length change on mathematical tasks.

What This Paper Is About

Thinking Mode Fusion lets a single language model answer simple questions concisely and also produce long step-by-step reasoning when asked, by training on two kinds of data distinguished by the presence or absence of reasoning content inside <think></think> tags. The paper asks a practical question that the original Qwen3 work left open: when both modes are trained into one model, how do the data ratio (how much thinking versus non-thinking data) and the training schedule (which mode comes first, or whether they are interleaved) affect final performance? To answer this, the authors build a controlled benchmark called Fusion Bench for mathematics and run a full grid of ratios and schedules.

Key Contributions

1. A systematic study of training schedules and data ratios in Thinking Mode Fusion, showing a trade-off between thinking and non-thinking modes in which increased non-thinking supervision degrades thinking performance, with a quantified negative correlation between the two.
2. A demonstration that training schedules modulate this trade-off: the optimal schedule depends on the data ratio, but the Mix (interleaved) schedule achieves strong performance in both modes when averaged across ratios.
3. The construction and release of Fusion Bench, a benchmark of thinking and non-thinking mathematical datasets at seven data ratios with three training schedules, enabling controlled and reproducible TMF analysis.
4. A quantified relationship between the two modes: a negative correlation of r = −0.58 with R² = 0.34 and a regression slope of −0.91, meaning a 0.01 gain in non-thinking performance is associated with roughly a 0.0091 drop in thinking performance.

Main Findings

• Both modes interfere with each other. Averaged over all data ratios, the Mix schedule reaches the highest accuracy on both test sets: 0.208 for thinking mode and 0.695 for non-thinking mode.
• Non-thinking supervision hurts thinking accuracy. The amount of thinking-mode training data is held fixed at 1,500 examples, so any change in thinking performance comes from adding non-thinking data. Thinking accuracy is lowest at the 1:4 ratio (highest non-thinking share) and highest at 4:1 — the Mean row shows thinking accuracy rising from 0.199 at 1:4 to 0.224 at 4:1.
• Non-thinking accuracy does not increase monotonically with more non-thinking data. Averaged over schedules, the non-thinking mode peaks at the 1:3 ratio (accuracy 0.705), not at the largest non-thinking ratio of 1:4 (0.701).
• Schedules matter strongly at a fixed 1:1 ratio. In the 1:1 setting, Mix gives the highest non-thinking accuracy, while T-NT gives the lowest and actually falls below the untuned base model's 0.683. For thinking mode at 1:1, NT-T gives the highest accuracy and T-NT the lowest, and all three schedules improve over the base model.
• The best schedule depends on the data ratio. For non-thinking accuracy, T-NT is best at 1:4 while Mix is best at 4:1, and Mix is best at 1:1. For thinking accuracy, Mix is best at 1:4 while NT-T is best at 4:1.
• Longer responses follow thinking data and late long-form training. Both modes produce their longest responses at the largest thinking ratio (4:1), and both produce their longest responses under the NT-T schedule when averaging over ratios.
• Near-total mode separation is not achieved. The authors report a clear negative correlation (r = −0.58, R² = 0.34, slope −0.91) between non-thinking and thinking accuracy, which they attribute to the two modes favoring competing objectives (concise answers versus long-form reasoning).
• Test-set length statistics illustrate the mode gap. Non-thinking test data have a mean length of 696 (min 298, max 1,863) while thinking test data have a mean length of 11,992 (min 1,840, max 69,268).

Methodology in Plain English

The authors start from Qwen3-4B, an open-weight model whose chat template natively supports optional <think> blocks. They pick one dataset for each mode: GSM8K (grade-school math with short solutions) for the non-thinking mode, and a subset of OpenMathReasoning for the thinking mode. From OpenMathReasoning's 3.2 million long chain-of-thought solutions, they sample 1,875 examples generated by DeepSeek-R1, keeping only problems with a pass rate above 0.96 under the 72B model, and split them 80:20 into 1,500 training and 375 test examples. GSМ8K's training set has 7,470 examples, and they subsample it down to whatever size the target ratio requires.

They fix the thinking data at 1,500 examples and sweep the non-thinking set size from 375 to 6,000, giving ratios of 1:4, 1:3, 1:2, 1:1, 2:1, 3:1 and 4:1 (thinking : non-thinking). For each ratio they build the supervised fine-tuning stream in one of three ways: T-NT (all thinking data first, then all non-thinking), NT-T (the reverse), or Mix (interleaved so the running ratio matches the target — one thinking example followed by about 1/ρ non-thinking examples when ρ ≤ 1, and about ρ thinking examples per non-thinking example when ρ > 1). This is a full factorial design of seven ratios × three schedules.

Training is continual SFT with LoRA (rank 16, alpha 64, dropout 0.0, no bias) applied to the q, k, v, o, gate, up and down projections. Optimization uses AdamW with a cosine schedule, learning rate 2×10⁻⁵, weight decay 0.001, 5 warmup steps, one epoch, per-device batch size 1 with gradient accumulation of 6, maximum sequence length 5,120, and results averaged over three runs. Runs used a single-node, single-GPU (NVIDIA H100) SLURM setup.

Evaluation uses 375 sampled test examples per mode. Generation is greedy, with max_new_tokens of 1024 for GSM8K and 5120 for OpenMathReasoning, and accuracy is exact match between the content extracted from the last boxed{...} tag and the gold answer. Both accuracy and response length are reported, since response length has been linked to reasoning quality in math problem solving.

Why This Matters

Impact on research. The paper turns an anecdotal engineering practice — "just mix reasoning and non-reasoning data" — into a measured trade-off with numbers attached. It distinguishes its setting from ordinary multi-task learning (math plus code) by noting that TMF unifies two response formats for the same task, and it shows this internal format conflict produces measurable interference. This gives follow-up work a concrete quantitative target (the r = −0.58 correlation and the −0.91 slope) to try to beat.

Real-world applications:

• Serving a single model that answers easy queries cheaply and reasons at length only when needed, reducing inference cost and deployment complexity.
• Deciding how much expensive long-reasoning data to collect relative to abundant short-solution data, given that adding more short-solution data degrades the reasoning mode.
• Choosing a training recipe (order/interleaving) for a target data budget, for example using T-NT when short-answer data dominates or Mix when reasoning data dominates.
• Reproducing or stress-testing TMF claims using the released Fusion Bench, which supplies both datasets, ratios and schedules.

Industry relevance. Teams that post-train reasoning models need to decide exactly these two things — data mix and training order — and the paper shows the decision is not free: the same data ratio performs best under different schedules, and the Mix schedule is the safest default when averaging across ratios. The finding that T-NT at 1:1 drops non-thinking accuracy below the untuned base model is a direct warning that a plausible-looking curriculum can be worse than doing nothing.

Future Directions

• Generalize beyond mathematics. The authors state their analysis is restricted to mathematical reasoning and that it is unclear whether the observed interference patterns transfer to domains such as code generation.
• Scale both modes jointly. The study fixes thinking-mode data at 1,500 examples to keep comparisons controlled; the authors note that regimes where thinking and non-thinking data scale together may reveal richer dynamics.
• Mitigate the trade-off. Quantifying the tension is not the same as resolving it — developing training strategies that explicitly reduce the conflict between concise answering and long-form reasoning remains open.
• Explain the mechanism. The paper measures the interference but does not identify why non-thinking supervision degrades reasoning; connecting the measured correlation to internal representations, and building on work showing that hybrid thinking LLMs achieve only partial mode separation, is a natural next step.

Target Audience

Researchers and engineers working on LLM post-training, reasoning-model training pipelines, or efficient inference with hybrid thinking modes. It is also useful for practitioners who must pick data mixtures and training orders under real budget constraints, and for those who want a small, reproducible benchmark (Qwen3-4B, LoRA, released data and code) for studying mode interference without large-scale compute.

Authors’ abstract

Thinking Mode Fusion (TMF) enables large language models to support both concise responses and long-form reasoning by unifying a non-thinking mode and a thinking mode within a single model. However, its training dynamics, including the \emph{data ratio} and \emph{training schedule} between the two modes, remain underexplored. In this work, we present a systematic study of TMF by analyzing the effects of the training schedule and data ratio between thinking and non-thinking modes. Focusing on mathematical problem solving, we construct a benchmark with multiple thinking-to-non-thinking data ratios and three training schedules. Our results reveal an asymmetric interaction between the two modes: increasing the ratio of non-thinking supervision reduces the accuracy of the thinking mode. We further show that different training schedules modulate this trade-off and that the optimal schedule depends on the data ratio. Finally, we quantify a negative correlation between non-thinking and thinking mode supervision, highlighting an inherent tension between these two modes. These findings provide practical guidance for designing effective TMF training settings. All code and data are released to support further research at: \href{https://github.com/caocongfeng/Fusion-Bench.git}{\textbf{Fusion Bench}}.