Enough is as good as a feast: A Comprehensive Analysis of How Reinforcement Learning Mitigates Task Conflicts in LLMs

AI summary

Overview

• Research area: Natural Language Processing, specifically post-training paradigms for large language models (supervised fine-tuning versus reinforcement learning) and training-free model merging.
• Technical level: Advanced. The empirical results are readable on their own, but Section 4 builds the argument through advantage-bound theorems, a convergence proof, and a convex-analysis argument (Bhatia–Davis inequality, Prokhorov's theorem, Krein–Milman theorem).
• Scope: A controlled empirical and theoretical comparison of how SFT-trained versus RL-trained LLMs behave when merged, using five tasks, four merging methods, three RL algorithms, and three base models.

What This Paper Is About

Model merging fuses the parameters of independently fine-tuned LLMs into one model without retraining or access to the original training data, but the merged model often loses performance because the parameters of different tasks interfere with one another (task conflicts). Existing merging research studies models that were already trained, and largely assumes they were trained with supervised fine-tuning (SFT). This paper asks whether the post-training paradigm itself — SFT versus reinforcement learning (RL) — determines how badly a model degrades after merging, and then explains why.

Key Contributions

1. Establishes that RL-trained models merge better than SFT-trained models. Across five representative tasks, four merging strategies, three RL algorithms (PPO, GRPO, Reinforce++) and three base models (Llama-3.2-3B, Llama-3.1-8B, Mistral-Small-3-24B), models trained with RL consistently preserve more of their original performance after merging than SFT counterparts.
2. Attributes the advantage to reduced task conflict, not to general robustness. Performance-landscape visualizations show both SFT and RL models tolerate random parameter perturbations equally well, but only SFT models degrade along the task-induced merge direction.
3. Proposes three mechanisms, each supported by theory and experiment: (1) on-policy training data produces smaller-magnitude gradient updates; (2) the RL objective adaptively shrinks updates as the advantage decays toward zero at convergence; (3) joint optimization over positive and negative samples yields unbiased, less conflicting task-specific parameter subspaces.
4. Provides a formal account of conflict reduction. The paper defines a "conflict norm" over element-wise products of task vectors and shows analytically that the expected conflict norm under RL is much smaller than under SFT.

Main Findings

• RL merging degrades far less than SFT merging. On unmerged models, SFT scores 61.9 (math), 60.5 (code), 63.9 (IF), 86.2 (puzzle), 52.8 (rank), averaging 61.5; GRPO-trained RL scores 64.6, 65.6, 90.0, 85.2, 55.7, averaging 72.2. After merging, SFT's average declines span roughly 18–22%, while RL's stay under 10% for most strategies.
• Puzzle is the most fragile task. SFT loses up to 65% on logical puzzles; RL-based models degrade less sharply there (for example, 24% under both TIEs and DARE) but the task remains the hardest to merge.
• The pattern holds across merging algorithms. With Averaging, SFT shows a 22% mean decline versus 17% for RL; with TIEs, RL falls only 7.1% versus 19% for SFT. Arithmetic and DARE follow the same pattern.
• The pattern holds across RL algorithms. Under TIEs merging, SFT models drop 28.7% on IF, while RL models lose almost nothing (GRPO -0.3%, PPO -2.5%, Reinforce++ -2.6%).
• The pattern holds across base models. Under TIEs merging, SFT models drop between -35.6% and -17.7% on IF, while RL-trained models range from -4.6% to +0.4%.
• SFT updates are entangled; RL updates are closer to task-orthogonal. In the performance landscape, random perturbations of matched norm barely change performance for either paradigm, but increasing the task-induced perturbation along Δθ noticeably degrades SFT models while leaving RL models stable.
• On-policy data yields much smaller parameter updates. The norm of Δθ on math/code/IF is 6.50 / 7.75 / 4.83 for SFT, 2.36 / 2.17 / 1.70 for RFT (rejection-sampling fine-tuning), and 0.78 / 0.71 / 0.64 for RL.
• RL's advantage shrinks toward zero as training converges. Theorem 1 bounds the expected absolute advantage by the square root of reward variance; Theorem 2 shows the expected absolute advantage tends to zero, so RL progressively down-weights updates while SFT keeps applying fixed-intensity updates.
• Negative samples matter. A controlled RL-Pos variant zeroing the advantage of all negative samples still beats SFT but underperforms full RL: 58.5 (math), 61.7 (code), 86.1 (IF), 68.8 average, versus full RL's 64.6, 65.6, 90.0, 73.4, and SFT's 61.9, 60.5, 63.9, 62.1. RL also suffers less cross-task merging degradation than RL-Pos under averaging and TIEs.

Methodology in Plain English

The authors fine-tune separate models for five tasks that can be automatically graded — math, code, instruction following, logical puzzles, and ranking — using either SFT or RL, then merge pairs of these models and measure how much each one loses. They repeat this with four merging methods (simple parameter averaging, TIEs, Task-Arithmetic, and DARE+TIEs), three RL algorithms (PPO, GRPO, Reinforce++), and three base models (Llama-3.2-3B, Llama-3.1-8B, Mistral-Small-3-24B), defaulting to Llama-3.1-8B with GRPO. Merging is always pairwise: two models are merged at a time, and for each task the reported number is the average performance of that task's model merged with every other task's model.

To find out why the gap exists, they use three probes. First, they visualize a two-dimensional performance landscape around a task's own parameters, with one axis along the merge-induced update direction and the other along a random perturbation of equal norm — if only the task-induced direction hurts performance, the problem is conflict rather than noise. Second, they track the size of parameter updates over training and the "conflict norm," defined as the L2 norm of the negative entries of the element-wise product of two tasks' task vectors. Third, they run a control experiment (RL-Pos) that keeps on-policy sampling and KL regularization but sets the advantage of all negative samples to zero, isolating the contribution of negative examples.

Why This Matters

Impact on research. Model merging has been studied almost entirely through the lens of the merging algorithm, with the training paradigm treated as a given. This paper flips that assumption and argues that how a model was post-trained is a first-order factor in whether it can be merged at all. It also gives the effect a mechanism rather than just a correlation, tying it to advantage decay, gradient magnitude, and the sign of parameter updates across tasks.

Real-world applications:

• Consolidating specialist checkpoints: Organizations that maintain separate math, code, and instruction-following models could merge RL-trained versions into one deployable model instead of serving several endpoints.
• Reducing serving cost: A unified merged model replaces multiple fine-tuned checkpoints, cutting memory and infrastructure overhead relative to keeping every specialist online.
• Multi-task post-training pipelines: Teams choosing between SFT and RL for downstream adaptation now have evidence that RL-trained checkpoints are the safer raw material if merging is planned later.
• Skill composition without retraining: Merging can combine capabilities when the original training data is unavailable or too expensive to re-collect, which fits privacy-restricted or proprietary-data settings.

Industry relevance. The paper's framing — "an underappreciated advantage of RL" and a "robust and scalable strategy for developing generalist models without retraining from scratch" — speaks directly to the cost calculus of frontier model development, where data access and retraining compute are the binding constraints. The involvement of Baidu researchers, alongside the Institute of Automation, Chinese Academy of Sciences, indicates industrial interest in this question beyond academic curiosity.

Future Directions

• Beyond pairwise merging. The experiments merge exactly two models at a time; whether the RL advantage compounds, saturates, or reverses when merging five or more task models is untested.
• Beyond verifiable-reward tasks. The theoretical treatment relies on bounded rewards in a verifiable setting (the analysis uses r ∈ {0, 1} and |A| ≤ 1/2). Whether the advantage-decay argument carries over to subjective or preference-based rewards is not established here.
• Opening the "why" further. The paper isolates three factors but does not report how much each contributes quantitatively — a decomposition of the total merge-degradation gap into on-policy data, adaptive objective, and negative samples would sharpen the explanation.
• Wider task and dataset coverage. Dataset sizes are not reported in the available content, and the five tasks are all automatically verifiable. Extending to open-ended generation, agentic, or safety-critical tasks is an open question.

Target Audience

Researchers and engineers working on LLM post-training, model merging, and multi-task adaptation — particularly those deciding between SFT and RL for downstream specialization, or those maintaining fleets of fine-tuned checkpoints they want to consolidate. Readers interested in the theory of advantage functions in RLHF-style training will find Section 4 and the appendices the most relevant part; practitioners who only want the empirical takeaway can read Sections 1 and 3 and the results tables.

Authors’ abstract

Model merging plays a crucial role in consolidating multiple specialized models into a single, unified model, especially in the era of large language models (LLMs). Recent research has primarily focused on developing strategies to enhance merging performance with the trained models, while the impact of training paradigms, such as supervised fine-tuning (SFT) and reinforcement learning (RL), on the effectiveness of model merging remains underexplored. In this study, we systematically explore the merging behavior of RL-trained LLMs compared to those trained with traditional SFT. Through comprehensive evaluations across five representative tasks, we find that RL significantly reduces task conflicts and results in less performance degradation after merging, making RL-trained models particularly well-suited for this process. To unearth the reasons behind the superior suitability of RL for model merging, we conduct extensive empirical experiments and theoretical analyses. Our findings highlight three key factors: (1) On-policy training data in RL control the gradient updates in a smaller magnitude, reducing the risk of overwriting existing knowledge for other tasks in the model. (2) The RL optimization objective, which favors ``\textit{enough is as good as a feast}", progressively reduces the magnitude and the number of conflict parameter updates as the model converges. (3) Joint optimization of positive and negative examples in RL steers the model towards an unbiased task-specific parameter subspace, ensuring robust performance while further preventing parameter conflicts.