Are we Merging the Right Models? Impact of Expert Training Duration on Model Merging for LLMs

AI summary

Overview

Research area: Multi-task model merging for large language models, specifically how the training duration of domain experts interacts with the choice of merging algorithm.

Technical level: Intermediate. The paper is readable for someone familiar with fine-tuning and basic weight-space merging, but the bias-variance-covariance decomposition and mode connectivity analysis assume some background in deep learning theory.

Scope: A systematic empirical study across three Qwen 3.5 model sizes (0.8B, 2B, 4B), five task domains, five merging methods, and eight expert training checkpoints ranging from 0.25x to 5x each expert's optimal validation checkpoint.

What This Paper Is About

Model merging combines separately fine-tuned expert models into one multi-task model by operating directly on weights. The standard convention is to freeze each expert at its own best validation loss before merging. This paper tests that convention by deliberately training experts well past their individual optimum and measuring what happens to the merged model — and finds that the right answer depends heavily on which merging method is used.

Key Contributions

1. A systematic study of how expert training duration affects merged decoder-only LLM quality, spanning three model sizes, five domains, and five merging methods, with checkpoints saved at {0.25, 0.5, 0.75, 1.0, 1.5, 2.0, 3.0, 5.0} × T* (the per-expert validation optimum).
2. The finding that the optimal training duration is method-dependent: Simple Averaging peaks with undertrained experts, Task Arithmetic and Greedy Soup sit in an intermediate regime, and sparsification-based methods (TIES, DARE+TIES) peak with overfitted experts.
3. A bias-variance-covariance decomposition and mode connectivity analysis explaining the pattern, drawing a parallel to random forests where high-variance individual learners are tolerated because the aggregation step suppresses their noise.
4. Practical guidance: with sparsification-based merging, practitioners should train each expert past its validation optimum rather than early-stopping, with the optimum shifting later as model size grows.

Main Findings

• Simple Averaging degrades with overfitting. It performs best in the undertrained regime (0.25–0.75 × T*) and loses 14–23 percentage points on average by 5 × T*, with losses up to 45 points on individual domains such as Math.

• Sparsification-based methods peak past the validation optimum. TIES-Merging peaks at 1.5–3 × T* for 0.8B and 2B and at 3–5 × T* for 4B, exceeding its T* score by 2.5–8.7 percentage points. DARE+TIES shows a similar but flatter profile, stable across durations with a slight preference for overfitted experts.

• Task Arithmetic occupies an intermediate regime. It degrades on some domains (Math peaks at 0.25 × T*) but tolerates moderate overfitting on others. It applies a global scale λ to task vectors, which partially attenuates overfitting noise but cannot resolve sign conflicts.

• Greedy Soup favors undertrained experts at small sizes. For smaller LLM sizes it tends to pick undertrained experts and is more balanced at 4B. Its curve is less smooth because the greedy selection includes different expert subsets at each training duration.

• Mechanism: interference resolution acts as variance reduction. Simple Averaging weights every parameter equally, so noisy or specialized parameters in overtrained experts pollute the merged adapter directly. TIES trims small-magnitude updates and resolves sign conflicts; DARE stochastically prunes deltas and rescales survivors. These steps remove idiosyncratic components before averaging, mirroring the random forest intuition (Breiman, 2001).

• Safety results echo the accuracy pattern. Simple Averaging peaks near T* (1 × for 2B and 4B, 2 × for 0.8B) but stays at or below 25% refusal even at 4B. Task Arithmetic peaks at 2 × for 0.8B (10%), reaches 30.5% at 3 × for 2B and 50% at 3 × for 4B. DARE+TIES peaks at 1.5 × for 0.8B and 5 × for both 2B and 4B. The 4B TIES merge reaches 68.5% refusal at 3 ×, more than 2.5 × higher than Simple Averaging at any duration.

• Greedy Soup is weak on safety. It peaks near T* (1 × for 2B and 4B at 12.5% and 23%, 2 × for 0.8B at 9%) and retains little refusal at extreme durations, because greedy selection optimizes accuracy-based validation scores and may actively exclude the safety expert when it hurts accuracy.

• Bias-variance-covariance decomposition explains the pattern. Past T*, bias drops (experts capture task-specific patterns) while variance and covariance rise (experts diverge). Simple averaging passes increased variance and covariance through to the merged model; sparsification filters out small or sign-conflicting values, reducing variance and covariance while preserving the low-bias signal.

• Overfitted experts stay in the same loss basin. The average loss barrier between expert pairs grows monotonically with training duration but remains moderate (below 0.35) even at 5 × T*, suggesting LoRA's low-rank constraint prevents experts from leaving the pre-trained basin.

• Best duration ranges vary by scale. Optimal ranges per method: Simple Averaging 0.25–1.0 × (0.8B), 0.25–0.5 × (2B), 0.25–0.75 × (4B); TIES 1.5–3.0 × (0.8B), 2.0–3.0 × (2B), 3.0–5.0 × (4B); DARE+TIES 2.0–5.0 × (0.8B), 0.25–5.0 × (2B), 3.0–5.0 × (4B); Greedy Soup 0.25–0.75 × (0.8B), 0.25–2.0 × (2B), 0.25–3.0 × (4B). Simple Averaging stays anchored to the undertrained regime regardless of scale.

• Every method beats its T score at some duration.* In the per-domain appendix tables, for every method and model size the best training duration outperforms the T* result. For example, at 4B, Best Greedy Soup reaches 61.1 average accuracy versus Expert @ T* at 59.2, and Best Task Arithmetic reaches 62.3.

• Task difficulty modulates sensitivity. Math shows the sharpest divergence: at 4B, Simple Averaging drops from 64.8% at 0.25 × T* to 20.3% at 5 × T*, while TIES improves from 60.3% at 0.25 × T* to a peak of 66.7% at 3 × T*. Code shows similar trends with smaller gaps because HumanEval is more saturated; Instruction Following and Multilingual are more stable.

• Partial agreement with prior work. The paper agrees with Horoi et al. (2025) that averaging-style methods favor undertrained experts, but diverges on sparsification: where that work reports TIES and DARE degrading with overtraining, this study finds they improve. The authors attribute the divergence to their larger LoRA rank (r = 16 vs. r = 8), larger decoder-only models with quantized adapters, and a constant learning rate.

Methodology in Plain English

The researchers fine-tuned separate experts on five domains using QLoRA — low-rank adapters at rank r = 16 on top of a 4-bit quantized base model, targeting all attention and MLP projection modules. They used a constant learning rate of 2 × 10⁻⁴ with a 100-step linear warmup, an effective batch size of 8, and the AdamW optimizer. The constant schedule with no decay keeps the learning rate active past the optimal validation checkpoint, which is what allows experts to overfit.

Domains and benchmarks follow the MergeBench setup (He and others, 2025):

Training datasets were subsampled to 10K training and 1K validation examples. For each domain and model size, the authors identified the validation optimum T* independently as the training step with minimum loss on that domain's held-out split (1K examples). Since T* differs per expert, all durations are expressed as multiples of each expert's own T* rather than absolute step counts. Training ran up to 5 × T*, with LoRA adapters saved at eight checkpoints.

Five merging methods were evaluated: Simple Averaging (equal-weight combination, w_i = 1/N), Task Arithmetic (task vectors scaled by λ ∈ {0.5, 1.0, 1.5} selected via grid search), TIES (density k = 0.5), DARE+TIES (density k = 0.5), and Greedy Soup (iteratively adds experts, keeping each only if it improves the held-out score of the merged model). Metrics are aggregated over the four accuracy-based domains; Safety is reported separately as a refuse-to-answer rate. For analysis, the authors applied the bias-variance-covariance decomposition from DiWA (Rame et al., 2022) and measured linear mode connectivity by interpolating θ(α) = αθ_A + (1 − α)θ_B for α ∈ [0, 1] and recording the loss barrier.

Why This Matters

Impact on research. The paper challenges a near-universal convention in the model merging literature — that experts should be frozen at their individual validation optimum before merging. It reopens a question recently raised by Horoi et al. (2025) for CLIP and T5 models and shows that for decoder-only LLMs fine-tuned with quantized low-rank adapters, the answer flips for sparsification-based methods. It also reframes merging method choice and training budget as a joint decision rather than two independent ones, and connects the empirical pattern to the same theoretical intuition that justifies growing trees deep in a random forest.

Real-world applications:

• Enterprise deployment of a single merged adapter replacing several task-specific models, reducing serving cost and latency.
• Fine-tuning pipelines for small open-weight models (below 8B) where customers want task-specific quality at a fraction of the cost of a larger general-purpose model.
• Training-budget planning: teams can skip early stopping when using TIES or DARE+TIES and redirect that compute into more training steps per expert.
• Safety-critical deployments, where the choice of merging method materially changes refusal behavior — the 4B TIES merge reached 68.5% refusal at 3 × T* versus a Simple Averaging ceiling at or below 25%.

Industry relevance. The study focuses deliberately on small models because enterprises commonly fine-tune below 8B to match or exceed task-specific quality of larger models at lower cost and latency (Belcak et al., 2025; Abdin et al., 2024). Deploying one merged model is typically cheaper than serving several specialized models. All authors are affiliated with Amazon Web Services, and the practical recommendation — train to 2–5 × T* with TIES or DARE+TIES for the Qwen 3.5 family — is directly actionable in production fine-tuning pipelines.

Future Directions

• Test whether the method-dependent pattern persists under full fine-tuning rather than QLoRA, since prior work finds full fine-tuning and LoRA respond differently to training budget.
• Repeat the study with learning-rate decay instead of a constant schedule, since decay regulates how sharply a model overtrains and the constant schedule may amplify the regime studied here.
• Extend to larger base models and other parameter-efficient fine-tuning methods beyond the fixed QLoRA rank of r = 16 on a 4-bit quantized base.
• Validate the bias-variance-covariance account as a proven mechanism rather than an explanatory hypothesis, and run non-deterministic methods (Greedy Soup and DARE's stochastic pruning) with multiple seeds instead of a single seed.

Target Audience

Practitioners building multi-task LLM deployments from small open-weight models, especially those running industrial fine-tuning pipelines below 8B parameters. Also useful for model merging researchers who want a systematic treatment of training duration as a variable, and for ML engineers deciding between merging methods and training budgets. Readers need working familiarity with fine-tuning, LoRA/QLoRA, and weight-space merging terminology to follow the analysis sections.

Authors’ abstract

Multi-task model merging combines separately trained expert models into a single model that handles all tasks without co-training. Standard practice merges experts at their optimal validation loss. We challenge this convention by systematically studying how training duration of domain experts affects the quality of the merged model. We fine-tune experts on five domains (Math, Code, Instruction Following, Multilingual, and Safety) across three model sizes (Qwen 3.5 0.8B, 2B, and 4B), saving checkpoints from 25\% to 500\% of the optimal training steps and evaluating five merging methods at each duration. Our findings reveal a striking method-dependent pattern: simple averaging degrades sharply with overfitting, while sparsification-based methods achieve their best performance well past the validation optimum. We formalize this through bias-variance decomposition analysis, drawing a parallel to random forests where averaging benefits from high-variance individual learners. These results suggest that training duration and merging method should be chosen jointly rather than independently.