Separating Common and Unique Directions for Model Merging

AI summary

Overview

Research area: Machine learning — specifically data-free model merging (combining multiple fine-tuned checkpoints into one multi-task model using only weights, without task data or retraining).

Technical level: Advanced. The paper assumes familiarity with task vectors, singular value decomposition, and subspace projection, though its core idea — separating shared from task-specific update directions — is describable in plain terms.

Scope: The paper introduces CUSD (Common–Unique Subspace Decomposition), a data-free pre-merging step that splits each layer's task vectors into directions shared across tasks and directions dominated by a few tasks, merges only the shared part with any existing arithmetic merging rule, and scales the rest with a single coefficient.

What This Paper Is About

Existing arithmetic merging methods (Task Arithmetic, TIES-Merging, DARE, Consensus Merging) apply one uniform element-wise rule to every entry of a task vector, treating all parameter coordinates the same way. The authors show that task vectors, viewed in a shared directional basis, actually contain two distinct regimes: a broad set of low-dispersion directions that all tasks use similarly, and a small high-dispersion tail whose energy is concentrated in only some tasks.

Because a single arithmetic rule cannot serve both regimes well, the paper's goal is to separate them before merging — so that the shared directions can be aggregated normally while the task-dominated ones are handled separately rather than being mixed into the common update.

Key Contributions

1. A quantified two-regime structure in task vectors. In an HO-GSVD-derived shared basis, the authors measure per-direction energy dispersion across tasks and find a low-dispersion common bulk plus a high-dispersion, task-dominated tail. In the two motivating layers shown, the highest-dispersion decile accounts for 53% (Llama-3.2-3B, mlp.gate_proj) and 77% (ViT-B/16, mlp.c_proj) of total directional energy.

2. CUSD, a data-free pre-merging module. For each matrix-structured layer, CUSD builds a shared orthonormal basis via a regularized, symmetrized HO-GSVD-derived Gram-quotient construction, computes a standard-deviation-based dispersion score per direction, and splits directions into a common set and a unique set using a KDE-detected valley threshold (Scott's bandwidth). The downstream operator is applied only to the common components, and the unique residuals are summed and scaled by a coefficient β ∈ [0, 1].

3. Operator-agnostic, one-time decomposition. The decomposition depends only on the task vectors, not on the merging rule, so it is computed once per model and reused across Task Arithmetic, TIES-Merging, DARE, and Consensus Merging. The projected decomposition is exact and orthogonal: C_i + R_i = A_i with zero Frobenius inner product between the two parts.

4. Broad evaluation. 11 base models across four model families (CLIP ViT-B/32, ViT-B/16, ViT-L/14; RoBERTa-Base and RoBERTa-Large; MergeBench Llama and Gemma models) and four arithmetic merging operators, plus comparisons against TSV-Merge, Iso-Merging, WUDI-Merging, and ACE-Merging.

Main Findings

• Broad improvement: CUSD improves 42 of 44 model–operator combinations, with one tie and one small decrease, reaching up to 22.5 normalized points over the corresponding baseline (TIES-Merging on RoBERTa-Large, 47.3 → 69.8).

• Encoder gains are large and consistent: All ViT and RoBERTa settings improve. Task Arithmetic on ViT-B/32 goes from 71.8 to 85.0 (+13.2) and on ViT-B/16 from 76.8 to 88.9 (+12.1); TIES-Merging on RoBERTa-Base improves by +18.2 and on RoBERTa-Large by +22.5. Average gains on ViT-B/32 and ViT-B/16 are about 12 points.

• LLM gains are real but smaller: CUSD improves 22 of 24 LLM settings. The only negative LLM change is a 0.3-point decrease for DARE on Llama-3.1-8B (94.1 → 93.8), a model the authors describe as having little headroom since plain merges already reach normalized averages around 94.

• Gains come from a better multi-task trade-off, not uniform uplift: With TIES-Merging, CUSD reduces an already strong MNIST score by only 0.1 points while improving three weaker tasks by 8.5–10.7 points. On GLUE, STS-B moves from negative plain-merge scores to positive CUSD scores across operators.

• Direction choice matters: Against matched controls selecting the same number of unique directions per layer (Random, Reversed, Top-energy), CUSD wins on both backbones — e.g., ViT-B/32: CUSD 76.48 vs Top-energy 73.78 and Random 59.20 ± 0.08; ViT-B/16: CUSD 81.74 vs Top-energy 81.07 and Random 65.64 ± 1.17.

• Merging the common part alone is best: Comparing merging applied to common only, unique only, or both separately, the "Common" design (i.e., CUSD) wins in all four reported settings — for example ViT-B/16 with TIES: 90.3 (Common) vs 73.0 (Unique) vs 72.4 (Both).

• Scoring choice is family-dependent: Relative dispersion scoring wins in six of eight LLM model–operator settings, while absolute scoring wins with Task Arithmetic on both LLMs tested. The authors attribute this to ViT having comparatively balanced directional energies, where relative scoring amplifies low-energy fluctuations.

• The β coefficient behaves differently by family: Encoders favor small β and β = 0 is a stable default; decoder LLMs prefer β ∈ [0.2, 0.5]. The reason given is structural — in encoders the unique subspace holds fewer than 1% of directions but roughly 12–20% of directional energy, whereas in decoder LLMs it holds 17.3% to 81.2% of the energy.

• Interference is measurably reduced: A normalized interference proxy drops consistently across blocks, with average reductions of 10.0%, 9.0%, and 10.2% on ViT-B/32, ViT-B/16, and ViT-L/14 respectively (DARE vs CUSD-DARE).

• KDE thresholding beats fixed cutoffs: The adaptive KDE valley detection outperforms fixed percentile thresholds on both ViT and LLM settings, and is stable across standard bandwidth choices (Scott's and Silverman's rules give nearly identical performance).

• Comparison with subspace baselines is mixed: CUSD+TA reaches 88.9 on ViT-B/16 versus Iso-CTS at 95.6, and 95.7 on Gemma-2-2b versus Iso-C at 97.7. It is highest on RoBERTa-Base (68.7 vs 45.9 for TSV-M) and Llama-3.2-3B (69.6 vs 65.7 for Iso-C).

• Preprocessing transfers to optimization-based methods unevenly: CUSD improves WUDI-Merging in three of four settings (notably RoBERTa-Base 69.3 → 80.7) and ACE on both LLMs (Gemma-2-2b 93.7 → 98.6), but encoder results are mixed.

• Cost is modest and amortized: ViT decompositions take 16.3–48.5 s with at most 1.9 GB peak GPU memory; Llama-3.1-8B takes 217.2 s with 15.4 GB peak memory. Per-layer times range from 0.340 s to 0.970 s.

Methodology in Plain English

The starting point is the task vector: for each fine-tuned model, subtract the base model's weights to get the "delta" that represents what that task taught the model. Arithmetic merging methods combine these deltas with simple rules.

The authors' observation is that these deltas are not a bag of independent numbers. Inside each weight matrix they form patterns — directions across many parameters at once. Some directions are used about equally by every task; others are used heavily by one or two tasks and barely by the rest. A rule like "average everything" or "keep the majority sign" cannot distinguish the two.

CUSD does the distinguishing as a preprocessing step:

1. Build one shared set of directions. Instead of running a separate decomposition per task (which gives directions that cannot be compared), CUSD uses a regularized variant of higher-order generalized SVD — a Gram-matrix quotient, symmetrized, with ridge term π = 10⁻³ — to get one orthonormal direction set that all tasks are measured against.

2. Score each direction for cross-task imbalance. For every direction, record how much energy each task puts into it, then take the standard deviation across tasks. High standard deviation means the direction is dominated by a few tasks.

3. Find the cut automatically. Rather than pick a fixed top-k percentage, CUSD fits a kernel density estimate to the spread of scores and takes the valley between the two modes as the threshold. Directions below it are "common"; above it are "unique."

4. Split and merge. Project each task's delta onto the common directions and the unique directions — an exact, orthogonal split. Apply the chosen arithmetic operator only to the common pieces, sum the unique pieces, scale them by β, and add the two results to the base model.

Because step 1–3 depend only on weights, the split can be computed once and swapped between merging rules, and β can be tuned independently of the operator's own hyperparameters.

Why This Matters

Impact on research. The paper reframes a per-coordinate engineering problem as a directional-geometry problem, and shows that a diagnostic observation (dispersion structure) can be turned into a cheap, plug-in intervention. Because CUSD is a pre-processing step rather than a competing operator, it provides a clean experimental lever: it can be bolted onto essentially any arithmetic merging method to test whether directional separation helps. The finding that gain comes from rebalancing weak tasks rather than lifting strong ones is a useful corrective to headline-average reporting.

Real-world applications:

• Multi-domain model deployment: Serving one merged model instead of one checkpoint per task — relevant where per-task serving is too expensive or where checkpoints come from different owners.
• Privacy-constrained or federated settings: Merging is possible when training data cannot be shared; CUSD adds no data requirement beyond weights.
• Open-weight model composition: Combining community fine-tunes of a shared open base model (e.g., Llama or Gemma variants) without re-training or access to their data.
• Vision and language adapters/backbones: The encoder results — ViT and RoBERTa families — map directly onto multi-dataset classification and GLUE-style NLU deployments.

Industry relevance. The one-time decomposition cost (16.3–48.5 s and up to 1.9 GB for ViT models; 217.2 s and 15.4 GB for Llama-3.1-8B) is small relative to training, and the decomposition is reusable across operators, so the practical barrier to adoption is low. The tuning burden is a single coefficient β, with the paper reporting stable ranges by model family.

Future Directions

• Per-layer or per-family automatic β. The paper sweeps a single global β and reports that encoders want near-zero while LLMs want 0.2–0.5, and that encoder performance is sensitive to β. Whether β can be set adaptively per layer or per model without a sweep is left open.

• Reconciling absolute and relative scoring automatically. The choice between scoring modes is made by model family in the main experiments, with a mechanistic explanation for why each fits. A unified criterion that selects the mode from the data rather than from the architecture remains an open question.

• Closing the gap to subspace baselines on ViT. CUSD+TA trails Iso-CTS on ViT-B/16 (88.9 vs 95.6) and Iso-C on Gemma-2-2b (95.7 vs 97.7), so where the pre-merging split is weaker than a post-merge correction — and whether the two can be combined — is unresolved.

• Extending beyond matrix-structured linear layers. CUSD is applied to matrix-structured linear-layer task vectors; the paper does not report results for other parameter types such as normalization or embedding parameters, leaving their treatment unspecified.

Target Audience

Researchers and engineers working on model merging, multi-task learning, and efficient model reuse — particularly those already familiar with task arithmetic and SVD-based merging who want a lightweight, operator-agnostic preprocessing technique. It is also useful for practitioners deploying merged open-weight models who need a low-cost method with a single tuning knob, and for readers interested in the empirical geometry of fine-tuning deltas. Beginners will find the high-level motivation accessible but will need background on task vectors and subspace projection to follow the method.

Authors’ abstract

Model merging combines multiple fine-tuned models without additional training, and arithmetic methods have emerged as a simple yet effective way to merge models without data. However, existing arithmetic methods treat task vectors in parameter space and apply a single element-wise rule to directions that can differ in mergeability. In a shared directional basis, we find a consistent two-regime structure: a low-dispersion common bulk shared across tasks and a high-dispersion task-dominated tail that carries disproportionate energy. These findings motivate Common--Unique Subspace Decomposition (CUSD), a data-free pre-merging method that separates task vectors into common and unique directional components, merges the common components with a downstream arithmetic operator, and retains unique residuals separately. Experiments across $11$ base models, four model families, and four arithmetic-based merging operators show that CUSD improves 42 of 44 model--operator combinations and achieves up to 22.5% improvement over the corresponding baselines. Our code is publicly available at https://github.com/stwang666/CUSD