When Shared Knowledge Hurts: Spectral Over-Accumulation in Model Merging

AI summary

Overview

Research area: Machine learning — specifically weight-space model merging, a technique for combining several fine-tuned models into one, analyzed here through the lens of the singular value decomposition (SVD) of task weight updates.

Technical level: Advanced. The paper is built on matrix spectral theory (singular vectors, singular values, subspace projections) and assumes familiarity with task vectors and CLIP-style vision-language models.

Scope: The paper identifies and explains a specific failure mode in model merging — repeated accumulation of shared spectral directions that inflates singular values — and proposes a training-free, data-free fix called Singular Value Calibration (SVC).

What This Paper Is About

Model merging takes several models fine-tuned from the same pre-trained backbone on different tasks and combines their weight updates into a single model, avoiding the cost of retraining. Earlier work mostly attacks this problem by resolving conflicts between task updates, but this paper argues that a different failure mode is overlooked: when tasks share aligned spectral directions (overlapping singular vectors), simply adding the updates counts that shared knowledge multiple times, inflating the corresponding singular values and pushing the merged model toward a few dominant shared subspaces. The goal is to detect this "spectral over-counting" without data or training, and correct it by rescaling only the singular values while leaving the spectral directions untouched.

Key Contributions

1. Identifying spectral over-counting as a failure mode. The authors show that redundant aggregation of shared knowledge across tasks induces singular-value inflation concentrated in a small number of dominant spectral subspaces, while suppressing task-specific components in the remaining subspaces.

2. Proposing Singular Value Calibration (SVC). SVC is a training-free, data-free post-processing method that measures output-space overlap in the merged spectral basis and rescales the corresponding singular values to restore spectral balance, without redesigning the merging rule or changing the merged singular vectors.

3. Providing theoretical and empirical validation. The paper gives a cross-term analysis (Lemma 3.2) and a projection-optimal calibration theorem (Theorem 3.3) showing that positive cross-task overlap inflates the merged singular value above the projection-optimal magnitude, supported by spectral diagnostics on an 8-task CV merging setup and across representative merging methods.

4. Demonstrating broad gains. SVC improves strong merging baselines across vision and language benchmarks, improving Task Arithmetic by 13.0% as stated in the abstract, and also enabling targeted improvement for a chosen task, such as preference optimization.

Main Findings

• Shared spectral directions are over-counted. Lemma 3.2 expresses the projection coefficient as s_i^r = 1 + sum_{j != i} <a_j^r, a_i^r> / ||a_i^r||^2. When many cross-task inner products are positive, s_i^r > 1, meaning multiple tasks accumulate along the same direction and the merged response over-counts that shared component. These overlaps cluster in the top spectral subspaces (Figure 3).

• Inflation shows up as a measurable singular-value gap. On the 8-task CV classification merging setup, comparing original singular values against calibrated values reveals a clear gap Δ = σ − σ* in top spectral subspaces, indicating systematic over-counting and inflation (Figure 2). The gap persists across representative merging methods, and some methods also show large variance with negative gaps in certain subspaces, i.e. under-accumulation (Figure 4).

• SVC improves CV merging across backbones. On 8 tasks, Task Arithmetic goes from 68.9 to 81.9 (+13.0) on ViT-B/32, 73.7 to 86.2 (+12.5) on ViT-B/16, and 84.3 to 91.3 (+7.0) on ViT-L/14. On 14 tasks it goes from 46.4 to 63.1 (+16.7), 57.1 to 72.0 (+14.9), and 57.7 to 76.7 (+19.0). TIES improves from 72.6 to 80.0 (+7.4) on 8-task ViT-B/32, DARE from 65.8 to 80.7 (+14.9), TSV-M from 84.0 to 84.8 (+0.8), Iso-C from 83.1 to 84.6 (+1.5), and Iso-CTS from 81.4 to 85.6 (+4.2). For reference on the same tables, the pre-trained model reaches 48.0 and the individual models 90.5 on 8-task ViT-B/32.

• SVC improves NLP merging as well. For Llama2-7B, SVC lifts Task Arithmetic from 49.1 to 51.7 (+2.5) on AlpacaEval and from 46.1 to 52.2 (+6.1) on GSM8K; Iso-C rises from 50.0 to 58.9 (+8.9) on AlpacaEval and 42.0 to 51.4 (+9.4) on GSM8K. On encoder-derived classification, Task Arithmetic improves from 56.9 to 69.0 (+12.1) for BERT, 41.5 to 46.3 (+4.8) for T5, and 53.5 to 65.8 (+12.3) for T0. For T0, IA³-based PEFT yields task vectors as vectors rather than full weight matrices, so SVD-dependent methods (TSV-M, Iso-C, Iso-CTS) are marked not applicable.

• Output-space calibration is the right side. Replacing left singular vectors with right singular vectors degrades performance and can fall below the uncalibrated baseline (for example, TIES drops from 72.6 to 65.7, a change of −6.9). The authors attribute this to right singular vectors describing input-side directions.

• Suppression-only calibration is the reliable setting. With α = 1, SVC is suppression-only and yields consistent gains. Lowering α into (0, 1) allows boosting under-accumulated subspaces and produces mixed outcomes, since it can disturb spectral balance. Unless otherwise stated, the default is α = 1/K; for TSV-M, α = 1 is used.

• Targeted calibration favors a chosen task. Using a target task's s_i^r directly to set γ^r = 1 / max(α, s_i^r), the diagonal of the task-by-task matrix (Figure 6) is usually the largest, meaning calibration targeted at task i mainly improves task i. Related tasks can also benefit — calibrating for Cars improves SUN397 — while tasks with large domain gaps may degrade.

• Overhead is modest. SVC requires SVD and runs once offline: 5.1 s and 1,027.4 MiB on ViT-B/32, 8.2 s and 1,082.8 MiB on ViT-B/16, 15.6 s and 1,488.5 MiB on ViT-L/14, 517.2 s and 1,898.7 MiB on LLaMA2 7B, and 249.3 s and 2,513.1 MiB on Qwen2.5 7B.

• Reporting caveat. All results are produced by the authors' own runs under a unified protocol; because checkpoints may differ from the original papers, absolute numbers can vary from previously reported results, including for Iso-C and Iso-CTS.

Methodology in Plain English

The researchers treat each layer's task matrix — the difference between a fine-tuned weight matrix and the pre-trained one — as a linear operator. They compute the SVD of the already merged task matrix and use its left singular vectors as a shared coordinate system, one axis per spectral subspace. Each individual task's update is then projected onto each of these axes, producing a per-task, per-subspace response vector. Because all tasks are now expressed in the same coordinates, they can be compared directly.

For each task and subspace, the merged response is projected back onto that task's own response direction. The resulting coefficient says whether the merged update amplified that direction (greater than 1) or attenuated it (less than 1). A coefficient above 1 means other tasks have piled onto the same direction — over-counting. These coefficients are aggregated across tasks into a single per-subspace factor, using a conservative harmonic-mean-like form with a floor α that prevents instability when some coefficients are very small. The merged singular value for that subspace is multiplied by this factor, and the merged matrix is rebuilt from the rescaled singular values and the original singular vectors. Nothing is retrained, no data is used, and the merge rule itself is left alone — SVC is a post-hoc correction applied on top of whatever merging method produced the update.

Why This Matters

Impact on research. The paper reframes model merging interference: conflict between task updates is not the only source of degradation, and aligned, seemingly "useful" shared knowledge can actively harm the merged model when it is counted repeatedly. The framing of merging as a spectral-balance problem gives a diagnostic (the singular-value gap between original and projection-optimal values) that can be run on any existing method, and a lightweight correction layer that complements rather than replaces existing merge rules.

• Multi-task vision deployment: combining separately fine-tuned CLIP-style visual encoders into one model that handles many classification tasks without retraining or holding several checkpoints.
• Large language model reuse: merging fine-tuned LLM variants, where the paper evaluates Llama2-7B on generation benchmarks and BERT/T5/T0 as encoders.
• Parameter-efficient fine-tuning pipelines: where adapters or task vectors are produced per downstream task and need to be consolidated.
• Targeted capability shaping: steering a merged model toward a specific task, as demonstrated in the preference optimization experiment, or away from undesirable knowledge, a use case the introduction cites for weight-space merging.

Industry relevance. Merging avoids the cost of joint retraining and of running multiple models at inference, and SVC adds only an offline SVD pass — seconds on ViT backbones, and on the order of minutes on 7B-parameter LLMs, with no gradient computation. For practitioners who already use Task Arithmetic, TIES, DARE, TSV-M, Iso-C, or Iso-CTS, it is a drop-in post-processing step. The paper notes that recent training-free methods often match or exceed training-based approaches such as AdaMerging++ and Surgery, which strengthens the case for spectral calibration as a practical default.

Future Directions

• Combining SVC with dynamic merging. Methods such as DaWin, EMR-Merging, TALL-Mask, and Twin-Merging depend on task labels or routing decisions at inference time; how static post-hoc calibration interacts with input-conditioned merging is an open question.
• Handling settings where SVD is undefined. The authors note that IA³-based PEFT produces task vectors as vectors rather than full weight matrices, so SVD-dependent methods including TSV-M and Iso-C cannot be applied; extending spectral calibration to this regime is unresolved.
• Balancing suppression and amplification. The α ablation shows suppression-only (α = 1) is consistently beneficial, while boosting under-accumulated subspaces (α < 1) is risky. How to safely rebalance both inflated and shrunken subspaces remains open.
• Systematizing targeted calibration. The per-task preference experiment shows diagonal gains and some cross-task spillover, but also degradation for tasks with large domain gaps; choosing which target task to calibrate for, and predicting which related tasks benefit, is not yet solved.

Target Audience

Researchers and engineers working on model merging, multi-task learning, and efficient adaptation of vision or language models, who are comfortable with linear algebra and spectral methods. It is most useful to those already using Task Arithmetic, TIES, DARE, TSV-M, Iso-C, or Iso-CTS and looking for a data-free way to improve an existing merged checkpoint, as well as to readers interested in the theoretical question of why shared knowledge across tasks can degrade a merged model rather than help it.

Authors’ abstract

Model merging combines multiple fine-tuned models into a single model by adding their weight updates, providing a lightweight alternative to retraining. Existing methods primarily target resolving conflicts between task updates, leaving the failure mode of over-counting shared knowledge unaddressed. We show that when tasks share aligned spectral directions (i.e., overlapping singular vectors), a simple linear combination repeatedly accumulates these directions, inflating the singular values and biasing the merged model toward shared subspaces. To mitigate this issue, we propose Singular Value Calibration (SVC), a training-free and data-free post-processing method that quantifies subspace overlap and rescales inflated singular values to restore a balanced spectrum. Across vision and language benchmarks, SVC consistently improves strong merging baselines and achieves state-of-the-art performance. Furthermore, by modifying only the singular values, SVC improves the performance of Task Arithmetic by 13.0%. Code is available at https://github.com/lyymuwu/SVC.