M-Loss: Quantifying Model Merging Compatibility with Limited Unlabeled Data

AI summary

Authors: Tiantong Wang, Yiyang Duan, Haoyu Chen, Tiantong Wu, Wei Yang Bryan Lim Affiliations: Nanyang Technological University (College of Computing and Data Science); Beijing Jiaotong University (School of Computing and Information Technology); Alibaba-NTU Global e-Sustainability CorpLab (ANGEL) arXiv: 2602.08564v1 [cs.LG], 09 Feb 2026 | Code: https://github.com/languangduan/mLoss | License: CC BY 4.0

Overview

• Research area: Model merging and model compression in deep learning, specifically the theory and evaluation of combining the weights of multiple fine-tuned models.
• Technical level: Advanced. The paper combines layer-wise neural network analysis, expected-value derivations for ReLU, GELU and Leaky ReLU activations, and large-scale Vision Transformer experiments.
• Scope: The paper introduces M-Loss (Merging-ensembling loss), a metric computed on limited unlabeled data that measures how closely a parameter-averaged model approximates an output-averaged ensemble, and uses it to both justify model merging theoretically and drive a dynamic pruning schedule called M-TIES.

What This Paper Is About

Model merging fuses several fine-tuned models into one by combining their weights, which avoids extra training and avoids needing labeled data, but it can fail when source models differ a lot in their weights. Model ensembling, which averages the outputs of all source models, is usually more stable and accurate, but it requires storing and running every model at inference time. This paper asks why simple weight averaging can approximate ensembling at all for nonlinear networks, and proposes M-Loss as a measurable, unlabeled-data-based answer to that question.

Key Contributions

1. A new evaluation metric with theoretical grounding. M-Loss quantifies the discrepancy between a weight-merged model and an ensembled model at layer and node levels, and the paper derives the expectation of this loss under three common activations: ReLU, GELU and Leaky ReLU.
2. A theoretical explanation of merging feasibility. The analysis shows that when the pre-activation range k of the pretrained model is much larger than the parameter shift σ between fine-tuned models (k ≫ σ), the expected M-Loss is small — providing justification for why models fine-tuned from a shared pretrained backbone can be merged by averaging.
3. M-Loss as a hyperparameter and pruning guide. Because M-Loss indicates parameter significance and inter-model conflict, the authors use node-wise M-Loss scores to set dynamic per-node keep rates, producing M-TIES, a variant of TIES with a data-driven pruning budget schedule.
4. Few-Layer M-TIES. Based on the observation that M-Loss is concentrated in the first layer and the last few layers, the authors propose applying dynamic pruning only to those layers, saving computation with almost no accuracy change.

Main Findings

• M-Loss has closed-form small expectations. Under the assumptions x ~ Uniform(−k, k), a, b ~ N(x, σ²), and k ≫ σ, the expected node-level M-Loss is approximately σ² / (√2 π k) for ReLU, (1 − α)σ² / (√2 π k) for Leaky ReLU with parameter α, and σ² / (4k) for GELU. These are all small when k ≫ σ.
• Mergeability has two levers. A larger range k for the pretrained model's intermediate representations (a more general pretrained model) predicts better merging, and a smaller σ (greater parameter similarity between source models) predicts better merging; the paper states mergeability is positively correlated with parameter similarity.
• M-TIES beats other merging baselines on average accuracy. On eight datasets with ViT-B/32, average accuracy is 73.23 for M-TIES versus 72.42 for TIES, 71.64 for DARE, 70.61 for Task Arithmetic, and 65.90 for Simple Average; the Ensembling baseline reaches 78.56. On ViT-L/14, M-TIES reaches 85.28 versus 85.04 for DARE, 84.94 for TIES, 83.62 for Task Arithmetic, 79.46 for Simple Average, and 85.56 for Ensembling.
• M-TIES can surpass ensembling on individual tasks. On ViT-B/32, M-TIES scores 97.62 on MNIST against Ensembling's 95.80, and 83.06 on SVHN against Ensembling's 82.15.
• Better stability across tasks. The paper reports the variance of accuracy across tasks for the top-3 merging methods under ViT-B/32 as 172.22 (M-TIES), 203.27 (TIES) and 197.61 (DARE), arguing M-TIES is less biased toward high-accuracy tasks.
• Extremely few unlabeled samples are enough. Computing M-Loss with 128, 256 or 512 samples gives nearly identical average accuracy (73.232, 73.227 and 73.227 on ViT-B/32), so the authors settle on a single batch of 128 samples — 16 per dataset on average.
• Stable under random sampling. Across random seeds 1, 2 and 42, per-dataset standard deviations range from 0.000 to 0.336, with an average standard deviation of 0.037.
• Out-of-domain generalization. On ViT-B/32, M-TIES averages 63.46 across Flowers102, FashionMNIST, Food101, STL10 and CIFAR100, versus 60.25 for DARE, 57.55 for TIES and 72.20 for Ensembling.
• Few-Layer M-TIES is nearly lossless. Pruning by M-Loss only in layers 0, 8, 9, 10 (ViT-B/32) yields an average of 73.216 versus 73.232 for all layers; for layers 8, 9, 10 the average is 73.221. For ViT-L/14, layers 0, 20, 21, 22 give 85.259 and layers 20, 21, 22 give 85.256, versus 85.278 for all layers.
• Modest overhead. M-TIES takes roughly 1 minute 30 seconds to merge ViT-B/32 and 3 minutes for ViT-L/14, compared with 30 seconds and 1 minute for TIES; evaluation takes about 5 minutes and 15 minutes respectively. The authors note that merging layer k only requires forwarding the input through (k − 1 + q) layers rather than k·q layers.
• M-Loss is not uniform across depth. The layerwise node-group analysis shows the first layer and the last few layers (excluding the output layer) carry higher M-Loss than the others.

Methodology in Plain English

The core idea is to compare two things that should, in principle, behave similarly: the output of a model built by averaging the weights of several fine-tuned models, and the average of the outputs of those same models. The difference between these two quantities is M-Loss.

To make this measurable, the authors break the network down layer by layer and node by node. For each node, they look at what that node's pre-activation value is in each source model, apply the activation function in two ways — once to the average of those values, once to each value and then averaged — and measure the gap. Summing these node-level gaps gives the layer-level score, and the layer scores accumulate into the overall merged-versus-ensembled discrepancy. A normalized version divides by the magnitude of the activated term (plus a tiny constant, typically 10⁻⁴ or smaller) so that layer scale differences do not distort the scores.

Because only weighted sums and activation functions are involved, each node's score depends on a small group of parameters whose partial derivatives are constant for a given input. The paper calls these Linearly Correlated Model Parameters, and this is what makes pruning practical: when a node has a high M-Loss, the paper prunes its associated parameters more aggressively.

To get the theoretical result, the authors model the pretrained model's pre-activation value as roughly uniform over a range [−k, k] and the fine-tuned models' values as normally distributed around it with variance σ², assuming k ≫ σ. Solving the expected discrepancy for ReLU, Leaky ReLU and GELU yields the small closed-form expressions listed above.

The practical method, M-TIES, applies this inside the standard TIES pipeline. For each layer it computes task vectors for all source models, computes node-level M-Loss on unlabeled data, converts those losses into per-node keep ratios between (k − e)% and k% (a base keep rate and a variation range), prunes each task vector accordingly, elects the dominant sign, merges the pruned vectors with the given weights, adds the increment back to the base layer, and synchronizes the updated layer back into the source models before moving on.

Experiments use OpenAI CLIP ViT-B/32 and ViT-L/14 backbones fine-tuned on eight datasets — RESISC45, Cars, MNIST, DTD, EuroSAT, GTSRB, SUN397 and SVHN — with equal aggregation weights, compared against Simple Average, Task Arithmetic, TIES, DARE and Ensembling. Hyperparameters are searched on a validation set with ViT-B/32 and then held fixed for ViT-L/14. All experiments run on a single NVIDIA RTX A6000 GPU.

Why This Matters

The paper's contribution is mostly conceptual: it turns a well-known empirical observation — that weight averaging sometimes mimics ensembling — into a quantity that can be computed, bounded, and acted on. That gives researchers a way to reason about merging before spending compute on it, and it gives practitioners a signal for deciding which parameters to keep when merging with no labels available.

Impact on research: The expected-loss expressions connect mergeability to two interpretable quantities, the pretrained model's representation range k and the inter-model parameter shift σ. That reframes model merging as something measurable rather than purely empirical, and M-Loss is proposed as a general criterion that could be used to evaluate future merging methods rather than only the ones tested here.

Real-world applications (implied by the techniques the paper studies):

• Consolidating many task-specific fine-tuned checkpoints into one deployable model without retraining or collecting new labels.
• Reducing inference and storage costs relative to ensembling, which the paper notes requires storing all models and computing their joint outputs.
• Deploying merged multi-task models in settings where only a tiny unlabeled sample (128 samples in this study, 16 per dataset on average) can be spared for calibration.
• Layer-selective merging for cheaper pipelines, via Few-Layer M-TIES, which largely reduces computation while barely moving accuracy.

Industry relevance: The paper ties directly to the practical economics of foundation models — fewer stored checkpoints, one forward pass instead of many, and a calibration step that needs no annotation. The work is supported by an NTU startup grant, the RIE2025 Industry Alignment Fund – Industry Collaboration Projects (IAF-ICP) (Award I2301E0026) administered by ASTAR, Alibaba Group and NTU Singapore through ANGEL, and ASTAR's Japan-Singapore Joint Call (Project ID: R24I6IR139), which signals industrial interest in the problem.

Future Directions

• Refining the theoretical assumptions. The derived expectations rely on x ~ Uniform(−k, k), a, b ~ N(x, σ²) and k ≫ σ. Testing how well the bounds hold when these approximations break down — for more divergent fine-tunes or architectures with layer normalization — is an open question.
• Extending M-Loss beyond linear layers and beyond the tested backbones. The paper uses vanilla TIES trimming for non-linear layers because computing M-Loss there is more complex, and evaluates only ViT-B/32 and ViT-L/14; both are stated limitations that invite follow-up.
• Broadening the pairing with other merging backends. The conceptual overview mentions integration with TIES Top-K and DARE, but experiments use TIES; systematically applying the M-Loss scheduler to other merging methods is a natural next step.
• Determining when M-Loss can replace validation data entirely. The paper positions M-Loss as a predictor of merged performance without a labeled test set, but the reported experiments still use a validation set for the initial hyperparameter search on ViT-B/32 — clarifying how far M-Loss alone can substitute for that search is unresolved.

Target Audience

• Machine learning researchers working on model merging, model compression, or transfer learning who want a theoretical handle on why merging works.
• Practitioners deploying multiple fine-tuned models who need a low-cost way to decide which parameters to keep without labeled data.
• Graduate students and advanced undergraduates with background in neural network theory and Vision Transformers who want a concrete example of connecting activation-level analysis to a usable engineering method.
• Engineers interested in the practical trade-offs between merging and ensembling in terms of accuracy, storage, and inference cost.

Authors’ abstract

Training of large-scale models is both computationally intensive and often constrained by the availability of labeled data. Model merging offers a compelling alternative by directly integrating the weights of multiple source models without requiring additional data or extensive training. However, conventional model merging techniques, such as parameter averaging, often suffer from the unintended combination of non-generalizable features, especially when source models exhibit significant weight disparities. Comparatively, model ensembling generally provides more stable and superior performance that aggregates multiple models by averaging outputs. However, it incurs higher inference costs and increased storage requirements. While previous studies experimentally showed the similarities between model merging and ensembling, theoretical evidence and evaluation metrics remain lacking. To address this gap, we introduce Merging-ensembling loss (M-Loss), a novel evaluation metric that quantifies the compatibility of merging source models using very limited unlabeled data. By measuring the discrepancy between parameter averaging and model ensembling at layer and node levels, M-Loss facilitates more effective merging strategies. Specifically, M-Loss serves both as a quantitative criterion of the theoretical feasibility of model merging, and a guide for parameter significance in model pruning. Our theoretical analysis and empirical evaluations demonstrate that incorporating M-Loss into the merging process significantly improves the alignment between merged models and model ensembling, providing a scalable and efficient framework for accurate model consolidation.