Do We Really Need Permutations? Impact of Model Width on Linear Mode Connectivity

AI summary

Overview

• Research area: Deep learning theory and model merging — specifically linear mode connectivity (LMC) and permutation symmetries in neural network loss landscapes.
• Technical level: Advanced (the paper assumes familiarity with loss barriers, permutation symmetries, and intermediate-layer representations), though the core message is conveyed plainly.
• Scope: An empirical and analytical study showing that sufficiently wide, independently trained networks can be linearly interpolated without any permutation, provided softmax temperature calibration is used, and explaining this via a new concept called layerwise exponentially weighted connectivity (LEWC).

What This Paper Is About

A widely held belief is that merging two independently trained neural networks into one requires finding a permutation of one model's neurons that preserves its input–output behavior, and that this only works for very wide models. The authors ask whether the permutation is really the essential ingredient, or whether width alone is doing the work. They find that once models are wide enough, simply averaging weights with no permutation at all produces a merged model whose accuracy matches the originals, and they offer a mechanistic explanation for why.

Key Contributions

1. Widening alone improves merged-model performance without permutations. The authors empirically show that increasing the width of independently trained models monotonically improves the accuracy of their merged model (plain weight averaging, no permutation), eventually matching the performance of the original models. The loss barrier can be reduced to nearly zero by calibrating the softmax with an appropriate inverse temperature.

2. A new theoretical concept: layerwise exponentially weighted connectivity (LEWC). LEWC states that the intermediate-layer outputs of the merged model equal an exponentially weighted sum of the corresponding outputs of the two original models, with depth-dependent coefficients λ^ℓ and (1−λ)^ℓ. This makes the merged model behave like an ensemble of the two models, explaining why LMC holds. Widening the model makes LEWC more likely to hold.

3. Identification of two sufficient conditions for LEWC. The authors prove (Theorem 5.3) that for bias-free models, weak additivity for ReLU activations plus reciprocal orthogonality imply LEWC, and they verify both conditions empirically in wide models.

4. The role of low-rank weight structure. The authors show that low-rank weight matrices are what make the sufficient conditions hold. Weakening weight decay (from the paper's default of 0.003 to 10⁻⁴) increases weight rank and breaks LEWC, weak additivity, reciprocal orthogonality, and LMC together.

Main Findings

• Permutations are not necessary at sufficient width. Ainsworth et al. (2023) reported needing a 32× width multiplier for ResNet-20 and 4× for VGG-16 to get permutations to reduce the loss barrier on CIFAR-10. The authors show that with no permutation, merged accuracy still approaches the originals as width grows.

• Barriers with and without permutation are both small at the tested widths. Table 1 reports barrier values at λ = 1/2 using calibrated losses, with width multipliers of 16× for MLP, 16× for VGG-11, and 32× for ResNet-20:

  • MLP / MNIST — without permutation: accuracy barrier 0.519 ± 0.225, loss barrier 0.013 ± 0.004; with permutation: −0.027 ± 0.139 and −0.003 ± 0.003.
  • MLP / FMNIST — without permutation: 2.467 ± 0.685 and 0.056 ± 0.014; with permutation: 4.925 ± 4.379 and 0.155 ± 0.111.
  • VGG-11 / CIFAR-10 — without permutation: 1.308 ± 1.594 and 0.066 ± 0.054; with permutation: 7.000 ± 3.694 and 0.177 ± 0.039.
  • ResNet-20 / CIFAR-10 — without permutation: 2.694 ± 1.493 and 0.087 ± 0.051; with permutation: 5.135 ± 2.935 and 0.173 ± 0.099.

• Accuracy rises with width, but raw loss does not. Test accuracy of merged models monotonically increases with width, yet uncalibrated test loss fails to reach the original models' level. The discrepancy comes from logit scaling: an inverse temperature changes cross-entropy loss without changing predicted labels.

• Temperature calibration closes the gap. The authors estimate an inverse temperature on 20% of the test set and compute calibrated losses on the remaining 80%. With calibration, the loss barrier approaches zero as width increases.

• Random permutations do not hurt. Applying random permutations before merging at λ = 1/2 leaves merged models at high accuracy, further indicating permutations are not the critical factor at sufficient width.

• Cosine similarity between merged and ensemble representations increases with width. For each layer, the cosine similarity between f_ℓ(x; (θ_a+θ_b)/2) and (f_ℓ(x; θ_a) + f_ℓ(x; θ_b))/2 rises with width, and at the last layer approaches 1 — the merged model's logits nearly match the ensemble's.

• ReLU activations become approximately linear in high dimension. Theorem 5.4 shows for Gaussian random vectors u, v ~ N(0, I_d), the cosine similarity between σ(u+v) and σ(u)+σ(v) converges in probability to (3/4 + 1/π)/√(1 + 1/π) ≈ 0.93 as d grows, with K = 32/3 and probability at least 1 − 3δ.

• Dimensionality alone is not enough — low rank matters. Observed cosine similarities exceed 0.93 for wide models, so the authors point to low-rank weights as an additional cause: low-rank weight matrices reduce overlap between the active (large-second-moment) dimensions of the two models, making ReLU act linearly. Figure 6 shows that as width increases, the fraction of small-moment dimensions grows and overlap decreases.

• Reciprocal orthogonality emerges with width. The ratio E‖W_ℓ^(a) z_ℓ−1^(b)‖ / E‖W_ℓ^(a) z_ℓ−1^(a)‖ decreases with width in all layers except the input, and the cosine similarity between W_ℓ^(a) z_ℓ−1^(c) and W_ℓ^(a) z_ℓ−1^(a) approaches 1.

• Weakening weight decay breaks everything. At weight decay 10⁻⁴, merged models show large barriers; LEWC fails in deeper layers at λ = 1/2, and neither weak additivity for ReLU activations nor reciprocal orthogonality holds. This ties LMC to the low-rank structure of SGD solutions.

• LLFC is not the right frame. Zhou et al. (2023) introduced layerwise linear feature connectivity (LLFC) with sufficient conditions of weak additivity for ReLU activations and a commutativity property. The authors show in Appendix C that the commutativity property does not hold in their setting, motivating LEWC as a distinct concept.

Methodology in Plain English

The authors take pairs of models trained independently from scratch on the same dataset and architecture, then:

1. Sweep width. They train MLPs, VGG-11, and ResNet-20 at a range of width multipliers, merge each pair by linearly interpolating weights at various coefficients λ, and record test accuracy and test loss. No permutation is applied. Results are averaged over three independent merges of two models, reported as mean and standard deviation.

2. Calibrate the loss. Because accuracy and loss can disagree, they fit an inverse temperature on 20% of the test set, then evaluate loss on the remaining 80% under that temperature. This separates genuine loss barriers from mere logit-scale effects.

3. Probe layer by layer. To test whether LEWC holds, they measure cosine similarity between the merged model's layer-ℓ output and the average of the two original models' layer-ℓ outputs, using test data. For the final layer they compare logits.

4. Check the sufficient conditions. They measure ReLU linearity by cosine similarity between σ((z̃_a + z̃_b)/2) and (σ(z̃_a) + σ(z̃_b))/2, and reciprocal orthogonality through norm ratios of cross-model weight–activation products. They also plot histograms of the square root of the second moment of pre-activations to see how many dimensions are effectively active.

5. Vary weight decay. As a controlled intervention, they retrain with weak weight decay (10⁻⁴ versus the default 0.003) to raise weight-matrix rank and observe that LEWC, its two conditions, and LMC all degrade.

6. Test permutation sensitivity. They apply random permutations before merging at λ = 1/2 to confirm that permutations are not driving the result.

Experiments are limited to relatively simple datasets (CIFAR-10 and MNIST, with CIFAR-100 in Appendix F.1) because LEWC needs larger width multipliers than permutation-based merging.

Why This Matters

Impact on research. The paper reframes the role of width in linear mode connectivity. Prior work treated width as a way to enlarge the space of candidate permutations; this work argues that width instead pushes the trained solutions themselves into a regime where weight averaging is already safe, without any alignment step. The LEWC framework, together with the theorem connecting it to weak additivity for ReLU activations and reciprocal orthogonality, gives a testable mechanistic account. It also reinforces the emerging picture that the low-rank structure of SGD solutions — known to be relevant to permutation-based merging via Ito et al. (2025a) and Yunis et al. (2024) — governs whether merging works at all, and shows that the same requirement applies to permutation-free merging.

Real-world applications.

• Model merging: Combining expert or task-specific models into one without solving a costly permutation-matching problem.
• Federated learning: Averaging client models trained on separate data when no shared initialization exists.
• Continual learning: Consolidating models trained sequentially while preserving performance.
• Model ensembling by construction: Because LEWC makes the merged model behave like a logit ensemble, weight averaging can serve as a cheap substitute for running two models at inference.

Industry relevance. Permutation search is computationally expensive and scales poorly. If width, weight decay, and softmax temperature calibration can substitute for it, deployment pipelines for merging large trained models become simpler. The finding that weight decay strength directly determines mergeability also gives practitioners a concrete training-time knob.

Future Directions

• Scale beyond simple settings. The experiments cover MLPs, VGG-11, and ResNet-20 on CIFAR-10, CIFAR-100, MNIST, and FMNIST. Whether LEWC extends to larger-scale models and other modalities such as language or other data types is untested.
• Reduce the width cost. LEWC typically requires larger width multipliers than permutation-based merging, so finding ways to achieve permutation-free LMC at lower width is an open problem.
• Design training procedures that promote the conditions. The authors suggest training or permutation-search procedures that actively encourage weak additivity and reciprocal orthogonality.
• Handle logit norm decay. LEWC causes logit norms to decay exponentially with depth; simple rescaling to offset this is suggested as a practical direction.
• Reconcile LEWC with LLFC. The commutativity property underlying LLFC (Zhou et al., 2023) does not hold here, leaving open how the two frameworks relate in general architectures.

Target Audience

Researchers working on loss landscape geometry, linear mode connectivity, and permutation symmetries; practitioners building model-merging, federated learning, or continual learning systems who need to know when weight averaging is safe; and anyone interested in how model width and weight decay shape the solutions that stochastic gradient descent finds. A background in neural network training dynamics helps, but the central argument — wider models merge without permutations once you calibrate the softmax — is stated plainly enough for readers with intermediate machine learning knowledge.

Authors’ abstract

Recently, Ainsworth et al. empirically demonstrated that, given two independently trained models, applying a parameter permutation that preserves the input-output behavior allows the two models to be connected by a low-loss linear path. When such a path exists, the models are said to achieve linear mode connectivity (LMC). Prior studies, including Ainsworth et al.(2023), have reported that achieving LMC requires not only an appropriate permutation search but also sufficiently wide models (e.g., a 32 $\times$ width multiplier for ResNet-20). This is broadly believed to be because increasing the model width ensures a large enough space of candidate permutations, increasing the chance of finding one that yields LMC. In this work, we empirically demonstrate that, even without any permutations, simply widening the models is sufficient for achieving LMC when using a suitable softmax temperature calibration. We further explain why this phenomenon arises by analyzing intermediate layer outputs. Specifically, we introduce layerwise exponentially weighted connectivity (LEWC), which states that the output of each layer of the merged model can be represented as an exponentially weighted sum of the outputs of the corresponding layers of the original models. Consequently the merged model's output matches that of an ensemble of the original models, facilitating LMC. To the best of our knowledge, this work is the first to show that widening the model not only facilitates nonlinear mode connectivity, as suggested in prior research, but also significantly increases the possibility of achieving linear mode connectivity.