Expert Merging in Sparse Mixture of Experts with Nash Bargaining

AI summary

Overview

• Research area: Machine learning — specifically sparse Mixture of Experts (SMoE) architectures, model merging, and cooperative game theory.
• Technical level: Intermediate. Readers should be comfortable with neural network training basics and the general idea of MoE routing; the game-theoretic portion is explained from first principles but involves some matrix algebra.
• Scope: The paper introduces a principled, game-theoretic method for merging the parameters of many experts in an SMoE layer into a single unified expert, then validates it across vision and language benchmarks up to 16B parameters.

What This Paper Is About

Sparse Mixture of Experts models contain many separate "expert" sub-networks, but sometimes you need a single compact model — for deployment, memory limits, or transfer. Existing techniques for collapsing experts into one model mostly use heuristic averaging, giving each expert a weight without any principled reason. This paper asks whether merging can instead be treated as a bargaining negotiation among experts, where each expert argues for its own contribution and the final merged model emerges from a fair, Pareto-optimal agreement.

Key Contributions

1. NAMEx (Nash Merging of Experts): A new expert-merging framework that recasts the merging of expert parameters as a Nash Bargaining problem among experts, where each expert's deviation from a shared base expert (its "domain vector") acts as a utility function. Solving the bargaining problem yields merging coefficients derived from first principles rather than heuristics.
2. NAMEx-Momentum: An extension that adds complex-valued momentum to the cross-layer propagation of the base expert, accelerating convergence and stabilizing training when experts have conflicting or adversarial interactions. The authors provide a convergence guarantee (Proposition 3.5) and a spectral-radius-based convergence-rate bound.
3. Quaternion momentum as a proposed direction: The authors identify quaternion-valued momentum as a promising route to further improving expert merging, framing it as future work rather than a fully validated method.
4. Large-scale validation: Comprehensive experiments on WikiText-103 language modeling, GLUE text classification, ImageNet-1K classification plus corrupted-image robustness (ImageNet-A/O/R), and deployment on Qwen1.5-MoE (14B) and DeepSeek-MoE (16B) in both zero-shot and fine-tuning settings.

Main Findings

• Language modeling gains: On WikiText-103, NAMEx-Full-Mom achieves the best perplexity in both small and medium pretraining regimes. Small-scale validation perplexity drops to 82.44 versus 83.53 for CAMEx and 86.64 for Top-1 SMoE; medium-scale validation reaches 34.25 versus 35.69 for CAMEx and 38.60 for SMoE.
• GLUE improvements across the board: NAMEx-Full-Mom is best on all seven GLUE tasks with a T5-Base backbone, including 95.06 on SST-2, 93.27 on MRPC, 60.13 on CoLA, 78.15 on RTE, and 87.45 on MNLI.
• Vision gains and robustness: On ImageNet-1K, NAMEx-Full-Mom reaches 84.52 top-1 accuracy versus 83.29 for CAMEx and 83.15 for SMoE. The robustness gap is larger: on ImageNet-O it reaches 51.34 versus 50.69 for CAMEx and 43.34 for SMoE; on ImageNet-A, 35.27 versus 25.45 for CAMEx and 23.72 for SMoE.
• Momentum matters most for robustness: Comparing NAMEx to NAMEx-Mom on corrupted images shows that complex momentum supplies substantial additional gains on ImageNet-A (25.32 to 35.05), suggesting it is particularly valuable when experts conflict.
• Bargaining captures cooperation and competition: The authors show analytically that when the interaction term between a given expert and the rest is positive (cooperative), that expert's weight decreases, and when the term is negative (adversarial), the weight increases to restore the Nash condition. This gives the method an automatic mechanism for handling expert disagreement.
• Merging behavior is layer-dependent: Analysis of expert output cosine similarity in Swin-MoE, Switch-Transformer, and Qwen-MoE shows that cooperative and competitive patterns vary by layer and architecture, motivating the dynamic, layer-wise weighting NAMEx uses.
• EP-CAMEx is largely a special case: The paper argues that prior propagation-based merging ignores inter-expert interaction and can be recovered as a scaled, interaction-free trivial solution of the Nash system.

Methodology in Plain English

The starting point is CAMEx, an earlier merging method that updates a shared "base expert" by nudging it toward each expert, weighted by routing scores and scaled by a curvature matrix. An extension, EP-CAMEx, carries that base expert across layers so information propagates from shallow to deep layers. The authors observed that EP-CAMEx actually performs worse than its static counterpart, and suspected the problem was that it averaged expert contributions without any notion of how experts relate to one another.

To fix this, they borrow from multi-task learning. In that literature, when several tasks propose conflicting gradient directions, you can treat each task as a player in a cooperative game and solve for a Nash Bargaining Solution — a set of weights that is fair and Pareto-optimal, meaning no player can be made better off without hurting another. The authors map each expert's deviation from the base expert (its "domain vector") onto a task gradient, and solve the same bargaining equation. The solution requires solving a small nonlinear system where the weight vector α satisfies GᵀGα = 1/α element-wise, with G the matrix of domain vectors.

Because this bargaining step can only happen once per layer, the base expert's adjustment is limited by the model depth. To compensate, the authors add complex-valued momentum to the accumulation of these update directions. Complex momentum has been shown to be more robust than ordinary momentum in adversarial game settings, because the imaginary component can carry information about oscillatory or conflicting dynamics. After each layer, only the real part of the momentum buffer is added to the parameters. The paper proves under mild conditions that this procedure converges and derives a bound on the rate.

Experimentally, they keep the computational budget matched to baselines by fixing the bargaining solver at 20 iterations per batch, and test two variants: NAMEx (solve α once at the first layer and reuse it) and NAMEx-Full (spread the budget evenly across layers). Results are averaged over five seeds on an 8×A100 server.

Why This Matters

The paper's core insight is that merging experts is not a neutral arithmetic operation — experts cooperate and compete, and the merging rule should reflect that. This shifts expert merging from a heuristic engineering choice into something with a game-theoretic justification, and the empirical results show that the principled version is measurably better, especially on distribution-shifted data.

Real-world applications:

• On-device and edge deployment: Collapsing a large SMoE into a single merged expert lets models fit into phone, browser, or embedded memory budgets without retraining from scratch.
• Robustness-critical systems: The large gains on ImageNet-A and ImageNet-O suggest the method helps models retain accuracy on corrupted, adversarial, or out-of-distribution inputs — relevant for autonomous driving, medical imaging, and industrial inspection.
• Cross-domain transfer: Merging experts trained on different domains into one model supports transfer learning and multi-domain serving, where a single system must handle heterogeneous inputs.
• Efficient model serving: A merged single-expert representation removes the routing overhead and memory footprint of holding many expert parameter sets live during inference.

Industry relevance: Most frontier open-weight MoE systems (Qwen, DeepSeek, Mixtral-style architectures) carry tens to hundreds of experts per layer. Merging is a direct lever for reducing serving cost and enabling distillation of large MoE checkpoints into smaller deployable models. The authors validate on Qwen1.5-MoE (14B) and DeepSeek-MoE (16B), signaling the method is intended for production-scale models rather than toy settings.

Future Directions

• Quaternion momentum: The authors explicitly flag quaternion-valued momentum buffers as an unexplored direction that may capture richer inter-expert interaction structure than complex momentum.
• Interaction structure at scale: The cosine-similarity analysis shows that cooperation and competition vary by layer and architecture, but the paper does not yet use that structure to design architecture-specific or adaptive bargaining schedules.
• Efficiency of the bargaining solver: Solving GᵀGα = 1/α requires an iterative solver capped at 20 iterations per batch; faster closed-form approximations or cached solutions could reduce this overhead further.
• Interaction with routing and load balancing: The paper analyzes expert interaction under load-balancing loss, but does not fully explore how NAMEx should be co-designed with routing policies in frontier systems where balancing pressure shapes expert specialization.

Target Audience

This paper is most useful to machine learning researchers and engineers working on Mixture of Experts architectures, model compression, or model merging. It is also relevant to readers interested in applications of cooperative game theory — particularly Nash Bargaining — to deep learning optimization, since the multi-task learning framing is cleanly transferred here. Practitioners deploying large MoE checkpoints on constrained hardware will find the empirical results directly actionable, while readers unfamiliar with MoE routing may want to start with the background section on CAMEx and SMoE before the bargaining formulation.

Authors’ abstract

Existing expert merging strategies for Sparse Mixture of Experts (SMoE) typically rely on input-dependent or input-independent averaging of expert parameters, but often lack a principled weighting mechanism. In this work, we reinterpret expert merging through the lens of game theory, revealing cooperative and competitive dynamics among experts. Based on this perspective, we introduce Nash Merging of Experts (NAMEx), a novel framework that incorporates Nash Bargaining into the merging process, enabling more balanced and efficient collaboration among experts. Additionally, we incorporate complex momentum into NAMEx to accelerate expert propagation with theoretical guarantees for convergence. Extensive experiments across language modelling, text classification, image classification, and zero-shot robustness under data corruption show that NAMEx consistently outperforms competing methods while integrating seamlessly with popular MoE architectures. Finally, we demonstrate NAMEx's scalability by applying it to large-scale systems, including Qwen1.5-MoE (14B) and DeepSeek-MoE (16B), where it proves effective in both zero-shot and fine-tuning settings. The code is publicly available at: https://github.com/anh147/NAMEx.