The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications

AI summary

Overview

• Research area: Sparse Mixture-of-Experts (MoE) architectures for large language models, covering routing and expert design, centralized versus decentralized training/inference paradigms, and vertical-domain applications.
• Technical level: Intermediate. The survey presents formal routing and load-balancing equations alongside systems-level parallelism and distributed-training concepts, so it assumes some familiarity with deep learning and model training infrastructure.
• Scope (one sentence): The paper surveys MoE from its algorithmic foundations through centralized and decentralized infrastructure to industry-specific applications, and identifies open challenges in the decentralized paradigm.

Note: the provided content is truncated. Sections 4 (vertical domain applications), 5 (challenges and opportunities), and 6 (conclusion) are only summarized by their stated scope; their detailed contents, and any benchmarks, datasets, or quantitative evaluations, are not reported in the available text.

What This Paper Is About

Sparse MoE models scale model parameters without proportionally scaling computation, because a routing network activates only a subset of expert subnetworks for each token. These models are now central to LLM development, but the paper argues that existing surveys have gaps: some omit recent breakthroughs, some ignore decentralized MoE paradigms, and none extensively explore vertical-industry applications. The goal of this survey is to fill those gaps by covering MoE core components, the shift from centralized to decentralized infrastructure, and domain-specific deployment, while identifying key challenges and future directions.

Key Contributions

1. Comprehensive and timely survey. The authors present what they describe as the most comprehensive review in the MoE field, systematically identifying work from algorithmic foundations to decentralized architectures and vertical domain applications, and analyzing motivations, technical principles, and key considerations in each.
2. In-depth exploration of the decentralized architecture paradigm. The survey delineates critical challenges of decentralized MoE, including hardware performance heterogeneity, communication efficiency, fault tolerance, and security and privacy protection, and reviews research addressing each. The stated aim is to evolve the MoE development paradigm "from 'competing for resources' to 'on-demand scalability.'"
3. Extensive exploration of vertical domain applications. The survey examines MoE applications in typical vertical domains, including medical diagnosis, autonomous driving, finance analysis, business intelligence, and blockchain.
4. Explicit positioning against prior surveys. Table 1 compares the survey with seven related surveys on three axes: comprehensive introduction of MoE core designs and recent advancements, delineation of the decentralized architecture paradigm, and extensive exploration of vertical domain applications. The survey is the only one marked as covering all three.

Main Findings

• MoE origins and mechanism. The concept dates to the 1991 Adaptive Mixture of Local Experts. An MoE divides the network into independent expert subnetworks and dynamically determines which experts process which tokens, activating only the most relevant few, so parameter count can grow substantially under the same computational budget.
• Model scale has grown sharply. Table 2 lists state-of-the-art open-source MoE models spanning DeepSeekMoE (2024.01) through MiniMax-M2 (2025.10). Examples include DeepSeekMoE with 64 routed plus 2 shared experts, 6 plus 2 activated, and 2.8B activated of 16.4B total parameters; DeepSeek-v3 with 256 plus 1 experts, 8 plus 1 activated, 37B of 671B parameters; KimiK2 with 384 plus 1 experts, 8 plus 1 activated, 32B of 1T parameters; and Qwen3-Next with 512 plus 1 experts, 10 plus 1 activated, 3B of 80B parameters.
• Open-source models now approach closed-source quality. A recent open-source MoE model (reference [54], listed as DeepSeek-v3 in the paper's model table) is described as having surpassed other open-source alternatives and demonstrated performance comparable to prominent closed-source models such as GPT-4o.
• Scale creates resource concentration. Some recent MoE models have scaled to 1T parameters with context windows exceeding 128K tokens, dramatically increasing computational demands. The authors state that high-performance computing clusters for advanced MoE work remain unaffordable for resource-limited individual researchers and small laboratories, leaving only a handful of large corporations and institutions able to develop such models and creating quasi-monopolies that stifle AI innovation.
• Two routing families. Token choice routing has each token select the top-k experts; expert choice routing has each expert pick the top-k tokens. The paper states expert choice allows tokens to be processed by varying numbers of experts while achieving perfect load balancing, resolving both expert-token assignment imbalance and the limitation of a uniform expert count per token.
• Load imbalance is a systemic risk. Because only a few experts are activated, some experts become overloaded and others underutilized, which can cause routing collapse where all tokens go to a few experts and the rest remain undertrained and ineffective. When experts are spread across devices, imbalance worsens computational bottlenecks.
• Balance is enforced through soft and hard mechanisms. Expert-Level Balance Loss and Device-Level Balance Loss are added as auxiliary losses, with the total training loss defined as L_total = L_moe + α·L_aux. A large α degrades performance, while a small α leads to training instability and resource underutilization. Expert capacity, defined as (T/E) × f, is the hard mechanism, where T is the number of tokens, E the total number of experts, and f a free hyperparameter called the capacity factor. Overflow tokens have their computation skipped and are passed to the next layer via residual connections or completely discarded.
• Capacity factor trade-off. If the capacity factor is too large, computational resources are wasted through excessive padding; if too small, model performance is sacrificed due to token discards. The paper's illustration uses T=8 and E=4, allowing up to 2 tokens per expert: at capacity factor 1.0 one token must be discarded, and at 1.5 excessive padding is added.
• Shared experts reduce redundancy. DeepSeekMoE introduces a fine-grained expert scheme dividing the expert network into shared experts, which process general features of all tokens and are always active, and routed experts, which are dynamically allocated. This division reduces model redundancy, improves computational efficiency, and prevents multiple experts from converging on the same shared knowledge.
• Interpretability is a natural property. MoE models lend themselves to interpretability studies because each input is processed by an identifiable, discrete subset of weights. Examples cited include experts specializing in punctuation marks or proper nouns, multilingual training, and a multimodal model observed to learn experts specializing in textual and visual data, including patches of textures, plants, eyes, and words.
• Centralized infrastructure relies on RDMA. Current training and inference run on high-performance dedicated clusters interconnected by high-speed RDMA networks, which in data centers can reach over 400 Gb/s. Table 3 lists efficient centralized frameworks from Switch (2022.01) through DeepEP (2025.02), combining expert, data, pipeline, and tensor parallelism.
• Expert parallelism has a communication cost. Expert parallelism assigns different experts to distinct devices and uses all-to-all communication to dispatch inputs and return outputs. As the scale and frequency of all-to-all communication grow exponentially, communication time increases significantly and overall training efficiency falls.
• Decentralized bandwidth is far lower. Decentralized training communicates mainly through LANs or even WANs with bandwidths under 10 Gb/s, making gradient exchange a bottleneck and leading to GPU underutilization, with the problem worse in geographically distributed hybrid clusters.
• Decentralized resources are heterogeneous. Devices aggregated across consumer-grade GPUs, nodes, or clusters vary in memory size and communication bandwidth, ranging from mobile GPUs to mid-end and high-performance GPUs, requiring techniques that prevent slower devices from becoming bottlenecks.
• Fault tolerance is built on distributed hash tables. Nodes can join at any time and existing nodes may exit. DHTs store key-value pairs across computers, with each computer needing to know only O(log N) of N computers and able to look up any key in at most O(log N) requests. The paper names Kademlia as the most popular DHT variant, used in BitTorrent, I2P, and Ethereum.
• Privacy requires cryptographic approaches. Risks include unauthorized data access and manipulation, malicious compute nodes, and membership inference. Mitigation strategies cited include secure multiparty computation, differential privacy, encryption, and Zero-Knowledge Machine Learning, which applies Zero-Knowledge Proofs to let data owners train models without sharing raw data.

Methodology in Plain English

This is a survey rather than an experimental study, so its method is structured literature organization rather than data collection. The authors first define MoE through its two core components, the routing network and the expert network, and trace how each has been designed across the literature. They present the standard mathematical formulations for token choice routing, expert choice routing, auxiliary balance losses, and expert capacity, and analyze the trade-offs built into each.

They then organize the systems landscape into two paradigms. The centralized paradigm covers work that trains and serves MoE models on homogeneous, high-performance clusters, describing expert parallelism and the hybrid combinations of data, pipeline, and tensor parallelism, with a table mapping which frameworks use which strategies and when they appeared.

The decentralized paradigm is organized around four challenges the authors identify, and for each they review specific systems: heterogeneity, where approaches range from assigning experts by device capacity to DAG splitting and clustering by RAM and bandwidth; communication efficiency, where approaches include dynamic grouping, fault-tolerant pipelining, low-frequency synchronization, gradient compression, and partial staleness; fault tolerance, where distributed hash tables provide the underlying substrate; and security and privacy, where zero-knowledge proofs and collaborative privacy-preserving MoE designs are proposed. A summary table maps each challenge to the works addressing it and its key features.

Finally, the authors shift from horizontally studied applications such as NLP, computer vision, and multimodal processing to vertical industries, then analyze challenges and opportunities.

Why This Matters

• Impact on research. The survey consolidates a fragmented literature into one reference spanning algorithms, infrastructure, and applications, and explicitly contrasts itself with seven prior surveys on three coverage axes. Its emphasis on decentralized training reframes MoE development as a scalability and access question rather than purely a model-architecture question.
• Democratization argument. The authors argue that decentralized clusters and consumer-level devices hold significantly more computing resources than centralized clusters, and that these resources are typically overlooked because of lower bandwidth and compute power. Mobilizing them is presented as a path to broader participation in MoE development.
• Real-world applications (as listed in the paper):
  • Medical diagnosis
  • Autonomous driving
  • Financial analysis
  • Business intelligence
  • Blockchain
• Industry relevance. Because only organizations with large-scale high-end GPU clusters can currently develop advanced MoE models, the paper frames infrastructure access as a competitive and innovation issue. The centralized-paradigm frameworks table (Switch through DeepEP, spanning 2022.01 to 2025.02) and the open-source model table (DeepSeekMoE through MiniMax-M2, spanning 2024.01 to 2025.10) together map the pace of both tooling and model release. The stated goal is to move from "competing for resources" to "on-demand scalability."

Future Directions

• Resolving the heterogeneity problem at scale. The survey identifies hardware performance heterogeneity as a critical challenge and reviews systems such as DeDLOC, Petals, ATOM, Ravnest, and HeterMoE, but the text does not report a unified solution, leaving open how to best schedule work across mobile GPUs, mid-end GPUs, and high-performance GPUs.
• Closing the bandwidth gap. With decentralized communication running under 10 Gb/s versus over 400 Gb/s for RDMA data centers, the paper's reviewed approaches (Moshpit's dynamic grouping, SWARM's fault-tolerant pipelining, DiLoCo's low-frequency synchronization, DiLoCoX's gradient compression, StellaTrain's partial staleness) indicate the problem is actively attacked but not settled.
• Making fault tolerance and heterogeneous training coexist. Decentralized nodes join and leave at any time, so the interaction between dynamic membership, DHT-based coordination, and training correctness remains a design tension the paper raises rather than resolves.
• Preserving privacy without sacrificing utility. The paper's discussion of PC-MoE is truncated mid-sentence, but the framing indicates the open question of whether privacy preservation always imposes a cost on utility, and vice versa. The paper says the technique begins to address this but the content cuts off before the conclusion is stated.
• Vertical domain best practices. The authors position extensive vertical-domain exploration as a core contribution and promise readers a reference for implementing MoE in specific tasks; the details of that analysis are not included in the provided content.

Target Audience

Researchers and practitioners working on large language models and efficient training infrastructure, especially those interested in MoE architecture design, routing and load-balancing mechanisms, distributed and decentralized training, and privacy-preserving collaborative learning. The survey is also aimed at readers evaluating how MoE applies to specific vertical industries, since the authors state they intend it as a resource for both researchers and practitioners to track the latest advancements and inspire new ideas. Readers without background in distributed systems or model parallelization will need to supplement the systems sections, since terms such as expert parallelism, pipeline parallelism, tensor parallelism, all-to-all communication, and RDMA are used with the assumption of prior familiarity.

Authors’ abstract

The sparse Mixture of Experts(MoE) architecture has evolved as a powerful approach for scaling deep learning models to more parameters with comparable computation cost. As an important branch of large language model(LLM), MoE model only activate a subset of experts based on a routing network. This sparse conditional computation mechanism significantly improves computational efficiency, paving a promising path for greater scalability and cost-efficiency. It not only enhance downstream applications such as natural language processing, computer vision, and multimodal in various horizontal domains, but also exhibit broad applicability across vertical domains. Despite the growing popularity and application of MoE models across various domains, there lacks a systematic exploration of recent advancements of MoE in many important fields. Existing surveys on MoE suffer from limitations such as lack coverage or none extensively exploration of key areas. This survey seeks to fill these gaps. In this paper, Firstly, we examine the foundational principles of MoE, with an in-depth exploration of its core components-the routing network and expert network. Subsequently, we extend beyond the centralized paradigm to the decentralized paradigm, which unlocks the immense untapped potential of decentralized infrastructure, enables democratization of MoE development for broader communities, and delivers greater scalability and cost-efficiency. Furthermore we focus on exploring its vertical domain applications. Finally, we also identify key challenges and promising future research directions. To the best of our knowledge, this survey is currently the most comprehensive review in the field of MoE. We aim for this article to serve as a valuable resource for both researchers and practitioners, enabling them to navigate and stay up-to-date with the latest advancements.