Mixture-of-Experts and Sparse Conditional Computation

A router scores several expert networks for each token or example. The system dispatches the representation to one or a few selected experts. It then combines their outputs.

Total parameter capacity can grow while active computation stays closer to the selected subset. That separation is older than the Transformer backbone. In 2017 a sparsely-gated mixture-of-experts layer went between stacked LSTM layers. Shazeer and six colleagues reported “greater than 1000x improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters”, from architectures “in which a MoE with up to 137 billion parameters is applied convolutionally between stacked LSTM layers”. Capacity and active computation were already being priced separately in 2017. The advantage still depends on efficient dispatch and balanced use across devices.

Routing decisions create both a learning problem and a distributed-systems problem. The steps below are one design, not the design. Step two and step three are a contested pair: tokens choose experts, then a fixed buffer decides which of those tokens are actually served.

In 2022 Expert Choice routing inverted that order. Instead of tokens picking experts, experts pick tokens. The abstract states the whole idea in one sentence: “Instead of letting tokens select the top-k experts, we have experts selecting the top-k tokens.” Every expert then receives a fixed bucket size by construction. The capacity step is removed rather than tuned.

On Google's research blog, Zhou is blunt about the ordering being replaced: “The independent token choice approach often leads to an imbalanced load of experts and under-utilization.” The post is equally blunt about what the inversion bought: “EC routing speeds up training convergence by over 2x in an 8B/64E model, compared to the top-1 and top-2 gating counterparts.” The paper measures more than 2x faster training convergence than Switch Transformer top-1 and GShard top-2 gating at equal compute. It blames load imbalance under token-choice top-k for under-trained and mis-specialized experts. It reports higher fine-tuned quality on 11 GLUE and SuperGLUE tasks. The step order was never neutral.

Parameter count, active FLOPs, communication, and memory must be reported separately — and for shipped models they are.

Top-2 routing is not a diagram. It is what Mixtral 8x7B runs: 8 feed-forward experts per layer, and a router that selects two of them for each token. Mistral put the arithmetic in the abstract: “As a result, each token has access to 47B parameters, but only uses 13B active parameters during inference.” The release note of 11 December 2023 gives the same split more precisely, as 46.7B total and 12.9B active. That note also put the model under Apache 2.0.

The shared-plus-routed row has an instance too. DeepSeek-V3's technical report, from December 2024, opens its abstract this way: “We present DeepSeek-V3, a strong Mixture-of-Experts (MoE) language model with 671B total parameters with 37B activated for each token.” Each MoE layer there holds 1 shared expert and 256 routed experts, of which 8 routed experts fire per token. Load balancing uses the auxiliary-loss-free bias strategy rather than a balancing loss. Full training took 2.788M H800 GPU-hours.

The sparse rows were demonstrated long before either model. The 2017 sparsely-gated layer already separated capacity from active computation. Mixtral and DeepSeek-V3 are what that separation looks like once it has to be served.

Router statistics reveal failures hidden by aggregate loss. Two of the five below have a name in the literature, a published fix, and a command-line flag in a production training framework.

Measure quality and systems behavior on the same workloads. Step 2 in particular is a configuration exercise rather than a research one. NVIDIA's Megatron-Core exposes the overflow mechanism as --moe-expert-capacity-factor, --moe-pad-expert-input-to-capacity and --moe-token-drop-policy, whose policies are probs and position. An engineer can set the capacity factor, decide which tokens lose their expert, and count what the setting cost.

Inactive expert weights still consume storage, optimizer state during training, and checkpoint bandwidth. Expert parallelism adds placement and communication constraints.

Report total parameters, active parameters, FLOPs, peak memory, network traffic, throughput, and quality. GLaM is a published model where those numbers appear together. Its 2022 abstract: “The largest GLaM has 1.2 trillion parameters, which is approximately 7x larger than GPT-3. It consumes only 1/3 of the energy used to train GPT-3 and requires half of the computation flops for inference, while still achieving better overall zero, one and few-shot performance across 29 NLP tasks.” Behind those figures sits a specific architecture: 64 experts per MoE layer, an MoE layer replacing the feed-forward block of every other layer of a 64-layer decoder — 32 MoE layers — and top-2 gating. Each token therefore activates a 96.6B-parameter subnetwork, 8% of the 1.2T; Google's blog rounds this to 97B. On average GLaM outperforms GPT-3 in 6 of 7 task categories.

Serving is the other half of the move, and it did not arrive free with the sparsity. Microsoft built a compression plus inference system to obtain it. DeepSpeed-MoE cuts MoE model size by up to 3.7x. It reports 7.3x better latency and cost than existing MoE inference solutions, and on that basis “DeepSpeed-MoE offers an unprecedented scale and efficiency to serve massive MoE models with up to 4.5x faster and 9x cheaper inference compared to quality-equivalent dense models.” Note what that last comparison is against. Quality-equivalent dense models, not models at matched active compute. Sparse models should be compared with dense baselines at matched serving budgets.