MoDES: Accelerating Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert Skipping

AI summary

Overview

Research area: Efficient inference for Mixture-of-Experts (MoE) Multimodal Large Language Models (MLLMs), specifically dynamic expert skipping.

Technical level: Intermediate — readers should understand transformer FFN layers, MoE routing (top-k gating), and the basic architecture of vision-language models, but no advanced mathematics beyond KL divergence is required.

Scope: This paper introduces a training-free framework (MoDES) that adaptively skips redundant experts during inference in MoE MLLMs, delivering large accuracy retention at aggressive skipping ratios (up to 88%) and measurable wall-clock speedups.

What This Paper Is About

Mixture-of-Experts multimodal models route each token through a small subset of "expert" subnetworks, but even that subset is often larger than necessary — many activated experts contribute almost nothing to the final output. Prior "expert skipping" methods that remove these redundant experts were designed for text-only LLMs, and the authors show that naively applying them to multimodal models causes accuracy drops exceeding 10%. This paper asks why those methods fail on MLLMs and builds a skipping framework that respects two properties the prior work ignored: layer-wise differences in expert importance and modality differences between text and vision tokens.

Key Contributions

1. Two diagnostic insights about MoE MLLMs. The authors show empirically that (i) experts in shallow layers contribute far more to final outputs than experts in deep layers, because errors introduced early are amplified by subsequent layers, and (ii) text and vision tokens behave differently inside FFNs — vision tokens are more orthogonal to FFN weights and are therefore updated much less, meaning vision experts can be skipped more aggressively.

2. Globally-Modulated Local Gating (GMLG). A scoring mechanism that multiplies each expert's local routing probability by an offline-calibrated, layer-specific global importance factor α^(l), derived from KL divergence between the original model's output distribution and the distribution produced when that layer's experts are removed.

3. Dual-Modality Thresholding (DMT). Two separate skipping thresholds — one for text tokens, one for vision tokens — instead of a single global threshold, directly implementing the modality-gap insight.

4. Frontier search for threshold optimization. An algorithm exploiting monotonicity of the accuracy loss and skip ratio with respect to the thresholds, reducing search complexity from O(ND²) to O(ND) — roughly a 45× reduction in search time, cutting optimization from days to hours.

Main Findings

• Prior methods collapse at high skip rates; MoDES does not. On Kimi-VL-A3B-Instruct, NAEE, MC-MoE, and DiEP lose more than 11% average accuracy when 83% of experts are skipped. MoDES retains 96.25% of the full model's accuracy at the same skip ratio.

• Largest gain on Qwen3-VL-MoE-30B-A3B-Instruct. Skipping 88% of experts yields a 10.67 percentage-point improvement over the best baseline (97.33% vs. 86.66%) on the aggregate benchmark score.

• Consistent across three model families. MoDES outperforms baselines by more than 5 percentage points on average across Kimi-VL, Qwen3-VL-MoE, and InternVL-3.5 backbones (including a GPT-OSS-based variant), with the pattern holding at multiple skipping ratios.

• Real speedups, not just theoretical sparsity. Roughly 2.16× faster prefilling and 1.26× faster decoding for Qwen3-VL-MoE-30B-A3B-Instruct on a single H200 GPU. The smaller decoding gain is attributed to decoding being memory-bound and involving only text tokens (which get skipped less).

• Complementary to quantization. Combined with mixed-precision weight quantization (2.5-bit), MoDES retains 94.43% of original performance on Qwen3-VL-MoE, compared with 89.58% for MC-MoE; at 1.5-bit, MoDES degrades by ~17% vs. >20% for MC-MoE.

• Skipping sometimes improves accuracy. On benchmarks such as RealWorldQA and VideoMME, skipping certain experts actually raises accuracy above the unmodified model — suggesting some experts actively interfere rather than merely being redundant.

• Search is cheap. For 20–30B parameter models, calibration plus frontier search completes in 20 minutes to under 4 hours on 8×H200 GPUs. Frontier search matches exhaustive grid search almost exactly (96.24% vs. 96.25% on Kimi-VL at 83% skip).

• Deployment pattern confirms the insights. Visualization shows MoDES skips progressively more experts in deeper layers and far more experts for vision tokens than text tokens, mirroring the two motivating observations.

Methodology in Plain English

The method has three stages.

First, offline calibration. The team runs a small set of calibration samples (1024 from GQA) through the model. For each MoE layer, they ask: how much does the model's output distribution change if I remove the experts in this layer? They measure this with KL divergence and average over the sample set. This gives a scalar α^(l) per layer — a measure of how globally important that layer is. Shallow layers get large α values; deep layers get small ones. This is computed once and adds no inference-time cost.

Second, a runtime importance score. During inference, when a token reaches layer l, each activated expert gets a score that is simply its local routing probability multiplied by the layer's calibrated α^(l). This blends "how much the router wants this expert for this token" with "how much this layer matters overall."

Third, modality-aware skipping. Two thresholds are chosen — one for text tokens, one for vision tokens. Any activated expert whose score falls below the relevant threshold is not executed. Because vision tokens get a higher threshold (or equivalently, are treated as more skippable), more vision experts are dropped, consistent with the observation that FFNs barely change vision token representations.

Finding the thresholds. Searching over all pairs of thresholds on a 100-point grid would take days. The authors note that both the accuracy loss and the skip ratio increase monotonically with the thresholds. They exploit this to walk a "frontier" — for each text threshold, find the smallest vision threshold that meets the target skip ratio — evaluating only O(D) pairs instead of O(D²). The pair on this frontier with the lowest accuracy loss is chosen.

Why This Matters

Impact on research. This is the first expert-skipping framework designed specifically for MoE MLLMs rather than adapted from text-only LLMs. It provides a general recipe for the modality- and layer-aware treatment of sparse computation, and it openly challenges the assumption that layers of a transformer are interchangeable for pruning purposes. The training-free nature means it can be applied as a drop-in to existing deployed models.

Real-world applications:

• On-device and edge vision assistants — running multimodal models on phones or embedded hardware where every FLOP and every memory access counts.
• High-throughput document and image processing — OCR, chart understanding, and visual QA pipelines that must serve many concurrent requests under latency SLAs.
• Video understanding platforms — long-video QA and surveillance analysis, where per-frame vision tokens dominate the token budget and are exactly the tokens MoDES skips most aggressively.
• Cost reduction in cloud inference — pairing with quantization to compress MoE MLLM serving costs, as demonstrated in the mixed-precision experiments.

Industry relevance. MoE architectures are now the default for frontier open-weight multimodal models (Qwen3-VL-MoE, Kimi-VL, InternVL). A training-free, calibration-only method that retains >95% accuracy while executing only 12–17% of experts directly translates into GPU-hour savings and lower serving costs, with no retraining or data-license complications.

Future Directions

1. Combining MoDES with orthogonal compression techniques. The authors explicitly flag pruning and distillation as future work, and the quantization experiments suggest stacking multiple efficiency methods can compound rather than conflict.

2. Deeper investigation of "harmful" experts. The finding that skipping experts occasionally improves accuracy opens a question about whether some experts systematically add noise, and whether this can be predicted from the α calibration.

3. Extending beyond vision and text. The dual-modality thresholding scheme generalizes in principle to three or more modalities (audio, depth, sensor streams), but this is untested.

4. Closing the decoding speedup gap. Decoding gains (~1.26×) lag prefilling gains (~2.16×) because decoding is memory-bound and processes only text tokens. Better skip scheduling or KV-cache-aware techniques may recover more of the decoding budget.

5. Reducing calibration cost further. The method currently needs a forward pass over skip-variants for every layer; scaling this analysis to 100B+ models, or eliminating the calibration entirely, remains open.

Target Audience

• Systems and inference engineers deploying MoE multimodal models who need concrete efficiency gains without retraining.
• Efficient-ML researchers working on sparsity, pruning, quantization, and dynamic computation, who will benefit from the layer- and modality-aware framing.
• Multimodal model developers interested in understanding where redundancy lives inside MoE MLLMs.
• Graduate students entering the efficient-inference field, since the paper's two motivating insights are clearly demonstrated with visualizations and ablation tables.

The paper is accessible to anyone comfortable with standard transformer and MoE terminology; the core algorithm is conceptually simple (a calibrated score plus a threshold) even though the engineering and evaluation are extensive.

Authors’ abstract

Mixture-of-Experts (MoE) Multimodal large language models (MLLMs) excel at vision-language tasks, but they suffer from high computational inefficiency. To reduce inference overhead, expert skipping methods have been proposed to deactivate redundant experts based on the current input tokens. However, we find that applying these methods-originally designed for unimodal large language models (LLMs)-to MLLMs results in considerable performance degradation. This is primarily because such methods fail to account for the heterogeneous contributions of experts across MoE layers and modality-specific behaviors of tokens within these layers. Motivated by these findings, we propose MoDES, the first training-free framework that adaptively skips experts to enable efficient and accurate MoE MLLM inference. It incorporates a globally-modulated local gating (GMLG) mechanism that integrates global layer-wise importance into local routing probabilities to accurately estimate per-token expert importance. A dual-modality thresholding (DMT) method is then applied, which processes tokens from each modality separately, to derive the skipping schedule. To set the optimal thresholds, we introduce a frontier search algorithm that exploits monotonicity properties, cutting convergence time from several days to a few hours. Extensive experiments for 3 model series across 13 benchmarks demonstrate that MoDES far outperforms previous approaches. For instance, when skipping 88% experts for Qwen3-VL-MoE-30B-A3B-Instruct, the performance boost is up to 10.67% (97.33% vs. 86.66%). Furthermore, MoDES significantly enhances inference speed, improving the prefilling time by 2.16$\times$ and the decoding time by 1.26$\times$. Our code is available at https://github.com/ModelTC/MoDES.