Modality-Aware Expert Pruning for MoE-Based Multimodal Large Language Models

AI summary

Overview

Research area: Model compression and efficient inference for Mixture-of-Experts (MoE) multimodal large language models (MLLMs) — sitting at the intersection of computer vision, efficient deep learning, and vision-language modeling.

Technical level: Intermediate. Readers should know what MoE routing (top-k expert selection), vision tokens, and calibration-based pruning are; the method itself is training-free and conceptually simple.

Scope: The paper proposes MAEP, a training-free framework that scores expert importance separately for text and vision tokens and globally prunes routed experts in MoE-based MLLMs, evaluated on three models and eleven benchmarks.

What This Paper Is About

MoE-based multimodal models only activate a handful of experts per token (8 of 128 in the models studied), yet every expert must stay in GPU memory — experts account for 93–94% of total model size (54 GB out of ~58 GB in BF16). Existing expert-pruning methods average importance over all tokens uniformly, ignoring that the same expert may play different roles for vision versus language inputs. The goal is to prune experts using modality-decomposed importance so that memory drops substantially without disproportionately damaging either modality.

Key Contributions

1. Delta-H Output (ΔH), a redistribution-aware importance metric. ΔH measures the squared L2 change in a MoE layer's output when an expert is removed and its routing weight is redistributed to the remaining experts. Unlike routing-frequency or isolated-contribution scores, it captures residual redundancy after co-activated experts absorb the removed expert's load, and it can be computed for all experts in a single forward pass.

2. Low-attention vision tokens as a dataset-stable vision-side pruning signal. Instead of using all vision tokens (or high-attention ones), the method scores ΔH only on the bottom q-th percentile of text-to-vision attention, which yields expert-importance rankings that stay stable across calibration datasets.

3. An asymmetric two-signal scoring rule with global pruning. Text-side ΔH acts as a preservation signal (high → keep) and low-attention vision-side ΔH as a pruning signal (high → remove); the combined score uses layer-wise z-score normalization and a closed-form, model-specific modality balance α, and experts are ranked globally across all layers rather than under a fixed per-layer quota.

4. A theoretical and empirical grounding for ΔH. Proposition 1 bounds the LM-head KL divergence between the unpruned and pruned distributions by a layer-dependent constant times the squared L2 MoE-output change, so within-layer ranking by ΔH is equivalent to ranking by that bound.

Main Findings

• Memory reduction: At 50% expert pruning, MAEP reduces the BF16 routed-expert weight footprint by ~27 GB (~47% of total weight memory), enough to fit a ~58 GB model's full weight footprint within a single 40 GB GPU.

• Best average retention at both pruning ratios: MAEP achieves the best or tied-best average retention on every model–ratio configuration in Table 1. Average retention across all 11 benchmarks: InternVL3.5-30B-A3B 97.7% at 25% and 91.2% at 50%; Qwen3-VL-30B-A3B 96.6% and 88.3%; Gemma4-26B-A4B 96.0% and 86.7%.

• Baseline comparison: REAP reaches 95.9% / 88.5% / 96.0% at 25% and 96.0% is where MAEP ties it on Gemma4; at 25% REAP stays within ~2 points of MAEP, but at 50% the gap widens to 2–6 points. HC-SMoE falls to 74.1% (InternVL3.5), 85.5% (Qwen3-VL), and 39.3% (Gemma4) at 50% pruning.

• Text-only trade-off: All methods degrade substantially on text-only benchmarks at 50% pruning; MAEP achieves the highest aggregate retention but trails REAP on the three Qwen3-VL text-only benchmarks (MMLU-Redux, IFEval, ARC-Challenge).

• Modality decomposition is essential: Removing the text/vision split drops InternVL3.5 at 50% pruning from 2160/63.3/76.7/75.6 to 2034/51.1/41.2/66.0 on MME/MMStar/ChartQA/TextVQA.

• Vision-signal choice matters more than almost anything else: High-attention subsets of {2, 5, 10}% cause collapse; top-20% high-attention gives 2226/55.5/54.5/76.1 and full-vision gives 2152/48.3/38.8/66.9, with a −22.2 ChartQA drop for high-attention relative to the default.

• Rankings genuinely differ by modality: Spearman ρ between text-based and vision-based expert rankings is between −0.11 and 0.05 at specific layers, and expert removal shifts the LM-head output distribution up to 29× more on vision-token positions than on text-token positions.

• Dataset stability hierarchy: Cross-dataset Spearman ρ (ShareGPT4V vs. ChartNet) is ≈0.0 for full vision, ≈0.2 for high-attention vision (top q≈20%), ≈0.5 for text, and ≈0.6 for low-attention vision. At size-matched top/bottom 2%, low-attention reaches 0.54 versus 0.33 for high-attention and ≈0 for full vision.

• Global beats uniform pruning: Even with α re-tuned for a per-layer quota (α = 0.5), global allocation leads on MME (2160 vs. 2104) and ChartQA (76.7 vs. 72.1) on InternVL3.5 at 50%.

• Calibration domain matters and can be exploited: A mixed 700 ShareGPT4V + 300 ChartNet calibration set recovers 13.9 points on ChartQA and improves MMStar by 4.1 points on Qwen3-VL at 50% pruning, with MME and TextVQA nearly unchanged.

• Calibration size is not critical: Reducing from 1,000 to 100 ShareGPT4V samples retains most of the default performance (e.g., MME 2006.0 vs. 2042.8), with some fluctuation on individual benchmarks such as ChartQA.

• Closed-form α lands in the stable region: The rule yields 0.75 / 0.33 / 0.08 for InternVL3.5 / Qwen3-VL / Gemma4, rounded to 0.8 / 0.3 / 0.1. Qwen3-VL's retention collapses above its operating point (e.g., 25.4% at α=0.5, 50% pruning), while InternVL3.5 tolerates α ≤ 0.7 but falls to 65.1% at α=0.9.

• ΔH correlates with KL in deeper layers: Per-layer Spearman between ΔH and actual KL exceeds 0.8 in deeper layers for vision and reaches ~0.89 for text, consistent with the bound's depth dependence.

Methodology in Plain English

The researchers start from a practical measurement: in these models only 8 of 128 experts fire per token, but all 128 must sit in memory. To decide which experts to throw away, they ask a counterfactual question for each expert, layer, and token: what would the MoE layer's output be if this expert were deleted and its routing weight shared among the surviving experts? The squared L2 size of that output change is ΔH. Because router weights and expert outputs are cached, every expert's counterfactual can be evaluated analytically in a single forward pass — no gradients, no weight updates. The authors contrast this with directly measuring KL divergence at the output, which would need N × L full forward passes per sample (128 × 48 = 6,144 for InternVL3.5-30B-A3B), and prove a bound connecting the cheap proxy to the expensive target.

The second idea is that token type changes the meaning of the score. ΔH computed on text tokens marks experts worth keeping. For the vision side, they find that scoring on all image tokens or on the most-attended image tokens gives rankings that shift with the calibration dataset — so instead they score on the least-attended 2% of vision tokens, which they argue reflect non-salient background content rather than query-specific objects. On that side, a high score marks an expert as a good pruning candidate.

The two scores are log-transformed, z-scored within each layer (raw ΔH varies across layers by 10³–10⁴×), and combined as (1−α)·text − α·low-attention-vision. The weight α is not hand-tuned per model: it is computed in closed form from layer-averaged Spearman correlations between the three ranking signals. Finally, all experts in all layers are ranked by this single score and the bottom ⌊r · N_total⌋ are removed, rather than deleting a fixed fraction per layer. Calibration uses 1,000 ShareGPT4V samples selected by text length.

Why This Matters

Impact on research. The paper argues that a compression criterion developed for text-only MoE LLMs is not automatically valid for MLLMs, and backs this with concrete evidence: near-zero rank correlation between text- and vision-derived expert rankings, a 29× asymmetry in output-distribution shift, and downstream degradation when the modality split is removed. It also reframes "which tokens to calibrate on" as a first-class design choice rather than a detail, showing that token-subset selection changes both dataset stability and final accuracy.

Real-world applications:

• Serving a ~58 GB parameter MoE-MLLM on a single 40 GB GPU by cutting ~27 GB of routed-expert weights.
• On-premises or edge deployment of vision-language assistants where GPU memory, not compute, is the binding constraint.
• Document and chart question answering in enterprise settings, where the authors show calibration can be deliberately targeted at a deployment domain by mixing in ChartNet data.
• Cost reduction in inference fleets, since expert parameters — not activations — dominate memory in these models.

Industry relevance. The work is explicitly framed around deployment economics: the headline claim is a memory-capacity threshold crossed (58 GB → under 40 GB), the method is training-free so it requires no retraining pipeline, and it needs only 1,000 calibration samples within a single forward pass — a practical profile for teams compressing an existing checkpoint.

Future Directions

• Layer-wise modality weighting. The paper fixes one α per model and explicitly names exploring layer-wise modality weights as a promising direction, since modality sensitivity may vary across layers.
• Pairwise and input-conditional redundancy. The stated limitations note that ΔH measures residual change after co-activated experts absorb the removed load, so it does not model joint removal of experts that back each other up, nor input-conditional sparsity; the prune mask is fixed per layer.
• Calibration-mixing strategies and generative benchmarks. The authors leave systematic calibration-mixing recipes and extension to reasoning-heavy generative benchmarks (coding, mathematical problem solving) to future work.
• Recovering residual gaps with fine-tuning. Fine-tuning (e.g., LoRA) is described as orthogonal to the training-free setting and a way to recover residual accuracy loss, particularly on text-only tasks at 50% pruning.

Target Audience

Practitioners and researchers working on efficient inference and compression of large multimodal models — especially engineers who need to fit MoE-MLLM checkpoints into fixed GPU memory budgets. It will also interest researchers studying modality-specific expert behavior and routing in vision-language MoE architectures, and readers of the expert-merging/pruning literature who want to know where modality-agnostic criteria break down. Readers unfamiliar with top-k MoE routing should read the preliminaries section first.

Authors’ abstract

Mixture-of-Experts (MoE) enables multimodal large language models (MLLMs) to scale capacity efficiently, but expert parameters dominate memory even though only a few experts are activated per token: they account for over 90% of model size in modern MoE-MLLMs (e.g., 54 GB out of about 58 GB in BF16 for InternVL3.5-30B-A3B and Qwen3-VL-30B-A3B). Expert pruning has emerged as an effective compression strategy, but existing methods aggregate importance uniformly across tokens, ignoring that the same expert may serve different roles for vision versus language processing. To address this, we propose MAEP (Modality-Aware Expert Pruning), a training-free framework that decomposes expert importance by modality in a single forward pass. MAEP introduces (1) Delta-H Output ($ΔH$), a redistribution-aware metric measuring the MoE output change when an expert is removed, and (2) low-attention vision tokens as a vision-side pruning signal that is dataset-stable: its expert-importance ranking remains stable across different datasets. Text-side importance serves as a preservation signal while vision-side importance serves as a pruning signal, and experts are ranked globally across layers. Experiments on three MoE-based MLLMs across multimodal and text benchmarks show that MAEP outperforms modality-agnostic baselines on the majority of multimodal benchmarks while remaining competitive on text-only tasks, reducing the BF16 expert weight footprint by about 27 GB (about 47% of total memory) so that the full weight footprint of an approximately 58 GB model fits within a single 40 GB GPU. Code is available at https://github.com/nota-github/MAEP.