Shared Low-rank Basis Factorization for Data-free Mixture-of-Experts Compression

AI summary

Overview

Research area: efficient inference and model compression for Mixture-of-Experts (MoE) large language models, specifically data-free compression of expert feed-forward network weights.

Technical level: Intermediate. The paper assumes familiarity with MoE routing, low-rank factorization, and tensor decomposition, but its central argument can be followed with a general machine learning background.

Scope: This paper provides a structural error analysis of three MoE compression families and introduces a data-free weight reconstruction method, Shared Low-rank Basis Factorization (SLBF), evaluated on five MoE architectures from 16B to 122B parameters.

What This Paper Is About

MoE models gain capacity without proportionally increasing compute, but every expert must still reside in accelerator memory even when rarely activated. The authors analyze three ways to shrink MoE expert parameters (pruning, merging, and weight reconstruction), show that pruning and merging carry structural errors that do not vanish, and then propose SLBF, a weight reconstruction method that shares many low-rank bases across experts to reduce reconstruction error under a fixed compression budget.

Key Contributions

1. A unified structural cost analysis of three MoE compression families, with new error bounds for expert pruning and for weight reconstruction, showing that weight reconstruction avoids the intervention-induced structural costs of pruning and merging.
2. Shared Low-rank Basis Factorization (SLBF), which replaces each full-rank shared basis with a rank-k factorization so that more bases fit under the same compression budget, plus a bilinear gauge fixing that removes redundant parameters at no representational cost.
3. An evaluation on five MoE architectures spanning 16B to 122B parameters against representative methods from all three compression families, showing consistent improvements.
4. Ablations and runtime analysis covering gauge fixing, choice of compression basis count m, choice of projections compressed, training cost, and runtime-decompressed serving.

Main Findings

• Structural costs are non-vanishing for pruning and merging. The derived merging cost depends on pair routing mass (g_i + g_j)^2, the variance of the input-dependent mixing ratio r(x), and the expert functional gap. The derived pruning cost is positive when expert j is active on a non-trivial subset of inputs and its output is not parallel to the substitute's.
• Weight reconstruction has no structural term. Its error bound is a router-weighted parameter reconstruction residual, with no additional term from removing experts or tying routing.
• Structural costs are empirically non-trivial. On Qwen3-30B-A3B-Instruct-2507 (n=128, top-8) and Mixtral-8x7B-v0.1 (n=8, top-2), both structural quantities are strictly above zero across every MoE layer, grow approximately 3-4 orders of magnitude with depth, and trace similar shapes across layers.
• Weight reconstruction shows the lowest output drift. At 25% MoE compression, SLBF achieves the lowest per-layer relative MoE-block output discrepancy across all layers on both architectures, compared with REAP (pruning) and HC-SMoE (merging). The ordering of REAP and HC-SMoE reverses between the two models.
• Many low-rank bases mix better than few full-rank bases. SLBF reaches MoBE's reconstruction loss in about 2k iterations and improves by more than 20% by 20k iterations on Qwen3-30B-A3B layer 7 gate, despite running fewer steps. Under the same budget, SLBF permits m = n while MoBE is restricted to m < n, and SLBF distributes reconstruction across multiple bases instead of concentrating each expert on one dominant basis.
• Uniform reconstruction gain on Mixtral. SLBF achieves lower reconstruction MSE across all 32 MoE layers of Mixtral-8x7B under matched budget.
• Multi-architecture results. At 24% compression on Qwen3-30B, SLBF preserves near-original accuracy (84.3 average) while MoBE drops 2.1 points and pruning/merging baselines drop over 8 points. On Gemma4-26B-A4B-it at 37%, SLBF reaches 81.8 average against MoBE's 67.5, a 14.3-point gap. On Qwen3.5-122B-A10B at 34%, SLBF reaches 88.0 against MoBE's 82.8, a 5.2-point gap. SLBF at 36% on Qwen3-30B still outperforms MoBE at the easier 32% ratio.
• Moonlight behaves differently but SLBF still holds. Moonlight-16B-A3B was trained end-to-end with the Muon optimizer. At 15% compression, SLBF's average score drops by 1.8 points, versus 9.9 points for MoBE and over 30 points for MoLAE.
• Mixtral cross-family comparison. At 24% compression, SLBF reaches 70.7 average accuracy and WikiText-2 perplexity 6.92, above the strongest pruning baseline (EAN: 70.2 / 8.28) and the strongest merging baseline (HC-SMoE: 69.1 / 8.54). SLBF's perplexity is 1.4 lower (16% relative) than the next-best method, with the accuracy gap within 0.5 points. At 30%, SLBF reports perplexity 7.78 against MoBE's 8.59.
• Gauge fixing and configuration ablations. SLBF without gauge fixing already outperforms MoBE at every compression ratio, with gauge fixing adding a secondary gain. Compressing all three projections outperforms two at matched budget (62.5 versus 58.2 on Moonlight-16B). Sweeping m at fixed budget suggests m = n lies near a strong operating region.
• Runtime and serving. On Mixtral gate projections, SLBF takes 40.9 minutes per (layer, projection) pair versus 94.3 minutes for MoBE, using only 40% as many training iterations. In runtime-decompressed serving on 2×H100, SLBF reduces model-weight memory by 30% and increases KV-cache capacity and maximum serving concurrency by 53%, while retaining 0.88× throughput.

Methodology in Plain English

The paper first sets up a simplified pairwise analysis, following prior work on merging, in which two experts with their respective routing weights are considered together. Each compression family is treated as a particular intervention on that pair: merging replaces two experts with one representative and tied routing, pruning deletes one expert and compensates through the surviving expert's router (even allowing an oracle router), and weight reconstruction keeps both experts and routing intact and only approximates their weight matrices. The minimum residual under each intervention is called the irreducible structural cost. These costs are then measured empirically on two models using activations from 128 sequences of 2048 tokens from the C4 corpus, and the realized per-layer output discrepancy is measured at 25% compression with one representative method per family.

For the method itself, the authors build on MoBE, which expresses each expert weight as a per-expert linear adapter multiplied by a nonlinear function of a weighted mixture of shared full-rank bases. SLBF replaces each full-rank basis B_j with a product U_j V_j, where U_j is r×k and V_j is k×d_2, so each basis is cheaper and more bases fit in the same budget. Because the pre-activation mixture of m rank-k terms generically attains rank min(mk, r, d_2), the mixture can still be high-rank. The product U V is invariant under (UQ)(Q^{-1}V) for invertible Q, so the authors fix the gauge by re-parameterizing U with a fixed identity block and folding the remaining factor into V, removing k^2 parameters per basis, with the rows selected by column-pivoted QR on U-transpose. Training minimizes the per-layer Frobenius reconstruction loss with AdamW, using matrix-sign initialization, separate learning-rate groups for {A, w} and {U, V}, and the SiLU activation for the nonlinearity. No calibration data and no fine-tuning are used. In the experiments, m = n and r = d_1, with the per-basis rank k chosen to match the target compression ratio.

Evaluation covers Moonlight-16B-A3B-Instruct, Qwen3-30B-A3B-Instruct-2507, Gemma4-26B-A4B-it, Mixtral-8x7B-v0.1 (base), and Qwen3.5-122B-A10B. Baselines include pruning (Frequency, REAP, EAN), merging (MC-SMoE, HC-SMoE, REAM), weight reconstruction (MoLAE, MoBE), and others (Sub-MoE, TD-MoE); calibration-based baselines use 1024 C4 sequences of 2048 tokens. Instruction-tuned models are evaluated on ARC-Challenge, GPQA-Diamond, IFEval, MMLU, CEval, CMMLU, GSM8K, MBPP, and HumanEval; Mixtral is evaluated on WikiText-2 perplexity and zero-shot accuracy on ARC-Challenge, ARC-Easy, HellaSwag, OpenBookQA, RTE, WinoGrande, and PIQA, using lm-evaluation-harness.

Why This Matters

Impact on research: The paper reframes MoE compression as a choice among intervention types rather than only a question of which heuristic works best, providing derived bounds that separate structural error (from removing experts or tying routing) from reconstruction error. It also connects shared-basis expert compression to rank-(k,k,1) block-term decomposition, positioning MoBE as the Tucker-1 case recovered at k = r.

Real-world applications:

• Serving memory-constrained MoE deployments, where expert weights dominate the serving budget and model distribution pipeline, such as Qwen3-235B-A22B with 128 expert FFNs per layer routing to 8, or DeepSeek-V3 (671B) distributed across dozens of GPUs.
• One-shot compression of already-released instruction-tuned checkpoints, since SLBF is data-free and requires no calibration data or fine-tuning.
• Reducing model distribution and transfer cost when shipping large MoE checkpoints across clusters.
• Increasing serving concurrency and KV-cache capacity under fixed accelerator memory, as demonstrated in runtime-decompressed serving on 2×H100.

Industry relevance: MoE has become the dominant paradigm for scaling production LLMs, and the paper measures 30% model-weight memory reduction with 53% higher KV-cache capacity and serving concurrency while retaining 0.88× throughput, which speaks directly to serving economics.

Future Directions

• Calibration-guided budget allocation and post-compression fine-tuning to recover residual accuracy loss, which the authors explicitly leave unexplored.
• Efficient forward computation through the factorized representation, noted as beyond the scope of the work; the authors report that directly loading the factorized representation can incur substantial reconstruction overhead for large MoEs in their current implementation.
• Adaptive per-layer or per-projection choices of (r, k, m), since the current work uses a fixed r = d_1 and m = n across models and layers, even though m = n was only identified as a strong operating point.
• Interaction with other compression axes, such as quantization or attention compression, and evaluation on more hardware and serving configurations with optimized MoE kernels, both listed as unexplored.

Target Audience

Researchers and engineers working on LLM efficiency, MoE inference, and model compression who want a principled account of why weight reconstruction differs structurally from pruning and merging. Practitioners deploying large MoE checkpoints under tight accelerator memory budgets will find the compression ratios, benchmark tables, and runtime numbers directly relevant, while readers interested in tensor decomposition will find the block-term decomposition framing useful.

Authors’ abstract

Mixture-of-Experts (MoE) large language models decouple capacity from compute through sparse routing, but their large parameter count creates storage and serving challenges. We analyze three MoE compression families: expert pruning, expert merging, and weight reconstruction, and derive structural error bounds showing that pruning and merging can incur non-vanishing errors tied to routing and expert heterogeneity. In contrast, weight reconstruction avoids these structural costs by preserving expert structure and routing. Motivated by the analysis, we propose Shared Low-rank Basis Factorization (SLBF), a data-free weight reconstruction method that uses rank-$k$ bases shared among experts, enabling richer cross-expert sharing, faster convergence, and lower reconstruction error. A post-hoc gauge fixing removes redundant parameters at no representational cost. Across five MoE architectures spanning 16B to 122B parameters, SLBF consistently outperforms methods from all three compression families.