What We Keep of What We Forget: Replacement Operators and Routing Semantics in Training-Free MoE Expert Pruning

AI summary

Overview

Research area: Machine learning — model compression for Mixture-of-Experts (MoE) large language models, specifically training-free expert pruning and the routing semantics that surround it.

Technical level: Intermediate. The paper is readable with a general understanding of transformer language models and MoE routing; the statistical protocol and the reduced-rank regression formulation require closer reading.

Scope (one sentence): The paper isolates one usually-implicit decision in training-free MoE pruning — what a layer computes when its router still selects a deleted expert — and compares seven replacement operators under a fixed pruning mask across OLMoE-1B-7B, DeepSeek-V2-Lite, Moonlight-16B-A3B and Granite-3.0-3B-A800M.

What This Paper Is About

When you delete experts from a Mixture-of-Experts language model without retraining, two things must be decided: which experts to remove, and what the layer should compute when its router still routes a token to an expert that no longer exists. The first decision has received nearly all the attention in prior work; the second is usually made implicitly, and different published methods make it differently, so a comparison of pruning methods is also an uncontrolled comparison of replacement rules. This paper holds the pruning mask fixed and varies only the replacement, in order to find out which replacement rules actually work, on which models, and whether standard evaluation even lets you tell.

Key Contributions

1. The trained-scale rule for routing gates. The authors show that after deletion, the gate scale given to surviving experts should match the convention the router was trained under: keep the unpruned router's gates on routers trained without top-k renormalization (OLMoE, DeepSeek-V2-Lite), and renormalize over the selected survivors on routers trained with it (Moonlight, Granite). At 50% pruning this holds on all four routers.

2. The tilted novice ("tilt"). A closed-form, low-rank linear map of a pruned expert's input, fitted from streamed calibration statistics by reduced-rank ridge regression, whose rank-0 case is exactly MoNE's constant mean output. It stores 2rd+2r parameters per pruned expert over the constant and adds 1.2–1.8% of the pruned model's parameters at 50% pruning.

3. A fixed-mask comparison across three replacement families. Seven operators — fill the slot from the pruned expert (constant, tilt), fill it from a survivor (MoRA-style proxy, ConMoE-style slot remapping), and drop the slot and reroute (keep-scale, renormalized, COMPEL weight compensation) — plus an equal-budget partial copy of each pruned expert, all evaluated on identical masks. At 50% pruning four selection criteria are crossed with the operators on OLMoE and DeepSeek-V2-Lite.

4. An evaluation and calibration audit, plus a reporting checklist. The paper quantifies how much the choice of benchmark and the choice of calibration domain change conclusions, re-implements three operators whose code was not public, reproduces MoNE within 0.32 points, and proposes a five-item reporting checklist for expert-pruning comparisons.

Main Findings

• The gate scale can matter more than the operator. On OLMoE and DeepSeek-V2-Lite, which are trained without top-k renormalization, renormalizing the survivors' gates (a common implementation choice) costs 0.8–8.5 points at every pruning ratio, lowers ranking ability on two-choice tasks, and reverses whether rerouting beats MoNE's constant. On DeepSeek-V2-Lite it costs more than any operator or competitive selection criterion in the authors' grid.

• Direction reverses on renormalizing routers. Table 1 reports trained-scale minus other-scale nine-task accuracy at 50% pruning: +3.39 points on OLMoE, +4.67 on DeepSeek-V2-Lite, +1.68 on Moonlight and +2.08 on Granite. At 25% and 75% the same sign pattern holds per model (Moonlight at 25% is −0.17, an interval containing zero). With each router at its trained scale, Granite orders five operators at 50% exactly as OLMoE does.

• No single operator dominates. Trained-scale rerouting is best on DeepSeek-V2-Lite at 50% and 75%; the tilt is best on Moonlight at both ratios and on OLMoE at 50%. Averaged over four masks, the OLMoE and DeepSeek-V2-Lite differences survive Holm correction (+2.39 and −1.22). In the main table at 50%, the constant/tilt/trained-scale-reroute scores are 49.09 / 52.54 / 51.48 on OLMoE, 59.84 / 60.32 / 61.19 on DeepSeek-V2-Lite and 60.24 / 61.61 / 60.06 on Moonlight, against dense scores of 66.76, 67.15 and 71.48.

• The tilt's gain over the constant is real but modest. At 50% pruning it gains 3.45 points on OLMoE and 1.37 on Moonlight, both surviving Holm correction over all comparisons; the gain moves by at most 0.20 points across three calibration draws. It also improves on the constant at 75% on all three models (Holm-significant on two), and on Granite it beats both the constant and trained-scale rerouting at every ratio (outside the Holm families). On OLMoE a rank-8 tilt beats keeping one more expert per layer with a sixth of the parameters, and at 75% pruning it lowers WikiText perplexity from 1,434 to 172.

• A partial copy is a strong, cheaper-to-explain alternative. Keeping each pruned expert's highest-energy neurons exactly, at the tilt's parameter budget, is statistically indistinguishable from the tilt on OLMoE and DeepSeek-V2-Lite (post-hoc bounds rule out differences beyond 0.81 and 0.73 points) and behind it on Moonlight. Whitened or plain SVD copies do not detectably beat the constant even at four times the budget.

• Survivor-based replacements underperform. Cross-expert proxies, slot remapping (nearest and functional variants) and Fisher-weighted weight compensation (COMPEL) do not reach trained-scale rerouting on any model in the authors' re-implementations.

• Fitted operators survive quantization better than rerouting. Under 4- and 3-bit GPTQ, the tilt's gain over the constant is not detectably reduced in 11 of 12 settings, whereas rerouting's gain falls in 11 of 12 (significantly in 5).

• Benchmark choice changes the verdict. At 25% pruning, MoNE's constant keeps 92–97% of dense multiple-choice accuracy but only 5–65% of GSM8K and 0–17% of code scores (MBPP, HumanEval). Generation can rank operators differently from multiple choice.

• Calibration domain moves the mask far more than resampling. Changing the calibration domain moves the pruned set 4–9 times as much as resampling general text at a matched token budget. In a crossed test on OLMoE, fitted operators inherit the domain of their fit.

• In-domain calibration recovers generation quality. OLMoE pruned by half with trained-scale rerouting, calibrated in-domain, keeps 97% of its dense GSM8K score (44.28 against 45.49) with no fitted parameters. On DeepSeek-V2-Lite, 25% pruning with the tilt and 4-bit weights beats uniformly quantized experts at matched storage on GSM8K by 4.25 points, though on general tasks uniform 3-bit expert quantization remains stronger at the tested budget.

• Deployment formats are already affected. In October 2026, vLLM's compression tooling added a checkpoint schema covering REAP-style pruned and MoNE-style novice experts. Because keep-scale rerouting needs the pruned experts' router rows, the trained-scale rule constrains what a pruned checkpoint must store.

Methodology in Plain English

Hold one thing fixed, change one thing. The authors first compute pruning masks once, using MoNE's released code and its "fusion score" over 1,000 calibration sequences of 4,096 tokens drawn from Zyda-2, pruning 25%, 50% or 75% of routed experts in every MoE layer. From that point the mask never changes. Only the rule that fills a deleted expert's routing slot changes, so any measured difference is attributable to the replacement rather than to a different set of deleted experts.

Seven replacement rules in three families. From the pruned expert: the constant (MoNE's mean output, the "novice"), and the tilted novice, a rank-32 input map solved in closed form from streamed statistics with ridge 10⁻² and features clipped at 4σ. From a survivor: a MoRA-style affine proxy of a retained expert used only when held-out R² exceeds 0.4, and ConMoE-style slot remapping. Drop the slot and reroute: keep-scale rerouting (each survivor gets the gate the unpruned router would give it) and renormalized rerouting (survivors rescaled as if the deleted experts never existed), plus COMPEL-style Fisher-weighted weight compensation with γ = 0.1. A seventh comparison keeps a partial copy of each pruned expert's highest-energy neurons at the tilt's exact parameter budget.

Calibration noise is separated from test noise. Calibration text is cached once and cut into three disjoint slices (A = sequences 0–999, B = 1,000–1,999, C = 2,000–2,999). The mask always comes from slice A; fitted operators are refitted on A, B or C, so the spread across draws measures calibration noise with the mask held fixed. Test-item noise is handled with a paired, task-stratified bootstrap (B = 10,000 resamples, seed 0, 95% percentile intervals), and multiplicity by Holm correction within each table and over all 111 specified contrasts, of which 47 survive. An effect is called established only if it has the same sign on every draw.

Evaluation is deliberately redundant. Nine zero-shot lm-eval multiple-choice tasks (ARC-c, ARC-e, BoolQ, COPA, MMLU, OpenBookQA, PIQA, RTE, WinoGrande), a "big-5" mean that does not let small tasks outweigh MMLU, 5-shot GSM8K with strict match, HumanEval pass@1, MBPP at 25% pruning, and WikiText word perplexity at 2,048 tokens. The authors also check that no token reaches a pruned expert in any run, and they verify each re-implementation against the model's own router code bit-exactly on an empty mask.

Why This Matters

Impact on research. Published MoE pruning methods differ in two decisions at once — which experts to remove and what replaces them — so headline comparisons conflate the two. This paper shows that one sub-decision inside the second (the gate scale) can be worth more than the choice of operator or of selection criterion, and that a widely used implementation default is actively harmful on routers trained without top-k renormalization. It also supplies the field with a checklist for reporting such comparisons, and a statistical protocol (paired item bootstrap, Holm over a declared contrast family, replication across calibration draws) that most pruning papers do not currently follow.

Real-world applications:

• Memory-constrained serving. Expert weights dominate deployment memory even though each token activates only a few; the paper's operators are aimed at reducing this storage with no retraining.
• Compressed checkpoint formats. Because keep-scale rerouting requires the pruned experts' router rows, the trained-scale rule directly affects what a pruned checkpoint must store to stay faithful — relevant to tooling such as vLLM's compressed-tensors schemas.
• Domain-specific deployment. The finding that general calibration collapses GSM8K and code scores, while in-domain calibration restores most of GSM8K (44.28 vs 45.49 on OLMoE), guides practitioners who prune for a known target domain.
• Stacks that combine pruning with quantization. The tilt's gain survives 4- and 3-bit GPTQ in 11 of 12 settings, and 25% pruning with the tilt plus 4-bit weights beats uniformly quantized experts on GSM8K by 4.25 points on DeepSeek-V2-Lite — useful when both levers are pulled together.

Industry relevance. MoE models from multiple vendors are now served at scale, and pruning plus quantization is the standard compression pair. The paper's central warning — that a one-line change in how surviving gates are rescaled can cost up to 8.5 points, more than any operator choice in the grid — is a practical audit item for anyone shipping a pruned MoE checkpoint.

Future Directions

• Gate scales between the two endpoints. The authors test only the two endpoint scales (keep and full renormalization) and explicitly scope their claim to those; intermediate or learned scalings, and routers with other gate functions or scaling factors, are untested. Moonlight's sigmoid router uses a routed scaling factor of 2.446, and DeepSeek-V2-Lite and Moonlight share an expert layout but not a router convention, so router-convention coverage is a natural next axis.
• Cheaper fitting. The tilt is fitted on 1,000 calibration sequences; with 100 the gain is not stable across slices. Reducing calibration cost while keeping the gain would widen applicability.
• Operators the study could not run. MoRA, ConMoE and COMPEL had released no code by 27 October 2026, so the authors re-implemented each from its equations; results for these are component comparisons, not evaluations of the full methods. Merging pipelines (MC-SMoE, HC-SMoE, Sub-MoE, REAM, PRIME) were not run at all, so pruning-versus-merging under a fixed mask remains open.
• Beyond multiple choice, and beyond these models. The authors state they do not infer preservation of open-ended generation from multiple-choice scores, and their claims are limited to one-shot, training-free pruning of the tested base models with 40–64 routed experts per layer. Extending the fixed-mask comparison to open-ended generation, to post-trained/instruct checkpoints, and to layer-adaptive selection is left open.

Target Audience

Researchers and engineers working on MoE inference efficiency, expert pruning, or model compression — particularly those implementing or auditing pruned checkpoints and comparing pruning methods. It is also valuable for practitioners combining pruning with quantization and for anyone who needs a template for statistically defensible compression evaluations; readers interested in the tilted novice as an estimator will find the authors themselves note that reduced-rank regression is classical (Izenman, 1975) and that the contribution is its use as a replacement operator, not a new estimator.

Authors’ abstract

Training-free expert pruning makes two decisions: which experts to remove, and what the layer computes when its router still selects a removed expert. Published methods change both at once. Holding the mask fixed, we compare replacement operators from three families on OLMoE, DeepSeek-V2-Lite and Moonlight, with Granite as a fourth router, using paired, multiplicity-corrected tests. The largest effect is a choice usually left implicit: the survivors' gate scale. On routers trained without top-k renormalization, renormalizing the survivors' gates, a common implementation choice, costs 0.8-8.5 points and reverses whether rerouting beats MoNE's mean-output replacement; on two routers trained with renormalization, the trained scale wins at 50% pruning. We introduce the tilted novice, a closed-form low-rank map of the pruned expert's input whose rank-0 case is MoNE's constant. At 50% pruning it adds 1.2-1.8% parameters and gains 3.45 and 1.37 points over the constant on OLMoE and Moonlight, stable across calibration draws and largely unchanged by 4- and 3-bit quantization. No operator dominates: trained-scale rerouting is best on DeepSeek-V2-Lite, and a partial copy of each pruned expert is statistically indistinguishable from the tilted novice on two models. Evaluation can change conclusions: at 25% pruning MoNE's constant keeps 92-97% of dense multiple-choice accuracy but only 5-65% of GSM8K and 0-17% of code scores, and the calibration domain moves the mask 4-9 times more than resampling. We propose a five-item reporting checklist.