SERE: Similarity-based Expert Re-routing for Efficient Batch Decoding in MoE Models

AI summary

Overview

• Research area: Machine learning systems and inference optimization for Mixture-of-Experts (MoE) large language models, specifically batch (multi-request) decoding efficiency.
• Technical level: Advanced. The paper assumes familiarity with MoE routing, top-k gating, memory-bandwidth-bound decoding, and inference frameworks such as vLLM. The core idea, however, can be explained without heavy math.
• Scope: A single-paper summary of SERE (Similarity-based Expert Re-routing), a method that re-routes tokens away from redundant experts during batched MoE decoding to cut latency while preserving output quality (arXiv:2602.07616v1, 07 Feb 2026).

What This Paper Is About

MoE models save compute by sending each token to only a few experts (for example, 8 out of 128 in Qwen3-30B-A3B), but when many user requests are batched together for hardware efficiency, the batch as a whole ends up activating a large share of the experts at once. Because decoding is memory-bandwidth-bound, this excessive expert activation slows token generation down. The paper's goal is to reduce the number of experts actually executed during batch decoding, in an input-aware way, without degrading the model's reasoning quality.

Key Contributions

1. SERE, a similarity-based expert re-routing method. SERE keeps the highest-ranked ("primary") experts for all tokens in a batch, then re-routes tokens assigned to lower-ranked ("secondary") experts to the most similar retained primary expert, using a pre-computed expert similarity matrix. This reduces the number of active experts while preserving the dominant contributors to the layer output.
2. Preservation of critical experts. Experts whose similarity to all candidates falls below a threshold ρ are treated as "critical" and are always retained, preventing the capability loss that naive skipping would cause. The threshold gives fine-grained control over the accuracy-versus-speed trade-off.
3. A custom CUDA kernel and vLLM integration. The implementation is model-agnostic, works inside the vLLM framework without modifying its core execution pipeline, and requires only a single additional line of code to enable. The CUDA version is reported to deliver approximately 1.5x speedup over the PyTorch version.
4. Extensive evaluation across three MoE models and multiple reasoning benchmarks, showing up to 2.0x speedup with minimal quality loss, plus ablations on similarity metrics, calibration data, and re-routing targets.

Main Findings

• Speedup: SERE achieves up to 2.0x speedup. For Qwen3 and DeepSeekV2 the reported speedup range is 1.2x to 1.6x; for Qwen1.5 the acceleration ratio reaches up to 2.0x at QPS = 24.
• Accuracy retention: With aggressive expert skipping (Top-2), SERE maintains over 97% of the original model's accuracy across all tasks, while reducing decoding latency by up to 1.6x on Qwen3 and 1.4x on Qwen1.5 and DeepSeekV2.
• Direct top-k reduction fails: Simply reducing top-k yields the lowest latency but causes severe performance degradation, reported as up to a 90% accuracy drop, indicating significant loss of model capacity.
• Baseline comparison: HC-SMoE and LYNX perform competitively on Qwen3 but drop significantly on Qwen1.5 and DeepSeekV2, particularly on math and code tasks. The paper attributes this to Qwen3 having more fine-grained, redundant experts, while Qwen1.5 and DeepSeekV2 have fewer, more specialized experts.
• Threshold behavior: Beyond an inflection point, raising ρ continues to improve accuracy but sharply reduces speedup. For Qwen3-30B-A3B, ρ = 0.5 is reported as a good balance. SERE performs well even without preserving critical experts (ρ = 0.0), and ρ > 0 adds further accuracy at negligible latency cost.
• Critical expert example: In Qwen3-30B-A3B, Layer-1 has the highest average similarity with nearly all expert pairs above 0.9, while Layer-6 has the lowest with most pairs below 0.4. Even in Layer 1, Expert 92 has similarity of less than 0.1 to all others.
• Activation scaling: Average activated expert count grows logarithmically with batch size for all Top-K values, and larger K consistently leads to more activations. Inter-layer differences in activated expert counts become more pronounced as K increases.
• Overhead is negligible: The computation cost breakdown for Qwen3-30B-A3B shows SERE overhead at 6 µs across batch sizes 16, 24, 32 and 64, compared with attention costs of 115, 117, 119 and 119 µs and MLP costs of 137, 186, 227 and 233 µs respectively.
• Activation-based similarity beats parameter-based: Parameter-based similarity methods (Concat and Logic combinations of weight matrices) perform significantly worse than activation-based methods, suggesting that dynamic activations capture functional similarity better than static parameters.
• Similarity metric cost: On Qwen1.5-MoE-A2.7B, calibration time cost is 28 seconds for Frobenius, 75 seconds for Cosine, 541 seconds for CKA-Linear, 13,459 seconds for CKA-Poly, and 16,064 seconds for CKA-RBF, with comparable accuracy across metrics. Frobenius is chosen as the fastest.
• Calibration robustness: At K = 2, performance remains highly consistent across calibration datasets (FineWeb-Edu, C4, WIKI, domain-specific sets, OpenCompass) and volumes (200×64, 400×128, 800×256). At K = 1, domain-specific calibration gives slightly better results than general calibration.
• Re-routing target matters: On Qwen1.5-MoE-A2.7B at K = 2, re-routing to the most similar expert gives an average of 47.15, random expert selection gives 41.68, and the least similar expert gives 28.74.

Methodology in Plain English

The approach has three moving parts.

First, measure how similar experts are. Before deployment, the authors feed a calibration dataset (FineWeb-Edu, 400 sequences of 128 tokens in the final setup) through the model and record the output activations of every expert in every layer. They compute pairwise similarity between expert activations within a layer, average over many batches, and store the resulting similarity matrix per layer. They compare several similarity functions (Frobenius norm, cosine similarity, and CKA variants) and settle on Frobenius norm because it is accurate and the fastest to compute (28 seconds). This step is done once, requires no retraining, and no task-specific tuning.

Second, re-route at inference time. For each batch, the router first picks its normal top-K experts per token. SERE then identifies the "primary" experts: the union of the top-S experts across all tokens in the batch (where S ≤ K, a hyperparameter controlling aggressiveness). Primary experts are always kept. Every remaining "secondary" expert is compared against the retained primary experts using the similarity matrix. If its best similarity score is at least ρ, all tokens routed to it are redirected to that most similar primary expert, and the original expert is skipped entirely. If its best similarity is below ρ, the expert is judged critical and stays active. Router weights themselves are not modified — only the token-to-expert assignment changes.

Third, make it fast in practice. The authors write a dedicated CUDA kernel for the re-routing logic so it can be dropped into the vLLM inference framework with a single line of code, rather than slowing the pipeline down with token-by-token Python operations as some prior dynamic methods do.

Evaluation covers three models (Qwen1.5-MoE-A2.7B-Chat, DeepSeekV2-Lite, Qwen3-30B-A3B), three baselines (HC-SMoE, Top-K reduction, LYNX), and reasoning benchmarks from OpenCompass across Exam (CMMLU, BoolQ, BBH), Math (Math, GSM8K, Math_401) and Code (HumanEval, MBPP). Acceleration is measured as Time per Output Token (TPOT) under varying Queries per Second (QPS) with vLLM, with fixed input/output lengths of 128/32 tokens, on NVIDIA H20 GPUs with each model deployed on a single GPU.

Why This Matters

Impact on research. The paper reframes MoE inference optimization as a re-routing problem rather than a pruning or merging problem. Instead of permanently deleting or fusing experts (which shrinks capacity and depends on calibration priors), SERE changes only the decoding-time routing target, leaving all parameters and layer structures intact. This suggests a general direction for dynamic expert selection, and the similarity-matrix analysis (redundant groups, cross-layer variation, critical experts) gives a reusable diagnostic tool for studying MoE internals.

Real-world applications:

• Large-scale LLM serving platforms that batch many concurrent user requests and need lower per-token latency at fixed hardware cost.
• Latency-sensitive interactive products such as chat assistants and coding copilots, where time per output token directly shapes the user experience.
• Cost reduction in production inference clusters, since faster decoding at the same quality means more throughput per GPU.
• Deployment of large MoE models on constrained or single-GPU setups, where reducing simultaneously active experts eases memory-bandwidth pressure.

Industry relevance. The work comes from the Taobao & Tmall Group of Alibaba together with the Shenzhen Graduate School of Peking University, and is explicitly oriented toward production serving. The single-line vLLM integration and the model-agnostic kernel are aimed at practitioners who need to adopt the method without reworking their serving stack. The paper also reports an open-source implementation at github.com/JL-Cheng/SERE.

Future Directions

• Where the threshold should sit, per model and per layer. The ablation shows DeepSeekV2 stays relatively stable across thresholds while Qwen3 and Qwen1.5 fluctuate noticeably. An open question is whether ρ (and S) can be set automatically rather than tuned per deployment.
• Whether skipping pressure should vary by layer. The paper finds inter-layer differences in activated expert count grow with K, and argues this favors more aggressive skipping in layers with higher activation. It does not report a layer-adaptive policy; an obvious next step is to build one.
• How far the redundancy observation generalizes. Similarity matrices are visualized for several MoE models, including ones with and without upcycling initialization, and the paper reports high expert similarity is common regardless. Extending the analysis to more architectures, larger models, and different routing designs is left open.
• Behavior outside decoding. The authors state that SERE's behavior in the prefill stage is examined in Appendix C.2, and the main text focuses on decoding. Broader integration questions — for example, combining re-routing with quantization or with other inference optimizations — are framed as future work.

Target Audience

This paper is most useful to ML systems engineers and inference-infrastructure researchers who deploy MoE models at scale and care about decoding latency and throughput. It is also relevant to researchers studying MoE routing, expert specialization, and dynamic expert selection, and to practitioners using vLLM who want a low-integration-effort optimization. Readers without a background in MoE architectures or serving systems will find the high-level idea accessible but the experimental tables and similarity analysis harder to interpret without context.

Authors’ abstract

Mixture-of-Experts (MoE) architectures employ sparse activation to deliver faster training and inference with higher accuracy than dense LLMs. However, in production serving, MoE models require batch inference to optimize hardware efficiency, which may cause excessive expert activation and thus slow the memory-bound decoding stage. To address the fundamental tension between batch decoding and expert sparsity, we present SERE, a Similarity-based Expert Re-routing method for Efficient batch decoding in MoE models. SERE dynamically reduces the number of active experts in an input-aware manner by re-routing tokens from secondary experts to their most similar primary counterparts. It also leverages similarity patterns to identify and preserve critical experts, thereby preventing capability loss. Notably, SERE avoids static expert pruning or merging, instead enabling dynamic expert skipping based on batch-level expert redundancy. Additionally, we provide an efficient custom CUDA kernel for SERE, enabling plug-and-play use in vLLM with only a single-line code change. Extensive experiments on various complex reasoning benchmarks demonstrate that SERE achieves up to 2.0x speedup with minimal quality loss, providing a practical solution for cost-efficient and latency-sensitive large-scale MoE deployment. Code implementation of SERE can be found in https://github.com/JL-Cheng/SERE.