Certain Head, Uncertain Tail: Expert-Sample for Test-Time Scaling in Fine-Grained MoE

AI summary

Overview

Research area: Inference-time scaling and Mixture-of-Experts (MoE) architecture for large language models; specifically, sampling strategies that generate diverse candidate solutions without retraining.

Technical level: Intermediate. The paper assumes familiarity with LLM inference, MoE routing (top-k gating, softmax-normalized router scores), and test-time scaling concepts such as pass@n, Best-of-N verification, and majority voting.

Scope: The paper empirically characterizes the router score distribution in fine-grained MoE models, proposes a training-free routing-level sampling method called Expert-Sample, and evaluates it across four MoE models and multiple math, knowledge-reasoning, and code benchmarks.

What This Paper Is About

Test-time scaling improves LLM accuracy by generating many candidate answers and picking the best one, but the usual way of producing that diversity — raising the sampling temperature at the token level — forces a trade-off between diversity and sample quality. The authors ask whether the expert-routing mechanism inside fine-grained MoE models can supply diversity instead, and they find that router scores split into a confident "certain head" of top-ranked experts and a flatter "uncertain tail." Their goal is a sampling method that keeps the confident experts fixed while randomizing the tail, thereby gaining diversity without destabilizing outputs.

Key Contributions

1. An empirical characterization of fine-grained MoE routing. The authors show that when activated experts are halved, single-run greedy decoding accuracy stays stable across five models and two benchmarks, while pass@n degrades substantially, and they link this asymmetry to a router score distribution with a sharp high-confidence head and a flat low-confidence tail.

2. Expert-Sample, a training-free routing-level sampler. At each MoE layer, the method deterministically keeps the top k_keep experts and samples the remaining k - k_keep slots from ranks k_keep+1 to r using temperature-scaled softmax and Gumbel-top-k, then renormalizes with the original gating weights. It requires no architectural change and no additional training.

3. Demonstration that the method scales and composes with verification. Across four fine-grained MoE models and three task categories, Expert-Sample improves pass@n scaling curves, and when combined with Best-of-N or Weighted Majority Voting it raises final accuracy.

4. Evidence that the approach is cheap and its diversity metric is robust. Per-path prefill and decode throughput changes stay within roughly ±1%, and LLM-judge diversity scores are validated across three independent runs and three different judge models.

Main Findings

• Expert reduction preserves greedy accuracy but hurts diversity. Cutting activated experts to half of the default top-k left greedy decoding accuracy "remarkably stable" across Qwen3-Next-80B-A3B-Instruct, Qwen3-30B-A3B-Instruct, GPT-OSS-20B, Ling-Lite-1.5, and DeepSeek-V2-Lite-Chat on GPQA-Diamond and AIME-120. But with k/2+1 experts, 64 parallel samples at T=0.7, and pass@n computed at various sample sizes on Qwen3-30B-A3B-Instruct and Ling-Lite-1.5, pass@n accuracy dropped substantially.

• Router scores show a certain head and an uncertain tail. For Qwen3-30B-A3B-Instruct (128 experts per layer, 8 activated per token by default), the gap in averaged router score from rank 1 to rank 5 exceeds 3.9%, while the cumulative difference from rank 5 to rank 32 is less than 1.5%. The boundary between head and tail roughly coincides with half of the default top-k.

• Stability is preserved on solvable problems. On the AIME-120 Correct Set, high-temperature token sampling caused the pass rate to drop by 2.6% and 2.1% on Qwen3-30B-A3B-Instruct and Ling-Lite-1.5 respectively. Expert-Sample caused only a 0.1% drop on Qwen3-30B-A3B-Instruct and no drop on Ling-Lite-1.5.

• Accuracy gains on the Medium Set with verification. With Best-of-N using Qwen2.5-Math-PRM as verifier and 32 responses, Expert-Sample achieved the highest BoN(32) accuracy on both models, improving over normal-temperature sampling by 7.1% on Qwen3-30B-A3B-Instruct and 7.2% on Ling-Lite-1.5.

• Larger diversity and coverage on the Hard Set. Diversity scores were 0.383 and 0.362 for Expert-Sample versus 0.288 and 0.299 for high-temperature sampling. On pass@32, Expert-Sample enabled Qwen3-30B-A3B-Instruct to solve 30.8% of previously unsolvable problems (versus 7.7% for high-temperature), and Ling-Lite-1.5 to solve 12.8%, meaning 6 of 47 previously impossible problems became solvable.

• Headline verification result. On Qwen3-30B-A3B-Instruct evaluated on GPQA-Diamond with 32 parallel samples, pass@32 rose from 85.4% to 91.9%, and accuracy improved from 59.1% to 62.6% with Best-of-N verification.

• Consistent scaling gains. At pass@64, Expert-Sample yielded an average improvement of 4.32% over normal-temperature token sampling across all 12 model-benchmark combinations, and pushed all four models to near-perfect pass@64 accuracy on GPQA-Diamond.

• Verification table results. Across 20 model-dataset combinations, Expert-Sample gave average accuracy improvements of 4.28% on top of BoN and 3.15% on top of Weighted Majority Voting. For example, on Qwen3-30B-A3B-Instruct with BoN, GPQA-Diamond went from 59.09% to 62.63%, and AIME2025 from 70.00% to 76.33%.

• Negligible overhead. At prompt length 1024, batch size 8, output length 1024, Qwen3-30B-A3B-Instruct prefill throughput changed from 136754 to 137121 tokens/s (up 0.27%) and decode from 666.7 to 662.1 (down 0.70%); Ling-Lite-1.5 prefill went from 214802 to 213463 (down 0.62%) and decode from 1024 to 1028 (up 0.40%). The paper states the relative change per path stays within ±1%.

• Robust diversity evaluation. DeepSeek-R1 pairwise diversity over three runs gave 0.228 ± 0.014 for t=0.7, 0.296 ± 0.006 for t=1.3, and 0.386 ± 0.015 for Expert-Sample. With GPT-4o as judge the values were 0.294 ± 0.007, 0.349 ± 0.008, and 0.422 ± 0.005; with Claude-Sonnet-4.6 they were 0.258 ± 0.010, 0.321 ± 0.008, and 0.403 ± 0.012.

Methodology in Plain English

Fine-grained MoE models hold hundreds of experts per layer but only activate a handful per token. The router assigns each expert a score, and the standard approach always picks the highest-scoring ones — a greedy choice analogous to greedy decoding.

The authors first ran a diagnostic: they artificially reduced the number of experts activated per token and measured what happened. Greedy accuracy barely moved, but multi-sample pass@n fell, implying that the lower-ranked experts matter mainly for producing varied reasoning paths. Inspecting the averaged router scores confirmed a structural split: a few very confident experts, then a long run of experts with nearly equal, low scores.

Expert-Sample then acts on that structure with three hyperparameters: k_keep (how many top experts to lock in), a temperature τ for the tail, and a candidate range r. The top k_keep experts are always selected. The remaining k - k_keep slots are drawn from the experts ranked k_keep+1 through r, using the router logits rescaled by τ, a softmax, and Gumbel-top-k sampling. The originally computed gating weights are restored for all selected experts before the weighted sum, so the model's learned weight magnitudes are respected. The default recommendation is k_keep = floor(k/2)+1, τ = 1.0, and r = 4k.

Because everything is vectorized and k_keep is constant across the batch, the added cost is tiny. The method is applied on top of normal-temperature token sampling (T=0.7, top-p 0.8, top-k 20 in the experiments) rather than replacing it.

Evaluation used four models — Qwen3-30B-A3B (30B total, 3B active, 128 experts, top-k 8), GPT-OSS-20B (21B total, 3.6B active, 32 experts, top-k 4, low think-budget), Ling-Lite-1.5 (16B total, 3B active, 64 experts, top-k 6), and Qwen3-Next-80B-A3B (80B total, 3B active, 256 experts, top-k 10). Scaling experiments covered AIME-120, GPQA-Diamond, and LiveCodeBench-V6-Lite; verification experiments covered AIME 2024, AIME 2025, MATH-500-Hard (Level-5), HMMT 2025, and GPQA-Diamond, with 5 independent trials on the smaller benchmarks. Baselines were normal-temperature token sampling, high-temperature token sampling (T=1.3, top-p 0.98, no top-k), and Entropy-based Dynamic Temperature (EDT) sampling. Tooling was LightEval for evaluation and vLLM for inference. Verification used Best-of-N with Qwen2.5-Math-PRM-7B and Llama3.1-8B-PRM-Deepseek-Data as reward models, plus Weighted Majority Voting.

Why This Matters

Impact on research. The paper opens a new axis for test-time scaling — diversity injected at the routing level rather than the token level. It also reframes the fine-grained MoE expert pool as an inference-time resource rather than purely a training-efficiency or serving-optimization concern, and it offers a concrete structural explanation (the head/tail router distribution) that other researchers can test, extend, or dispute.

Real-world applications:

• Mathematical and competition problem solving, where Best-of-N over many samples with a process reward model is already standard practice and additional diversity directly raises the fraction of solvable problems.
• Graduate-level knowledge reasoning such as GPQA-Diamond, where the reported pass@32 and Best-of-N gains translate into higher answer accuracy without extra model training.
• Code generation and competitive programming evaluation on LiveCodeBench-V6-Lite, where multiple candidate solutions must be generated and filtered.
• Cost-controlled deployment, since the method is reported to add negligible per-path latency and can be enabled as a drop-in flag in an existing vLLM serving stack.

Industry relevance. Because Expert-Sample is training-free, requires no architectural modification, and composes with existing verification pipelines, it is directly applicable to any deployed fine-grained MoE model — a family that includes DeepSeek-R1, GPT-OSS, and the Qwen3-MoE series. Teams already paying for N parallel samples can get more out of that budget through routing diversity rather than temperature tuning.

Future Directions

1. Training-side implications. The conclusion explicitly hopes the work "provides insights for the training side of fine-grained MoE models." An open question is whether routers could be trained to make the head/tail boundary even more useful for inference-time sampling, or whether the default top-k should be rethought.

2. Hyperparameters beyond the recommended defaults. The paper reports robustness across a wide range of k_keep, τ, and r settings and recommends a single unified configuration, but sensitivity analysis is deferred to Appendix B, leaving open whether task-specific tuning could yield further gains.

3. Interaction with the broader diversity toolkit. The paper positions Expert-Sample as complementary to token-level sampling and orthogonal to verification. It does not report combinations with prompt rephrasing, question modification, beam search, or tree-based reasoning, so the additive benefit with those methods is not reported.

4. Extension to other architectures and modalities. The evaluation covers only fine-grained MoE models on text benchmarks. Behavior on coarse-grained MoE, on dense models, or on non-text tasks is not reported. The paper also does not report statistical significance testing for the main verification table.

Target Audience

Researchers and engineers working on LLM inference, test-time scaling, and Mixture-of-Experts architectures. It is most useful for practitioners who deploy fine-grained MoE models and want to improve multi-sample generation without retraining or significant latency cost, and for researchers studying routing behavior, sampling diversity, or verification pipelines. Readers should already understand top-k gating, pass@n, and Best-of-N selection to follow the experimental design.

Authors’ abstract

Test-time scaling improves LLM performance by generating multiple candidate solutions, yet token-level sampling requires temperature tuning that trades off diversity against stability. Fine-grained MoE, featuring hundreds of well-trained experts per layer and multi-expert activation per token, offers an unexplored alternative through its rich routing space. We empirically characterize fine-grained MoE routing and uncover an informative pattern: router scores exhibit a certain head of high-confidence experts followed by an uncertain tail of low-confidence candidates. While single-run greedy accuracy remains stable when fewer experts are activated, multi-sample pass@n degrades significantly-suggesting that the certain head governs core reasoning capability while the uncertain tail correlates with reasoning diversity. Motivated by these findings, we propose Expert-Sample, a training-free method that preserves high-confidence selections while injecting controlled stochasticity into the uncertain tail, enabling diverse generation without destabilizing outputs. Evaluated on multiple fine-grained MoE models across math, knowledge reasoning, and code tasks, Expert-Sample consistently improves pass@n and verification-based accuracy. On Qwen3-30B-A3B-Instruct evaluated on GPQA-Diamond with 32 parallel samples, pass@32 rises from 85.4% to 91.9%, and accuracy improves from 59.1% to 62.6% with Best-of-N verification.