Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE

AI summary

Overview

• Research area: Artificial intelligence — efficient large language model architecture, specifically Mixture-of-Experts (MoE) parameter efficiency along the depth dimension.
• Technical level: Intermediate. The empirical sections are approachable; the Section 4 analysis of residual pathways and gradient accumulation is more advanced.
• Scope: An empirical and theoretical comparison of layer-private "Local Experts" versus cross-layer shared "Meta Experts" in MoE Transformers, covering expert ratios, topology, scaling behaviour, downstream accuracy, and a formal account of why sharing helps.

What This Paper Is About

Standard MoE models give every Transformer layer its own private pool of experts, even though interpretability work suggests different layers store similar knowledge. This redundancy means a deep model pays to store the same content many times over. The paper asks whether pooling some experts across all layers — a shared "Meta Expert" pool that any layer can route to — can deliver the same quality with fewer stored experts and no extra computation per layer.

Key Contributions

1. A systematic empirical comparison of Meta and Local Experts across three axes: the ratio of Meta to Local experts in total and per activation, whether every layer should contain Meta Experts, and whether the Meta pool should itself be hierarchically organized into sub-pools.
2. A demonstration of storage efficiency: with activated expert count and expert granularity held fixed, models with Meta Experts match Vanilla MoE using only 70–80% as many experts (or 75–90% as many parameters, per Figure 1), with savings growing as models get deeper and larger.
3. A redundancy metric built from a linear-CKA similarity kernel on experts' centered covariance responses, summarized by a Gini coefficient over the kernel's eigenvalue spectrum, giving a threshold-free and cross-architecture-comparable measure of expert redundancy.
4. A theoretical account of Meta Experts as bidirectional residual pathways: the shared expert block lets deep-layer training signals reach shallow layers as activation-level content rather than only as parameter updates, and vice versa, through a path of O(1) length independent of the layer distance.

Main Findings

• Fewer experts, same loss: Meta Expert models match Vanilla MoE loss with 70–80% of the expert budget (75–90% of parameters), and the advantage grows with depth and scale.
• Higher Meta ratio, lower loss and lower redundancy: sweeping total-number proportions of 0%, 25%, 50%, 75%, 100% at any activation ratio, higher Meta proportions consistently reduced both cross-entropy loss and expert redundancy ρ.
• Activating more Meta Experts helps loss; activating more Local Experts helps redundancy but hurts loss: the authors suggest Local Experts, each activated in only one layer, may be under-trained, and this under-training may give rise to redundancy.
• 75% Meta ratio is the workhorse configuration: with per-layer activation fixed at 4 Meta Experts and 4 Local Experts, the 75% total-number ratio beat all other Partial-Meta settings at every compute budget, though it remained slightly worse than the Pure-Meta (100%) setting; the authors chose 75% partly because a Meta-only model may suffer training instability.
• Lighter topological intervention beats none: restricting only the first two and last two layers to Local Experts (F2L2), or looping the Meta pool only twice (Loop2), slightly improved loss relative to plain sharing — while heavier versions of both interventions generally degraded it.
• Shallow and deep layers are specialized: the F2L2 result and the redundancy measurements together indicate that experts in the shallowest and deepest layers serve more differentiated functions, with essentially no mutual redundancy between these layers.
• The combined design is best: MR75 + F2L2 + Loop2 outperformed each individual refinement at every compute budget in both loss ratio and redundancy ratio versus Vanilla, and the gap widened as training compute grew.
• Downstream results are mixed but Meta Experts never lose overall: across five 6B-A1.6B variants trained on 200B tokens, different Meta variants won on different benchmarks, but every Meta variant beat the Vanilla baseline on at least some tasks.
• Per-benchmark detail (6B-A1.6B, 200B tokens), Vanilla / MR75 / MR75+Loop2 / MR75+F2L2 / MR75+F2L2+Loop2:
  • MMLU: 0.4087 / 0.4120 / 0.4175 / 0.3928 / 0.4052
  • C-Eval: 0.4354 / 0.4443 / 0.4487 / 0.4160 / 0.4302
  • CMMLU: 0.4203 / 0.4161 / 0.4344 / 0.4198 / 0.4249
  • TriviaQA: 0.1924 / 0.1964 / 0.1932 / 0.2014 / 0.2042
  • BBH: 0.2972 / 0.2912 / 0.3030 / 0.3061 / 0.2984
  • OpenBookQA: 0.3500 / 0.3380 / 0.3640 / 0.3320 / 0.3540
• Deeper models want more hierarchy: in the scaling ladder, as depth increased, using more Local-Experts-only layers and a larger hierarchy of Meta Experts performed better — suggesting the fraction of specialized boundary layers stays roughly constant with depth, while the Vanilla architecture is overly hierarchical.

Methodology in Plain English

The researchers take a standard MoE Transformer layer — an attention block plus a feed-forward "MoE block" with a router and pool of experts — and change one thing: instead of each layer having its own router and its own expert weights, a shared router and shared expert pool are used by every layer. Each token still picks only its top-k experts, so the compute per layer and the number of activated parameters are unchanged; only storage changes, because the pool is stored once rather than once per layer.

They then run ablations. First, a sweep on a 2B-A0.5B model over how many experts are shared (0% to 100% of the total) and how many shared experts are activated per layer (25% to 75%), while holding total expert count and total activation count fixed so all models have similar FLOPs. Second, a topology sweep — forcing some layers to use only private experts, or splitting the shared pool into 2, 4, or 8 sub-pools accessed in a chunked or interleaved loop. Promising configurations are then validated on a scaling ladder of four depths (L ∈ {12, 16, 20, 24}), trained from scratch on NVIDIA H200 GPUs, with every model using the training-optimal configuration of the Vanilla MoE rather than one tuned for their design. Finally, the best configurations are trained at 6B-A1.6B for 200B tokens and evaluated on MMLU, C-Eval, CMMLU, TriviaQA, BBH, and OpenBookQA.

To measure redundancy, they feed a shared probe set of tokens through each expert, compute each expert's centered output covariance, and compare experts via cosine similarity in that covariance feature space — which is exactly linear CKA on functional responses. A Gini coefficient over the resulting similarity matrix's eigenvalues gives ρ ∈ [0,1], where 0 means experts are fully complementary and values near 1 mean similarity energy has collapsed onto a few directions. For a uniform rank-k spectrum, Gini(λ) = 1 − k/E, so ρ reads directly as the fraction of expert directions that have collapsed.

Why This Matters

The paper reframes inter-layer redundancy in deep MoE models — usually treated as an unavoidable cost of depth — as convertible into effective capacity. It also shows that a shared expert block is not merely a parameter-saving trick: because every layer writes to and reads from the same storage, the shared block acts as a residual pathway that existing architectures lack, giving hidden states access to expert content across depth within a single forward pass.

Real-world applications:

• Memory-constrained deployment: with expert storage reduced to 70–80% of the Vanilla budget for equal quality, larger MoE models become viable where total parameter storage is the binding constraint.
• Cost-efficient training and serving at scale: the savings in experts and parameters grow with depth and size, so the benefit compounds exactly where serving costs are highest.
• Multilingual and knowledge-heavy assistants: the paper evaluates Chinese and English knowledge benchmarks (C-Eval, CMMLU, MMLU) alongside reasoning and open-ended generation, domains where stored factual knowledge dominates.
• Architecture design for future foundation models: the finding that deeper models want a larger hierarchical shared-expert topology, and that the boundary-layer fraction stays roughly constant with depth, is directly actionable for configuring new models.

Industry relevance: the design keeps per-layer FLOPs and activated parameters unchanged and requires no extra recursions, so it preserves the parallelism of standard Transformers — a low-friction change for teams already training MoE models on standard frameworks (the authors build on Megatron-LM).

Future Directions

• Scale to larger models and broader architectures, which the authors name explicitly as future work.
• Resolve the hierarchy paradox: mildly hierarchical Meta Experts were found to introduce redundancy without degrading performance; why this redundancy does not translate into higher loss is described as an open question.
• Stabilize the Pure-Meta configuration: the 100% Meta setting performed best in the pilot scaling ladder but was not adopted because a Meta-only model may suffer training instability — making it trainable is a clear next step.
• Push beyond the expert block: the paper positions Meta Experts as a finer-grained form of weight tying than Looped Transformers or attention/FFN weight sharing; extending the analysis to other reused modules and to longer training budgets (the authors note their 200B-token downstream budget was limited, causing variants to trade wins across datasets) remains open.

Target Audience

Machine learning researchers and engineers working on MoE architectures, LLM pretraining efficiency, or parameter-sharing schemes. It is most useful to practitioners who configure large MoE training runs and care about the storage-versus-compute tradeoff, and to researchers interested in the interpretability-adjacent question of what different depth layers actually store. Readers focused on inference-time KV-cache sharing will find the Related Work section's contrast with their setting useful.

Authors’ abstract

As model parameters scale up, sparsely activated Mixture-of-Experts (MoE) approaches have become essential for model scaling. However, mainstream models currently adopt hierarchical architectures, where experts in different layers, although topologically independent, exhibit substantial similarity and redundancy in their stored content. Compared to the hierarchically arranged \textbf{Local Expert} in existing mainstream models, we hypothesize that inter-layer shared \textbf{Meta Expert} hold greater potential. To this end, we conduct a multi-dimensional empirical comparison between Local and Meta Experts, and find that models incorporating Meta Experts exhibit lower parameter redundancy, together with lower cross-entropy loss and higher downstream task accuracy. With the activated quantity and the expert granularity held fixed, \textbf{our model matches the performance of Vanilla MoE using only 70--80\% of the experts}, substantially improving the model's scaling potential. Beyond removing inter-layer redundancy among experts and improving information storage efficiency, our theoretical analysis further shows that, compared with Local Experts, Meta Experts provide hidden states with more flexible residual pathways, thereby improving information retrieval efficiency.