MoEMeta: Mixture-of-Experts Meta Learning for Few-Shot Relational Learning

AI summary

Overview

Research area: Few-shot relational learning (FSRL) on knowledge graphs, combining meta-learning with mixture-of-experts (MoE) architectures.

Technical level: Intermediate. The paper assumes familiarity with knowledge graph embeddings, meta-learning (specifically MAML-style episodic training), and the basic MoE gating formulation, though each is introduced.

Scope in one sentence: The paper proposes MoEMeta, a meta-learning framework that separates globally shared relational knowledge (learned by a mixture-of-experts over relational prototypes) from task-specific local context (learned by lightweight per-task projection vectors), and evaluates it on three KG benchmarks.

What This Paper Is About

Knowledge graphs are incomplete, and many relations have only a handful of known triplets, so models must reason about new relations from very few examples. Existing few-shot relational learning methods use meta-learning to adapt quickly, but they learn meta-knowledge within each task in isolation and rely on a single shared initialization, missing patterns that recur across related tasks and missing the local context each individual task needs. MoEMeta's goal is to disentangle these two kinds of knowledge so that a model can both generalize broadly and adapt rapidly under few-shot supervision.

Key Contributions

1. A task-level mixture-of-experts meta-learner. MoEMeta introduces a sparsely-gated MoE with a shared pool of expert networks that dynamically learns "relational prototypes" — common relational patterns shared across tasks — rather than learning each task's meta-knowledge in isolation. The authors position this as distinct from prior KG MoE work (Song et al., 2022), which fuses relation semantics at the triplet level, whereas MoEMeta operates at the task level.

2. A task-tailored local adaptation mechanism. Each task receives three randomly initialized, task-specific projection vectors (p_h, p_r, p_t) that project head entity embeddings, the relation-meta, and tail entity embeddings into a task-specific subspace. This operates at the embedding level, which the authors contrast with task-aware modulation methods (Vuorio et al., 2019; Abdollahzadeh et al., 2021) that rely on task mode prediction. Only these local parameters, plus the relation-meta, are updated during meta-testing.

3. An explicit split between global and local parameters. MoEMeta has globally optimized parameters Φ = {W, β, {θ}ᵢ₌₁^M, θ_g} for neighbor aggregation and MoE-based meta-knowledge learning, and locally optimized task parameters η = {p_h, p_r, p_t}.

4. State-of-the-art results and ablations on three benchmarks. Experiments on Nell-One, Wiki-One, and FB15K-One show consistent improvements over baselines, with ablation studies isolating the contribution of attentive neighbor aggregation, the MoE, and local adaptation.

Main Findings

• Nell-One (1-shot and 5-shot): MoEMeta achieves MRR 0.322 / Hits@1 0.228 / Hits@5 0.429 / Hits@10 0.501 in the 1-shot setting, and MRR 0.339 / Hits@1 0.236 / Hits@5 0.440 / Hits@10 0.523 in the 5-shot setting. Against HiRe (the second-best performer), the paper reports gains of 11.8% MRR, 23.9% Hits@1, 6.4% Hits@5, and 6.1% Hits@10 under 1-shot, and 10.8% MRR, 14.0% Hits@1, 2.3% Hits@5, and 0.6% Hits@10 under 5-shot.

• Wiki-One (1-shot and 5-shot): MoEMeta reports MRR 0.325 / Hits@1 0.276 / Hits@5 0.385 / Hits@10 0.434 in the 1-shot setting and MRR 0.377 / Hits@1 0.324 / Hits@5 0.420 / Hits@10 0.473 in the 5-shot setting, outperforming the listed baselines including HiRe (1-shot MRR 0.322; 5-shot MRR 0.371) and RelAdapter (1-shot MRR 0.247; 5-shot MRR 0.305).

• FB15K-One (3-shot): MoEMeta reports MRR 0.423, Hits@1 0.302, and Hits@10 0.651 in Table 3. The paper states it outperforms the second-best method by 4.4% MRR, 0.3% Hits@1, and 13.2% Hits@5 (the text refers to Hits@5 while Table 3 reports Hits@1 and Hits@10 columns). The second-best MRR is RelAdapter at 0.405.

• Largest gains concentrate on Nell-One's Hits@1. The paper attributes this to Nell-One triplets being constructed in natural language, giving richer semantics that the relational prototypes capture more effectively, and notes that Hits@1 measures ranking the correct tail entity as the top candidate.

• Statistical testing. Paired t-tests were run between MoEMeta and the second-best performers on all datasets, with improvements significant at the 0.05 level marked with ∙. Results are averaged over 5 independent runs with different random seeds.

• Ablation — removing attentive neighbor aggregation (w/o N.A): Moderate performance decrease across all settings (Nell-One 1-shot MRR 0.311, 5-shot 0.328; FB15K-One 3-shot MRR 0.413), which the authors say shows the importance of neighboring information.

• Ablation — removing MoE (w/o MoE): The largest degradation among the ablated variants (Nell-One 1-shot MRR 0.291, 5-shot 0.293; FB15K-One 3-shot MRR 0.401).

• Ablation — removing local adaptation (w/o L.A): Performance drop (Nell-One 1-shot MRR 0.301, 5-shot 0.315; FB15K-One 3-shot MRR 0.408), indicating the task-specific projections contribute to adaptation.

• Full-model ablation row. Table 4 lists MoEMeta at Nell-One 1-shot MRR 0.322 / Hits@1 0.228 / Hits@5 0.429 / Hits@10 0.501, 5-shot 0.339 / 0.236 / 0.440 / 0.523, and FB15K-One 3-shot 0.422 / 0.298 / 0.652; these FB15K-One ablation-row values differ slightly from the 0.423 / 0.302 / 0.651 reported in Table 3.

Methodology in Plain English

A few-shot relational learning task is built around one relation: a support set of a few known triplets (K = 1, 3, or 5) and a query set where the tail entity must be ranked among candidates.

Step 1 — Enrich entity embeddings. For each head and tail entity in the support set, MoEMeta aggregates its one-hop neighbors. Each neighbor tuple (relation, neighboring entity) is concatenated, passed through a linear layer with ReLU, scored by a sigmoid gate, weighted, aggregated, and added back to the entity's own embedding. This gives the model local structural context rather than relying on the entity embedding alone.

Step 2 — Build the relation-meta with a mixture of experts. A pool of M = 32 expert MLPs exists. For each support triplet, a gating network scores all experts, softmax normalizes the scores, and only the top N = 5 experts are kept (the rest are zeroed). The selected experts' outputs are weighted and combined into a relation representation for that triplet, and the relation-meta is the average of these representations across the support triplets. Because the experts are shared globally and selectively activated, they specialize into reusable relational patterns — the paper's "relational prototypes."

Step 3 — Adapt locally per task. Inspired by TransD's per-entity and per-relation projections, but avoiding over-parameterization in few-shot settings, each task gets three randomly initialized projection vectors. Each one is dotted with the relation-meta to produce a scalar alignment score, which scales the relation-meta into an offset added to the corresponding embedding. This produces projected head, relation-meta, and tail embeddings in a task-specific subspace.

Step 4 — Score and learn. Triplets are scored by the L2 norm of h′ + R′ − t′, and a margin-based loss (margin γ = 1) pushes true triplets below corrupted ones. Loss on the support set updates the projection vectors and the relation-meta (the inner loop); loss on the query set updates the global parameters (the outer loop). At meta-testing, global parameters are frozen and only the local parameters are fine-tuned on the support set.

Experimental setup. Embedding dimension 100 for Nell-One and FB15K-One, 50 for Wiki-One, initialized with TransE-pretrained weights from GMatching. Neighbors per entity capped at 50. Each expert and the gating network are two-layer MLPs with hidden dimension 64 and ReLU (gating output dimension 1). Adam optimizer, batch size 1,024, learning rate 0.001. Implemented in PyTorch, trained on a single Tesla P100 GPU, evaluated with MRR and Hits@k (k = 1, 5, 10).

Datasets. Nell-One and Wiki-One (Xiong et al., 2018) and FB15K-One (Ran et al., 2024). Train/validation/test task splits are 51/5/11 for Nell-One, 133/16/34 for Wiki-One, and 75/11/33 for FB15K-One. Reported statistics: Nell-One has 358 relations, 68,545 entities, 181,109 triplets, 67 tasks; Wiki-One has 822 relations, 4,838,244 entities, 5,859,240 triplets, 183 tasks; FB15K-One has 237 relations, 14,541 entities, 281,624 triplets, 119 tasks.

Baselines. KG embedding methods (TransE, TransD, DistMult, ComplEx, reproduced with OpenKE) and FSRL methods — metric-learning (GMatching, FAAN, FSKGC) and meta-learning (MetaR, GANA, HiRe, RelAdapter). CSR and NP-FKGC were excluded because they use a different training and evaluation protocol. MetaP and RelAdapter results are noted as not fully available due to unreleased pre-trained information.

Why This Matters

Impact on research. The paper challenges the implicit i.i.d. assumption behind MAML-style FSRL: KG relations are heterogeneous and semantically clustered (the paper's example is that FatherOfPerson and BrotherOf share family-tie semantics on Nell-One, while ColorOf concerns physical properties). By treating shared structure as something to be learned globally and task idiosyncrasy as something to be adapted locally, it offers a template that could transfer to other non-i.i.d. few-shot settings. It also repositions MoE as a task-level meta-learner rather than a per-instance fusion or scaling mechanism.

Real-world applications:

• Question answering over knowledge graphs, where queries touch rare or newly emerging relations with few recorded facts.
• Web search and entity-centric search, which rely on structured knowledge that is often sparse for long-tail relations.
• Recommendation systems that use knowledge graph structure, where new item categories or relation types appear with minimal interaction data.
• Knowledge graph completion pipelines for long-tail or emerging relations, the exact data-scarcity setting the paper targets.

Industry relevance. The method is deliberately lightweight at adaptation time: meta-testing freezes the global parameters and fine-tunes only three projection vectors per task, which suits deployments where new relations must be onboarded cheaply. The paper states that code is available at https://github.com/alexhw15/MoEMeta. The experiments were run on a single Tesla P100 GPU, and the paper includes complexity and runtime analyses in its appendices, though the truncated content does not report those numbers.

Future Directions

1. Richer projection functions. The paper notes that the three projection vectors could functionally be merged into one perturbation term, but keeping them separate preserves modularity so that more expressive or heterogeneous projection functions incorporating richer inductive biases could be added in future extensions.

2. Extending beyond relational learning. The global/local disentanglement is described as targeting a challenge in the "unique, non-i.i.d. context of relational learning," but the same idea of learning shared patterns across semantically clustered tasks could plausibly be tested in other few-shot domains where tasks are not independent.

3. Scaling and efficiency of the expert pool. The paper fixes M = 32 experts and N = 5 selected experts; the complexity analysis is placed in Appendix A and the runtime analysis in Appendix B.2, neither of which is available in the truncated content, leaving the behavior of larger expert pools or different sparsity levels as an open question.

4. Broader relation-pattern coverage and evaluation. Only three benchmarks are used (Nell-One, Wiki-One, FB15K-One), with 1-shot and 5-shot evaluation on the first two and 3-shot on the third. Whether the gains hold on other KG families, other shot counts, or relations with more diverse interaction patterns remains untested in the content provided. The provided content also does not report an explicit limitations section.

Target Audience

Researchers and graduate students working on knowledge graph representation, few-shot learning, or meta-learning who want a concrete architecture for separating shared from task-specific knowledge; practitioners building KG completion or KG-backed search and recommendation systems for long-tail relations; and readers interested in mixture-of-experts designs applied at the task level rather than the token or instance level.

Authors’ abstract

Few-shot knowledge graph relational learning seeks to perform reasoning over relations given only a limited number of training examples. While existing approaches largely adopt a meta-learning framework for enabling fast adaptation to new relations, they suffer from two key pitfalls. First, they learn relation meta-knowledge in isolation, failing to capture common relational patterns shared across tasks. Second, they struggle to effectively incorporate local, task-specific contexts crucial for rapid adaptation. To address these limitations, we propose MoEMeta, a novel meta-learning framework that disentangles globally shared knowledge from task-specific contexts to enable both effective model generalization and rapid adaptation. MoEMeta introduces two key innovations: (i) a mixture-of-experts (MoE) model that learns globally shared relational prototypes to enhance generalization, and (ii) a task-tailored adaptation mechanism that captures local contexts for fast task-specific adaptation. By balancing global generalization with local adaptability, MoEMeta significantly advances few-shot relational learning. Extensive experiments and analyses on three KG benchmarks show that MoEMeta consistently outperforms existing baselines, achieving state-of-the-art performance.