MoEMeta: Mixture-of-Experts Meta Learning for Few-Shot Relational Learning

Figure 1 lacks clarity.

Thank you for your advice. The figure layout is constrained by space limit. we will revise Figure 1 to improve clarity and provide more details to illustrate the main components of the proposed method.

Incremental contribution. While the integration is reasonable, it does not offer significant innovation beyond existing approaches.

We understand that our work might initially appear as a combination of existing components. However, we respectfully argue that our technical novelty lies in proposing a principled meta-learning framework to solve a non-trivial, previously unaddressed challenge in FSRL: how to disentangle globally shared patterns across tasks from local task specifics to facilitate fast adaptation to unseen tasks under few-shot settings.

We would like to reiterate the novel aspects of our work. First, we have correctly identified valid research limitations in the existing few-shot relational learning. These strongly motivate a compelling need for novel methodology to address these limitations. Second, we have proposed two key components to explicitly address these important research gaps:

• A novel MoE for learning shared relational prototypes: Unlike existing methods that learn meta-knowledge only within individual tasks, our MoE-based meta-learner is explicitly designed to capture common relational patterns shared across similar tasks for learning generalizable meta-representations. This is a novel formulation that explicitly models the compositional nature of relations in KGs, but also a clear distinction from previous work.

• A novel task-tailored adaptation mechanism: Unlike prior FSRL methods that relies on a single set of global parameters for initialization (a uniform MAML approach). Our work introduces a task-tailored adaptation mechanism that fine-tunes the meta-representations and entity embeddings tailored to the unique structural context of each task. Our adaptation is a novel, lightweight solution, specifically designed to capture the diverse interactions between entities and relations in individual tasks.

Therefore, our key innovation is a novel unified framework that enables global generalization across similar tasks and local adaptability. Our experimental evaluation demonstrates the state-of-the-art performance of our proposed solution, affirming the novelty of our contributions. We will revise the paper to clarify our contributions more explicitly.

The discussion on limitations should incorporate more relevant references.

Thank you for your valuable comments. Below, we provide further justification for the two limitations identified in prior work and will incorporate this discussion into the final version of our paper.

First, we argue that prior FSRL methods typically learn relation-specific meta-knowledge separately for each task, but lack mechanisms to capture patterns that are globally shared across similar tasks. This issue has also been echoed in the broader few-shot and meta-learning literature. For example, works like [4] and [5] have demonstrated the benefits of modeling task similarities or intra-class commonalities within each task. However, this principle has not been systematically explored for FRSL. Our work is the first to introduce a novel MoE for explicitly learning relational prototypes and leverage these shared patterns across tasks for knowledge transfer.

Second, we identify that existing meta learning-based FSRL methods typically adopt the generic Model-Agnostic Meta-Learning (MAML) framework without accounting for complex relation patterns in KGs. The KG literature has long established that relations exhibit complex, heterogeneous patterns (e.g., 1-to-N, N-to-N mappings) [6, 7], which pose a core challenge for relational learning. A standard MAML-style approach, which learns a single global initialization for all tasks, is ill-equipped to handle this structural diversity. Our work addresses this by introducing a task-tailored adaptation mechanism that explicitly uses the local context of the few-shot examples to adapt the model, a crucial step for handling the diverse nature of relations that has been overlooked for FRSL.

[4] P. Zhou et al., "Task similarity aware meta learning: theory-inspired improvement on MAML." In UAI 2021, PMLR 161:23-33. [5] H. Li et al., "Finding task-relevant features for few-shot learning by category traversal." In CVPR 2019: 1-10. [6] Z. Wang et al., "Knowledge graph embedding by translating on hyperplanes." In AAAI 2014: 1112-1119. [7] G. Ji et al., "Knowledge graph embedding via dynamic mapping matrix." In ACL/IJCNLP 2015: 687-696.

The "Related Work" section should reflect a summary of methodological lineages (including the problems each addresses).

Thank you for the valuable feedback. In our Related Work section, we have aimed not simply to list prior papers but to summarize the key limitations of existing methods, which clearly motivates the research gaps that our work targets. The discussion around methodological lineages is kept concise in the main paper mainly due to space limit. Yet, we have provided a more detailed discussion in Appendix C.3. We will further strengthen the Related Work section to ensure the narrative clearly emphasizes these aspects.

At the methodological level of MoEMeta: 1) Do the "attentive neighbor aggregation" and the "gated and attentive neighbor aggregator" in GANA differ in implementation? 2) Does "Task-Tailored Local Adaptation" differ from TransD in implementation?

Thank you for the insightful questions. We address both points as follows:

1). Attentive neighbor aggregation vs. GANA’s gated and attentive aggregator: Our attentive neighbor aggregation module is designed as a lightweight mechanism to incorporate local entity context, using a simple attention scheme over one-hop neighbors. This component is not the focus of our work; thus, we intentionally adopt a simple design to ensure that improvements stem from our core contributions (i.e., the MoE-based meta-knowledge and task-specific adaptation). In contrast, GANA's main contribution lies in designing a more sophisticated gated and attentive aggregation module with multiple gating and transformation layers to better handle sparse neighbours. We do not use the gated structure or GANA’s architecture in our implementation.

2). Task-Tailored Local Adaptation vs. TransD: Our task-specific adaptation mechanism is inspired by TransD in using projections to model the interaction between entities and relations. However, there is a key difference in how these projection vectors are treated:

• In TransD, the projection vectors are learned globally as part of the shared model parameters, and remain fixed across tasks or relations once trained.
• In contrast, our projection vectors are randomly initialized for each task and optimized only within that task, allowing for more flexible and task-specific adaptation in the few-shot setting. This design enables MoEMeta to better capture task-level semantics without relying on global sharing, which may not generalize well to unseen relations.

A reasonable comparison process between MoEMeta and the latest methods is required.

Thank you for your comment. We would like to clarify that RelAdapter is a recent 2024 method, and we have included comparative results with it on both FB15K-One and Wiki-One, as shown in Table 2 and Table 3. As RelAdapter does not report results on NELL-One, we compare MoEMeta against the strongest available baseline on NELL-One, HiRe. Despite being a 2023 method, HiRe remains a competitive baseline in meta-learning based FSRL literature, especially on NELL-One.

We acknowledge that meta-learning based FSRL has seen limited updates in recent years. We will continue to track and include future methods as they become available, and are prepared to include additional comparisons in an updated version of the paper if new implementations or results are released.

Marginal improvements in "Hits@X" on the Wiki-one dataset.

Thank you for valuable suggestion. We agree that the improvements of MoEMeta on the Wiki-One dataset are less pronounced compared to NELL-One. A key reason lies in the nature of relation representations in Wiki-One: In Wiki-One, relations are represented using non-semantic identifiers that lack natural language descriptions. In contrast, NELL-One uses textual relation names, which often exhibit explicit semantic regularities and correlations. Since MoEMeta is designed to model and leverage commonalities across relations through expert sharing, its advantage is more evident when relation semantics can be meaningfully captured. We will add this analysis to the revised version to clarify the differences across datasets.

We respectfully reiterate that MoEMeta is NOT a simple extension of existing methods such as MetaR or GANA. Instead, we have proposed a novel unified framework that enables both generalization across similar tasks and fine-grained local adaptation. Below, we would like to clarify the technical novelty of our approach, which addresses two fundamental limitations of prior methods.

• Capturing shared knowledge across tasks: Unlike existing methods that learn meta-knowledge only within individual tasks, our MoE-based framework is explicitly designed to capture common relational patterns shared across similar tasks. This is a key departure from previous work and allows for better generalization.

• Task-tailored adaptation: Prior methods often use a single set of global parameters for initialization (a uniform MAML approach). Our work introduces a task-tailored adaptation mechanism that fine-tunes the shared knowledge using the unique structural context of each task.

Importantly, our work is NOT a simple introduction of MoE into few-shot relational learning. It is a new framework carefully designed to disentangle what to generalize (globally shared patterns across tasks) from what to adapt (local task specifics) --- a distinction that prior work does not address. We believe that this is not only novel, but also opens up a new research direction in few-shot relational learning, encouraging future work to explore structured task relationships, rather than treating each task as i.i.d.

How the proposed model performs on traditional datasets such as FB15K237 and WN18RR?

Thanks for your question. We have provided experimental results on FB15K-One, a few-shot version derived from FB15K237, to evaluate the proposed MoEMeta. In the literature on few-shot relational learning, NELL-One and Wiki-One are commonly used benchmarks, whereas WN18RR is rarely adopted. One key reason is that WN18RR contains only 11 relations, making it unsuitable for constructing meaningful few-shot tasks with sufficient diversity.

Why the amount of experts is set as 32 on all the three datasets?

Thank you for this thoughtful question. To clarify, we did not uniformly fix the number of experts across different datasets. Instead, we treated the number of experts as a key hyperparameter and performed a sensitive analysis, as detailed in Section 5.4. On each dataset, we tuned the number of experts in the range of $\{16, 32, 64\}$ based on practical guidance from the number of relations and task diversity on that dataset. Our empirical results reveal a consistent finding: performance on all three datasets peaks or plateaus when the number of experts was set to 32, striking a good balance between model capacity and generalization.

It is suggested to conduct the analysis of the model's complexity.

Thanks for this valuable suggestion. We would like to draw your attention to our detailed analysis of model complexity and runtime in the appendix.

• Complexity Analysis (Appendix A.2): We analyze the complexity of all MoEMeta components with respect to the number of parameters and the number of multiplication operations in each epoch. The table below summarizes this analysis.

The MoE's parameter overhead of $\mathcal{O}(d^2)$ ($d$ is the embedding dimension) is on par with the complexity of the neighbor aggregator, which is an essential component in most SOTA methods. Thus, this additional parameter overhead is marginal relative to the model's overall complexity.

• Runtime Performance (Appendix B.2): We provide a runtime comparison with baselines in Appendix B.2. Table 7 reports the total runtime for meta-training and the inference time per triplet during meta-testing in seconds. The table below summarizes this comparison.

During meta-training, MoEMeta is only slightly slower than MetaR and RelAdapter, but faster than other baselines such as GANA and FSKGC. On NELL-One, MoEMeta requires 26,277 seconds to train, compared to 22,691 seconds for MetaR and 24,085 seconds for RelAdapter. This moderate increase in training time mainly comes from the two MLPs in the MoE. During meta-testing, MoEMeta, like all baselines, is highly efficient, with an inference time of milliseconds per triplet.

We sincerely thank the reviewer for providing supportive comments and acknowledging the strengths of our work. We respond to your questions and provide clarifications as follows:

In meta-learning community, such task-specific modulation has long been studied since 2018. It's a bit surprising that no prior work has studied such task-specific modulation in the context of relational learning area.

We agree with the reviewer that while task-specific modulation has been well studied in meta learning, its application to relational learning has been surprisingly limited. We believe that this gap exists precisely because a direct application of modulation from general meta-learning is inapplicable or insufficient for the structured, non-i.i.d. domain of knowledge graphs (KGs) for few-shot relational learning (FSRL), where only a handful of training triplets are available per task. Our work is thus proposed to fill this critical gap by proposing a novel unified framework that enables both generalization across similar tasks and fine-grained local adaptation.

To our best knowledge, RelAdapter (published in 2024) is the only baseline using task-specific adaptation. However, its simple approach--using an MLP to generate a single adaptation vector--fails to account for diverse interactions between entities and relations tailored to individual tasks. Our proposed approach addresses this limitation through explicitly projecting entity and relation embeddings into a task-specific subspace, offering a more principled solution for task-specific adaptability.

More specifically, it's surprising that no prior work has considered aggregating the neighbor information when processing each entity or relation. Task-specific MoE-like gating is not a new idea in meta-learning as well. The only difference seems whether it's for relational learning or not. Task-specific adaption is also not a new idea if we speak broadly, but the details seem novel.

Thank you for this insightful feedback. We would like to clarify the distinct contributions of our work by addressing each point.

• On neighbor aggregation: We clarify that neighbor aggregation is not novel; it is an essential component used by SOTA methods. Our work builds upon this standard practice, which is why we did not claim it as a contribution.

• Technical novelty of our work: We agree that the high-level concepts of task-specific modulation and MoE are not new in isolation in general meta-learning. Our core technical novelty lies in designing a principled framework that solves a critical, non-trivial challenge unaddressed by prior FSRL work: How to disentangle shared, global knowledge from task-specific, local context to enable effective generalisability and fast adaptation under few-shot settings.

Our MoEMeta framework provides a principled solution to this challenge:

• Global knowledge (what to generalize): This is achieved by our MoE-based relational prototypes that capture common relational patterns shared across similar tasks for learning relation-meta representations. Unlike Meta-DMoE [1] where each expert is specialized for a specific task domain, our MoE is designed to learn common relational patterns shared across tasks for better generalization. Our experts are trained to learn distinct relational prototypes. The gating network then learns to represent each new, unseen relation as a weighted combination of these shared, fundamental prototypes. This is a novel formulation that explicitly models the compositional nature of relations in KGs.

[1] Zhong, Tao, et al. "Meta-DMoE: Adapting to domain shift by meta-distillation from mixture-of-experts." NeurIPS 2022: 22243-22257.

• Local knowledge (what to adapt): This is achieved by our task-tailored adaptation mechanism that leverages task-specific structural context--unique to each task--to fine-tune relation-meta and entity embeddings. As the reviewer noted, "the details seem novel." and this distinction is crucial. Unlike general task-specific modulation methods [2, 3], which focus on predicting task mode to modulate the meta-learned prior parameters, our adaptation is a lightweight solution operating at the embedding level, specifically designed to capture the diverse interactions between entities and relations in individual tasks. For each task, we locally learn three randomly initialized projection vectors, allowing for highly specific and efficient adaptation.

[2] R. Vuorio et al., Multimodal Model-Agnostic Meta-Learning via Task-Aware Modulation, NeurIPS 2019: 14632-14644.

[3] Mi. Abdollahzadeh et al., Revisit Multimodal Meta-Learning through the Lens of Multi-Task Learning, NeurIPS 2021: 1-12.

This dual approach forms a novel unified framework that enables both generalization across similar tasks and fine-grained local adaptation. We will revise the paper to make these points more explicit and better contextualize our contributions with respect to the broader meta-learning literature.

More baselines and more public datasets.

We appreciate your valuable suggestion. We aimed to provide a thorough comparison against the most relevant and current methods. To clarify, our evaluation includes RelAdapter (published in 2024), which is, to our best knowledge, the latest SOTA baseline within the meta-learning paradigm for this task.

For fair comparison, our use of Nell-One and Wiki-One is consistent with the standard evaluation protocol used by the majority of prior work in few-shot relational learning. To further validate the generalizability of our method, we have also expanded our experiments to include a third dataset, FB15K-One, adopting the same setup introduced by the SOTA method, RelAdapter. The results are reported in Table 3.