Dear Reviewer SDXC,

We appreciate your constructive feedback. We are pleased to provide detailed responses to address your concerns.

W1:The novelty of the paper is limited.

A1: The novelty of Link-MoE: Our Link-MoE model stands out due to its innovative approach, leveraging the complementary strengths of different GNN4LP models, resulting in substantial performance gains. The core of our model's success lies in the following findings and innovations:

1. GNN4LP Models are Complementary: Our preliminary studies show that the overlapping between different GNN4LP models is notably low, shown in Figure 3, 9 and 10 in our paper, indicating these models offer complementary insights for link prediction. Different node pairs may be best addressed by different models. The critical challenge, then, is determining the most effective way to assign node pairs to the appropriate GNN4LP models.

2. Heuristic-Informed Gating Mechanism: Our preliminary studies revealed that different GNN4LP models perform differently across different heuristic groups. By designing a gating model that intelligently applies these heuristics, we facilitate the optimal assignment of node pairs to their most suitable predictors, which is a significant departure from traditional MoE applications, which use the same input features for the gating and experts. We compare the Link-MoE with traditional gating, which only leverages the node features in gating. The results are shown in Table 1 (Traditional MoE) in the global response. Traditional MoE only results in comparable performance to the best single experts, while our approach yields superior performance. This phenomenon demonstrates the effectiveness and rationality of the designed gating model.

W2:The baselines considered in the experiment are not comprehensive.

A2: Thank you for pointing out the related papers. Despite the Mean-Ensemble and Global-Ensemble method in our paper, we test two more ensemble methods. [1] and [2] utilized a random forest and MLP to ensemble various link predictors, respectively. The results are presented in Table 2 in the global response. Link-MoE outperforms all ensemble methods. The superior performance of our approach is attributed to the dynamic nature of our gating module, which assigns customized weights to each expert for every node pair.

[1] Stacking Models for Nearly Optimal Link Prediction in Complex Networks, PNAS'20

[2] An Ensemble Model for Link Prediction based on Graph Embedding, Decision Support Systems'22

W3:The overall computational efficiency of the mixture method is concerning. The idea of MoE, especially conditional computation, is not leveraged at all to reduce the computational cost.

A3: We would like to highlight that our framework differs from traditional MoE methods. Our goal is to leverage different GNN4LP models to address various node pairs effectively. We utilize a two-step training strategy to train the experts and a lightweight gating model separately. While training multiple experts does increase computational cost, our approach only needs a few experts to achieve superior performance compared to the baselines.

To demonstrate this, we conducted experiments on ogbl-collab and Pubmed datasets using only 3 or 4 experts. The detailed settings and results are shown in Table 1 (3 Experts & 4 Experts) in the global response.

We observe that using only 3 or 4 experts can achieve comparable performance to using all experts, indicating the computational cost of Link-MoE is not prohibitively high.

W4:The authors can further benchmark the Link-MoE on [3].

A4: We conduct experiments on ogbl-collab and PubMed under HeaRT setting [3]. The results shown in Table 4 of global response demonstrate that our model significantly outperforms all individual experts on both datasets. This highlights the effectiveness of our approach in leveraging the strengths of multiple experts.

Q1:Do authors consider involving the expert models as inputs to the gating models?

A5: We use the heuristic as input because the GNN4LP models are aligned with the heuristic methods, as investigated in Section 3.2 in our paper. To investigate the impact of involving expert model predictions as input to the gating model, we conducted experiments on the ogbl-collab and Pubmed datasets by concatenating the prediction results of experts with the heuristic features. The results are shown in Table 1 (With Experts as Input) of global response.

We observed that involving the experts' prediction results as additional input did not lead to improvement. In fact, this approach may result in lower performance compared to using heuristics alone. This phenomenon suggests that the outputs of the expert models may not effectively reflect their importance to specific node pairs, highlighting the effectiveness and rationality of using heuristics as the gating input.

Q2:Is there any ablation study on training the gating and expert models jointly?

A6: We explored an end2end training strategy that trains several experts alongside the gating module from scratch. We conducted experiments on Cora and ogbl-collab to evaluate this approach. The results and experimental settings are detailed in Table 3 of global response.

The end2end results were significantly worse than Link-MoE. A potential reason is that different experts have varying convergence rates, leading to the collapse problem. Figure 1(a) of global response illustrates some strong experts, such as NCN and NCNC, converge faster and dominate the performance. Strong models are assigned higher gating weights while MLP and GCN are assigned zero weights, as shown in Figure 1(b) of global response, which limits its ability to leverage additional information from MLP and GCN. While our end2end approach did not yield improved results, it remains an interesting and challenging idea to explore the conditional computation of MoE for efficient end2end training.

Dear Reviewer 758a,

Thank you so much for your support and recognition of our framework. We are pleased to provide detailed responses to address your concerns.

W1: The figures are hardly readable sometimes (e.g., Fig. 8), the font size (in the legend) should be increased.

A1: Thank you for pointing out this issue. We will enlarge the font size of the axis labels, legend, and axis ticks for all figures to ensure they are more readable.

W2: Notation problem: in Eq. 1 the gating function G has only one parameter while in Eq. 2 G takes both $x_{ij}$ and $s_{ij}$. Please uniform it as this may be a little confusing for the reader.

A2: In Eq. 1, we initially used $h_{ij}$ to represent all the heuristics, including both the pairwise node features $x_{ij}$ and the structural heuristics $s_{ij}$. To alleviate potential confusion, we will rewrite Eq. 1 to match Eq. 2 as follows:

$$Y\_{ij} = \sigma \left( \sum\_{o=1}^m G(\mathbf{x}\_{ij}, \mathbf{s}\_{ij})\_o E\_o(\mathbf{A}, \mathbf{X})\_{ij} \right)$$

W3: The difference between the proposed Link-MoE and the considered baseline Global-Ensemble should be further discussed in the main paper given the strong similarity between the two approaches.

A3: We will further discuss the Global-Ensemble method in the main text. Specifically, we will include a discussion of this method at the end of Section 3. We will first include the definition of the Global-Ensemble method (given by lines 570-575 in the appendix). We will then discuss the weaknesses of this method, mainly that the weights are non-adaptive and are the same for all links. Our empirical results in Section 3.1 and 3.2 also show that this kind of design is unable to model all links. The deficiency of this formulation serves as a strong motivation for our proposed method Link-MoE in Section 4.

W4: To evaluate the effectiveness of the proposed method different metrics are reported for different datasets.

A4: In Table 1, we use MRR for evaluating Cora, Citeseer, and Pubmed datasets, while Hits@50, Hits@100, and MRR are used for the ogbl-collab, ogbl-ppa, and ogbl-citation2 datasets. This approach is consistent with prior research, where notable methods [13, 14, 15, 16, 27, 53] have been designed for link prediction tasks. Additionally, we provide the results of other metrics in the Appendix, specifically in Tables 5, 6, 7, and 8.

Q1: Why reporting different metrics for different datasets?

A5: For the training of Link-MoE, we adhere to the experimental settings outlined in prior research [53], which serves as a benchmark for the link prediction task. During the training, we select and save the best model based on the validation performance. However, there are multiple evaluation metrics, which might be not aligned well. In this paper, we select the best model based on MRR for the Cora, Citeseer, and Pubmed datasets. For the ogbl-collab, ogbl-ppa, and ogbl-citation2 datasets, we used Hits@50, Hits@100, and MRR as selection metrics following [53]. As a result, our method might not perform optimally on all individual metrics due to the misalignment of different metrics. Nevertheless, we consistently rank within the top 3 across almost all datasets and metrics. These observations indicate that Link-MoE is capable of delivering strong performance across various metrics.

In the revision, we plan to fix all the typos and add another experiment that selects the best results based on each specific metric to provide a more comprehensive evaluation.

We hope that we have addressed the concerns in your comments, and please kindly let us know if there is any further concern, and we are happy to clarify.

Dear Reviewer fyCt,

We sincerely appreciate your recognition of our framework and insightful comments. We are pleased to provide detailed responses to your questions.

W2: It solves for a very specific task - Link Prediction in an inductive setting only. Not clear to me how it could be leveraged for a transductive setting as well.

A1: We'd like to clarify that our paper actually focuses on the transductive setting, where the same graph is used during both the training and testing phases. In contrast, the inductive setting involves unseen nodes and connections in the test graphs. Since the vast majority of methods for link prediction tasks are designed specifically for the transductive setting (please refer to [14, 15, 25, 26, 13, 16, 27] in our paper), we have limited our focus to it as well.

However, due to the generality of our Link-MoE method, it should have no issue adapting to the inductive setting when the individual expert models are themselves inductive. In the inductive setting, we would train the gating model based on the heuristics calculated from the training graph. During inference, we would simply recalculate the heuristics based on the test graph before applying Link-MoE. We believe this approach would allow Link-MoE to perform effectively in inductive settings, and we intend to explore this in future work.

Q1: Did you face any convergence issues with training the gating network?

A2: We did not encounter any convergence issues with training the gating network in our Link-MoE model. Our model successfully converges on the datasets we used, as evidenced by the smooth training loss curves on Pubmed presented in Figure 2 in the global response.

Q2: What happens when you use a graph network which is transductive in nature? Can that information be leveraged in some manner?

A3: As discussed in A1, while our Link-MoE focuses on the transductive setting, it can be generalized to the inductive setting. During the training phase, we train the gating model based on the heuristics calculated from the training graph. During inference, we would recalculate the heuristics based on the test graph before applying Link-MoE.

Q3: In the situations when the graph is a KG on language. Can LLM's outperform this method?

A4: We appreciate the inspiring question.

Due to the uniqueness of KGs, the methods used for KGs differ from those used on non-KGs (i.e., the graphs used in our paper). This is because the type of structural information needed to accurately predict new links depends on the graph. On KGs, research shows that path-based information that connects the two nodes are necessary for strong performance [16]. On the other hand, for link prediction on non-KGs, heuristics such as common neighbors or feature similarity tend to be more important [17]. Because of the disparity in inductive bias, in our paper we limit our focus to link prediction on non-KGs only. However, due to the generality of our method Link-MoE, it should have no issue adapting to the link prediction on KGs, as we only need to use KG-specific methods as the experts.

Lastly, as of now there is no consensus on whether LLMs can outperform graph-based methods for link prediction on KGs. However, recent work has shown that for inductive link prediction, LLM-based methods can potentially outperform SOTA graph-based methods on KGs (see [1] below).

[1] Wang, Kai, et al. "LLM as Prompter: Low-resource Inductive Reasoning on Arbitrary Knowledge Graphs." arXiv preprint arXiv:2402.11804 (2024).

Dear Reviewer bwf6,

We appreciate your constructive feedback. We are pleased to provide detailed responses to address your concerns.

W1:Despite its empirical effectiveness, the technical novelty appears limited.

A1: The novelty of Link-MoE. Our Link-MoE model stands out by intelligently leveraging the complementary strengths of different GNN4LP models, resulting in substantial performance gains. The core of our model's success lies in the following findings and innovations:

1. GNN4LP Models are Complementary: Our preliminary studies show that the overlapping between different GNN4LP models is notably low, shown in Figure 3, 9 and 10 in our paper, indicating these models offer complementary insights for link prediction. Different node pairs may be best addressed by different models. The critical challenge, then, is determining the most effective way to assign these node pairs to the appropriate GNN4LP models.

2. Heuristic-Informed Gating Mechanism: Our preliminary studies revealed that different GNN4LP models perform differently across different heuristic groups, shown in Figure 4, 11 and 12 in our paper. By designing a gating model that intelligently applies these heuristics, we facilitate the optimal assignment of node pairs to their most suitable predictors. This strategic use of heuristics marks a significant departure from traditional MoE applications, which use the same input features for gating and experts [1]. We compare the MoE with traditional gating with Link-MoE. The results are shown in Table 1 (Traditional MoE) in the global response. Traditional MoE only results in comparable performance to the best single experts, while our approach yields superior performance.

[1] Towards Understanding Mixture of Experts in Deep Learning. NeurIPS'22

The Collapse problem. In traditional MoE models, the collapse problem [37] can occur when a single expert is consistently selected, resulting in the under-utilization and inadequate learning of other experts. However, we employ a two-step training strategy for the Link-MoE. First, we train each expert separately. Then we train the gating model to leverage the strengths of each expert effectively. This approach ensures that each expert is well-trained before the gating model is introduced. We empirically observe that there does not exist collapse problem in our model. As illustrated in Figure 8 of this paper, different experts are activated to model different node pairs.

We also tested the end2end training of the experts and gating models, as shown in Table 3 and Figure 1 in the global response. Our results show that the convergence speed of different experts varies significantly, which often leads to the model collapsing to a single expert. As a result, the performance of end2end training is not as good as the proposed Link-MoE. Despite this, it remains an interesting and challenging idea to explore the effective end2end training.

W2.1:How does Link-MoE determine the appropriate number of experts?

A2: There are mainly three types of pairwise structural information in link-prediction tasks: local structure proximity (LSP), global structure proximity (GSP), and feature proximity (FP). Different GNN4LP models leverages different heuristic information. Based on these, we classify these methods into three groups: LSP (NeoGNN, NCN, NCNC, n2v), GSP (Seal, Buddy, NbfNet, PEG), and FP (MLP, GCN).

For the selection of experts, it is suggested to include methods that cover all groups due to the complexity of connection patterns across different datasets. To verify this, we conducted an experiment on the ogbl-collab dataset. When using all the models, the Hits@50 score is 71.32. However, if we remove all LSP experts, the Hits@50 score drops to 67.93. Interestingly, if we include only one LSP expert, the Hits@50 score can reach 71.25. This demonstrates the necessity of local structure information for the ogbl-collab dataset, but it also indicates that only one model may be sufficient to capture this information effectively.

Furthermore, our experiments show that using just a few experts (3 or 4) can achieve similar performance levels to using all experts, as shown in A4.

W2.2:On heterophilic graphs, how the experts selected by Link-MoE differ from those selected on homophiles graphs?

A3: We analyzed the weights generated by the gating mechanism on two heterophilic graphs, Chameleon and Squirrel. The results are shown in Figure 3 in the global response. We observed that feature proximity-based experts, such as MLP and GCN, are rarely employed for both datasets. This is consistent with the characteristics of heterophilic graphs, where connect nodes tend to have dissimilar features. These findings demonstrate the effectiveness of our gating design, as it accurately selects the appropriate experts for different types of graphs.

W3:The complexity of the proposed method can be high.

A4: The proposed Link-MoE model employs a two-step training strategy, which first trains the single experts and then trains a lightweight gating model. The most time-consuming part is the training of the single experts. However, as discussed in A2, there is no need to use all available experts, which can effectively reduce complexity and enhance scalability. we conduct experiments on ogbl-collab and Pubmed by only using 3 or 4 experts (please refer to Different number of experts for the expert selection details in the global response). The results shown in Table 1 (3 Experts & 4 Experts).

From the results, we can find that only 3 or 4 experts can achieve comparable performance with using all experts in the paper. Additionally, the two-step training strategy of our Link-MoE model introduces another layer of efficiency by allowing the integration of pre-trained models. In real-world scenarios where models are continually updated, this approach is highly efficient: when a new model arrives, we only need to train the gating model.