Node-wise Filtering in Graph Neural Networks: A Mixture of Experts Approach

Dear Reviewer N224,

We appreciate your constructive feedback. We are pleased to provide detailed responses to address your concerns.

W1: A comparable framework, Link-MoE [1], was previously proposed for link prediction and bears a close resemblance to the overall framework of Node-MoE. However, this work is neither cited nor discussed in the related literature. Given the presence of this recent work, the novelty of the proposed method appears limited.

A1: Thank you for pointing out Link-MoE [1], which utilizes the MoE framework for link prediction tasks. While both Link-MoE and Node-MoE leverage the MoE concept, they differ significantly in their motivations, expert designs, gating mechanisms, and training paradigms:

1. Motivation: The primary motivation of Link-MoE is to use different GNN4LP models for different node pairs, as different node pairs may require distinct heuristics for effective link prediction. In contrast, the motivation of Node-MoE is to address the diverse structural patterns exhibited by nodes within a graph.

2. Experts design: Link-MoE employs different GNN4LP models as experts to specialize in various link prediction heuristics. Node-MoE, on the other hand, uses the same base model as experts, allowing them to focus on learning distinct filters tailored to different node-level patterns.

3. Gating design: Link-MoE uses predefined heuristics as inputs for the gating model, while Node-MoE leverages the feature distance to distinguish nodes exhibiting different structural patterns.

4. Model Training: Link-MoE follows a two-step training process, where the expert models are trained first, followed by the training of the gating model. In contrast, Node-MoE adopts an end-to-end training approach, simultaneously training the gating model and experts, which ensures better integration and optimization.

We have updated our paper to include a discussion of Link-MoE in the related work section, highlighting these differences and clarifying the unique contributions of Node-MoE.

W2: While the paper argues that applying different filters to homophilic and heterophilic separately can lead to linear separability, a similar effect might also be achieved by increasing the representation capacity of global filters. For instance, by increasing the order in polynomial filters, or by learning piecewise polynomial filters[2]. Moreover, [3] explores node-wise filters via polynomial graph transformers. A comparison with these methods is necessary to evaluate the effectiveness of Node-MoE.

A2: Thank you for pointing out these related works. While high-order polynomial filters can address complex graph structures, the optimal order of polynomial filters is the number of nodes $n$, which is infeasible for large graphs. Moreover, the lack of sufficient labels for training such high-order filters and the need to ensure generalization further limit their practicality. PP-GNN [2] addresses the over-smoothing issue of the Monomial basis by approximating with piecewise polynomial filters. However, the approximation is suboptimal due to the complexity of the optimal filter, which still requires an order of $n$. PP-GNN [2] applies the same filter to all nodes, whereas our theoretical analysis shows that a single filter optimized for one type of node may negatively impact the performance of other node types. PolyFormer [3] applies multiple filters (where each polynomial basis can be viewed as a filter) to all nodes with an attention mechanism to combine the filtered results, which has a similar idea to ACM-GCN and AutoGCN. In contrast, the proposed Node-MoE employs a gating model to adaptively select appropriate filters for different nodes.

We further conducted experiments to compare the performance of Node-MoE with PP-GNN [2] and PolyFormer [3]. The following results further highlight the effectiveness of Node-MoE, particularly on heterophilic datasets.

Q3: What are some limitations of this work?

A6: While the proposed Node-MoE demonstrates strong performance and flexibility, we acknowledge the following limitations and propose future directions for improvement:

1. Gating Model: The gating model is a critical component of Node-MoE, as it determines the assignment of experts to different nodes. In this work, we leverage the feature distance between a node and its neighbors as the input for the gating model, and we have theoretically demonstrated that this feature distance can differentiate homophilic and heterophilic nodes under the CSBM model (as shown in the revised Theorem 1). However, real-world graphs may exhibit more complex features, where the feature distance might not effectively separate homophilic and heterophilic nodes. Although Node-MoE consistently outperforms the single-expert baseline under the noise feature setting, as shown in Table 6, designing more sophisticated gating models could further improve performance.

2. Expert Model: In this work, we use the same expert model with different learned filters. However, exploring diverse expert architectures tailored to specific node characteristics could lead to further improvements. For instance, if a node's feature alone can determine its label, an MLP-based expert could be more efficient. Investigating hybrid expert architectures could enhance both efficiency and performance.

3. Applicability to Other Tasks: The primary focus of this work is on node classification. However, extending Node-MoE to other graph-based tasks, such as node regression and clustering, is a promising direction for future research. These tasks could benefit from the flexibility and adaptability of the Node-MoE framework.

We hope that we have addressed the concerns in your comments. Please kindly let us know if there are any further concerns, and we are happy to clarify.

W3: Table 1 compares Node-MoE with ACM-GCN [4]. However, [4] also includes additional results for ACM-GCN+ and ACMII-GCN++, which achieve 76.08% (Node-MoE: 73.64%) on the Chameleon dataset and 69.98% (Node-MoE: 62.31%) on the Squirrel dataset, with similar improvements across other datasets. GCN+ and GCN++ incorporate MLP(A) as additional features. Moreover, [5] demonstrated that incorporating structural information (A) as additional features alongside simple graph filters can yield notable improvements—reaching 77.48% on the Chameleon dataset and 74.17% on the Squirrel dataset. In light of these results, the necessity of an MoE-based approach remains unclear.}

A3: Thank you for pointing out the variants of ACM-GCN. As noted, ACM-GCN+ incorporates MLP(A) as additional features, which provides an enhancement over the original ACM-GCN. However, since this trick was not utilized by other baselines in our experiments, we did not initially include the variants of ACM-GCN in the comparison.

We would like to point out that the reported performance of ACM-GCN+ and ACMII-GCN++, which achieve 76.08% and 69.98% on the Chameleon dataset, respectively, is based on a random 60%/20%/20% split. In our experiments, we use the fixed data split as the same with [6, 7].

To ensure a fair comparison, we extended our approach by incorporating MLP(A) into the experts of Node-MoE, resulting in a variant denoted as Node-MoE+. Using the same experimental setup and hyperparameter tuning (based on the official code of ACM-GCN paper), we compared Node-MoE+ against ACM-GCN+ and ACM-GCN++. The results show that Node-MoE+ also benefits from the inclusion of MLP(A) on heterophilic datasets, achieving competitive performance improvements. However, on homophilic datasets, both ACM-GCN+ and Node-MoE+ exhibit minimal performance gains.

HLP [5] considers learning filter coefficients as equivalent to learning a new graph and uses SVD to generate these new graphs. However, applying SVD to large graphs is computationally expensive, which may not be efficient for practical applications. Additionally, the data splits of the Chameleon and Squirrel datasets used in HLP [5] differ from those in our work and ACM-GCN, making direct performance comparisons challenging. Since HLP [5] does not provide code, we are unable to include numerical performance comparisons with their method. Nonetheless, the proposed Node-MoE demonstrates strong performance on both homophilic and heterophilic datasets, validating its robustness and effectiveness.

[6] Pei, Hongbin, et al. "Geom-gcn: Geometric graph convolutional networks." ICLR'20.

[7] Lim, Derek, et al. "Large scale learning on non-homophilous graphs: New benchmarks and strong simple methods." NeurIPS'21.

Q1: The average training time per epoch for ChebNetII and Node-MoE-3/5 are quite similar. What is the end-to-end training time comparison between these methods?

A4: The training time per epoch for ChebNetII and Node-MoE-3/5 is quite similar, as shown in Table 2, because we adopt a Top-1 gating mechanism in Node-MoE, which ensures that each node is processed by only one expert. We adopt end-to-end training for Node-MoE with the same maximum number of epochs and select the results with the best epoch.

Q2. In Appendix C.2, the strategies for polynomial coefficients include both increasing/decreasing powers of $\alpha^k$ and uniform values. How would initializing each filter randomly impact performance?

A5: Thank you for the suggestion. We have tested the performance of the proposed Node-MoE with random filter initialization, denoted as Node-MoE-Random. The following results demonstrate that Node-MoE with our filter initialization strategy outperforms Node-MoE-Random across all datasets. However, it is noteworthy that Node-MoE-Random still performs better than the single expert ChebNetII, highlighting the robustness of the Node-MoE framework.

I would like to thank the authors for their proactive engagement during the discussion period and for providing additional experiments and clarifications. Based on the presented results, the ACMII-GCN+/++ variants demonstrate competitive performance compared to Node-MoE, which somewhat diminishes the appeal of an MoE-based approach in this context. As such, I will maintain my current rating of 5.

That said, the current setup utilizes identical expert models differentiated only by filter initializations. To further strengthen the contribution of this work, I suggest exploring the use of diverse expert models—such as a combination of specialized GNN architectures tailored for homophily and heterophily. Demonstrating improvements with such an approach could significantly enhance the paper’s impact and practical relevance.

Q2: Regarding "ensuring that neighboring nodes are likely to receive similar expert selections", does this apply to both homophilic and heterophilic graphs? If so, how would the approach differ between them? In heterophilic graphs, neighboring nodes are more likely to differ in features or labels, so how would similar expert selections across neighbors work?

A5: As shown in Figure 3 in Section 2.1, the community phenomenon exists in both homophilic and heterophilic graphs. In the proposed Node-MoE, we do not explicitly distinguish between homophilic and heterophilic graphs, as the gating model operates uniformly across both types.

It is true that neighboring nodes are more likely to differ in features or labels in a heterophilic graph. However, the gating model's objective is to assign nodes to different experts, where each expert is capable of handling nodes from different classes. Although neighboring nodes have different features or labels, they can still be processed by the same filter. For instance, as shown in Figure 1(b) of our paper, although the heterophilic nodes have different features and labels, the same high pass filter can still distinguish them.

We hope that we have addressed the concerns in your comments. Please kindly let us know if there are any further concerns, and we are happy to clarify.

W3: The proposed method appears to be a combination of a gating function and a mixture of experts. When multiple GNNs with different filters are used, the model grows in complexity, introducing additional learnable parameters and hyperparameters (such as $\lambda$ for adjusting the influence of the filter smoothing loss, and K for the number of experts). This added complexity typically leads to performance improvements, so the modest gains shown in the experimental results are not surprising.

A3: The number of learnable parameters of the proposed method are similar to several baselines. To ensure a fair comparison, we assume that all models have a feature dimension and hidden size of $d$, and the number of layers is $l$. We omit a small number of parameters, such as the learnable filter parameters in models like GPR-GNN and ChebNetII, which are typically around 10 and do not significantly impact the comparison. For the MLP, GCN, APPNP, GCNII, GPR-GNN, ChebNetII, AutoGCN, WRGCN, and PC-Conv, the trainable parameters are approximately $ld^2$. For GAT and ASGAT, with $K_1$ attention heads, the number of learnable parameters becomes $K_1ld^2$. ACM-GCN uses three filters with separate transformation matrices, resulting in $3ld^2$ and LinkX includes three MLPs, also leading to $3ld^2$. For the MoE-based methods, GMoE has $K_2ld^2$ parameters, where $K_2$ is the number of experts. Mowst has $2ld^2$ learnable parameters, which combines MLP and GCN. For the proposed Node-MoE, the number of learnable parameters is also $K_2ld^2$, and $K_2 = 3$ for most of the results in Table 1. Thus, the proposed Node-MoE has a comparable number of learnable parameters to several baselines. Furthermore, as shown in Table 2, the training time per epoch for Node-MoE with top-1 gating is comparable to that of a single expert model, ensuring that the additional parameters do not significantly inflate training costs.

We further conducted experiments with Node-MoE using 3 experts and a significantly reduced hidden dimension size, decreasing it from 64 to 21. This adjustment ensures that the proposed Node-MoE has a comparable number of learnable parameters to models like MLP and GCN. Despite the lower hidden dimension, the results demonstrate that Node-MoE maintains similar performance and continues to outperform the baselines.

Compared to the base expert model, the additional hyperparameters introduced by Node-MoE include the filter smoothing loss weight $\lambda$, the number of experts, and the top-k gating mechanism. As shown in Figure 9, the model demonstrates robustness to the choice of $\lambda$, achieving stable performance across a wide range of values. Similarly, Table 4 shows that the proposed method achieves competitive performance even with as few as 2 experts, and Table 5 indicates that good performance is maintained with a top-1 gating mechanism. These results suggest that the proposed method is robust to its additional hyperparameters.

Q1: Could you elaborate on why GIN has strong community detection capabilities?

A4: GNNs are widely utilized in community detection and graph clustering tasks due to their powerful ability to represent graph-structured data effectively [1, 2, 3, 4]. Due to the community properties of nodes with different patterns as shown in Section 2.1, we choose GIN as an example for the gating model. As demonstrated in A1, GCN can also achieve comparable performance to GIN and both outperform MLP, further validating the effectiveness of GNN-based models in this context.

[1] Shchur, Oleksandr, and Stephan Günnemann. "Overlapping community detection with graph neural networks." arXiv preprint arXiv:1909.12201 (2019).

[2] Souravlas, Stavros, Sofia Anastasiadou, and Stefanos Katsavounis. "A survey on the recent advances of deep community detection." Applied Sciences 11.16 (2021): 7179.

[3] Bianchi, Filippo Maria, Daniele Grattarola, and Cesare Alippi. "Spectral clustering with graph neural networks for graph pooling." International conference on machine learning. PMLR, 2020.

[4] Tsitsulin, Anton, et al. "Graph clustering with graph neural networks." Journal of Machine Learning Research 24.127 (2023): 1-21.

Dear Reviewer s73g,

We appreciate your constructive feedback. We are pleased to provide detailed responses to address your concerns.

W1: The observation of varying structural patterns (homophilic and heterophilic patterns within the same graph) is not highly original. While the paper provides a well-illustrated demonstration in Section 2.1, analyzing these patterns from two perspectives: node homophily density and homophily across different communities, this does not significantly enhance the originality. These structural variations are commonly known and have been addressed in previous works using various approaches as discussed in Section 5 Related Works.

A1: We acknowledge that several prior studies have highlighted that real-world graphs contain nodes exhibiting variations in homophilic behavior, as we have also cited in Section 2.1. The previous methods usually apply the same filter or multiple filters to all nodes. Through our theoretical analysis, we demonstrate that applying the same filter to all nodes can lead to suboptimal performance, as a single filter optimized for one type of node may negatively impact others. This observation inspired our node-wise filtering approach, where different filters are applied to different nodes.

In Section 2.1, we further illustrate that homophily levels vary significantly across different structural communities, and nodes with varying homophily levels are not uniformly distributed across the graph. This phenomenon, to the best of our knowledge, has not been explicitly studied in prior works. This insight motivated the choice of our gating model, which is tailored to capture community properties. We demonstrate experimentally that GNN-based gating models usually outperform MLP-based ones, as shown in the table below:

W2: The proposed method assumes that a heterophilic pattern is present when a node’s features significantly differ from those of its neighboring nodes. While this assumption is reasonable, it restricts the method's applicability. Node features do not always correlate with node labels in terms of homophilic and heterophilic patterns, which may reduce the method's effectiveness when feature-label alignment is weak. Although the authors argue that the proposed method is distinct from existing works because those approaches pass all nodes through all filters, these prior methods do not make specific assumptions about heterophilic patterns, instead allowing the filters to adaptively learn from data. Thus, the distinction between the current work and these prior methods seems more like a trade-off rather than a clear improvement.

A2: Thank you for raising the issue of feature-label alignment. While it is true that the effectiveness of our method depends on some level of alignment, this is a common assumption in graph-based models. For example, the CSBM model assumes that nodes from different classes exhibit different feature distributions. To further strengthen our approach, we have added a new theorem in the revised version of our paper to address the classification of homophilic and heterophilic nodes. The theorem and proof can be found in Section 2.2 and Appendix A of our paper. For your convenience, the new theorem is as follows:

Homophilic and heterophilic nodes can be separated based on the feature distance between a node and the average feature vector of its neighbors, given by $\|X_i - \sum_{j \in \mathcal{N}(i)} \frac{X_j}{D_{ii}}\|$ with probability $P = 1 - o_d(1)$.

Specifically, we demonstrate that the feature distance $\|X-D^{-1}AX|$ can be used to effectively separate homophilic and heterophilic nodes in the CSBM model.

To address weaker feature-label alignment scenarios, we have conducted additional experiments by introducing varying levels of Gaussian noise to the node features. The results, shown in Table 6 of our paper, reveal that as the noise level increases, the performance of both MLP and learnable filter methods like ChebNetII drops significantly. The performance gap between NODE-MOE and the single-expert ChebyNetII decreases. However, NODE-MOE consistently outperforms the single expert, even under high noise levels. This is because, when noise is excessively high, the gating model may randomly assign nodes to different experts, causing the learned filters to converge to a performance similar to the single-expert model. Nevertheless, Node-MoE remains robust and effective across various noise levels, demonstrating its adaptability to weaker feature-label alignment scenarios. Furthermore, the proposed Node-MoE is highly flexible and it can incorporate existing methods as experts, which also differ from the current methods.

Dear Reviewer VQRg,

We appreciate your constructive feedback. We are pleased to provide detailed responses to address your concerns.

W1: Figure 1 serves as a poor illustrative example, as it may lead to confusion. The introduction in Section I states that “nodes tend to connect with others that share similar labels, reflecting homophilic patterns,” yet Figure 1 shows both solid and dashed edge nodes connecting to nodes labeled and 1 on the right side, which contradicts this description.

A1: Thank you for pointing out the potential confusion regarding Figure 1. In the figure, we assume that a node exhibits a homophilic pattern if the majority of its neighbors share the same label as the node itself; otherwise, it exhibits a heterophilic pattern. For the solid-edge nodes, 2 out of their 3 neighbors have the same label, indicating homophilic patterns. Conversely, for the dashed-edge nodes, 2 out of their 3 neighbors have different labels, indicating heterophilic patterns. Thus, Figure 1 is consistent with the description in Section I. To avoid further confusion, we have revised the paper to clarify the assumptions and explanation associated with Figure 1, ensuring it aligns clearly with the text.

W2: The message and patterns presented in Figure 2 lack clarity regarding their relevance to the key contributions of the work. The statement that “the majority of nodes in homophilic graphs predominantly exhibit homophilic patterns” raises questions about the definition of homophilic graphs. According to the authors' definition in the Homophily metrics section, this concept only applies to graphs with different labels; otherwise, h(v) will always equal 1.

A2: It is true that the homophily metric applies only to graphs with nodes having different labels. The homophily metric for graphs is a global property, where a high graph-level homophily indicates that nodes generally tend to connect with others that share the same label. However, at the node level, it is not guaranteed that the majority of neighbors of every individual node in a homophilic graph will share the same label. Figure 2 illustrates the density of node-level homophily across the graph, showing that while most nodes in homophilic graphs predominantly exhibit homophilic patterns, some nodes still exhibit heterophilic patterns. This observation inspires our node-wise filtering approach, which applies different filters to nodes with homophilic and heterophilic patterns. To address potential confusion, we have revised the description of Figure 2 in the paper for greater clarity.

W3: The paper suffers from overloaded notations that can lead to confusion for readers. For instance, h(v_i) is used in Section 1 to denote the label similarity between a node v_i and its neighbors, while in Section 3.1, the same notation is repurposed to refer to a node classifier. This inconsistency may hinder comprehension.

A3: Thank you for pointing out the issue with overloaded notations. To address this, we have updated the notation for the classifier in Section 3.1 to Classifier for clarity. Additionally, we have reviewed the other notations throughout the paper to ensure consistency and avoid potential confusion.

We hope that we have addressed the concerns in your comments. Please kindly let us know if there are any further concerns, and we are happy to clarify.

Dear Reviewer qWvW,

We sincerely appreciate your recognition of our work and insightful comments. We are pleased to provide detailed responses to answer your questions.

W1: Limited novelty of the proposed problem setup (Section 2.1 and 2.2). It is well known that real-world graphs contain nodes that exhibit variations in homophilic behavior. [ref - 1,2]. Limited technical novelty since, much of the techniques used in the proposed method is based on earlier work in the MoE area like Top K sampling, use of gating mechanism [3].

A1: Thank you for pointing out the related works. We acknowledge that several prior studies have highlighted that real-world graphs contain nodes exhibiting variations in homophilic behavior, as we have also cited in Section 2.1. The previous methods usually apply the same filter or multiple filters to all nodes. Through our theoretical analysis, we demonstrate that applying the same filter to all nodes can lead to suboptimal performance, as a single filter optimized for one type of node may negatively impact others. This observation inspired our node-wise filtering approach, where different filters are applied to different nodes.

In Section 2.1, we further illustrate that homophily levels vary significantly across different structural communities, and nodes with varying homophily levels are not uniformly distributed across the graph. This insight motivated the choice of our gating model, which is tailored to capture community properties. We demonstrate experimentally that GNN-based gating models usually outperform MLP-based ones, as shown in the table below:

While it is a common practice to leverage MoE models with top-k gating mechanisms to reduce computational complexity, the technical novelty of Node-MoE lies in the design and selection of the gating model. Specifically, we innovate in the input design for the gating model and the choice of the gating model itself. As shown in Figure 8 in our paper, the proposed gating model significantly outperforms traditional gating models, demonstrating the effectiveness of our approach.

W2: The gains in performance due to the proposed method "Node-MoE" is not clear compared to other non MoE baselines given the significant overlap of mean standard deviation. Moreover, the heterophilic datasets chosen are known to be problematic [4]. It would a stronger argument, if additional new datasets from [4] are used for the experiments.

A2: To address your concern, we have updated Table 1 to include a significance test (t-test) for our results compared to the best-performing baselines. The results demonstrate that the proposed Node-MoE outperforms all baselines on 7 datasets, with 5 of these showing statistically significant improvements ($p<0.05$). These findings provide stronger evidence for the effectiveness of Node-MoE.

For the Chameleon and Squirrel datasets, [4] demonstrates that there are some nodes with the same features and neighbors. To address this, we provide results for filtered versions of these datasets in Table 8 of our paper. The proposed Node-MoE continues to outperform the baselines on these filtered datasets. Additionally, we have included results for two widely used large heterophilic datasets, Penn94 and Pokec, in our experiments, where Node-MoE also demonstrates superior performance compared to the baselines.

Following your suggestion, we have conducted further experiments on the amazon-ratings and tolokers datasets from [4]. The results show that the proposed Node-MoE not only outperforms the single expert model but also achieves better performance than the best model reported in [4]. These additional evaluations reinforce the robustness and generalizability of Node-MoE across diverse datasets.

W3: A clear algorithm detailing the different techniques can be presented for better clarity to the reader.

A3: Thank you for the great suggestion. We have added the algorithm in Appendix C.2 in the revised paper.

Q1: Given that the performance gains in Table 1 are not very clear (due to the overlap in std. dev. with non MoE baselines) what were the winning hyper parameters from the search space? The reason I ask is because the paper claims that due to efficiency reasons one can reduce the number of experts (say 2) and utilize gating with Top K = 1. But since that come with the trade-off with accuracy, I am curious about the setting used for the main results presented.

A4: For the results presented in Table 1, we used 3 experts without sparse gating to simplify the experimental setup. To provide a more comprehensive analysis, we present the performance with varying numbers of experts and different Top-K gating settings in Table 4 and Table 5, respectively.

The results demonstrate that accuracy is not always a trade-off with the number of experts or the Top-K gating mechanism. For instance, while using 5 experts outperforms 3 experts on the Cora dataset, it performs worse on the Chameleon dataset. Similarly, Top-1 gating on the Cora dataset yields even better results than using all experts. However, the differences in performance across these configurations are not substantial. These findings suggest that the relationship between the number of experts, the gating mechanism, and performance is dataset-dependent, and the proposed Node-MoE is robust across a range of configurations.

We hope that we have addressed the concerns in your comments. Please kindly let us know if there are any further concerns, and we are happy to clarify.

Dear Reviewer AtKX,

We appreciate your constructive feedback. We are pleased to provide detailed responses to address your concerns.

W1: The abstract and introduction of this paper require revision to reflect the existing literature more accurately.

A1: Thanks for your great suggestion, we have revised our paper following your suggestions. Here are some changes for your reference:

Abstract: Most GNNs employ a uniform global filter. ... While a few methods have introduced multiple global filters, they often apply these filters uniformly across all nodes, which may not effectively capture the diverse structural patterns present in real-world graphs.

Introduction: There are a few methods, such as ACM-GNN, AutoGCN, PC-Conv, and ASGAT that leverage different filters to alleviate this issue. Specifically, they apply multiple filters to all nodes and then combine the predictions of different filters. However, applying a uniform type of filter, tailored for just one of these patterns, across all nodes may hurt the performance of other patterns. Combining the predictions of different filters may inevitably introduce noise.

Instead of applying one or multiple filters to all nodes, we propose a node-wise filtering method that apply different filters to different nodes based on their specific structural patterns.

W2: The paper claims that the proposed model offers better computational complexity than previous work discussed in the related section. However, no rigorous analysis is provided to substantiate this claim. Given the highly parallelizable nature of some existing “post-fusion” architectures, these prior models may, in fact, achieve lower computational costs.

A2: To provide a fair comparison, we assume identical model architectures and training methods, differing only in the use of filters. For post-fusion methods, each node must be processed by all filters, resulting in higher computational demands. However, for the proposed node-wise filtering method Node-MoE, each node only needs to be processed by one filter with a top-1 gating mechanism. Consequently, the total computational cost of Node-MoE is lower than that of post-fusion methods. While it is true that some existing post-fusion architectures can be highly parallelized, enabling similar training times, their overall computational cost remains high due to the necessity of processing all filters for every node. Additionally, the proposed Node-MoE is flexible and can also benefit from existing parallelization techniques, further enhancing its computational efficiency.

W3: Theorem 1 only provides proof for linear models with lower expressivity than typical GNNs. Moreover, the proof in part 2 relies on a strong assumption that every homophilic and heterophilic node can be perfectly classified and assigned the appropriate filter.

A3: In our theoretical analysis, we separate the graph-filtering operator from the classifier, similar to decoupled GNNs like SGC, which is a common practice for analyzing GNNs [1, 2, 3]. Theorem 1 demonstrates that applying different filters to nodes with varying patterns can achieve linear separability, providing a foundational understanding of the advantages of node-wise filtering in such scenarios.

We appreciate your feedback on the assumption in part 2 of the proof. To address this, we have included additional proof in the revised version of our paper to handle the classification of homophilic and heterophilic nodes. The theorem and proof can be found in Section 2.2 and Appendix A of our paper. For your convenience, the new theorem is stated as follows:

Homophilic and heterophilic nodes can be separated based on the feature distance between a node and the average feature of its neighbors, given by $\|X_i - \sum_{j \in \mathcal{N}(i)} \frac{X_j}{D_{ii}}\|$ with probability $P = 1 - o_d(1)$.

Specifically, we demonstrate that the feature distance $\|X-D^{-1}AX|$ can be leveraged to effectively separate homophilic and heterophilic nodes. This refinement not only mitigates the strong assumption but also provides a rationale for the design of the gating model, further validating our approach.

[1] Baranwal, Aseem, Kimon Fountoulakis, and Aukosh Jagannath. "Graph convolution for semi-supervised classification: Improved linear separability and out-of-distribution generalization." arXiv'21

[2] Ma, Yao, et al. "Is homophily a necessity for graph neural networks?." ICLR'22.

[3] Wu, Xinyi, et al. "A non-asymptotic analysis of oversmoothing in graph neural networks." arXiv'22

W4: The proposed NODE-MoE seems to use a larger number of learnable parameters compared to other baselines. How many learnable parameters are used for both the baselines and NODE-MoE, as shown in Table 1? Is the same performance improvement observed, even if the number of parameters for NODE-MoE is adjusted to be on a similar scale as the baselines?

A4: Different baselines have varying configurations across datasets, such as the number of layers, hidden dimensions, and attention heads. To ensure a fair comparison, we assume that all models have a feature dimension and hidden size of $d$, and the number of layers is $l$. We omit a small number of parameters, such as the learnable filter parameters in models like GPR-GNN and ChebNetII, which are typically around 10 and do not significantly impact the comparison. For the MLP, GCN, APPNP, GCNII, GPR-GNN, ChebNetII, AutoGCN, WRGCN, and PC-Conv, the trainable parameters are approximately $ld^2$. For GAT and ASGAT, with $K_1$ attention heads, the number of learnable parameters becomes $K_1ld^2$. ACM-GCN uses three filters with separate transformation matrices, resulting in $3ld^2$ and LinkX includes three MLPs, also leading to $3ld^2$. For the MoE-based methods, GMoE has $K_2ld^2$ parameters, where $K_2$ is the number of experts. Mowst has $2ld^2$ learnable parameters, which combines MLP and GCN. For the proposed Node-MoE, the number of learnable parameters is also $K_2ld^2$, and $K_2 = 3$ for most of the results in Table 1. Thus, the proposed Node-MoE has a comparable number of learnable parameters to several baselines.

We further conducted experiments with Node-MoE using 3 experts and a significantly reduced hidden dimension size, decreasing it from 64 to 21. This adjustment ensures that the proposed Node-MoE has a comparable number of learnable parameters to models like MLP and GCN. Despite the lower hidden dimension, the results demonstrate that Node-MoE maintains similar performance and continues to outperform the baselines.

W5: The proposed NODE-MoE seems to have a broader hyperparameter search space than other baselines. Since its training time per epoch is longer, this broader search space would likely result in a significantly longer training time, including the validation time required to find the best hyperparameter set. Is the same performance improvement observed, even if the total training time, including hyperparameter search, is adjusted to be comparable to the baselines?

A5: Compared to the base expert model, the additional hyperparameters introduced by Node-MoE include the filter smoothing loss weight $\lambda$, the number of experts, and the top-k gating mechanism. As shown in Figure 9, the model demonstrates robustness to the choice of $\lambda$, achieving stable performance across a wide range of values. Similarly, Table 4 shows that the proposed method achieves competitive performance even with as few as 2 experts, and Table 5 indicates that good performance is maintained with a top-1 gating mechanism. These results suggest that the proposed method is robust to its additional hyperparameters. Furthermore, as shown in Table 2, the training time per epoch for Node-MoE is comparable to that of a single expert model, ensuring that the additional hyperparameters do not significantly inflate training costs.

W6: Figures 6 and 15 seem inconsistent with the underlying motivation.

A6: In Figure 5 of our paper, we demonstrate the two learned filters. It's important to note that we did not constrain the shape of these filters. Filter 0 is approximately a low-pass filter but also has a relatively high response at high frequencies. Conversely, Filter 1 is primarily a high-pass filter but also responds at low frequencies. Additionally, the real homophily level of each node in the gating model is estimated based on the node features, which can introduce some errors. Consequently, both filters have weights in each homophily group. As a result, both filters have weights in each homophily group. Despite this, there is a clear trend in Figure 6: as the node homophily level increases, the weight of the high-pass filter consistently decreases. The overall trend supports our motivation that nodes with higher homophily levels tend to rely less on the high-pass filter. For Figure 15, the CiteSeer dataset exhibits a high homophily level, and the two learned filters as shown in Figure 14 are low pass filters. However, Filter 1 has a higher response for the high-frequency signals. Therefore, the weight of filter 1 decreases with the increase of homophily level. Therefore, it also consistents with our motivation.

W7: Minor: There appears to be an error in y-tick (density) values in Figure 2, as they are not within the 0-1 range.

A7: Thank you for pointing this out. Figure 2 illustrates the probability density function (PDF), where the area under the entire curve equals 1. As a result, the y-axis (density) values are not constrained to the 0-1 range. To clarify this, we have updated the figure description in the revised paper to explicitly state that it represents a PDF.

Q1: How did the authors find GIN and ChevNetII as an gating and expert models respectively? What alternatives are considered and why these combination performs the best?

A8: Gating model selection:

As discussed in Section 2.1, structural patterns are not uniformly distributed across the graph, and distinct communities often exhibit varying structural characteristics. To account for these differences, we aim to use a gating model capable of capturing community properties. Given the strong community detection capabilities of GNNs, we selected GIN as an example. We also experimented with different gating models, and the results show that GNN-based models, such as GCN and GIN, generally outperform MLP, particularly on heterophilic datasets.

Expert model selection:

Rather than using fixed filters, we aim for each expert to learn its own filter dynamically. Therefore, we selected expert models from the category of learnable filter GNNs. In approximation theory, Chebyshev polynomials achieve near-optimal error when approximating functions, making ChebNetII an excellent choice for its strong filter learning capabilities. Additionally, we experimented with GPR-GNN as an expert model. The results indicate that Node-MoE with GPR-GNN significantly improves the performance of the base GPR-GNN model, further demonstrating the flexibility and effectiveness of our approach.

We hope that we have addressed the concerns in your comments. Please kindly let us know if there are any further concerns, and we are happy to clarify.

Thank you for the detailed response. While the authors address several points raised in the review, there are still notable gaps and concerns. For example, the robustness of the proposed model against different combinations of hyperparameters is unclear, given the narrow differences between the performance of different models. Several analyses provided in the manuscript are also inconsistent regarding the dataset used for the analysis. Based on the other reviews and responses, I have decided to keep my original score.