Edge-free but Structure-aware: Prototype-Guided Knowledge Distillation from GNNs to MLPs

It appears there has been a misunderstanding regarding my question. "A crucial piece of information that is missing from the paper is the comparative performance of PGKD and a standard MLP. This comparison is vital as it establishes a baseline to understand the necessity of using PGKD. For instance, Lim et al. have demonstrated that a vanilla MLP can achieve a performance of 60.92 on the twitch-gamers dataset [1], which is slightly higher than what PGKD manages to achieve."

My inquiry was aimed at comparing the performance of PGKD with that of a standard MLP, not PGKD versus LINKX. The figure 60.92% pertains to the MLP's performance as documented in [1], which surpasses that of the PGKD. The performance of the MLP serves as a critical benchmark. Despite your previous note that PKGD achieves statistically significant improvements over GLNN for the larger Twitch-gamer dataset, it nonetheless falls short of the results obtained with a basic MLP. This raises questions about the actual efficacy of PKGD.

I would strongly urge the authors to include a comparison with a vanilla MLP as a baseline within the performance evaluation. Without this, justifying the use of PKGD over simpler models becomes challenging.

Thanks for your reviews. For your concerns, our responses are as follows:

R4-Q1: Concerns about the effectiveness of PGKD.

In this paper, we study the GNN2MLP distillation based on an edge-free setting. In the paper "Large scale learning on non-homophilous graphs: New benchmarks and strong simple methods", both LINK and LINKX employ the adjacent matrix as input which is not edge-free.

For effectiveness, PGKD outperforms GLNN with higher average scores and lower standard deviations. In particular, for large datasets (Arxiv and Twitch-gamer) with more than 160,000 nodes, PGKD statistically significantly (p-value < 0.05) outperforms GLNN. Also, as shown in Figure 2, PGKD shows strong robustness to noisy features.

R4-Q2: Missing important related works.

Thanks again for your kindness. We will add these papers to the next version.

R4-Q3: The authors emphasize latency reduction during inference as a key motivation for their work. Nevertheless, the paper lacks a crucial comparison of runtime between PKGD, GLNN, MLP, and GNN. Including such a comparison is vital to substantiate the authors' claims and showcase the practical benefits of PKGD in real-world applications.

For PKGD, GLNN, and MLP methods, the models we get and deploy are all MLPs of the same size. For PGKD, we do not add any parameters compared to GLNN. For the latency of the trained student model, GNN > GLNN==PGKD==MLP.

Thanks for your reviews. We are glad to see that you find our PGKD provides a novel aspect of the analysis of the effect of edges from inter-class effect and intra-class effect. And our paper is well-motivated and clear. For your concerns, our responses are as follows:

R2-Q1: Could you clarify the rationale behind selecting SAGE and GCN for small and large datasets? Are there specific reasons for this choice?

We just select them randomly to test the generality of different GNN teachers. In Figure 2, we show the performance of SAGE, GAT, GCN, and APPNP for more results.

R2-Q2: While the hyperparameters \lambda 1 and \lambda 2 play a pivotal role in obtaining optimal results, the current search space for these parameters is rather constrained. What might be the impact of expanding this search space to encompass a broader range of values?

Thanks for your suggestions. We do believe that searching hyperparameters in larger space could lead to better results. However, for larger datasets such as Arxiv and Twitch-gamer, the search cost is too high to afford. For more results, please refer to Table 10 and Table 11 in the Appendix.

R2-Q3: Why does the inductive test of GNN not need the graph structure according to Table 2?

Under the inductive setting, the goal is to test the general ability of unseen graph nodes since we often meet with unseen nodes in realistic scenarios.

R2-S1: Other suggestions.

We sincerely thank you for your insightful suggestions. We would consider adding them to our next version.

Thanks for your reviews. For your concerns, our responses are as follows:

R3-Q1: The motivation for introducing the prototype is not clear, it does not seem to tackle the problem of "structure-aware distillation". Although it can somehow satisfy the edge-free requirement, the advantage in distilling the structural information is not clear, which is more critical in GNN distillation.

Why choosing prototypes is based on the impact of edges we study. As explained in Section 4.1, the intra-class edges help capture the homophily property for nodes from the same class, and the inter-class edges determine the pattern of class distances. The class distances are defined as the distances between class prototypes. Also, the homophily property can be better mimicked by prototypes (Please refer to sec 4.2 for pros of prototypes). Therefore, we chose the prototype to design the losses.

R3-Q2: The so-called "the impact of graph structures (i.e. graph edges) on GNNs has been studied" is over-claimed. First, the conclusion based on the inter and intra edges is trivial. Second, the analysis and conclusion are based on some small datasets, which is not sufficient.

The conclusion for inter-class edge is non-trivial. We analyze the results on cora, citeseer, and A-computer. However, the A-computer dataset contains 7,650 nodes and 119,081 edges with 745-dimensional features.

R3-Q3: The overall model only introduces prototypes for more information distillation, rather than overcoming the critical "structure-aware distillation" problem.

Structure-aware means whether the MLP students learn the graph structural information. We analyze the graph structural information and then try to distill such information to MLP students without edges. Therefore, our PGKD method achieves the goal of learning structure-aware MLPs.

R3-Q4: The evaluated dataset is too small, where the distillation is not necessary.

We evaluate our dataset on 7 datasets and the size can be found in Appendix A.1, including Twitvh-game (168114 nodes and 6797557 edges) and Arxiv (169343 nodes and 1166243 edges). Both are large datasets.

R3-Q5: The baselines are weak and many important baselines are missed.

We have discussed many related methods in the related work section. As mentioned in section 5.2,

For baselines, we do not compare with the regularization methods since these methods utilize the graph edges as extra inputs. In real-world applications, these graph edges may be unavailable. Please refer to Appendix A.2 for specific scenarios. Therefore, we conduct all experiments in the edge-free setting. For fairness, we select the edge-free GLNN (Zhang et al., 2022) as our baseline, which adapts vanilla logit-based KD from GNNs to MLPs.

As far as we know, the only edge-free baseline till now is GLNN.

R3-Q6: How can the model distill the structure information?

Please refer to Section 6.1. We show the empirical results in Table 5 and Table 6. Also, we show the node distributions in Section 6.5 for visualization.

R3-Q7: What's the advantage of introducing prototypes?

Please refer to Section 4.2 for more details.

The intra-class edges in GNNs capture the homophily property for nodes from the same class. One intuitive idea is to draw closer for any two nodes from the same classes in the edge-free setting. However, this strategy has two drawbacks: 1) It is easily influenced by the outliers, viz. noisy points; and 2) A high complexity O(n^2 ) where n denotes the quantity of a given class. To tackle these, we design the intra-class loss analogous to InfoNCE (van den Oord et al., 2018), whose goal is to draw a selected node closer to its corresponding prototype.

Thanks for your reviews. We are glad to see that you find our problem formulation interesting and proposed PGKD brings forth a creative combination of knowledge distillation and prototype learning. For your concerns, our responses are as follows:

R1-Q1: Why PGKD is structure-aware but GLNN is not.

As mentioned in Section 5.8 of the GLNN paper,

For example, when every node is labeled by its degree or whether it forms a triangle. Then MLPs won’t be able to fit meaningful functions, and neither will GLNNs. We can see that GLNN only learns from the node distributions of GNN and ignores the edge information, and thus we consider GLNN as not structure-aware.

For PGKD, the MLP students do not only learn the node distributions from GNN teachers but also the impact of inter-class edges and intra-class edges.

R1-Q2: Why compare to GLNN only.

As mentioned in section 5.2,

For baselines, we do not compare with the regularization methods since these methods utilize the graph edges as extra inputs. In real-world applications, these graph edges may be unavailable. Please refer to Appendix A.2 for specific scenarios. Therefore, we conduct all experiments in the edge-free setting. For fairness, we select the edge-free GLNN (Zhang et al., 2022) as our baseline, which adapts vanilla logit-based KD from GNNs to MLPs.

As far as we know, the only edge-free baseline till now is GLNN.

R1-Q3: Why prototypical cluster helps distillation? is it only work for graphs or can it be generalized to other datasets? Can the authors provide some explanation? How is this work different from some prototypical contrastive learning: https://openreview.net/pdf?id=KmykpuSrjcq and https://arxiv.org/abs/2012.12533

Why choosing prototypes is based on the impact of edges we study. As explained in Section 4.1, the intra-class edges help capture the homophily property for nodes from the same class, and the inter-class edges determine the pattern of class distances. The class distances are defined as the distances between class prototypes. Also, the homophily property can be better mimicked by prototypes (Please refer to sec 4.2 for pros of prototypes). Therefore, we chose the prototype to design the losses.

PGKD works for the situation aiming to distill the knowledge from GNNs to struct-aware MLPs under the edge-free setting. For other scenarios, it may work if the motivation and setting are similar.

Differences:

i) Compared to PCL, the overall idea is similar and they design the fine-grained and coarse-grained prototypes. For PGKD, we only need to calculate one prototype for each class. Both methods employ the idea of contrastive learning. We would consider adding it to the Related Work section.

ii) MICRO-Graph is graph level but our PGKD is node level. Also, the prototypes in PGKD are related to node labels but the MICRO-Graph is self-supervised.