Demystifying GNN Distillation by Replacing the GNN

Thank you for your detailed review. We appreciate that you found our paper interesting. We address each of your questions below:

W1. “The analysis is mainly focused on homophily graphs”

As discussed in the limitations section, neither the GNN distillation methods nor the direct regularization technique we study, yield strong performance in scenarios characterized by heterophily. This study focuses on identifying the factors contributing to the success of distillation methods. Accordingly, we aim to use the same priors as distillation methods.

---

W2. “Are GNN distillation loss and consistency loss equivalent?”

The GNN distillation loss and the consistency loss are (potentially) not mathematically equivalent in the non-linear activation. However, using a linearized network to analyze properties of DNNs is a common practice [1]. Furthermore, we would like to note the strong connection between our proof and our results. This theory motivated the connection between the consistency loss and the GNN; even though it is not necessarily a mathematical equivalence in the non-linear case. Empirical results support the connection in the non-linear case in Tab. 2.

[1] Arora, Sanjeev, et al. "A convergence analysis of gradient descent for deep linear neural networks." arXiv preprint arXiv:1810.02281 (2018).

---

W3. “it is necessary to validate the effectiveness on larger-scale datasets”

In Tab. 3 in the original manuscript, we compare our approach on much larger datasets: ogb-arxiv and ogb-products. We also include the results in a table below.

---

W4. “Lack of comparison with important baselines”

We provided comparisons to additional methods in the Appendix Tab. 7 in the original manuscript.

---

W5. “Combining them is indeed effective but an incremental contribution.”

Although the individual components - consistency loss, iterative pseudo-labeling, and label histograms - may have similarities with existing methods, our work provides the novel insight that the success of GNN distillation methods hinges on regularization rather than expressivity. Through our analysis, we demonstrate that integrating these techniques into an MLP model produces results that closely align with those achieved through GNN distillation. This finding highlights that GNNs contribute to optimizing MLPs by inducing homophilic regularization.

---

Thank you very much, once again, for your excellent comments. We respectfully ask that if your concerns are addressed, please consider updating your score. If not, please let us know what can be further improved; we are happy to continue the discussion any time until the end of the discussion period. Thank you!

Thank you for your detailed review. We appreciate that you found our paper interesting and important. We address each of your questions below:

W1. "... further analysis of the experimental results”

Q3. “Can there be more analysis in the experiment”

Following the reviewer’s comment, we extend in the revised version the empirical analysis with two additional experiments:

1. Analysis of accuracy vs. node degree: App. Fig. 5 illustrates the model's accuracy across nodes with varying edge degrees. The findings suggest that our optimization process for MLPs exhibits patterns closely aligned with those observed in GNN distillation. Additionally, we extend this analysis to scenarios involving noisy labels. For this, we evaluate the model’s accuracy for nodes with varying edge degrees when 20% of the train and validation labels are replaced with random (noisy) labels. The results, presented in App. Fig. 6, indicate that the impact of noise is relatively consistent across nodes of varying degrees. Furthermore, this effect qualitatively resembles the patterns observed when labels are not noisy.

2. Correlation between accuracy and node homophily: App. Fig. 7 examines the correlation between confidence assigned by the model to the true label and the homophily rate of the node. This analysis reveals that the correlation observed for our MLP is more similar to that of GNN distillation than to a standard MLP, further supporting our hypothesis.

---

W2. "these nodes appear to come from outside the training set”

Q1.“where does the node with pseudo-labels come from?”

We would like to clarify that our evaluations are in line with other works, and do not use the test labels. Similarly to most of the GNN models and distillation methods, we leverage un-labeled nodes that are available during the training phase. These nodes, also known as observed nodes, are incorporated into the training process by assigning pseudo-labels to the unlabeled nodes based on our own predictions (and not the ground-truth labels). Namely, our node predictions for which the model generates highly confident predictions.

---

Q2. “is the consistency loss correlated with the datasets?”

We analyze how incorporating homophily priors influences model performance across various datasets in Sec. 6. The effectiveness of the consistency loss demonstrates variability depending on the dataset (see Tab. 2). Yet, it is helpful on almost all of them. Analysis of the dataset variability with the consistency loss combined with pseudo-label smoothing is shown in Tab. 4.

Q2. “Can you compare [homophily prediction encouraged by GNN during distillation] with [ homophily prediction encouraged by your suggested loss during MLP training]?”

We added to the revised version an analysis on the correlation between confidence assigned by the model to the true label and the homophily rate of the node. App. Fig. 7 shows that the correlation observed for our MLP is more similar to that of GNN distillation than to a standard MLP, confirming our hypothesis. We include the results in the table below.

---

Q4. “Line 343, "As our goal is to understand GNN distillation, we want to optimize the MLP model such that during inference we do not utilize neighboring nodes. Thus, we apply prediction smoothing only to the pseudo-labels within the iterative training algorithm." What is the causal relationship here?”

Thank you for asking. We see how this sentence might be unclear and would like to clarify. As GNN-distillation methods aim to avoid using the graph-structure during inference, we also want to avoid using it. Therefore, we perform label smoothing (that relies on the graph structure) only during training. We clarify this in the revised manuscript.

---

Thank you very much, once again, for your excellent comments. We respectfully ask that if your concerns are addressed, please consider updating your score. If not, please let us know what can be further improved; we are happy to continue the discussion any time until the end of the discussion period. Thank you!

Thank you for your very detailed review. We appreciate that you found our paper clear and interesting. We address each of your questions below:

W1. “The authors could provide the train-test performance gaps of GNNs and MLPs to support this claim.”

We indeed claim that the few-labels setting significantly induces a performance gap between train and test sets. In the manuscript Fig. 1 shows that MLPs have similar accuracy to GNNs when the training set is bigger, while in smaller training sets MLPs are much worse. This indicates that MLPs overfit more than GNNs for small training sets. According to the reviewer's suggestion, to better support this claim, we add a comparison between the accuracy on the test and train sets in App. Tab 8. This table shows that MLPs have nearly 100% accuracy on the training set, while other models have lower training accuracy and higher testing accuracy. We also include the table below.

W1. “experiment in which GNN pseudo-labels are replaced with MLP's own pseudo-labels (in a self-training way) may help”

Our work indeed ablates the usage of pseudo-labels with experiments where we use the MLP’s own pseudo-labels in a self-training approach, without any reliance on GNN-generated labels. As presented in Sec. 4.3.2, Table 2, this comparison demonstrates that using such MLP pseudo-labels improves the model's accuracy. Furthermore, additional supporting ablation studies are provided in App. Tab. 9 to reinforce this finding.

---

W2. “Some related works are missing”

Thank you for bringing these works to our attention. We discuss them in the revised manuscript.

“the smoothing-regularization loss seems equivalent to the contrastive loss of [4] … The authors should make a distinction between these”

We thank the reviewer for this suggestion. Indeed there is a distinction, our loss only encourages similarity of neighboring nodes, and is not contrastive. Yet, we agree there is a relevant similarity and added an explanation on the distinction to the revised manuscript.

---

W3. “topic of this paper is distillation, not few-labels learning”

“The authors should provide more insights on why this technique is effective in distillation, especially in a normal setting where labels are sufficient.”

The topic of our paper is indeed understanding GNN distillation. GNN distillation is studied mostly for the few-labels setting [1, 2].

GNN distillation is mostly needed in cases where labels are scarce. In Fig. 1 we show that as the size of the training set increases, the performance gap between MLPs and GNN distillation diminishes. We show that part of the effectiveness of GNN distillation can be attributed to the utilization of unlabeled nodes. In our work, we directly emulate the key factors contributing to the success of GNN distillation on MLPs by using pseudo-labels. Specifically, we demonstrate that iterative pseudo-labeling achieves comparable performance to GNN distillation (see Tab. 2) when combined with a consistency loss that replicates the distillation loss - without requiring the GNN.

---

W4. “the authors put this important inductive setting in the appendix“

Following previous works [1, 2], we first report the results on the transductive settings, and later on the inductive settings. Due to space limitations, we put this experiment in the appendix. We added a clearer reference to it from the main text.

W4. “the proposed method is even surpassed by the weak baseline GLNN in some datasets”

The main goal of this paper is to analyze the key factors that make GNN distillation methods work. This includes, first, identifying the unique strengths of GNNs, namely that they act as homophily regularizers and that they benefit from unlabeled data. Second, we demonstrate that the performance of an MLP model that uses these properties is highly correlated with the performance of distillation. While we demonstrate practical techniques, we do not aim to propose a new method, but rather to provide a unique perspective and insights on training non-GNN graph classifiers.

---

[1] Zhang, Shichang, et al. "Graph-less neural networks: Teaching old mlps new tricks via distillation.", 2021.

[2] Tian, Yijun, et al. "Nosmog: Learning noise-robust and structure-aware mlps on graphs.", 2022.

---

We continue our response in the following comment.

Thank you for your detailed review. We appreciate that you found our paper good and interesting. We address each of your questions below:

W1. “add homophily ratio for each dataset”

Thank you for your suggestion. We added to Tab. 4 two homophily metrics: edge and node homophily rates. We note that datasets with stronger homophily benefit more from the homophilic priors. We also include these metrics in the table below.

W2. “Why is only GraphSage taken as base GNN”

Similar to previous works [1, 2], we compare to GraphSage as it is the teacher used in distillation baselines methods. Yet, following the reviewer request we are happy to report similar results for the GNN models GCN, GAT and APPNP. We revised the manuscript to include these models in App. Tab. 7. We show the table below:

[1] Zhang, Shichang, et al. "Graph-less neural networks: Teaching old mlps new tricks via distillation." arXiv preprint arXiv:2110.08727 (2021).

[2] Tian, Yijun, et al. "Nosmog: Learning noise-robust and structure-aware mlps on graphs." arXiv preprint arXiv:2208.10010 (2022).

W3. “What is the impact of gamma in eq.3”

In the distillation loss (Eq. 3), gamma determines the relative importance assigned to the teacher GNN predictions and the ground truth labels. In our consistency loss (Eq. 5), gamma controls the weighting of the consistency regularization term. We added to the revision App. Fig. 8, which depicts the accuracy of the model vs. the value of gamma. We find that our method is robust to a large range of gamma values.

W4. “Does the degree of a node have some role to play?”

“Are there any studies on noisy neighbors?”

We incorporated an analysis examining the impact of node degree on the model accuracy. In App. Fig. 5 in the revised manuscript, we present the model's accuracy for nodes with varying edge degrees. The results indicate that our optimization process for MLPs closely aligns with the patterns observed in GNN distillation. In the a-computer dataset, the accuracy appears to exhibit a stronger positive correlation with node degree compared to other datasets. Yet, generally there is no strong connection between the performance and the node degree across datasets.

Following the reviewer's suggestion, we did a similar experiment when replacing 20% of the train and validation labels with random (noisy) labels. We see that the effect is roughly similar for nodes of various degrees, and is qualitatively similar to the non-noisy labels. Please see the results in the revised paper (App. Fig. 6).

---

Thank you very much, once again, for your excellent comments. We respectfully ask that if you feel even more positive about our paper, to please consider updating your score. If not, please let us know what can be further improved; we are happy to continue the discussion any time until the end of the discussion period. Thank you!