Thank you for appreciating the strengths of our work. Below are our responses to address your important comments.

C1. The justification of using Dirchlet energy ratio as the quantity for knowledge transfer, and the reason why it is more effective than previous SOTA methods.

Response: Thank you for the insightful comment. Please find our justifications below. (i) The goal of distillation is to enable the student MLP to preserve the GNN teacher's knowledge as much as possible. Our analysis reveals a crucial finding: the Feature Transformation (FT) and Graph Propagation (GP) operations in a GNN layer often exhibit opposing smoothing effects. GP tends to be aggressive, while FT is more restrained in smoothing, as shown in Figure 1 of the paper. These distinct smoothing patterns represent important teacher knowledge that the student should preserve. We achieve this by leveraging the proposed Dirichlet Energy Distillation (DED), where we use the well-recognized smoothness measure, Dirichlet Energy, to design the DE ratio. This DED technique is validated to be effective in the experiments. (ii) Moreover, in our design, the parameters of FT operations in teacher GNN have been directed injected into the student by the teacher injection technique (TIN), while the GP operations are not compatible with the FC layers in student MLP, and the DED technique with Dirichlet Energy ratio helps on the distillation of GP operations. Therefore, our methods, combining TIN and DED, work together to achieve better performance than existing methods, as validated by extensive experimental results and ablation studies.

C2. I believe the proof is correct. The error bound can be loose on graphs with node degree skewed.

Response: Thank you for the insightful comment. As stated in our paper, the value of our analysis lies in its attempt to establish a theoretical relationship for using simple MLP layers to approximate complex GNNs. Our theoretical analysis focuses on the general case across all possible graphs, with the error being bounded by the largest eigenvalue. We acknowledge that this bound can be large in scenarios where the node degree distribution is skewed. In experiments, the observed actual error is typically much smaller than the theoretical result, as validated in Table 3 of the paper.

C3. The experiments are comprehensive, and most recent baselines are included. Compare with a recent baseline VQGraph, which has been mentioned in the related work section.

Response: Thank you for your feedback on the comprehensiveness of our experiments. As suggested, we have included a comparison with VQGraph. We thoroughly searched all the hyperparameter spaces specified in the original VQGraph paper (Table 12 of VQGraph). Specifically, VQGraph has the following hyperparameter search space for the teacher SAGE with codebook: max_epoch $\in$ {100, 200, 500}, hidden_dim $=$ 128, dropout_ratio $\in$ {0, 0.2, 0.4, 0.6, 0.8}, learning_rate $\in$ {0.01, 1e-3, 1e-4}, weight_decay $\in$ {1e-3, 5e-4, 0}, codebook_size $\in$ {8192, 16384}, lamb_node $\in$ {0, 0.01, 0.001, 1e-4}, and lamb_edge $\in$ {0, 1e-1, 0.03, 0.01, 1e-3}. For distillation, the hyperparameter search space of VQGraph is: max_epoch $\in$ {200, 500}, norm_type $\in$ {“batch”, “layer”, “none”}, hidden_dim $=$ 128, dropout_ratio $\in$ {0, 0.1, 0.4, 0.5, 0.6}, learning_rate $\in$ {0.01, 5e-3, 3e-3, 1e-3}, weight_decay $\in$ {5e-3, 1e-3, 1e-4, 0}, lamb_soft_labels $\in$ {0.5, 1}, and lamb_soft_tokens $\in$ {1e-8, 1e-3, 1e-1, 1}. The table below reports the results of VQGraph and our method TINED+. Observe that our method outperform VQGraph on the datasets. We will include this comparison in the paper.

Table A Results of VQGraph and our TINED+ with SAGE as teacher on transductive setting. The best result is in italics.

Thank you for acknowledging the strengths of our work. Below are our responses addressing your important comments.

C1. The hyperparameter search space.

Response: We clarify that we mainly adopt the conventional hyperparameter search space for the teacher model and the student model, following existing studies. For the hyperparameters of our method, we also employ grid search within a limited space. Detailed information about the hyperparameters is provided in the appendix.

C2. Compare with vanilla MLP with the same number of layers as TINED.

Response: As suggested, in addition to the vanilla MLP with 2 layers, we also compare with a 4-layer MLP, denoted as MLP*. In Table A and B below, under both transductive and production settings, our method, TINED, outperforms MLP* across all datasets. Moreover, compared to the MLP with 2 layers, MLP* shows degraded performance, indicating that additional layers cause the MLP to overfit on these datasets.

Table A: Compare with MLP* with 4 layers under transductive setting. The results are averaged over 10 runs and reported with standard deviation

Table B: Compare with MLP* with 4 layers under prod(ind&tran) setting

C3. Clarify fine-tuning with $\eta$ (Equation 6).

Response: In Eq (6), the parameter $\eta$ controls the balance between the inherited teacher knowledge and learned parameters during fine-tuning. A small $\eta$ tends to let the student make less changes to the inherited teacher parameters, and as $\eta$ increases, the fine-tuning will update the parameters aggressively. As shown in the experiments in the response to comment C1 of Reviewer qs85, when varying $\eta$, a mild $\eta$ setting can strike a good balance, leading to better performance.

C4. This work focuses primarily on performance optimization rather than efficiency.

Response: Our goal is to achieve better effectiveness than existing distillation methods, ensuring that the student MLP operates significantly faster than the teacher GNN, while maintaining comparable efficiency to existing methods. As shown in Figure 3 of the paper, our methods TINED and TINED+ achieve the best trade-off between accuracy and inference time. For example, on CiteSeer, while the teacher requires 153.14 milliseconds (ms) for inference, our methods, TINED and TINED+, achieve the highest accuracy at the cost of just 1.63 ms. All the distillation methods maintain the same order of efficiency, around 1-2 ms.

C5. Can TINED effectively handle dynamic graphs? If not, what modifications would be necessary?

Response: In this work, we focus on static GNNs, but we agree that distillation on dynamic GNNs is a promising direction. We believe that the proposed Teacher Injection (TIN) and Dirichlet Energy Distillation (DED) techniques are compatible with dynamic GNNs, such as DynGNN, EvolveGCN, and TGAT. For dynamic GNNs, TIN can be adapted to periodically inject temporal parameters from the teacher into the student, and DED can be performed adaptively since the DE ratio can be efficiently calculated on new graph snapshots. We leave a detailed investigation of these adaptations for future work.

C6. How does TINED ensure that the computed DE ratios accurately reflect the intended smoothing/diversification effects of specific operations?

Response: We clarify that the operation considered by the DE ratio in Definition 4.3 is the GP operation in Eq. (4), which performs aggregation and concatenation, excluding subsequent softmax and batch normalization. Our formulation precisely controls the computation boundary to prevent conflation with other factors, ensuring that the DE ratio accurately reflects only the smoothing/diversification effect of the GP operation itself. DE is a widely adopted measure in the literature for assessing the smoothing effect of GNNs [1], making it a reasonable choice for designing our DE ratio.

[1] T Konstantin Rusch, Michael M Bronstein, and Siddhartha Mishra. A survey on oversmoothing in graph neural networks. 2023.

Thank you for recognizing the strengths of our work. Here are our responses to your important comments.

C1. Elaboration on deployment with and without graph dependency (line 185-190)

Response: When performing inference on a new node with limited connections to a graph, such as a new user joining a social network or a new product being added to an online shopping platform, the node has restricted access to the existing graph structure. In these cases, the graph structure is unavailable to the node (without graph dependency), so inference relies solely on node features. Conversely, when conducting inference on nodes that have sufficient connections within the graph structure (with graph dependency), it is possible to leverage both node features and graph features. This allows for a graph-dependent inference approach, utilizing the rich information available from the node's connections within the graph.

C2. Generalize to student MLP with various layer counts.

Response: Thank you for the insightful comment. As shown in Appendix A.3, our approach for decoupled GNNs, such as APPNP in Eq (17), consolidates all graph propagation (GP) operations into a single FC layer in MLP, which can be easily extended to multiple layers. For traditional GNNs like SAGE, our design focuses on capturing layer-to-layer GNN knowledge within MLPs through the proposed techniques, Teacher Injection and Dirichlet Energy Distillation. Consequently, the number of layers in our method is related to the teacher model. One potential way for generalization is to explore layer fusion techniques, where multiple teacher layers are compressed into fewer student layers while maintaining their overall behavior. We will include this discussion in the revised paper.

C3. The performance of TINED on heterophilous graphs remains unexplored.

Response: Please find the experiments on heterophilous graphs (Squirrel and Amazon-ratings) in the response to comment C3 of Reviewer qs85. The results show that TINED and TINED+ outperform existing approaches. As future work, we plan to develop dedicated techniques to further improve the effectiveness of heterophilic GNN distillation.

C4. (Teacher Injection Mechanism) How to handle when teacher is suboptimal.

Response: We clarify that the goal of knowledge distillation is to create a student model that closely mimics the teacher model, while whether the teacher model is suboptimal or not is not the focus. After injecting teacher parameters on FT operations into the student, the MLP layers in the student are further fine-tuned in Eq. (6) and trained with Dirichlet Energy Distillation and the overall loss in Eq. (8,9). These distillation training steps on the student MLP also mitigates the mentioned risk, and improves student performance.

C5. (Distillation of parameterized GP operations) Explanation on the handling of the parameterized GP operations.

Response: As shown in Eq. (4) in Section 4.1 and Eqs. (15, 16, 17) in Appendix A.3, different GNNs have distinct GP operations, such as GAT with attention. Therefore, for these various GP operations, we choose to distill them into fully-connected (FC) layers in MLP, without directly using these GP parameters, but employing carefully designed techniques including layer-wise Dirichlet Energy distillation. On the other hand, the FT operations in different GNNs share a similar formulation that is compatible with FC layers in MLPs, as shown in Eq. (4) and Appendix A.3. Thus, we can directly inject the parameters of FT operations into the student model.

C6. The impact of $\zeta$ on large datasets to approximate DE ratio

Response: The parameter $\zeta$ controls the subgraph sampled to estimate the DE ratio, helping to avoid running out of memory on large datasets. We vary $\zeta$ and report the results on the large ogbn-arxiv dataset in Table A below. Observe that our TINED maintains stable performance as $\zeta$ changes, and we do not need to consider $\zeta$ larger than 0.8, to reduce computational overhead, while maintaining distillation quality.

Table A: Vary $\zeta$ on large graph ogbn-arxiv

C7. The implementation of the "with graph structure" version and NOSMOG.

Response: The phrase "with graph structure" refers to conducting inference on nodes that have sufficient connections within the graph, allowing the use of both node features and graph features. In this setting, we use DeepWalk node embeddings for observed nodes, and for unobserved nodes, we use the average positional encodings of their observed neighbors as their encoding, consistent with the ways in NOSMOG. We will include this clarification in the paper.

We appreciate your effort in reviewing our paper. We have conducted extensive experiments to address your key comments. Thank you for your consideration.

C1. Could the author provide the sensitivity analyses on the datasets mentioned in this paper?

Response: In our method, the parameter $\eta$ controls the degree of fine-tuning in Eq. (6), while $\beta$ controls the importance of Dirichlet Energy distillation in Eq. (9). We have varied them with results on A-computer and Cora datasets in Tables 4 and 5 of the paper. Below, Tables A and B present the accuracy results of varying $\eta$ and $\beta$ across more datasets. In Table A, as $\eta$ increases across all datasets, TINED exhibits a distinct pattern where performance first improves and then declines, with the best results in italics. This trend underscores the trade off controlled by $\eta$. A similar pattern is observed for $\beta$ in Table B. Note that hyperparameter tuning is essential in machine learning research. Appendix A.7 of the paper details the search space for our parameters.

Table A: Vary $\eta$. Results are averaged over 10 runs with standard deviation

Table B: Vary $\beta$

C2. I am not very familiar with this field but I remember there are many works focusing on distill GNN into MLP. I hope other reviewers check the novelty of this work.

Response: In other reviews, the reviewers acknowledged the comprehensive experimental evaluations compared to existing methods, as well as the originality and significance of our methods. Thank you.

C3. Could the author add experiments on heterophilic dataset.

Response: As suggested, we have conducted new experiments on representative heterophilic datasets, Squirrel and Amazon-ratings under the production setting. In Tables C(1) and C(2) with and without graph dependency, our TINED and TINED+ outperform existing approaches under all settings, often with a significant margin.
Note that our work, as well as existing methods, does not explicitly focus on graphs with heterophily. As future work, we plan to conduct an in-depth investigation and develop dedicated techniques to further improve the effectiveness of heterophilic GNN distillation.

Table C(1): prod (ind & tran) setting on heterophilic dataset (without graph dependency)

Table C(2): prod (ind & tran) setting on heterophilic dataset (with graph dependency)