We thank the reviewer for the positive comments! We hope that our response can resolve your concerns. Please feel free to ask any follow-up questions.

---

W1:

Notation Definitions:

• $N$: Number of nodes
• $|E|$: Number of edges
• $D$: Dimension of node features
• $H$: Dimension of hidden layers
• $R$: Number of teachers
• $k$: Parameter for $k$-pop subgraphs

Space Complexity:

• Node feature matrix: $O(N \cdot D)$
• Adjacency matrix: $O(|E|)$ (assuming the graph is sparse)
• Representations of hidden layers: $O(R \cdot N \cdot H)$
• Optimized distribution of local subgraph representations: $O(N \cdot k \cdot |E|)$

Overall space complexity: $O(N \cdot D + |E| + R \cdot N \cdot H + N \cdot k \cdot |E|)$

Time Complexity:

• Forward propagation: $O(N \cdot H^2 + |E| \cdot H)$
• Local subgraph generation: $O(N \cdot k \cdot |E|)$
• Structured-WiseKnowledge Transfer (SWKT) module: $O(N \cdot k \cdot H)$
• Similarity measurement computation: $O(N \cdot k)$
• Geometric Embedding Optimization (GEO) module: $O(N \cdot H^2)$

Overall time complexity: $O(N \cdot H^2 + |E| \cdot H + N \cdot k \cdot |E| + N \cdot k \cdot H + N \cdot k)$

Besides, we also conducted a computational efficiency study comparing the teacher and student models from both theoretical and empirical perspectives in our response to W1 commented by Reviewer GamE. In Table 2 of our paper, we present the parameter ratios (compression rates) between the teacher and student models for each knowledge distillation (KD) method. Additionally, we provide information on distillation training times and student inference times in Tables 2, 10, and 11.

---

W2:

We fully agree with your suggestion that exploring the necessary modifications for handling multiple geometries would broaden the applicability of our method. In our paper, we have simply attempted experiments incorporating spherical geometry.

In Table 1, the Cross method represent simplified variants of our framework. The rows marked $\mathbb{E}$ and $\mathbb{S}$ correspond to the use of both Euclidean and spherical geometries, while rows $\mathbb{E}$, $\mathbb{B}$, and $\mathbb{S}$ represent the use of Euclidean, hyperbolic, and spherical geometries simultaneously. The results indicate that spherical geometry does not provide significant additional benefits within our framework. We apologize for any confusion caused by unclear descriptions about Table 1 of our paper. We will improve the presentation of the experimental section to enhance clarity.

In future work, we will explore feasible approaches to extend our framework to other potential geometries. For example, we may consider integrating more complex Riemannian geometries or other non-Euclidean geometries to further enhance the performance and applicability of the method. We appreciate your valuable suggestion and look forward to exploring these aspects in our future research.

---

W3:

Yes, moving preliminary definitions that are not directly related to the core method to the appendix will enhance the clarity of the paper.

After carefully review, we found that the expression for the Poincaré Disk Model in Definition 1 is not directly used in the presentation of our method, so we will move it to the appendix. The description of our method requires the projection formula between the tangent space and hyperbolic space in Definition 3, and this conversion involves hyperbolic operations described in Definition 2. Therefore, we will retain Definitions 2 and 3 in the main paper.

Thank you for your constructive suggestion. We will make the necessary adjustments based on your feedback to improve the clarity and readability of the paper.

---

W4:

In fact, we have conducted some experiments using spherical teacher model in our paper. In Tables 1,7,8 of our paper, the rows labeled $\mathbb{S}$ indicate results obtained using spherical models either alone or as one of the teacher models. Experimental results indicate that the use of spherical teacher models provides limited improvement in the results. This situation may be improved after further refining the framework for spherical space.

We will explore using teacher models from other geometric spaces to enhance the adaptability of our framework. We will also provide more detailed descriptions of the experimental setups in the paper to avoid similar misunderstandings and improve clarity.

Dear Reviewer rr98,

We thank again for your thorough review and constructive feedback. In our response to your comments, we included an analysis of our method's time and space complexity and noted that our original work have explored integrating other geometric models into the framework and presented the results. We genuinely hope you could check our responses, and kindly let us know your valuable feedback. We would be happy to provide any additional clarifications that you may need.

Best regards,

Authors

We sincerely thank the reviewer for the time and effort in reviewing our paper. We hope that our response can resolve your concerns. Please feel free to ask any follow-up questions.

---

S3:

In fact, we provided sensitivity analysis of the weight coefficient $\beta$ in Figure 3 (middle). We performed a joint analysis of the hyperparameters $\lambda$ and $\beta$ to evaluate their combined effects on our method. Lines with different markers denote different $\beta$ amd the best value of $\beta$ is around 3.

---

W1:

Theoretically：

GCN's time complexity is: $O(\sum^L_{l=1}(|E|H + NH^2))$, where $N, |E|, L$ denote the number of nodes, edges, and layers, respectively, $H$ denote hidden dimension. In our paper, teacher and student models' hidden dimension is 128 and 8, respectively. Assuming graphs is sparse, which means $|E| \approx O(N)$, student is theoretically 229x faster than their teacher models.

Empirically:

The inference times (ms) of the teacher and student models measured on our device are shown in the table below:

Our method achieves an average acceleration of 232×.

Additionally, we provided time and space complexity analysis of SWKT and GEO modules in the response to W1 commented by reviewer rr98. In our paper, we provide the parameter ratios (compression rate) between teacher and student for each KD method in Table 2, distillation training times and student inference time in Tables 2, 10, 11.

---

W2:

The largest dataset used in our experiments was Coauthor Physics (495,924 edges and 34,493 nodes). We conduct further experiments on larger datasets ogbn-arxiv (1,166,243 edges, 169,343 nodes) and ogbn-proteins (39,561,252 edges, 132,534 nodes). Results show that our method consistently achieves best distillation performance on larger datasets.

---

W3:

Yes, some remarkable pervious works focused on MLP-based students[1][4]. Meanwhile, there are also significant studies in graph KD that use GNNs as the student networks, such as [2] and [3].

Our motivations for choosing GNNs as student: We selected GNNs as the student model to explore a new trade-off among performance, inference speed, and compression rate. GNNs are designed for graph-based data, so GNN students can achieve performance close to their teachers even with a much small size (e.g., our method uses a GNN student model with a hidden dimension of only 8 and ~2% of the teacher model's size). Furthermore, GNN students demonstrate notable efficiency in inference acceleration, achieving an average speed-up of 232x compared to their teacher models in our method.

---

Q1:

Our student model takes full graphs as input. The sampled subgraphs you mentioned are used by the SWKT module to enable it to extract local structure features and corresponding manifolds.

Differences between MLPs and GNNs students:

• GNN students are smaller, with a ~2% size of their teachers (see Table 2). MLP students and their teachers have similar sizes due to having the same number of layers and hidden dimension (see appendixes in [1][4]).
• GNN students achieve better performance on graph data since they leverage adjacency information.
• MLP students have faster inference speeds due to their lack of graph dependencies, with [1] showing a speed increase of 273×, our GNN students is 232× faster than their teachers.

In summary, GNN students are suitable for scenarios with limited space resource, whereas MLP students are better suited for scenarios requiring higher inference speed.

Additional experiments with MLP-based student: Our distillation method is model-agnostic. Using the same settings as [1] and [4], we test our method with GCN teachers and MLP students. Following results show that our KD method has better or similar performance with MLP students compared to GLNN [1] and VQG [4].

(part of results from [4])

---

Q2:

In fact, except our method, all baseline methods do not incorporate various manifolds, as we are the first to introduce a KD framework using multiple geometric teachers. In Table 1, rows marked as $\mathbb{E}$ show results for baseline methods in their original versions without additional manifolds information, while rows marked as $\mathbb{B}$ and $\mathbb{S}$ show results after adapting to hyperbolic and spherical spaces, respectively. This adaptation ensures a fair comparison and evaluates the SWKT and GEO modules. We apologize for any confusion caused by unclear descriptions about Table 1. We will improve the presentation of the experimental section to enhance clarity.

---

References:

[1] Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation, ICLR’22

[2] Distilling Knowledge from Graph Convolutional Networks, CVPR’20

[3] Boosting Graph Neural Networks via Adaptive Knowledge Distillation, AAAI’23

[4] VQGraph: Rethinking Graph Representation Space for Bridging GNNs and MLPs, ICLR’24

Dear Reviewer GamE,

We appreciate your thorough review and constructive feedback. We have provided an analysis of the computational efficiency of our method and further validated its performance on the ogbn-arxiv and ogbn-proteins datasets. We have analyzed the differences between GNN and MLP student models and clarified potential misunderstandings about the baseline methods. We would be grateful for the opportunity to discuss whether your concerns have been fully addressed. Please let us know if you still have any unclear part of our work.

Best regards,

Authors

We sincerely thank the reviewer for the time and effort in reviewing our paper. We take all comments seriously and try our best to address every raised concerns. Please feel free to ask any follow-up questions.

---

Q1:

Yes, our experimental settings are strictly following the original papers. We used same pre-trained teachers, which were the best-performing models obtained by extensive hyperparameter tuning.

MSKD[1] is an amazing baseline method that introduces multi-scale teachers, providing more comprehensive information to enhance distillation efficiency. Regarding the impressive performance of the teacher model in MSKD  (GCN 89.67%, GAT 95.47%) on Citeseer, despite extensive efforts, we were unable to make our GCN teacher has the similar performance. Since the authors did not release Citeseer dataset used, the significant performance differences may be due to variations in dataset preprocessing or dataset versions.

We believe that we have obtained a well-trained and effective GCN teacher model on Citeseer, since our GCN teacher have a comparable performance (73.97%) with some latest works which use Citeseer. VQG (one of our baseline method, ICLR'24) [2] shows that the GCN teacher model of performs at 70.49% on Citeseer. Additionally, reported F1 scores for node classification with GCN on Citeseer are 71.0% in [3] (NIPS’23) , 72.9% in [4] (NIPS’23), and 73.18% in [5] (WWW’23).

---

Q2:

We also believe that we have obtained a well-trained and effective GAT teacher on Citeseer. Reported F1 scores with GAT on Citeseer are 73.00% in [4] (NIPS’23) ,76.63% in [5] (WWW’23) . These results are comparable to our GAT teacher's performance (75.24%)

---

Q3：

Unfortunately, due to a lack of detailed experimental information and specific dataset versions, we could not replicate MSKD teachers' performance on Citeseer.

We agree that using a well-trained teacher model is very important in KD. Therefore, we conducted extensive hyperparameter tuning to ensure our teacher models achieved near-optimal performance on each dataset, comparable to the best results reported in latest other graph-related studies [2][3][4][5].

Moreover, all methods employed the same student architecture. Thus, the concern that insufficient training of the teacher model may result in illusory performance of the student model should not appear in our experiments.

---

Q4:

All runtime data were recorded using Python's datetime library. In our experiments, we replaced the student model of VQG with GCN to maintain consistency with other baseline methods This adjustment is likely the primary reason why all students, including in VQG, exhibit similar inference times.

We analyzed the speed-up of the student compared to their teachers in our method:

Theoretically：

GCN layer's time complexity is: $O(\sum^L_{l=1}(|E|H + NH^2))$

where $N, |E|, L$ denote the number of nodes, edges, and layers, respectively, $H$ denote the hidden layer dimension. The teacher and student models have hidden dimension of 128 and 8, respectively.

Assuming the graph is sparse, which means $|E| \approx O(N)$. the student model is theoretically 229x faster than the teacher model about.

Empirically:

The inference times (ms) of the teacher and student models measured by datetime library are shown below:

Our method achieves a notable acceleration with an average speedup of 232×. While the different student architectures result in lower speedup compared to VQG, our method's student achieves a high compression rate (~2%, see Table 2), whereas VQG's MLP students and teachers are similar in size (see appendix in [2]). We chose GNN as the student model due to considerations of the trade-off between performance and inference speed. Please refer our responses to Reviewer GamE's W3 and Q1 for more details.

---

Q5:

Yes, VQG[6] was originally proposed to address the GNN-to-MLP conversion. In our experiments, we replaced the student model of VQG with GCN to maintain consistency with other baseline method.

We use GCN as the teacher model and replaced the student model with MLP same as VQG's student for a comparison with VQG and another MLP-based graph KD method GLNN[7]. We keep same settings with [6], and the results shown in the following table demonstrate that our method achieves effective distillation even with MLP-based student networks.

(part of results from [6])

---

In the responses, we explained the reasons why our teacher models are being well-trained. Besides, all methods used same settings and pre-trained teacher models, with results averaged over 10 trials. Based on these solid results, our method achieved sota performance, we hope this can change your opinion.

---

References:

[1] Multi-Scale Distillation from Multiple Graph Neural Networks , AAAI’22

[2] VQGraph: Rethinking Graph Representation Space for Bridging GNNs and MLPs, ICLR’24

[3] On Class Distributions Induced by Nearest Neighbor Graphs for Node Classification of Tabular Data, NIPS’23

[4] Re-Think and Re-Design Graph Neural Networks in Spaces of Continuous Graph Diffusion Functionals, NIPS’23

[5] GIF: A General Graph Unlearning Strategy via Influence Function, WWW’23

[6] VQGraph: Rethinking Graph Representation Space for Bridging GNNs and MLPs, ICLR’24

[7] Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation, ICLR’22

Dear Reviewer Z3B3,

We sincerely thank you for your time and effort in reviewing our paper. In responses to your comments, we reviewed relevant works and demonstrated that our pre-trained teacher models are sufficiently trained on the Citeseer dataset. We analyzed the acceleration ratio of the student model compared to the teacher model from both theoretical and practical perspectives. Additionally, we replaced our student model with an MLP model of the same architecture as used in the VQG method and conducted comparative experiments. We would be grateful for the opportunity to discuss whether your concerns have been fully addressed. Please let us know if you still have any unclear part of our work.

Best regards,

Authors

Thank you for your positive feedback and we are glad to know that our rebuttal and new experiments have addressed most of your concerns!

Graph data may exhibit geometric heterogeneity (e.g., grid or tree structures) across different local structures. Therefore, we are committed to matching appropriate embedding geometric spaces through meticulous analysis and extraction of local structural features to enhance graph representation quality. We validated the effectiveness of this paradigm through knowledge distillation tasks.

We are pleased to learn that you found our paper is well-written and that the experimental results validate the effectiveness of our agnostic cross-geometric framework and modules. In our initial rebuttal, we analyzed the speedup achieved by our method from both theoretical and practical perspectives and compared our method with GNN-to-MLP methods by replacing the student model of our method with MLP. We will ensure that these important details are included in the revised version. Thank you again for your insightful comments and valuable suggestions!

Best regards,

Authors

We sincerely thank the reviewer for the time and effort in reviewing our paper. We hope that our response can resolve your concerns. Please feel free to ask any follow-up questions.

---

W1 & Q1:

It is true that the failure of one or more teacher models could potentially impact the student model’s performance, we have implemented several mechanisms in our method to mitigate this risk:

• Ensemble Learning: Using multiple teacher models that capture different geometric properties provides redundancy and robustness. If one model fails, the others still contribute valuable insights, minimizing the impact on the student model.
• Geometric Optimization Network: GEO dynamically adjusts the weight of information from each teacher model based on the loss function, reducing the influence of any underperforming model and ensuring the student model receives the most reliable information.

Additional experiments： We designed various experimental strategies to assess the impact of failing teachers on students:

S1: Train student models without KD.
S2: Train student models with the best-tuned teacher model.
S3: Train student models with an underperforming teacher model.
S4: Train student models with an untrained teacher model.
S5: Train student models with all untrained teacher models.
Note: All methods except MSKD and ours use a single teacher model,so their S5 f1 scores are N/A.

Teachers' F1 score (%) of different experimental strategies:

NC f1 score (%) of student models on Cora under different experimental strategies:

Except S5, which teachers have an average performance of only 30%, our method’s distilled student models consistently maintain stable performance even when some teacher models fail.

---

W2 & Q2:

Yes, the student model can operate in other geometric spaces. At first, we chose a Euclidean student model to combine hyperbolic accuracy benefits with Euclidean efficiency and stability. Our framework is model-agnostic, allowing replacement of the student model with other neural networks.

To validate student on various geometric spaces, we tested NC F1 scores (%) on the Cora dataset. Euclidean and hyperbolic teachers' F1 score is 86.98% and 90.90%.

---

W3 & Q3:

Our framework is model-agnostic and works with various model architectures. In fact, we have conducted experiments with alternative architectures. We apologize if our results for replacing GCN with GAT, originally presented in Table 7 in the appendix due to space constraints, escaped the reviewer's attention.

We also followed your suggestion and replaced the Euclidean teacher model with the Graph Transformer Network (GTN)[1].

GTN teacher has 4 layers and a hidden dimension of 128. Specifically, during distillation, student model's $l$ layers match the last $l$ layers of teacher accordingly. The GTN and HGCN teacher output intermediate representations from each layer to the SWKT module for local subgraph structure extracting and selection. These distributions are then optimized by GEO module. Then, these features extracted from the optimized cross-geometric intermediate representations are transferred to students via the corresponding loss function. Additionally, traditional KD loss is computed from the logits output by both the teacher and student models.

---

W4 & Q4:

As your suggestion, we conducted experiments using student models with the same architecture as the teacher models in our method. Following results are NC F1 scores (%) on the Cora dataset. Euclidean and hyperbolic teachers' F1 score is 86.98% and 90.90%.

Compared to the results presented in Table 1 of our paper, student models now even outperform some teacher models, but using the same architecture as the teacher makes student models larger and slower, limiting their suitability for resource-constrained devices.

---

References:

[1] Graph Transformer Networks. NeurIPS'19

Dear Reviewer ogwd,

We sincerely thank you for your time and effort in reviewing our paper. In responses to your comments, we have carefully designed and conducted experiments and analyses, which we believe have covered your concerns. We would be grateful for the opportunity to discuss whether your concerns have been fully addressed. Please let us know if you still have any unclear part of our work.

Best regards,

Authors

We sincerely thank the reviewers for their time and effort in reviewing our paper and discussing with us. We appreciate the AC, SAC, and PC for their diligent efforts in organizing and overseeing the paper review process. We are delighted to see that during the discussion phase, Reviewer ogwd, Z3B3, GamE stated that our responses and new experiments have successfully addressed most of their concerns, and that all reviewers ultimately have a positive attitude toward our paper. We will ensure that these analyses and experimental results are included in the revised version of the paper.

Best regards,

Authors