An Instance-Level Framework for Multi-tasking Graph Self-Supervised Learning

Thanks for your insightful comments and constructive suggestions! We mark the contents that already existed (but were missed by reviewers) in red, and those revised or newly added contents in blue in the revised manuscript. Next, we will address the concerns you raised, one by one, as described below:

---

Q1: The main concern is the experiments in this paper are only related to the classification task in a multi-tasking setting, while the proposed method is task-irrelevant. Although the some vision tasks are involved in the appendix, the graph-related tasks, such as node clustering and link prediction, should be investigated.

A1: We have added performance comparisons of MGSSL, AutoSSL, and ParetoGNN on various graph-related downstream tasks (graph regression, node clustering, link prediction) in Tables A2/A3/A4 and Lines 642-654 in Appendix A.10. These experiments demonstrate the applicability of MGSSL to various downstream tasks and its superiority over AutoSSL and ParetoGNN.

---

Q2: The Eq. (4) is quite misleading. The proposed distillation is in an offline fashion, and the optimality of teacher has already in the objective. As a result, the constraint is a little bit unnecessary. Also, the parameters for optimization are not clearly presented in the objective.

A2: The parameters to be optimized in Eq. (4) during KD are the student model ($\{\theta, \omega\}$) and the weighting function $\lambda_\gamma(k,i)$ (parameterized by $\gamma$). Therefore, although each teacher model is frozen before KD, the integrated teacher (the optimality of teacher) $\sum_{k=1}^K\lambda_\gamma(k,i) \widetilde{\mathbf{h}}\_i^{(k)}$ changes as $\lambda_\gamma(k,i)$ is updated during KD. As a result, Eq. (4) is essentially more like an online KD, and perhaps we have not made it clear what the parameters of optimization are, leading you to misinterpret it as "an offline fashion". To make it clear, we have added the above explanation to Lines 127-130 of the revised manuscript.

---

Q3: In page 2, you claim “Secondly, balancing multiple tasks by loss weighting during the pre-training phase makes it hard to scale the pre-trained model to emerging tasks. To incorporate new tasks, it requires to re-pretrain a new model from scratch.” Is it an over- pessimistic claim for the existing methods? Which may implicit exaggerate the contribution of the proposed method.

A3: Thanks for your insightful comment. We have modified the previous expression (somewhat over-pessimistic),

"To incorporate new tasks, it requires to re-pretrain a new model from scratch."

to a more moderate expression,

"To incorporate new tasks, it requires to re-weight the losses of new tasks and existing tasks to re-pretrain the model." in the revised manuscript.”

---

Q4: How do you perform vision task in Appendix 10? Please provide more details for the graph construction. Maybe you can discuss the performance difference between different methods to build the graph.

A4: To adapt MGSSL to vision tasks, we conduct graph construction by taking images as nodes and connecting the k-Nearest Neighbors (kNN) of each image to build edges. As you suggested, we have also provided the results of constructing the graph by thresholding in Table A5 of Appendix A.10, where two images with cosine similarity greater than 0.7 will be connected by an edge. The results show that constructing the graph by kNN outperforms thresholding, and we speculate that this is because kNN guarantees the balance of node degrees in the constructed graph and prevents the over-squeezing problem that is common in graph learning. We have added the above explanation to Lines 659-669 of the revised manuscript.

---

Q5: Is there more evidence for Figure 4? From Guidance 1, the integrated teacher probability is as close as possible to the true Bayesian probability. What is the specific reason for the failure of the other two schemes?

A5: Among the three schemes in Figure 4, only "Adaptive (ours)" follows Guideline 1, while "Averaged" and "Weighted" are two heuristic (vanilla) schemes that have nothing to do with Guideline 1. The reasons for the failure of "Averaged" and "Weighted" are that they cannot differentiate important teachers from irrelevant ones, and may mislead the student in the presence of low-quality teachers, as the manuscript described. By contrast, "Adaptive (ours)" follows Guideline 1, which enables the integrated teacher probability to be as close as possible to the true Bayesian probability by adaptive teacher weighting. To make it clear, we have added the above explanation to Lines 186-193 of the revised manuscript.

---

In light of these responses, we hope we have addressed your concerns. If we have left any notable points of concern unaddressed, please do share and we will attend to these points. We sincerely hope that you can appreciate our efforts on responses and revisions and raise your score.

Dear Reviewer,

Thank you for your insightful and helpful comments once again. We greatly appreciate your feedback. We have carefully responded to each of your questions point-by-point.

Unlike previous years, there will be no second stage of author-reviewer discussions this year. As the deadline for the discussion is approaching, we kindly request you to inform us if you have any additional questions.

Best regards,

Authors.

Thanks again for your insightful review and keeping a positive score for acceptance. If there is anything else that needs clarification, please don't hesitate to contact us!

Best regards, Authors.

Thanks for your insightful comments and constructive suggestions! We mark the contents that already existed (but were missed by reviewers) in red, and those revised or newly added contents in blue in the revised manuscript. Next, we will address the concerns you raised, one by one, as described below:

---

Q1: I wonder what knowledge is integrated into the student models. I think the knowledge may be the common knowledge from the multiple teacher models. The authors may present more analysis.

A1: We follow previous work [1] in decomposing knowledge into high- and low- frequency components, which are measured by cosine similarity and KL-divergence, respectively (see Appendix D of [1] for details on how to measure high/low frequency knowledge). We provide a comparison of high- and low-frequency knowledge for the five teacher and student models in Figure A2 and Lines 711-721 of Appendix A.16, from which it can be observed that (1) low-frequency knowledge (a.k.a., common knowledge) from multiple teachers can be well learned by the student, and (2) the student model differently learn high-frequency knowledge from each teacher.

[1] Wu L, Lin H, et al. Extracting Low-/High-Frequency Knowledge from Graph Neural Networks and Injecting it into MLPs: An Effective GNN-to-MLP Distillation Framework[J]. AAAI, 2023.

---

Q2: I find the paper not easy to follow. The symbols maybe complex for me to understand. As a result, I suggest the authors to simplify and well structure the used symbols.

A2: Thanks for your constructive suggestions. We have simplified and reorganized the symbols used in this paper and provided a list of symbols in Table A10 of Appendix A.17. We hope this helps you understand and appreciate our contributions. If anything is still unclear, please do not hesitate to contact us.

---

In light of these responses, we hope we have addressed your concerns. If we have left any notable points of concern unaddressed, please do share and we will attend to these points. We sincerely hope that you can appreciate our efforts on responses and revisions and thus raise your score.

Dear Reviewer,

Thank you for your insightful and helpful comments once again. We greatly appreciate your feedback. We have carefully responded to each of your questions point-by-point.

Unlike previous years, there will be no second stage of author-reviewer discussions this year. As the deadline for the discussion is approaching, we kindly request you to inform us if you have any additional questions.

Best regards,

Authors.

Thanks for your insightful comments and constructive suggestions! We mark the contents that already existed (but were missed by reviewers) in red, and those revised or newly added contents in blue in the revised manuscript. Next, we will address the concerns you raised, one by one, as described below:

---

Q1: Experiments on other downstream tasks (e.g. node clustering, link prediction, and partition prediction as in AutoSSL and ParetoGNN) are missing.

A1: We have added performance comparisons of MGSSL, AutoSSL, and ParetoGNN on various graph-related downstream tasks (graph regression, node clustering, link prediction) in Tables A2/A3/A4 and Lines 642-654 in Appendix A.10. These experiments demonstrate the applicability of MGSSL to various downstream tasks and its superiority over AutoSSL and ParetoGNN.

---

Q2: It seems better to add the discussion and comparison on related works about graph knowledge distillation and recent self-supervised graph learning.

A2: Thanks for your valuable suggestion. We have added the discussion on graph knowledge distillation and recent graph self-supervised learning in Lines 591-602 and Lines 626-640 of Appendix A.9, and the six papers you mentioned are all included, cited, and discussed. Moreover, we have included GraphMAE as an additional baseline in Table 2.

---

Q3: Typo: In the line2 of page6, “…this paper proposes two…” is miswritten as “…this paper propose stwo…”.

A3: All typos have been corrected in the revised manuscript.

---

In light of these responses, we hope we have addressed your concerns. If we have left any notable points of concern unaddressed, please do share and we will attend to these points. We sincerely hope that you can appreciate our efforts on responses and revisions and thus raise your score.

Dear Reviewer,

Thank you for your insightful and helpful comments once again. We greatly appreciate your feedback. We have carefully responded to each of your questions point-by-point.

Unlike previous years, there will be no second stage of author-reviewer discussions this year. As the deadline for the discussion is approaching, we kindly request you to inform us if you have any additional questions.

Best regards,

Authors.

Thanks for your insightful comments and constructive suggestions! We mark the contents that already existed (but were missed by reviewers) in red, and those revised or newly added contents in blue in the revised manuscript. Next, we will address the concerns you raised, one by one, as described below:

---

Q1: An evaluation of the scalability and computational costs of the proposed framework, especially in large-scale settings, would be a valuable addition.

A1: The discussion and analysis on the computational complexity and overhead of MGSSL has been provided in Lines 694-709, Appendix A.15 of the original manuscript. Did you miss this part of the experiment? Following your suggestion, we have added the computational cost of MGSSL on a larger ogbn-products dataset (consisting of 2,449,029 nodes and 61,859,140 edges) to demonstrate its excellent scalability in Table A9 in the revised manuscript.

---

Q2: .The author only compares their work with AutoSSL in Table 1. Since AutoSSL is a work from 2021, I suggest that the author should compare their work with more recent studies. This would further enhance the quality of the paper.

A2: One latest work, ParetoGNN, published in ICLR'2023, has been included in the comparison in Table 1. Additionally, we provide more comparisons of MGSSL with AutoSSL and ParetoGNN on various graph-related tasks in Tables A2/A3/A4 and Lines 642-654 in Appendix A.10.

---

In light of these responses, we hope we have addressed your concerns. If we have left any notable points of concern unaddressed, please do share and we will attend to these points. We sincerely hope that you can appreciate our efforts on responses and revisions and thus raise your score.

Dear Reviewer,

Thank you for your insightful and helpful comments once again. We greatly appreciate your feedback. We have carefully responded to each of your questions point-by-point.

Unlike previous years, there will be no second stage of author-reviewer discussions this year. As the deadline for the discussion is approaching, we kindly request you to inform us if you have any additional questions.

Best regards,

Authors.