Decoupling Weighing and Selecting for Integrating Multiple Graph Pre-training Tasks

W3.&Q2. Besides the empirical results, the theoretical analysis is still needed to answer why both selecting and weighing is needed, and why there is compatibility issue (i.e., why some tasks shouldn't be selected). Any in-depth understanding of the relationship for the selected tasks? Are they independent?

A3.

We sincerely appreciate your suggestions for enhancing the clarity of our manuscript, which will make it stronger! We have provided relevant analysis in Fig.1 of the Introduction, where Fig.1b illustrates the importance of a reasonable weighting scheme, and Fig.1c demonstrates the compatibility issues between different tasks. You may want us to include more detailed explanations regarding these issues.

We will address your concerns in three steps: 1. Are different tasks relatively independent? 2. Why do compatibility issues arise? 3. Theoretically speaking, what are the advantages of incorporating a selection mechanism as opposed to simply using a weighting approach? For the figures and proofs mentioned in the following content, please refer to the Appendix G.

• [Are tasks independent? Different teachers acquired different knowledge during pre-training, and this knowledge remains different after migration to downstream tasks] In Fig. A2, we analyze the knowledge acquired by different teachers. Different pre-training methods, e.g., GraphCL performs contrastive learning between instances, while AM masks node attributes on a single instance, are able to acquire different knowledge. Fig. A2 visualizes the representations obtained by different teachers after training on the same pre-training dataset by T-sne. It can be seen that they are distributed in different spaces, which means that they do not acquire the same knowledge. We can conclude that the knowledge they acquire is essentially independent.
• [Why do compatibility issues arise? Different instances may require different knowledge.] Then, for downstream tasks, what differences do different teachers' acquired knowledge make? Fig. A3 visualizes the graphs from the test set of BACE, where blue dots represent molecular graphs that can be correctly predicted for all seven teachers, while red dots represent graphs that can only be correctly predicted by some of the seven teachers. The red dots make up the majority, which means that different teachers will be able to predict different graphs. Therefore, for a given instance, if a teacher whose knowledge is not suitable for them is chosen, this may cause conflicts and could be worse than not having that teacher at all. This is the compatibility issue.
• [What are the advantages of incorporating a selecting mechanism? Having only the weighting module can lead to an inability to improve performance.] In the Appendix G, we provide a theoretical analysis of the performance without the selecting module (only the weighting module). Please refer to the Appendix G. Through this analysis, we can conclude that the selection module is a crucial factor for the model to achieve continuous performance improvement from an ever-expanding task pool.

More detailed content (figures and theoretical analysis) has been added to the Appendix G. Please refer to Appendix G for further information.

In light of these responses, we hope we have addressed your concerns, and hope you will consider increasing your score. If we have left any notable points of concern unaddressed, please share and we will attend to these points.

Dear Reviewer 66Ho:

Thank you for your valuable feedback and constructive suggestions. We sincerely appreciate your acknowledgment of effectiveness and writing quality. In the revised manuscript, we have highlighted existing content (which may have been omitted, but which we would like to bring to your attention) in red and revised or new content in blue. Our detailed response to your concerns is listed as follows:

W1.&Q1. The motivation of focusing on graph-structured data is unclear. The proposed method does not treat graph-structured data specially, nor is it tailored for graph modeling. Why is it positioned for pre-training on graphs, instead of for general pre-training?

A1.

The reason for your confusion may be because we have not added too many graph-specific priors to our method. This is due to the diversity of downstream tasks in the graph domain, where different downstream tasks contain different instance levels, e.g., graph classification and node classification. This leads to the fact that imposing too many restrictions on a certain method can make it difficult to handle diverse downstream tasks, since the tasks employed in the method already provide domain-specific a priori (e.g., node masking, edge prediction, etc.). For example, the previous work AutoSSL[1], used graph homophily as a priori knowledge, which caused it to only be able to handle node-level tasks and not graph-level tasks. We note that this series of subsequent work (ParetoGNN[2], AGSSL[3]) follows the idea of not imposing a priori on methods. Our starting point is common, i.e., imposing too many a priori on combinatorial methods can limit the scenarios in which the methods can be used.

As with previous tasks, the reason we are a framework positioned on graph-structured data is that the phenomena we observe (sharp performance degradation due to conflicts among graph pre-training tasks, etc.) are graph-based. In addition, our method does show potential for extension to other domains, but we do not believe that this should be one of the drawbacks, and we will consider extensions of our method as meaningful follow-on work.

[1]Wei Jin, Xiaorui Liu, Xiangyu Zhao, Yao Ma, Neil Shah and Jiliang Tang, Automated Self-Supervised Learning for Graphs, ICLR2022

[2] Mingxuan Ju, Tong Zhao, Qianlong Wen, Wenhao Yu, Neil Shah, Yanfang Ye and Chuxu Zhang, Multi-task Self-supervised Graph Neural Networks Enable Stronger Task Generalization, ICLR 2023

[3] Lirong Wu, Yufei Huang, Haitao Lin, Zicheng Liu, Tianyu Fan and Stan Z. Li. Automated graph self-supervised learning via multi-teacher knowledge distillation. arxiv2210.02099

W2. Compared to the baseline methods for multi-task learning on instance-level, the improvement from WAS is marginal. On the largest graphs in the experiment (ogbn-arxiv), it achieves worse performance compared to baseline method.

A2.

We kindly point out that "the improvement from WAS is marginal" might be somewhat inappropriate.

Firstly, the major contribution of our method lies in its ability to continually achieve performance gains from an expanding task pool, a capability not possessed by previous methods. Besides this primary contribution, our method outperforms previous multi-task learning methods in terms of performance on seven node-level datasets, even when the task pool is kept the same. Additionally, our method is capable of handling graph-level tasks.

Dear Reviewer 66Ho,

We sincerely thank you for taking the time to review our manuscript and providing valuable suggestions.

Unlike previous years, there will be no second stage of author-reviewer discussions this year, and a recommendation needs to be provided by November 22, 2023. Considering that the author-reviewer discussion phase is nearing the end, we would like to be able to confirm whether our responses have addressed your concerns.

We have provided detailed replies to your concerns a few days ago, and we hope we have satisfactorily addressed your concerns. If so, could you please consider increasing your rating? If you still need any clarification or have any other concerns, please feel free to contact us and we are happy to continue communicating with you.

Best,

Authors

Having thoroughly reviewed the responses and comments from other reviewers, I've decided to keep my original score.

Dear Reviewer pHBL:

Thank you for your valuable feedback and constructive suggestions. We sincerely appreciate your acknowledgment of effectiveness and writing quality. In the revised manuscript, we have highlighted existing content (which may have been omitted, but which we would like to bring to your attention) in red and revised or new content in blue.

Our detailed response to your concerns is listed as follows:

W1. In Table 1, 2, and 3, the authors do not compare the proposed WAS framework with existing frameworks that only focus on weighing (i.e., AutoSSL, ParetoGNN, AUX-TS, and AGSSL). However, in Table 4, WAS is compared with these methods. This makes the experimental settings not consistent.

A1.

We kindly point out that, in Table 1, we have compared these works.

In Table 2 and 3, we did not compare these works because these methods are primarily focused on node-level tasks, rather than graph-level tasks. Methods like AutoSSL struggle with the graph-level tasks that we addressed in Table 2 and 3, making a comparison unfeasible. Since Table 4 concerns node-level tasks, we were able to compare our method with these approaches.

W2. The complexity of the proposed has not been studied. Although the proposed method can achieve some performance improvement, it may take more training/inference time, due to combining multiple pre-training tasks. The authors need to have some analysis about the complexity of the proposed method.

A2.

We have provided an analysis of additional parameters, extra storage space, additional time consumption, and extra time complexity in Appendix C (complexity analysis), and mentioned this at the end of Sec. 4.2.3 (Line 267).

W3.&Q1. The authors also need to perform experiments to analyze how many tasks are selected for each instance on average. On average, how many tasks are usually selected for each instance?

A3.

We sincerely appreciate your suggestions for enhancing the clarity of our manuscript, which will make it stronger! In the following Table, we count the average number of selected teachers number, and then count the ratio of selecting Top-1/Top-[number] important teacher.

From the table, we can draw the following conclusions:

1. teachers with the highest weights have a higher probability of being selected
2. the selection module does not necessarily always select the higher weight teachers

This table is added to the updated Appendix F.

Q2. From Figure 6(d), we can observe that the following 4 tasks have the largest weights, i.e, EP, AM, IG, and CP. How about we only use these 4 tasks and learning the their weights for prediction?

A4.

Figure 6(d) represents only the result of one instance. The tasks with higher weights in this instance may not necessarily have high weights in other instances.

Using only these four tasks could potentially lead to a decline in performance. In the table below, "7-tasks" indicates that the task pool contains all 7 tasks, while "4-tasks" means the task pool includes only four tasks (EP, AM, IG, and CP).

With the reduction of tasks, there was a decrease in performance, but it still remained better than using a single task.

In light of these responses, we hope we have addressed your concerns, and hope you will consider increasing your score. If we have left any notable points of concern unaddressed, please share and we will attend to these points.

Dear Reviewer pHBL,

We sincerely thank you for taking the time to review our manuscript and providing valuable suggestions.

Unlike previous years, there will be no second stage of author-reviewer discussions this year, and a recommendation needs to be provided by November 22, 2023. Considering that the author-reviewer discussion phase is nearing the end, we would like to be able to confirm whether our responses have addressed your concerns.

We have provided detailed replies to your concerns a few days ago, and we hope we have satisfactorily addressed your concerns. If so, could you please consider increasing your rating? If you still need any clarification or have any other concerns, please feel free to contact us and we are happy to continue communicating with you.

Best,

Authors

W3. The compatibility issue, or interferences among tasks have been observed in previous work in other areas or problem settings. Some discussion on this aspect, and its particular challenges in graph context, would further strengthen the motivation of the paper.

A3.

We sincerely appreciate your suggestions for enhancing the motivation of our manuscript, which will make it stronger! Among the works you mentioned[a][b][c], [a] is specifically designed for transformers. However, the dominant architecture in the graph domain is still GNN, which limits its applicability.

[b] differs in focus from our method. We focus on extracting and combining knowledge from various pre-trained tasks to enhance the performance of downstream tasks, while [b] is concerned with handling different downstream tasks when using a single model. [c] combines tasks through gradient surgery, a method similar to AutoSSL and ParetoGNN mentioned in our paper. These approaches combine task losses through various weighting methods. Firstly, they are unable to provide a customized combination for each instance. Secondly, they overlook compatibility issues, making it difficult to achieve continual performance improvement as the task pool expands. Moreover, these works do not address the concept of "compatibility". Their solutions differ from ours as their main focus remains on various weighting methods, overlooking the necessity of selection.

We have updated Sec.2 to include these more discussions.

[a] Zeyuan Wang, Qiang Zhang, HU Shuang-Wei, Haoran Yu, Xurui Jin, Zhichen Gong, and Huajun Chen. 2022. Multi-level Protein Structure Pre-training via Prompt Learning. In The Eleventh International Conference on Learning Representations.

[b] Sen Wu, Hongyang R Zhang, and Christopher Ré. 2020. Understanding and improving information transfer in multi-task learning. arXiv preprint arXiv:2005.00944 (2020).

[c] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. 2020. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems 33 (2020), 5824–5836.

In light of these responses, we hope we have addressed your concerns, and hope you will consider increasing your score. If we have left any notable points of concern unaddressed, please share and we will attend to these points.

Dear Reviewer C6hu:

Thank you for your valuable feedback and constructive suggestions. We sincerely appreciate your acknowledgment of compelling motivation, thorough analysis and well-organized manuscript. In the revised manuscript, we have highlighted existing content (which may have been omitted, but which we would like to bring to your attention) in red and revised or new content in blue.

Our detailed response to your concerns is listed as follows:

W1. The concept of instance is not well defined. Different pre-training tasks may have different definitions of instances fundamentally (eg. at a node-level, subgraph/graph-level, edge-level etc). Having a common and converged definition may not work well. Furthermore, downstream task could also have different definitions of instances.

A1.

We kindly point out that your concerns are the same as those mentioned in our manuscript.

" The concept of instance is not well defined. "

We provided an example of instances on page 1,Line 36-37 ("e.g., nodes in a social network or graphs in a molecular dataset").

" Different pre-training tasks may have different definitions of instances fundamentally "

We completely agree with your statement and we discussed this in Line 191-193 (Original text: "For example, GraphCL performs contrastive learning between instances, while AM masks node attributes on a single instance"). For a graph-level task, GraphCL defines an instance as a graph, whereas for AM, the instance is still a node.

It might be that our expression was not clear enough. We will modify it to "For example, GraphCL performs contrastive learning between graphs, while AM masks node attributes on a single graph".

"downstream task could also have different definitions of instances."

We also fully agree with this. Therefore, we conducted experiments at both the node-level and graph-level to demonstrate the effectiveness of our method.

W2. While 4 questions are clear in the identification of the 4 steps/issues to address, the solutions are quite standard. E.g. using Gumbel-Softmax sampling for Bernoulli distribution, the weighting scheme and the Siamese network architecture are all well known tools.

A2.

We sincerely thank you for appreciating our work "4 questions are clear in the identification of the 4 steps/issues to address".

The Gumbel-Softmax, weighing scheme, and the Siamese architecture are seen as a tool rather than a contribution.

Our contributions are:

1. examining the shortcomings of various existing multi-task graph learning methods (they mainly based on weighing-only method), i.e., not considering the compatibility issue.
2. distinguishing between the compatibility issue and the importance issue, and using decoupling frameworks to deal with these two issues.

Considering that we are already achieving what the framework was designed for (1. good performance 2. the ability to sustain performance growth) by using those classic components, we kindly point out that we think that there is no need to force modifications to the classic components just to appear that we are different.

Dear Reviewer C6hu,

We sincerely thank you for taking the time to review our manuscript and providing valuable suggestions.

Unlike previous years, there will be no second stage of author-reviewer discussions this year, and a recommendation needs to be provided by November 22, 2023. We hope we have satisfactorily addressed your concerns. If so, could you please consider increasing your rating? If you still need any clarification or have any other concerns, please feel free to contact us and we are happy to continue communicating with you.

Best,

Authors

Dear Reviewer fcHp:

Thank you for your valuable feedback and constructive suggestions. We sincerely appreciate your acknowledgment of the effectiveness of our work. In the revised manuscript, we have highlighted existing content (which may have been omitted, but which we would like to bring to your attention) in red and revised or new content in blue. Our detailed response to your concerns is listed as follows:

W1. The authors should add more ablation experiments to demonstrate the effectiveness of the proposed model, such as the role of different updating methods in decoupling the selecting and weighting outputs, and the role of reweighting after selection.

A1.

You mentioned "the role of different updating methods", and we speculate that you hope to further understand the relationship between the momentum-update method and the decoupling of the selecting and weighting modules. It's about our design intuition, as we want the result of selecting to utilize the information from the importance, not the other way around. Considering that a traditional ablation (removing the momentum update method) would render our method completely ineffective, we use a comparative approach. In the table below, Method A represents gradient-based updating for weighing and momentum-based updating for selecting (i.e., same as WAS); Method B represents gradient-based updating for selecting and momentum-based updating for weighing. It can be seen that Method B performs much worse than Method A (WAS). This indicates that our update method is more effective and aligns better with the design concept.

Regarding "the role of reweighting after selection," this is done to standardize the magnitude of knowledge distilled to the student model. For instance, if we select teachers with weights of 0.1 and 0.2 from three teachers weighted as 0.1, 0.2, and 0.7, but do not perform reweighting, the final sum of the weights will not equal 1. This would result in the number of selected teachers significantly impacting performance. As can be seen from the experiment, eliminating this would make the results very poor, as the teacher would be completely unable to guide the students.

This additional result has already been added to the Table.5. Thank you again for your suggestion!

W2. The comparison model in A5 needs more description to distinguish it from A1, such as whether Random-Select and ALL use learned weights.

A2.

We sincerely appreciate your suggestions for enhancing the readability and clarity of our manuscript, which will make it stronger!

Both Random-Select and ALL use learned weights, but Random-Select randomly select teachers, and ALL select all the teachers. We add corresponding explanations in the updated manuscript to better clarify this point.

In light of these responses, we hope we have addressed your concerns. If we have left any notable points of concern unaddressed, please share and we will attend to these points.

Dear Reviewer fcHp,

We sincerely thank you for taking the time to review our manuscript and providing valuable suggestions.

Unlike previous years, there will be no second stage of author-reviewer discussions this year, and a recommendation needs to be provided by November 22, 2023. Considering that the author-reviewer discussion phase is nearing the end, we would like to be able to confirm whether our responses have addressed your concerns.

We have provided detailed replies to your concerns a few days ago, and we hope we have satisfactorily addressed your concerns. If you still need any clarification or have any other concerns, please feel free to contact us and we are happy to continue communicating with you.

Best,

Authors

Dear Reviewers,

We have updated the revised version of our manuscript. The new version includes more experiments, analyses, theories, and explanations to address your concerns. If you have any concerns that have not been resolved, please let us know and we are happy to continue communicating with you.

Best,

Authors