GraphECL: Towards Efficient Contrastive Learning for Graphs

Dear reviewer GMYZ, we appreciate your positive feedback on our paper regarding its reproducibility, good accuracy, fast inference time, and theoretical analysis. Please find our detailed responses below:

Q1. The term 'trade-off' makes me confused.

A1. Thank you for your comments. Although we study a novel problem of achieving satisfactory efficiency and accuracy trade-offs in GCL, our GraphECL excels in both efficiency and accuracy. We apologize for any confusion and have changed 'trade-off' to 'simultaneously.'

Q2. I feel that Equation (4) is redundant in the paper.

A2. Thank you for your suggestion. We will remove it from the main paper.

Q3. In Table 1, the result of SUGRL (30.31) on Actor is not marked, which is also the second-best value, and the order of Table 3 and Table 4 is reversed

A3. Thanks for pointing out the typos! We have corrected it.

Q4. There are concerns regarding the assumptions relied upon by GraphECL.

A4. Thanks for your comments. We believe there are some important misunderstandings. Our GraphECL is not based on the one-hop homophily assumption. As illustrated in Figure 1(d) and expressed in Equation (4) or Equation (5), GraphECL aims to push cross-modal neighboring node representations closed, i.e., $\left(f_M\left(v\right), f_G(u)\right)$. Thus, it's crucial to emphasize that GraphECL doesn't necessarily imply the learned MLP representations $\left(f_M\left(v\right), f_M(u)\right)$ become identical.

Consider a pair of 2-hop neighbors, $v$ and $v'$, both neighboring the same node $u$. Intuitively, by enforcing $f_{M}(v)$ and $f_{M}(v')$ to reconstruct (not align) the context representation of the same neighborhood $f_{G}(u)$, we implicitly make their representations similar. Thus, the 2-hop neighbors ($f_{M}(v)$ and $f_{M}(v')$) with the same neighborhood context serve as positive pairs that will be implicitly aligned. This alignment is supported by our Theorem 4. Additionally, as depicted in Figure 9, GraphECL is not based on the one-hop homophily assumption but automatically captures graph structures based on different graphs beyond homophily.

Q5. Lacking some baselines.

A5. Thanks for bringing up related works, CGD [1], SimGCL [2], and FastGCL [3]. Following your suggestions, we have compared GraphECL with [1, 2, 3]. Using the same data splits as outlined in our paper, we present the results in Table 8 in the updated submission (we also provide the results on some datasets in the following table). From the results, it's evident that GraphECL outperforms [1, 2, 3], particularly on heterophilic graphs. These additional results, combined with the comparison against 10 baselines using our published code, strongly validate the effectiveness of GraphECL.

We gratefully appreciate your time in reviewing our paper and your insightful comments. We sincerely hope our rebuttal has carefully addressed your comments point-by-point. The baselines [1, 2, 3] have less technical overlap in design with our approach and do not impact our major contributions. Moreover, these baselines can be easily added in the camera-ready version, and we believe that the reviewer's minor comments can be easily addressed in the camera-ready submission. We sincerely hope that the reviewer can consider increasing the score. Thank you!

[1] Zheng Y, et al. Rethinking and scaling up graph contrastive learning: An extremely efficient approach with group discrimination. NeurIPS 2022.

[2] Yu J, et al. Are graph augmentations necessary? simple graph contrastive learning for recommendation. SIGIR 2022.

[3] Wang Y, et al. Fastgcl: Fast self-supervised learning on graphs via contrastive neighborhood aggregation. arXiv.

Dear Reviewer GMYZ,

We gratefully appreciate your time in reviewing our paper and your insightful comments.

The reviewer's comments are not fatal to the major contributions of our manuscript, we believe that the reviewer's great comments can be easily and effectively addressed in the camera-ready.

We made our greatest efforts to address your concerns in the rebuttal. We would appreciate it if you could consider increasing your score. Thank you very much once again, and we are happy to follow up with any additional questions you may have!

Dear ICLR Reviewer GMYZ,

We gratefully appreciate your time in reviewing our paper. Since the discussion period ends soon, we sincerely hope our rebuttal has carefully addressed your comments point-by-point and hope you consider increasing your score. In particular, we clarified some misunderstandings and compared our GraphECL with additional baselines, as suggested by you. If you have any other comments or questions, we will be glad to answer them.

The Authors of "GraphECL: Towards Efficient Contrastive Learning for Graphs"

Dear Reviewer eihc,

Thank you for your time. The review is about Quantum Computing and is definitely not related to the content of our submission. Thus, we believe that the reviewer accidentally submitted the wrong review, which appears to be generated by ChatGPT.

Could you please submit your correct review? Thank you very much!

The Authors

Dear Reviewer eihc,

I hope this message finds you well. We have carefully reviewed your comments, and there appears to be a potential discrepancy. Your comments reference a paper titled "GrapheCL: Towards Efficient and Expressive Device-Independent Quantum Computing," while the current discussion pertains to a different submission.

To ensure accuracy and relevance in the rebuttal phase, could you kindly verify that your comments are correctly aligned with the manuscript under consideration? If there has been an inadvertent error in submission or if the comments were intended for another manuscript, please provide clarification. It is important for us to accurately attribute and address the comments during the review process.

Your cooperation in resolving this matter is highly appreciated. If necessary, please resubmit the correct comments for the appropriate manuscript.

Bests, AC

Thank you for your time and feedback. We apologize for any confusion and would like to clarify the term 'scalability' in our work and respond to your first two comments.

The message passing of the GNN encoder involves fetching topology and features from numerous neighboring nodes to perform inference on a target node, making it time-consuming and computation-intensive.

In our work, we specifically focus on reducing the inference time for inferring representations with a graph-aware MLP [3, 4, 5, 6], rather than aiming to reduce training time or memory consumption, as in [1, 2]. The term “scalability” in our paper refers to inference scalability, aligning with previous works [3, 4, 5, 6]. While we exclude the reduction of training time or memory [5, 6] from the scope of this work, we acknowledge it as a compelling direction that merits exploration in future research.

Thus, our studied problem is entirely different from [1, 2]. [1, 2] are great works which focus on reducing the training time and training memory, but they still need to utilize GNN to conduct inference, which is time-consuming and computation-intensive.

Nevertheless, we completely agree with the reviewer that also including an empirical comparison with CGD [1] would be beneficial. Following your suggestions, we have compared GraphECL with CGD [1]. Using the same data splits as outlined in our paper, we present the results in Table 8 in the updated submission (we also provide the results on some datasets in the following table). From the results, The results show that GraphECL outperforms CGD, particularly on heterophilic graphs.

[1] Rethinking and Scaling Up Graph Contrastive Learning: An Extremely Efficient Approach with Group Discrimination. NeurIPS 2022

[2] Large-Scale Representation Learning on Graphs via Bootstrapping. ICLR 2021

[3] Graph-less Neural Networks: Teaching Old MLPs New Tricks Via Distillation. ICLR 2021

[4] Learning MLPs on Graphs: A Unified View of Effectiveness, Robustness, and Efficiency. ICLR 2023

[5] Quantifying the Knowledge in GNNs for Reliable Distillation into MLPs. ICML 2023

[6] VQGraph: Graph Vector-Quantization for Bridging GNNs and MLPs. Arxiv.

We greatly appreciate the reviewer's insightful suggestion, which we believe will improve our work. We kindly want to remind the reviewer that we have conducted an extensive evaluation on 11 datasets Thus, we believe our experiments can strongly corroborate the effectiveness and scalability of GraphECL. Nevertheless, we completely agree with the reviewer that including an empirical comparison on more large datasets can further improve our paper.

Following your suggestions, we have compared GraphECL with baselines on your suggested datasets. We present the results in Table 9 in the updated submission (we also provide the results in the following table). The results show that our simple GraphECL can still achieve better (or competitive) performance on suggested datasets compared to elaborate methods, especially on large graphs.

We also totally agree with the reviewer's comments: "On the small graph, I think the training time is more heavily weighted compared to the inference time, while the authors only focus on the inference time. Conversely, in the large graph inference time is much more heavily weighted and the author rarely mentions it." We will explicitly state and mention this in the revision.

We thank you for your insightful questions, and we would like to address your questions in what follows.

Q1. Why did you not do scalability experiments similar to those in articles [1] and [2].

A1. Thank you for your questions. As we mentioned in Response 1/3, in our work, we specifically focus on reducing the inference time for inferring representations with a graph-aware MLP [3, 4, 5, 6] rather than aiming to reduce training time or memory consumption, as in [1, 2]. The term "scalability" in our paper refers to inference scalability, aligning with previous works [3, 4, 5, 6]. While we exclude the reduction of training time or memory [5, 6] from the scope of this work, as [3,4,5,6] also did, we acknowledge it as another compelling direction that merits exploration in future research.

Q2. GraphECL compared to Graph-MLP, the positive pairs are obtained as first-order neighbors instead of nth-order. Combining this with Eq. 4 improves the performance of heterophilic graphs. I didn't understand the principle illustrated at the bottom of Eq. 4, can you explain it in detail?

A2. Thank you for your question. As illustrated in Equation (4), GraphECL aims to bring cross-modal neighboring node representations closer together, i.e., $\left(f_M\left(v\right), f_G(u)\right)$. Thus, it's crucial to emphasize that GraphECL doesn't necessarily imply that the learned MLP representations $\left(f_M\left(v\right), f_M(u)\right)$ become identical, as Graph-MLP did.

Consider a pair of 2-hop neighbors, $v$ and $v'$, both neighboring the same node $u$. Intuitively, by enforcing $f_{M}(v)$ and $f_{M}(v')$ to reconstruct (not align) the context representation of the same neighborhood $f_{G}(u)$, we implicitly make their representations similar. Thus, the 2-hop neighbors ($f_{M}(v)$ and $f_{M}(v')$) with the same neighborhood context serve as positive pairs that will be implicitly aligned (not the one-hop neighbors). This is also supported by our Theorem 4. Additionally, as depicted in Figure 9, GraphECL is not based on the one-hop homophily assumption but automatically captures graph structures based on different graphs beyond homophily. We also highlight the above clarification (marked with blue text) in the updated submission.

Q3. Is it possible to compare the more recent Contrastive Learning methods in 2023 years and not only limited to within 2022 years.

A3. Thank you for your suggestions. We kindly want to remind the reviewer that we have conducted an extensive evaluation of 10 methods, including many state-of-the-art and solid graph contrastive learning methods. Thus, we believe our experiments can strongly corroborate the effectiveness of GraphECL. Nevertheless, we completely agree with the reviewer that including an empirical comparison with more baselines in 2023 would be better. Given the above, we further compare our GraphECL with MA-GCL [7], which provides the source code. From the following table, we can observe that our efficient GraphECL can still achieve better (or competitive) performance compared to MA-GCL proposed in 2023.

[1] Rethinking and Scaling Up Graph Contrastive Learning: An Extremely Efficient Approach with Group Discrimination. NeurIPS 2022

[2] Large-Scale Representation Learning on Graphs via Bootstrapping. ICLR 2021

[3] Graph-less Neural Networks: Teaching Old MLPs New Tricks Via Distillation. ICLR 2021

[4] Learning MLPs on Graphs: A Unified View of Effectiveness, Robustness, and Efficiency. ICLR 2023

[5] Quantifying the Knowledge in GNNs for Reliable Distillation into MLPs. ICML 2023

[6] VQGraph: Graph Vector-Quantization for Bridging GNNs and MLPs. Arxiv.

[7] MA-GCL: Model Augmentation Tricks for Graph Contrastive Learning. AAAI 2023

Dear Reviewer ZTuD,

We gratefully appreciate your time in reviewing our paper and your insightful comments.

We have provided additional results in response to your latest comments. We believe this evidence successfully addresses all your concerns.

With only four days left in the discussion period, we would like to confirm whether the reviewer has seen our latest comment and if there are any final clarifications they would like. If the reviewer's concerns are clarified, we would be grateful if the reviewer could update their review and score to reflect that. This way, we will know that our response has been seen. Once again, many thanks for your time and dedication to the review process; we are extremely grateful.

The Authors of "GraphECL: Towards Efficient Contrastive Learning for Graphs"

Dear reviewer ezu8, we appreciate your positive feedback on our paper's soundness, novel insights, and contribution. Please find our detailed responses below:

Q1. The method is mainly evaluated on node property prediction tasks. However, additional evaluation on graph property prediction will be essential.

A1. Thank you for your question and suggestion. For graph classification, we can use a non-parameterized graph pooling (readout) function, such as MeanPooling, to obtain the graph-level representation. In our experiments, we focus on graph classification using three benchmarks: PROTEINS and MUTAG. We follow the same experimental setup as GraphCL [1]. The results are presented in the following table. From the table, we observe that our simple GraphECL performs well on the graph classification task and achieves better performance compared to the baselines.

Q2. The authors have not provided an analysis of how the number of hidden layers impacts the method's performance.

A2. Thank you for your valuable comment! In the following table, we analyze the effect of hidden layers. From the table, it is evident that our GraphECL is not very sensitive to hidden layers. However, having more layers generally leads to better performance for both homophilic and heterophilic graphs.

Q3. The proposed method's effectiveness in capturing inter-neighborhood information will be further highlighted by the evolution of LRGB datasets.

A3. Following your suggestions, we compare GraphECL with two graph contrastive learning methods, BGRL [2] and CCA-SSG [3], on PascalVOC-SP [4]. For all methods, we employ GCN as the backbone. From the table below, we can observe that GraphECL performs well on PascalVOC-SP, achieving better performance compared to the baselines. This further strengthens our contribution.

In light of these responses, we sincerely hope our rebuttal has addressed your comments, and believe that your comments do not affect our key contributions and can be easily addressed in the revision. We also genuinely hope you will reconsider increasing your score. If you have any other comments, please do share them with us, and we will address them further. Thank you for your efforts!

[1] Graph Contrastive Learning with Augmentations. NeurIPS 2020

[2] Large-Scale Representation Learning on Graphs via Bootstrapping. ICLR 2022

[3] From Canonical Correlation Analysis to Self-supervised Graph Neural Networks. NeurIPS 2021 [4] Long Range Graph Benchmark. NeurIPS 2022

Thanks for your time and feedback! We would like to clarify some misunderstandings regarding our approach.

Q1. In Lim et al, MLP achieves 31%, GCN 45%, both being higher than the reported numbers (GraphECL 27%)

A1. Thank you for pointing out (Lim et al., 2021). However, it's essential to note that their evaluation setting differs from ours. Our paper focues on evaluting GraphECL in the unsupervised setting, whereas (Lim et al., 2021) employed the fully supervised setting, leading to reported higher performance.

Q2. many of the chosen datasets are too small or the variance is too high across splits

A2. Thank you for your suggestions! We'd like to remind the reviewer that we have tested GraphECL on 11 widely-used datasets, as detailed in Table 6, including larger datasets such as Snap-patents and Flickr, in comparison to (Cora, Citeseer, Pubmed, Cornell, Texas, etc.). While we acknowledge the reviewer's point about the importance of evaluating on more large datasets, following your suggestions, we have compared GraphECL with baselines on more datasets. We present the results in Table 9 in the updated submission (we also provide the results in the following table). The results show that GraphECL can still achieve better (or competitive) performance on suggested datasets compared to elaborate methods, especially on large graphs.

Q3. it might be a good idea to include the simple baselines of an MLP and a GCN.

A3. Thank you for your suggestions! We believe there might be some misunderstandings. Our primary focus is on designing a self-supervised algorithm without labels, rather than specific GNN or network architectures. Consequently, an empirical comparison with an MLP and a GCN is relatively less straightforward. Instead, we have compared our self-supervised learning algorithm with various others based on MLP (Graph-MLP, SUGRL) and GNN (BGRL, DGI, GCA, SUGRL, AFGRL, etc.).

Q4. How would the complementary negative pairs influence learning?

A4. Thanks for your insightful questions! We introduce an explicit negative pair to enhance representation diversity. As demonstrated in our Theorem 1, the inter-model negative pair $\left(f_M\left(v^{-}\right), f_G(u)\right)$ is crucial for capturing the 1-hop neighborhood distribution. Moreover, complementary negative pairs are essential for GraphECL. Our ablation studies indicate that GraphECL without both negative pairs results in a performance drop. Therefore, the addition of complementary negative pairs is crucial for improving representation diversity and increasing inter-class variation, ultimately leading to robust generalization. We will explicitly state this in the main text.

Dear Reviewer xAqE,

We gratefully appreciate your time in reviewing our paper and your insightful comments.

We made our greatest efforts to address your concerns in the rebuttal. We would appreciate it if you could consider increasing your score. Thank you very much once again, and we are happy to follow up with any additional questions you may have!

Dear ICLR Reviewer xAqE,

We gratefully appreciate your time in reviewing our paper. Since the discussion period ends soon, we sincerely hope our rebuttal has carefully addressed your comments point-by-point and hope you consider increasing your score. In particular, we clarified some misunderstandings and compared our GraphECL on additional datasets, as suggested by you. If you have any other comments or questions, we will be glad to answer them.

The Authors of "GraphECL: Towards Efficient Contrastive Learning for Graphs"