Effective Graph Representation Learning via Smoothed Contrastive Learning

We would like to thank the reviewer for his/her time and effort spent on evaluating our work and their constructive comments. We find the suggestions to be very helpful in improving the quality of our work, making it clearer and more convincing. We are pleased to hear that the reviewer found our idea novel and interesting and is convinced that the paper is well-motivated.

The writing of the paper should be improved

We appreciate your insights and acknowledge the need for improvement in the writing of the paper. We will carefully review and address the minor mistakes to enhance the overall quality of the manuscript.

A persuasive demonstration of why misaligning negative pairs is harmful should be provided

We present the following paragraphs in Section 2.1 to demonstrate the detrimental effects of misaligning negative pairs.

In contrastive learning, the misalignment of negative pairs adversely affects the learning process due to its inadvertent impact on the objective function. Consider the following contrastive loss function, designed for each anchor node $v_t^{(i)}$ with feature embedding $\mathbf{h}_t^i$.

The objective is to minimize the distance between embeddings of positive pair $v_t^{(i)},v_t^{(i)}$ and simultaneously maximize the distance between embeddings of negative pairs $$ v_t^{(i)},v_q^{(j)}$_{q=1,q\neq t}^{N-1}$ (Please see Eq. 1 in the paper).

Misalignment in negative pairs $v_t^{(i)},v_k^{(j)}$ detrimentally impacts the learning process by introducing errors in the loss computation. The misalignment leads to an undesired increase in the loss, hindering the optimization process. Specifically, the GCL model increases the distance between misaligned negative pairs, and inadvertently separates semantically similar samples, leading to a degradation of overall performance.

A time analysis should be provided to verify the efficiency of the proposal.

The computational cost of graph contrastive learning models is analyzed through two distinct components: pre-training and downstream task evaluation. In the pre-training phase, the process involves augmentation generation, encoder computation, and computation of the contrastive objective for each batch. In the downstream task phase, the model learns two input/output MLP layers and evaluates the model for node classification.

We conducted the computation analysis to evaluate the runtime performance of three variants of the SGCL model and compared these variants with several baseline methods on graphs of different scales, ranging from small to medium and large-scale graphs. The results of these experiments are summarized in the following table.

The table results indicate that during pre-training, SGCL-T on the Cora dataset outperforms MVGRL in running time. However, in other experiments, the computational cost of the proposed model is slightly increased compared to the baselines, primarily attributed to the computation associated with the smoothing approach. Specifically, its computational load is approximately twice that of MVGRL. It's noteworthy to highlight that the computational costs in the downstream evaluation phase across all models are nearly identical on each benchmark. This implies that, despite the more computations during the pre-training phase, our model demonstrates efficiency during the downstream evaluation phase.

The smoothing method only employs the original graph information. However, the node relationship can be highly changed after augmentation. For example, two highly related nodes can be dissimilar when one of them is dropped. In such cases, is the proposal still efficient?

According to the proposed framework, the smoothing method incorporates graph information from each augmented view. Following the generation of two augmented views, we apply the smoothing method to each view independently, aiming to capture every change in the original graph. We sincerely apologize for any confusion that may have arisen from the initial presentation of the framework. In the new version, we have made efforts to provide a clearer explanation.

The proposal seems incremental....

Respectfully, we adhere to the common framework established for GCL models. Inspired by the DGI framework that proposed an objective based on MI maximization in the graph domain, the majority of GCL models adopt this consistent framework, which encompasses graph augmentation, decoder computation, contrastive objective computation in the pre-training step, and subsequent evaluation in downstream tasks. However, what distinguishes our proposed framework from other GCL models is the intuitive incorporation of proximity information between nodes in positive and negative pairs.

We would like to emphasize that in the conventional GCL framework, there is a lack of information about the proximity of positive/negative pairs, and all N−1 negative pairs are uniformly treated. In other words, the conventional GCL approach treats all misclassified nodes equally, regardless of whether the misclassification occurs near the true positive or at a significant distance from it (since a 1-hop error is just as “expensive” as a k>>1 hop error). A common approach for considering proximity involves computing pairwise distances among all nodes in the graph. However, our approach is more efficient, avoiding the need to compute or store large dense matrices. This efficiency is achieved through the integration of three developed smoothing approaches embedded in our loss function.

Can previous contrastive objectives be used in the proposal?

We can further generalize the smoothed positive/negative pairs to some of the other contrastive objectives with similar settings, which apply positive/negative pairs at the node level. Additionally, the augmented generation strategy should maintain the nodes from the original graph to serve as corresponding positive pairs. For example, we applied the smoothed positive/negative matrices to the contrastive objective used in the GRACE method. However, considering that GRACE utilizes two types of negative pairs (inter-view and intra-view), we constrained it to only inter-view negative pairs. While results on some benchmarks showed improvements over the original GRACE, our approach consistently outperformed it.

There is a lack of ablation studies to exclude the effect of the proposed contrastive objective.

To perform an ablation study on the contrastive objective, we evaluate the significance of each individual term and subsequently combine them with hyperparameter $\lambda$. The following table provides the accuracies of different variants of SGCL achieved by different components of the contrastive objective on three benchmarks of varying scales: small (Cora), medium (CoauthorCS), and large (ogbn-arxiv). Initially, we observe that the exclusion of any term from our loss function results in deteriorated or collapsed solutions, aligning with our expectations. Subsequently, we investigated the influence of the combination of two individual terms using an optimal value of $\lambda$.

To select the value of $\lambda$, we initially set it as $\lambda =1/2N$​. However, in the experiments, we determined its optimal value through grid search. For instance, on the Photo dataset, the optimal value for $\lambda$ was found to be around $2.3e−4$. This value aligns with our first initialization when considering the batch size of $N=2000$ in the experiments.

We express our gratitude to the reviewer for dedicating time and effort to evaluate our work and providing constructive comments.

The main idea to smooth out weights is simple.

We would like to strongly emphasize that adapting smoothing correspondences to graph contrastive loss is highly non-trivial. Note that in GCL models, in the absence of labeled information, numerous incongruent nodes for each anchor node are potentially treated as false negatives. Specifically, nodes that are close to the anchor and thus have the potential to be semantically similar are inevitably categorized as negative pairs. However, in the conventional GCL methods, which lack information about the proximity of these nodes, all negative pairs are handled uniformly. In other words, the conventional contrastive learning approach treats all misclassified nodes equally, regardless of whether the misclassification occurs near the true positive or at a significant distance from it. As outlined in the paper, our approach addresses this limitation by promoting local consistency. A common approach for considering proximity involves computing pairwise distances among all nodes in the graph. However, our approach is more efficient, avoiding the need to compute or store large dense matrices. We effectively integrated three developed smoothing approaches embedded in our loss function. Subsequently, the loss function intuitively incorporates proximity information between nodes in positive and negative pairs.

What are the final embedding dimensions in the experiments?

In our experiments, we utilize a 512-hidden channel as the dimension of feature embedding learned using the encoder. Consequently, the final embedding dimension in the pre-training step is 512. However, during the evaluation step in downstream tasks, the final embedding dimension aligns with the number of classes in each benchmark. For instance, in the CoauthorCS dataset, it corresponds to 15.

Typo

We appreciate your insights and recognize the need for corrections, including typos in the writing of the paper. We will carefully review and address the minor mistakes to enhance the overall quality of the paper. $D_{ij}$ is the degree matrix computed using the augmented adjacency matrix $\hat{A}=A+I$, which $\hat{A}$ represents the adjacency matrix with a self-loop for each node.

The experiments in this paper seem overly simplified...

In our experiments, we employed the PyGCL library to explore a mini-batch scenario, aligning with the setting of our proposed model. Considering the diverse settings across various models developed for contrastive learning and the absence of comprehensive experiments on several Ogbn benchmarks in the published paper, to ensure fair comparisons, we reproduced the results of the state-of-the-art models available within this library, as detailed in Table 1 of the paper. We will publish the code of our model as well as the SOTA models for easy reproducibility of the results once the paper is accepted.

For further investigation, we conducted additional experiments involving a range of state-of-the-art methods, notably CGRA [1] and GRLC [2]. To establish a fair and robust comparison with the existing SOTA methods, these experiments are conducted in a full-batch scenario, adhering to the commonly employed data split in self-supervised learning as provided in the OGB benchmarks. The results are summarized in the following table.

[1 H. Duan et al., "Self-supervised contrastive graph representation with node and graph augmentation," Neural Networks, (67), 2023

[2] L. Peng et al., "GRLC: Graph Representation Learning With Constraints," in IEEE Transactions on Neural Networks and Learning Systems, 2023

Additionally, we have extended the evaluation of the model by conducting new experiments for the graph classification task, using five commonly used graph classification benchmarks: MUTAG, PTC, IMDB-BINARY, PROTEINS, and ENZYMES. In this experiment, we followed the InfoGraph setting for graph classification and compared the accuracy with the self-supervised state-of-the-art methods, including InfoGraph, GraphCL, MVGRL, BGRL, AD-GCL, LaGraph, and CGRA (Please see section 4.3).

The results reported in the following table (Table 3 in the paper) indicate that, in comparison to the best-performing state-of-the-art methods, the proposed approach demonstrates enhanced accuracy for IMDB-BINARY, PROTEINS, and ENZYMES, while maintaining comparable accuracy on other benchmarks.

It's worth mentioning that the accuracies of all models are reported from their respective published papers, except for the BGRL results, which we reproduced under the same experimental setting.

I believe that the choice of lambda is crucial, but the author does not explain how lambda is selected.

To select the value of $\lambda$, we initially set it as $\lambda =1/2N$​. However, in the experiments, we determined its optimal value through grid search. For instance, on the Photo dataset, the optimal value for $\lambda$ was found to be around $2.3e−4$. This value aligns with our first initialization when considering the batch size of $N=2000$ in the experiments.

Additionally, we performed an ablation study on the contrastive objective. We evaluated the significance of each individual term and subsequently combined them with hyperparameter $\lambda$. The following table provides the accuracies of different variants of SGCL achieved by different components of the contrastive objective on three benchmarks of varying scales: small (Cora), medium (CoauthorCS), and large (ogbn-arxiv). Initially, we observe that the exclusion of any term from our loss function results in deteriorated or collapsed solutions, aligning with our expectations. Subsequently, we investigated the influence of the combination of two individual terms using an optimal value of $\lambda$.

We express our gratitude to the reviewer for dedicating time and effort to evaluate our work and providing constructive comments. The suggestions are invaluable for enhancing the quality of our work, improving clarity, and making it more convincing. We are pleased to learn that the reviewer finds the theoretical foundation of our model to be novel and is convinced that the proposed method is intriguing and innovative.

The structure and organization of this paper appear quite impractical

Your insights are appreciated. We moved section 2 (related work) to the appendix and revised the paper accordingly to ensure a better balance between sections which allocates more space to the experimental aspects as suggested.

The author fails to provide the rationale and intuition behind using the loss function

and

Question1: .... I would recommend the author to double-check this issue.

As per your insightful recommendation, matrix $C \in \{0,1\}^{(N\times N)}$ accurately represents the similarity between node pairs in two augmented graphs, despite our inadvertent reference to it as a cross-correlation matrix. We sincerely apologize for any confusion resulting from the unclear presentation and typographical error in Eq. 4. In the first term of this equation, please note that $**I**$ should be substituted with the scalar "1," and the operator between matrices should be interpreted as the element-wise operator.

Kindly refer to the accurate formulation (Eq. 5 in the new paper) provided below:

$L_{SGCL}^{(i,j)} ={ \parallel {\tilde{\Pi}_{p}^{(i,j)}\odot (1-\mathbf{C}^{(i,j)})} \parallel_F^2 +\lambda \parallel (1-{\tilde{\Pi}_p^{(i,j)}})\odot \mathbf{C}^{(i,j)} \parallel_F^2 }$

It appears to us that the rationale behind utilizing the loss function has become more insightful now.

The intuition is straightforward. Generally, matrix $C^{(i,j)}$ comprises similarity values between pairs of nodes from two augmented graphs. Our expectation is to maximize $C^{(i,j)}$ for positive pairs and minimize $C^{(i,j)}$​ for negative pairs, equivalent to simultaneously minimizing $1−C^{(i,j)}​$ for positive pairs and $C^{(i,j)}$​ for negative pairs.

The first term minimizes the discrepancy between '1' and each element of $C^{(i,j)}$, aligning with the values in the smoothed positive pairs matrix $\hat{\Pi}_{p}^{(i,j)}$. This term enforces stability and preservation in the embeddings of positive pairs.

Likewise, the second term minimizes each element of $C^{(i,j)}$ in accordance with the values in the smoothed negative pairs matrix $\hat{\Pi}_{n}^{(i,j)}=1-\hat{\Pi}_p^{(i,j)}$. This term actively promotes a substantial diversity in the embeddings of negative pairs.

Questions 2:

According to our definition, $\hat{\pi}_{p} (i,j) = 1$ for each $i=j$.

To clarify, if we hypothetically consider that two nodes $i$ and $j$ ($i \neq j$) are very likely to be false negatives, such that $\hat{\pi}_{p}(i,j) = 1$, then the first term assigns a weight of '1' to minimize $1 - C^{(i,j)}$ which is equivalent to maximizing $C^{(i,j)}$, as you rightly mentioned. However, the second term is redundant since it assigns a weight of '0' to minimize $C^{(i,j)}$.

Smoothing for graph contrastive learning seems to be a little trivial to me.

We would like to strongly emphasize that adapting smoothing correspondences to graph contrastive loss is highly non-trivial. Note that in GCL models, in the absence of labeled information, numerous incongruent nodes for each anchor node are potentially treated as false negatives. Specifically, nodes that are close to the anchor and thus have the potential to be semantically similar are inevitably categorized as negative pairs. However, in the conventional GCL methods, which lack information about the proximity of these nodes, all negative pairs are handled uniformly. In other words, the conventional contrastive learning approach treats all misclassified nodes equally, regardless of whether the misclassification occurs near the true positive or at a significant distance from it.

As outlined in the paper, our approach addresses this limitation by promoting local consistency. A common approach for considering proximity involves computing pairwise distances among all nodes in the graph. However, our approach is more efficient, avoiding the need to compute or store large dense matrices. We effectively integrated three developed smoothing approaches embedded in our loss function. Subsequently, the loss function intuitively incorporates proximity information between nodes in positive and negative pairs.

The computational cost of graph contrastive learning models is analyzed through two distinct components: pre-training and downstream task evaluation. In the pre-training phase, the process involves augmentation generation, encoder computation, and computation of the contrastive objective for each batch. In the downstream task phase, the model learns two input/output MLP layers and evaluates the model for node classification.

We conducted the computation analysis to evaluate the runtime performance of three variants of the SGCL model and compared these variants with several baseline methods on graphs of different scales, ranging from small to medium and large-scale graphs. The results of these experiments are summarized in the following table.

The table results indicate that during pre-training, SGCL-T on the Cora dataset outperforms MVGRL in running time. However, in other experiments, the computational cost of the proposed model is slightly increased compared to the baselines, primarily attributed to the computation associated with the smoothing approach. Specifically, its computational load is approximately twice that of MVGRL. It's noteworthy to highlight that the computational costs in the downstream evaluation phase across all models are nearly identical on each benchmark. This implies that, despite the more computations during the pre-training phase, our model demonstrates efficiency during the downstream evaluation phase.

We would like to thank the reviewer for his/her time and effort spent on evaluating our work and their constructive comments. We find the suggestions to be very helpful for improving the quality of our work, making it more convincing. We are pleased to learn that the reviewer is interested in the problem and has found the presentation of our method to be clear.

The newest baseline is published in 2022. Therefore the paper misses a lot SOTA methods.

In our experiments, we employed the PyGCL library to explore a mini-batch scenario, aligning with the setting of our proposed model (https://github.com/PyGCL/PyGCL). Considering the diverse settings across various models developed for contrastive learning and the absence of comprehensive experiments on several Open Graph Benchmarks in the published paper, to ensure fair comparisons, we reproduced the results of the state-of-the-art models available within this library, as detailed in Table 1 of the paper. We will publish the code of our model as well as the SOTA models for easy reproducibility of the results once the paper is accepted. For further investigation, we conducted additional experiments involving a range of state-of-the-art methods, notably CGRA [1] and GRLC [2]. To establish a fair and robust comparison with the existing SOTA methods, these experiments are conducted in a full-batch scenario, adhering to the commonly employed data split in self-supervised learning as provided in the OGB benchmarks. The results are summarized in the following table.

[1] H. Duan et al., "Self-supervised contrastive graph representation with node and graph augmentation," Neural Networks, (67), 2023

[2] L. Peng et al., "GRLC: Graph Representation Learning With Constraints," in IEEE Transactions on Neural Networks and Learning Systems, 2023

Can the proposed methods be adopted for graph-level tasks [1,2]?

We acknowledge that our proposed model is able to be adopted for graph-level tasks. We have evaluated the proposed model for the graph classification task using five commonly used graph classification benchmarks: MUTAG, PTC, IMDB-BINARY, PROTEINS, and ENZYMES. In this experiment, we followed the InfoGraph setting for graph classification and compared the accuracy with the self-supervised state-of-the-art methods, including InfoGraph, GraphCL, MVGRL, BGRL, AD-GCL, LaGraph, and CGRA.

The results reported in the following table (Table 3 in the paper) indicate that, in comparison to the best-performing state-of-the-art methods, the proposed approach demonstrates enhanced accuracy for IMDB-BINARY, PROTEINS, and ENZYMES, while maintaining comparable accuracy on other benchmarks. It's worth mentioning that the accuracies of all models are reported from their respective published papers, except for the BGRL results, which we reproduced under the same experimental setting.

We would like to mention that among the two references you recommended, we compared the accuracies using the second reference, CGRA, while the first reference was supervised, making the comparison inherently unfair.