Learn Beneficial Noise as Graph Augmentation

Response to Reviewer iJqw

We greatly thank you for the detailed and valuable comments. Please find our responses to the comments as follows:

W1&W2: As is mentioned in 'Experimental Designs or Analyses', the detailed hyperparameter of the training is not listed. Moreover, as is mentioned in Figure 2, the model can be trained with a batch size to decrease memory burden, which is not discussed in Table 1, where some methods report OOM on some datasets.

Reply: Thank you for your valuable feedback. We will add detailed hyperparameter settings to improve clarity and reproducibility. Regarding batch size and memory efficiency, all GCL methods were trained using the same batch size to ensure a fair comparison. For smaller datasets, we do not apply batch processing, for large datasets, we set the batch size of 256. Our experiments are conducted on an NVIDIA 4090 GPU (24 GB memory) for most datasets and on an NVIDIA A100 GPU (40 GB memory) for OGB-arxiv. However, we recognize that this was not explicitly stated in Table 1. We will update Table 1 with a clarification that all methods follow the same batch size setting and discuss the impact of batch size adjustments on memory consumption. We appreciate your insightful comments and will revise the paper accordingly.

Q1: JOAO is mentioned as 'learnable' in Table 1. However, to the best of my knowledge, JOAO randomly selects augmentation methods, which are all randomly applied. The projection head of each augmentation method differs.

Reply: Thank you for your insightful comment. While JOAO does not learn augmentations directly, it does adaptively select augmentation strategies based on a similarity-based policy. Referring to it as "learnable" in Table 1 may not be the most accurate description. Instead, "adaptable" better captures its mechanism. We will revise it in the paper and provide a clearer explanation of this distinction.

Q2: The further discussion of 2.3 is placed in Appendix C after the detailed discussion of 3.3 and 4.2, which is kind of disordered.

Reply: We appreciate your feedback regarding the structure of the paper. We will reorganize the content to ensure a more intuitive and coherent presentation. Thank you for your valuable suggestion!

Q3: The performance of the method on Wiki-CS in Table 2 is different from that in Table 1, why? The difference also appears between Table 7 and Table 3.

Reply: Thank you for your question. The differences in performance arise because Table 1 and Table 3 report the best results, while in the ablation study, we re-run the experiments to ensure a fair comparison across different settings. The results in the ablation study reflect newly obtained experimental outcomes under controlled conditions.

Q4: The results of the ablation study are kind of hard to read. Maybe the average value of columns and rows can be added to better show the trend of difference between w/o Aug., Random, and Learnable.

Reply: Thank you for your helpful suggestions. We will update Table 2 to better highlight the best results for improved clarity.

Q5: Can the attribute noise also be visualized?

Reply: Thank you for your insightful suggestion. Similar to our response to Reviewer PRCX, visualizing attribute noise is challenging because node representations are high-dimensional and lack the structured spatial relationships found in visual data. Even if represented as a matrix or heatmap, the absence of spatial correlations between adjacent values could lead to misleading interpretations.

Response to Reviewer J7B2

We greatly thank you for the detailed and valuable comments. Please find our responses to the comments as follows:

W1&2&3&4: The relationships between the proposed model and the learnable methods are not discussed in Section 1. The graph contrastive learning task should be introduced in more detail in Section 1. The connections and differences between PINGDA and some current GCL models based on learnable strategies should be discussed in details. In contributions, the authors should clarify the merits of the proposed model from the Pi-Noise perspective.

Reply: Thank you for your valuable suggestions. Due to space limitations, we did not provide an in-depth discussion of these aspects in Section 1. We will provide a more comprehensive introduction to the graph contrastive learning task in the revised version. Additionally, we will provide a more detailed introduction to the graph contrastive learning task to improve clarity and comprehensiveness. For the specific differences with AD-GCL, please refer to the response to Reviewer PRCX.

W5: The authors could add the ablation study such as extending the proposed noise augmentations to more current GCL models.

Reply: We have included the requested ablation study by extending our proposed noise augmentations to more GCL models. Specifically, we selected both a classical GCL model (GRACE [1]) and a more recent approach (Sp²GCL [2]) to demonstrate the effectiveness of our method. The results are summarized in the table below:

These results highlight the impact of our augmentation approach across different GCL models.

[1]Zhu, Yanqiao, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. "Deep graph contrastive representation learning." arXiv preprint arXiv:2006.04131 (2020).

[2]Bo, Deyu, Yuan Fang, Yang Liu, and Chuan Shi. "Graph contrastive learning with stable and scalable spectral encoding." Advances in Neural Information Processing Systems 36 (2023): 45516-45532.

W6: Apart from the Memory-Times experiments, Memory-Performance or FLOPs-Performance should be introduced to validate efficiency.

Reply: The detailed tables are shown below. The results show how our method compares to existing baselines in terms of both computational cost and accuracy improvements. A negative value in ΔFLOPs or ΔMem. indicates a reduction of our method in computational cost or memory usage, while a positive value in ΔAcc% reflects our accuracy improvement over the respective method.

Our results demonstrate that while some methods, such as GRACEIS and AD-GCL, achieve efficiency gains by significantly reducing FLOPs and memory usage, our approach strikes a balance between efficiency and performance.

W7: There are some typos. For example, in line 19, “. Where” should be “, where”. In Figure 3, it lacks explanations about the dashed line between two nodes.

Reply: Thank you for your careful review. We will correct the typos. Regarding the dashed lines in Figure 3, they represent connections between nodes of different classes. We apologize for the unclear representation and will revise the figure or improve the explanation to ensure better clarity. We appreciate your valuable feedback and will make the necessary improvements.

Response to Reviewer smN2

We greatly thank you for the detailed and valuable comments. Please find our responses to the comments as follows:

W1&Q1: The theoretical analysis in Section 3.3 is interesting, but the definition of task entropy are unclear.The theoretical analysis in Section 3.3 is intriguing. However, I am unclear about the definition and role of task entropy. Could you provide a more detailed explanation or an example to clarify its significance?

Reply: In our framework, task entropy is used to measure the difficulty of a task. The key idea is that adding noise can help simplify the task, which is quantitatively measured by the task complexity, making it easier for the model to learn meaningful representations. This aligns with our theoretical analysis, where learned beneficial noise can lead to better performance. We hope this explanation can answer your question and we would like to further clarify it if the concern is still well-addressed.

W2&Q2: The paper does not detail the hyperparameters used in the experiment, such as learning rate batch size, etc., which will affect the reproducibility of the research. While the experimental setup is well-documented, the hyperparameter settings are not explicitly detailed. Could you provide more information on these aspects, either in the main text or in supplementary materials?

Reply: Thank you for your helpful suggestions. The hyperparameters were selected within the following ranges: For smaller datasets, we do not apply batch processing, for large datasets, we set the batch size of 256. The number of epochs varies from 500 to 2000 depending on the dataset. The learning rate is set between 0.0005 and 0.01, while the weight decay is kept constant at 0.0001 across all datasets. The feature dropout rates range from 0.0 to 0.3, and the edge dropout rates vary between 0.1 and 0.4. The temperature is typically set to 0.3 for most datasets, with an exception for Coauthor-Phy, where it is set to 0.5. This is a simple description, the detailed version will be added later.

Q3: The computational efficiency in Figure 2 does not show much advantage of PiNGDA. Can you explain why this is the case and provide analyze of the complexity?

Reply: The computational efficiency of PiNGDA may not appear significantly advantageous in Figure 2 because our method involves adding noise to all data points, which introduces additional computations. In the two noise generation modules, since each edge and each node feature is calculated by MLP, the computational complexity of the noise module is $\mathcal{O}(\mathcal{E} \cdot d) + \mathcal{O}(N \cdot d)$, where N is the number of nodes, $\mathcal{E}$ is the number of edges in the graph, and d is the dimension of the node feature. However, this process is crucial for improving the robustness and generalization of the model. To provide a more comprehensive comparison, we have included detailed memory-performance analyses in the tables. Please refer to response to Reviewer J7B2.

Response to Reviewer LdjD

We greatly thank you for the detailed and valuable comments. Please find our responses to the comments as follows:

W1: Innovation of application of $\pi$-noise to GCL.

Reply: Although $\pi$-noise has been explored in other fields, its adaptation for graph data and integration within the GCL framework offers a unique perspective. The core challenge of π-noise—how to define task entropy—remains highly difficult, especially for graph data. This is not merely incremental work, but rather a fundamental challenge, which forms the core contribution of our research. Our work achieves unified augmentations on both graph topology and node features within a theoretical framework, which prior GCL methods have not been able to accomplish. This novel application allows us to leverage noise in a way that improves the quality of graph representations.

W2: Lacks detailed experimental settings.

Reply: Thank you for your suggestion. Our experiments are conducted on an NVIDIA 4090 GPU (24 GB memory) for most datasets and on an NVIDIA A100 GPU (40 GB memory) for OGB-arxiv. For our proposed method, we employ a two-layer GCN network with PReLU activation, where the hidden layer dimension is set to 512, and the final embedding dimension is 256. Additionally, we utilize a projection head, consisting of a 256-dimensional fully connected layer with ReLU activation, followed by a 256-dimensional linear layer. Edge noise generator uses an MLP to process node features and then applies Gumbel-Softmax sampling method. Feature noise generator uses two MLPs to estimate the mean and variance of feature noise and then uses the reparameterization trick. We appreciate your feedback and will ensure these details are clearly presented in the revised version.

W3: Compared to AD-GCL, PiNGDA does not show advantages in terms of time and space efficiency.

Reply: Our method indeed introduces some additional computational overhead because we have two augmentation modules—one for graph structure and one for node features. This results in a slight increase in memory and time consumption. However, as shown in the table, this increase is not substantial, and more importantly, our method achieves significantly better performance, demonstrating its effectiveness.

C1&2&3:

Reply: Thank you for your helpful suggestions. We will restructure Section 3 to improve readability by separating the discussion of existing research. Additionally, we will correct minor grammatical issues. To enhance reproducibility, we will also provide more details on the experimental settings. We appreciate your valuable feedback and will make the necessary revisions.

Q: How does PiNGDA compare to other baselines that also focus on adaptive or learnable graph augmentations, such as those mentioned in the "Essential References Not Discussed" section?

Reply:

Since the official code for Paper 1 was not available, we replicated the results from the supplementary materials submitted with Papers 2 and 3 from OpenReview. The performance difference between the reported results and our replication could be due to the absence of specific hyperparameters in the papers. We base our results on our own replication.

Also, as suggested by Review J7B2, we also added PiNGDA to different GCL backbones and achieved good results.

Our method consistently outperforms existing approaches across all datasets, achieving significant improvements in accuracy. This demonstrates the effectiveness of our adaptive graph augmentation strategy, which better captures the underlying structure of the data and enhances model generalization.

Response to Reviewer PRCX

We greatly thank you for the detailed and valuable comments. Please find our responses to the comments as follows:

W1: The depth of the discussion of the experimental results can be further improved, such as the explanation of certain experimental phenomena and the possible differences between different datasets. For example, in the Table 2, the learnable methods seem not to perform as well as the other two datasets.

Reply: Regarding the observation that learnable augmentation performs worse than random augmentation on the WikiCS dataset, we believe this may be due to the following reasons. Compared to other datasets, WikiCS has a relatively higher number of average edges. The learnable augmentation method may not effectively enhance model performance when modifying the edge structure. Moreover, the node features in WikiCS have relatively low dimensionality. It indicates that edges may be more important than features in WikiCS. As a result, modifying features without adjusting edges might lead to suboptimal performance. However, if the learnable edge augmentation cooperates learnable feature augmentation, it may yield better results as can be found in the last column of Table 2. We will include a more detailed analysis in the paper to further investigate the differences across datasets. Thank you again for your insightful feedback!

W2: The mathematical derivation in the method section can be more detailed to enhance readability and understanding. For example, the random variable alpha is a bit confusing. Why is this variable necessary? And why do you assume the probability distribution showed in equation 4? It will be very helpful if they are explained clearly.

Reply: Thank you for your comments. In our paper, the random variable $\alpha$ is introduced to indirectly measure the difficulty of the task, closely tied to the concept of task entropy. We adjust the variance of the auxiliary Gaussian distribution, which directly influences the calculation of task entropy.

Q1: How is your method different from AD-GCL[1]? AD-GCL uses an adversarial method to enhance contrastive learning of graphs. It also uses a learnable method to drop edges which is quite similar as your method. Please clearly explain the key differences in the design of the two methods.

Reply: Our method differs from AD-GCL primarily in its augmentation strategy and focus. From the aspect of motivation, AD-GCL applies adversarial graph augmentation to minimize information from the original data which tries to maximize the loss, our approach enables the model to learn the augmentation simplify the representation learning. From the aspect of augmentation, AD-GCL mainly targets graph structure, whereas our method considers both graph structure and node attributes, leading to a more comprehensive augmentation. This advantage is reflected in our superior results in node classification, as shown in Table 1.

Q2: There are two main components proposed by PiNGDA, however the visualization part only shows the edge noise, how does the node attribute noise look like? The authors should provide more detailed explanation on the part.

Reply: Thank you for your insightful suggestion. From a contrastive learning perspective, methods like SimCLR and BYOL commonly use Gaussian noise as an augmentation, making it a natural choice for adaptation to non-vision data (tabular data). This is also an advantage of our method which serves as a unified framework for both graph and tabular node features. Regarding visualization, node attributes are typically high-dimensional, making them difficult to represent intuitively. Even if converted into a matrix or heatmap, unlike in vision tasks, there is no inherent spatial relationship between adjacent values, which could lead to misleading interpretations. We appreciate your valuable feedback and will add further explanations to clarify this point.

Response to Reviewer 65ie

We greatly thank you for the detailed and valuable comments. Please find our responses to the comments as follows:

W1: The introduction of positive-incentive noise and the discussions with the existing papers of noise are not enough, especially in the main paper.

Reply: Thank you for your valuable suggestion. Due to space limitations, we only provide a simple and brief discussion in the main paper. We will consider expanding this discussion in the supplementary material or refining the main text to provide a clearer comparison.

W2: Figure 1 can be further improved. It is hard to distinguish how to simultaneously generate augmentations on both graph topology and node features. Meanwhile, $\mathcal{N}(0,I)$ in the figure is inconsistent with the notations appearing in the main paper.

Reply: Thank you for your valuable feedback. We will refine the figure to enhance clarity. As for the notation inconsistency, we apologize for any confusion. In Appendix [B], we explicitly describe the reparameterization trick used in our method, where the covariance matrix is defined as $\Sigma = \text{diag}(\sigma^2)$, ensuring a diagonal structure. Here, $\sigma\sim \mathcal{N}(0, I)$, aligning with the notations in the figure.

W3&4: In Table 2, the best results should be further highlighted. The current version is not intuitive. The caption of Table 4 is quite confusing. The authors fail to clarify that the datasets are heterogeneous graphs in the caption.

Reply: Thank you for your helpful suggestions. We will update Table 2 to better highlight the best results for improved clarity and we will revise the caption of Table 4 to clearly specify that the datasets are heterogeneous graphs.

Q1: Why do the authors assume that the topological noise is a Bernoulli distribution?

Reply: Thank you for your question. We assume that the topological noise follows a Bernoulli distribution because the simplest way to model edge perturbation is to learn whether an edge should be kept or removed, which naturally aligns with a 0/1 binary decision process. This makes Bernoulli distribution a straightforward and effective choice for modeling edge retention or deletion in graph structures.

Q2: I cannot completely understand the meaning of Figure 3. How are the nodes visualized? t-SNE? Why does it only contain dozens of nodes? As we all know, there are thousands of nodes in a graph. The authors should clarify this in the rebuttal and the main paper.

Reply: Sorry for the lack of clarity in Figure 3. To improve visualization, the nodes shown in the figure are a subset selected from thousands of nodes for a case study. This selection helps make the visualization more interpretable. The node distribution in the figure is based on their connectivity relationships, rather than a dimensionality reduction method like t-SNE. We will clarify this in the main paper. Thank you for your valuable feedback.

Q3: What’s the additional computational and space complexity? Will it lead to a significant burden on both computation and memory?

Reply: Thank you for your insightful question. The computational and space complexity of noise learning depends on the design of the noise network. In the two noise generation modules, since each edge and each node feature is calculated by MLP, the computational complexity of the noise module is $\mathcal{O}(\mathcal{E} \cdot d) + \mathcal{O}(N \cdot d)$, where N is the number of nodes, $\mathcal{E}$ is the number of edges in the graph, and d is the dimension of the node feature. To minimize the burden, we have carefully simplified the network structure, such as reducing the number of hidden layers. The memory overhead as shown in Figure 2, is not significant. While our method introduces some additional computational costs, it provides clear performance improvements, making the trade-off worthwhile. The detailed table can be found in the response to Reviewer J7B2. We appreciate your feedback and will clarify this further in the paper.