Weakly Supervised Graph Contrastive Learning

We thank the reviewer for their feedback and suggestions.

1. The weak/noisy label setting appears to be confined to a limited context, especially on graphs. While I acknowledge the contribution of this paper in this specific area, its applicable generality to real-world graph datasets is questionable.

Response: We believe that such a weak supervision approach will benefit graph learning especially in text-attributed graphs where it is easier to obtain weak labels based on keyword matching, etc. Even with the recently growing popularity of text-attributed graphs such as in recommender systems[2] and fake news detection[1], where obtaining ground truth labels is challenging, the proposed weak supervision approach is useful.

2. The analysis to the baseline GCL methods performance under noisy settings are shallow to some extent. The authors may consider some further analysis. For example, how well would each baseline perform under different types of classifier? What are the samples in common that are "robust" to such weak/noisy labels for each baseline?

Response: We agree that experimenting with different types of classifiers is interesting and we will consider it for our future work. In the current paper, since we are interested in studying the robustness of the learned GCL embeddings to label noise in downstream classification, we think that the choice of the classifier would have relatively less effect on the final conclusion as the same classifier is used for comparing all baseline methods. It might be challenging to identify the common samples that are robust to weak labels for each baseline. Do you mean to consider robustness to be based on low noise or based on correct prediction? Please clarify the question.

3. An MLP is widely-used classifier as well. The authors may consider adding it to the experiments.

Response: Typically in contrastive learning, the learned embeddings are evaluated for linear separability using a simple linear classifier such as logistic regression. Although MLPs are generally better than LR, they further transform the learned node representations to another embedding space (if multiple layers). Having said that, it is an experiment that can be added but we believe tells us little more about the effectiveness of our method.

[1] Adrien Benamira, Benjamin Devillers, Etienne Lesot, Ayush K Ray, Manal Saadi, and Fragkiskos D Malliaros. 2019. Semi-supervised learning and graph neural networks for fake news detection. In Proceedings of the 2019 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining. 568–569.

[2] Jason Zhu, Yanling Cui,Yuming Liu, Hao Sun, Xue Li, Markus Pelger, Tianqi Yang, Liangjie Zhang, Ruofei Zhang, and Huasha Zhao. 2021. Textgnn:Improving text encoder via graph neural network in sponsored search. In Proceedings of the Web Conference 2021.2848–2857.

We thank the reviewer for their feedback and suggestions.

1. How does the weak label be generated? The author only state that the weak label is generated by the labeling function but without no more elaboration. If the label is not given, then how the weak label can be generated with a certain accuracy? A comparison on different labeling function could also be helpful.

Response: In our paper, the weak labels are generated using Algorithm 3 which requires the ground truth label as it is a synthetic experiment setup. In real-world problems, when the ground truth is not available, LFs look for clues within the data. For example, in citation networks like Cora, the abstract of the papers (nodes) contains certain keywords related to a class label. So if these keywords are present in a paper abstract, we can assign the corresponding class as its weak label.

2. The definition of robustness of GCL to label noise is very confusing. 1) Is label noise meaning the inaccuracy of true label or generated weak label? 2) It follows a logical intution that when there is a high level of lable noise in a dataset, the accuracy of a model trained on this dataset is likely to be low. If the label noise in a dataset as high as 53%, ahieveing high accuracy could be somewhat meaningless.

Response: To clarify, the weak label or label noise is only on the generated weak label and not the ground truth label. The proposed WSNet model is trained using the noisy labels but evaluated with the ground truth. Despite training the model with 50% noisy labels, it achieves a 68% F1 score (on Cora) when the predicted labels are compared to the ground truth labels (which have no noise). In other words, the train labels are always noisy and the test labels used for evaluating the predictions are clean.

3. The idea of this paper is very similar to cluster-based graph contrastive learning such as [1] which utilize the cluster or community information as auxiliary information for learning objective. Thus, the author should concentrate on comparining with these baselines.

Response: Thank you for sharing this reference. This work is focused on graph prediction whereas we are focused on node-level tasks. However, we will try our best to compare WSNet with this method for the future versions of our paper due to the time constraint and inaccessibility of their code.

5. The presentation of the paper should be improved. For example, the caption of table should apperaove above the table; It is better not to place any context between two tables.

Response: Thank you for the suggestion. We will update this in the revised version.

Q5. In PI-GNN [5], the authors employ noise ratios of 0.0, 0.2, 0.4, 0.6, and 0.8, which appear more intuitive. Despite PI-GNN focusing on image datasets and WSNET on graph datasets, The chosen noise ratios in this study (High 53%, Medium 32%, Low 10%) are notably specific. Could you elucidate the rationale behind these values and explain your approach to introducing noise to the original labels?

Response: Since we are interested in the setting where each node is associated with multiple weak labels we varied both the number of LFs ($m$) and individual LF accuracies ($p_a$) pairs as ($m$, $p_a$) = [(5, 0.45), (10, 0.65), (50, 0.55)]. These weak labels are aggregated using majority vote to obtain, 53%, 32% and 10% accuracies. In other words, we only control the $m$ and $p_a$ parameters and not the final accuracy/noise of the aggregated labels. We will highlight this better in the revised version of the paper.

Q6. How do you account for the enhanced performance on non-homophilous graphs? Might this improvement be ascribed to the community detection algorithm?

Response: Yes, we believe the improved performance on non-homophilous graphs may be attributed to positive sampling from communities. We will highlight this in the revised version.

Q7: In the paper, it is mentioned that weak labels have accuracies set at 47%, 68%, and 90%. I am curious about how these weak labels are generated and how their accuracies are controlled.

Response: Individual $m$ weak labels are generated using Algorithm 3 and are then aggregated using majority vote. We have clarified this in the revised version.

The grammatical errors and typos have been noted with thanks and corrected in the revised version.

Q2. The Louvain algorithm [1] identifies exclusive communities, implying each node associates with only one community. This exclusivity is often incongruent with real-world scenarios. For instance, individuals in social networks typically affiliate with multiple groups, such as family, friends, and colleagues. Similarly, in biological contexts, genes or proteins often participate in multiple pathways. Furthermore, the Louvain method can yield poorly connected communities [2]. Could you elucidate further on the algorithm's application, such as the number of communities detected versus class count and how effective it helps with “finding nodes with a similar graph structure”? Why not opt for other superior methods, such as the Leiden algorithm?

Response: Thank you for your suggestion. We acknowledge that Leiden is a good alternative to Louvain especially for scalability and we will add this discussion to the revised version of the paper. However, we would also like to highlight that our proposed method is not dependent on the community detection algorithm. Our contribution is more on the idea of sampling positives from detected communities for robustness to weak labels. We also experimented with using Leiden instead of Louvain and reported the results in Tables 1 (Cora) and 2 (Pubmed) in the common response. We observe that there is not much difference in the number of detected communities and mean entropy of class labels within communities for both the methods. We do note that Leiden has a slightly higher NMI and ARI score with the ground-truth labels compared to Louvain which could be the reason why WSNet performs slightly better using Leiden on Cora. The NMI and ARI between the detected community labels of the two methods is also high for Cora but quite low for Pubmed. “Could you elucidate further on the algorithm's application, such as the number of communities detected versus class count and how effective it helps with “finding nodes with a similar graph structure”?” Do you mean to ask if matching the number of communities and number of classes would result in better performance? Clarification on this question will be helpful.

Q3. Hard samples, particularly hard negatives, are pivotal for representation learning under the Contrastive framework [3]. The Supervised Contrastive Loss study [4] further emphasizes the significance of hard samples over easy ones. Could you provide further insight into L-supcon, given its centrality in your method?

Response: Thanks for bringing up this point. We agree that hard negatives are important for contrastive learning and this is an interesting future work for consideration. As our work is a preliminary study of weakly supervised GCL, we simply sample positives and negatives based on the aggregated weak labels. Nodes from different classes are considered negatives and nodes from the same class are positives, similar to the original L_{SupCon} loss.

Q4. The proposed loss combines Self-Supervised Contrastive Loss L-s and Supervised Contrastive Loss L-supcon. This combination suggests equal influence from both losses, yet intuitively, L-supcon seems more potent than L-s. Do you think it is necessary to account for their respective contributions to representation learning, perhaps by introducing and searching for a hyperparameter?

Response: We actually had experimented with this idea and did not see improvement, in particular in the high noise setting. When the quality of labels is low, weighting $L_{SupCon}$ higher, negatively impacts the overall performance. This is expected as $L_{SupCon} relies entirely on the weak labels for sampling positives and negatives. Here we report the performance of WSNet with (40, 60), (30, 70) and (20, 80).

We thank the reviewer for their feedback and suggestions.

1. Identifying nodes belonging to the same community from the entire graph is a computationally intensive operation. Including an analysis of efficiency would enhance the completeness of this work.

Response: This is true that there is a computational overhead, however, this can be implemented such that it only incurs a one-time computation before the model training. The model training itself is pretty fast. Specifically, we need to first find the communities. Community detection is a one-time computation that happens before the model training. Since we use the Louvain algorithm for detecting communities, there is an overhead of O($nlogn$) for graphs with $n$ nodes. Next, we identify and sample all the positives and negatives we need during the training based on the communities, and the similarities of their weak label distributions. This has a runtime of $O(np^2)$, where $p$ is the size of the largest community in the graph and $p << n$ in most real world graphs. We also report the runtime used for these pre-training computations as well as the actual training on Cora and Pubmed in Tables 1 and 2 using Louvain in the common response.

2. The paper lacks diagrams/figures, training details, or efficiency analysis.

Response: We will add the training details and run-time analysis to the revised paper. All the experiments were run locally on an 8-cpu core Macbook M2 computer. WSNet was trained for 50 epochs and the results averaged over 5 runs is reported in the paper. For all the other GCL methods, we used the official code released by the authors of the respective papers and tried our best to use the recommended hyperparameters wherever relevant.

5. The paper proposes a way to improve the learning of node embeddings learned with graph contrastive under weak supervision. The paper also provides some experiments that show that their model can achieve good performance. However, the experiments are not sufficient, and the innovativeness is not up to ICLR standards.

Response: We would like to highlight that the main novelty of our work is to introduce weakly supervised graph contrastive learning. There are no other works, to the best of our knowledge, that do GCL with weak labels. We also believe that weakly supervised graph learning is an important field of study for solving graph-based real world problems such as in recommender systems or misinformation detection where only weak labels may be available. Secondly, we systematically study existing GCL methods in the framework of weak supervision which is novel and our results show that there is scope for further research in this direction.

Q1. Which labeling functions (LFs) were used to derive the weak label matrix, Lambda? As far as I remember, the famous citation network triplet (Cora, Citeseer, and Pubmed) does not come with such weak labels. Furthermore, in the context of Majority Voting, how is a tie among LFs resolved?

Response: The LFs were synthetically generated as per Algorithm 3. For a given accuracy/coverage, we flip the labels (given the ground truth) to get weak labels. We repeat this process $m$ separate times to obtain $m$ weak labels for each node. In case of majority voting tie, one of the tied labels is selected uniformly at random.

We thank the reviewer for their insightful comments.

1. The authors spend too much space on Research Question 1, which evaluates the robustness of existing GCL methods. This is only a process of running baselines and cannot be the main innovation and contribution of the paper. The authors should focus more on the proposed method.

Response: We would like to clarify that although RQ1 is not the main innovation point in the paper, there is still value in studying how different GCL methods perform in the presence of label noise as this comparative study has not been done before. We clarify that these experiments are not only baselines but also constitute the first study on the robustness of GCL methods to weak labels. Moreover, we observe that no single method consistently outperforms others showing that there is scope for further research in this direction. We will discuss this better in the revised version of the paper.

2. Even with the amount of space devoted to RQ1, I don't think the authors' analysis is deep enough. For example, the authors conclude from the experimental results that baselines that use neighborhood-based sampling of positive pairs perform better than others in the high noise setting. What is the reason for this, and the authors should provide some analysis.

Response: GNNs typically implement neighborhood aggregation for learning efficient and robust node representations and this supports our observations that neighborhood-based sampling makes the embeddings more robust to weak labels compared to node augmentations. On contrasting views of the same nodes versus neighboring nodes, the learned embeddings contain more information about the neighbors in the latter than the former. We will expand on this discussion in the revised version.

3. The notations are inconsistent, which makes it difficult for readers to understand the algorithm. For example, in the first line of Algorithm 1, $\lambda_i$ looks like a scalar, but Section 3 says that \lambda is a label function. Additionally, are $\Lambda_i$ and $\Lambda[V_i]$ representing the same vector?

Response: Thank you for pointing these out. We will clarify these notations in the revised version of the paper. Mainly that $\lambda_i$ is not a scalar but the output vector obtained by applying labeling functions $\lambda$ on $i$th node and $\Lambda_i$ and $\Lambda[V_i]$ do represent the same vector.

4. Will the algorithm's results differ significantly depending on the community search algorithm used and the label aggregation method used? The authors should provide more experimental results to compare the possibilities.

Response: Both the community search algorithm and the label aggregation methods are modules that can be replaced in our proposed WSNet framework. Our method is not tied to these modules. For our experiments, we chose the simplest and most popular method in both cases.

a) More specifically, we used the simple majority vote label aggregator to demonstrate the effectiveness of our proposed WSNet. On using better label aggregation methods the accuracy of the aggregated labels would increase resulting in the improvement of the overall performance of WSNet. In our work we show that even using a simple majority vote label aggregation, WSNet outperforms other baselines. We repeated our experiment using the Snorkel[1] label aggregator and reported the results in Table below. We observe that the Snorkel accuracy of the aggregated labels is lower than MV for all noise levels and as a result WSNet trained using the corresponding labels has a lower performance than MV.

b) Similarly, for community detection, we address the comments in the common response.

5. Will the size of the negative samples r have a significant impact on the results? The authors should provide a sensitivity analysis of the parameters.

Response: Generally, the higher the value of $r$, the better the performance. However, with higher values of $r$, the computational complexity also increases as the denominator term in the $L_S$ (Eq 3 in the paper). We ran experiments on the Cora dataset for varying values of $r$ and reported the results below. We will add these results and discussion to the appendix of the revised paper due to space constraints. We observe a similar performance for $r=10$ and $r=20$ with slight increase in runtime. Given that the performance is similar, we used $r=10$ as in our experiments in the paper.

Q1. It is unclear why only the node with the highest similarity is sampled as a positive sample for the L_S loss. It would be interesting to investigate whether using a top-k sampling strategy would be feasible and potentially improve the results.

Response: We used only one positive sample keeping with the norm in contrastive learning. That is the standard for InfoNCE loss because it is considered to be similar to cross-entropy loss for classifying the correct positive sample from the remaining negative ones. Additionally, since the labels are noisy, by considering the top-k samples with similar weak label distribution, there is a higher chance of the noisy labels affecting the embeddings. Moreover, this can also become an issue for communities with only 1 node. However, we think this is an interesting setup to investigate. We ran an experiment to see if top-5 improves the results. As expected, for the higher noise settings, $k=1$ does better.

Q2. It is unclear why WSNET performs better in the High Noise setting than in the Medium Noise setting on the Texas dataset.

Response: Considering that Texas is the smallest dataset we have with only 183 nodes, we observe a high variance in the results and although on average it seems the performance on high noise setting is better compared to medium noise setting, considering the variance, there is overlap in the ranges. This is also only observed with linear regression and not the random forest classifier, which might highlight this is partly due to simplicity of the downstream logistic regression classifier, more than the quality of the embeddings from WSNet. WSNet + RF shows the expected increase in performance with decrease in noise. We will expand the discussion in the paper on this.

[1] Ratner A, Bach SH, Ehrenberg H, Fries J, Wu S, Ré C. Snorkel: Rapid Training Data Creation with Weak Supervision. Proceedings VLDB Endowment. 2017 Nov;11(3):269-282. doi: 10.14778/3157794.3157797. PMID: 29770249; PMCID: PMC5951191.

We ran our method WSNet on a larger OGB dataset - ogbn-arXiv (169,343 nodes and 1,166,243 edges).

The ogbn-arXiv dataset is a directed graph and it represents the citation network between all CS arXiv papers indexed by MAG [1]. Each node is a paper and each directed edge indicates that one paper cites another one. Each paper is represented using a 128-dimensional feature vector obtained by averaging the skip-gram embeddings of words in its title and abstract. Each paper belongs to 1 of 40 CS subject areas so the task is a 40-class classification problem.

We ran WSNet on arXiv and report the F1 classification score averaged across 5 runs for different noise levels. We also note the time taken in seconds for community detection, positive sampling and training. We note that 40-class classification is a much harder problem especially in a weak learning setup and consequently we observe low classification scores.

We tried running MVGRL, PI-GNN and Joint-Training baselines on arXiv but they all resulted in OOM and we let them run for over (1930.4 + 46.2 + 40886.3) seconds. We will try our best to run the remaining baselines for the revised version of the paper.

[1] Kuansan Wang, Zhihong Shen, Chiyuan Huang, Chieh-Han Wu, Yuxiao Dong, and Anshul Kanakia. Microsoft academic graph: When experts are not enough. Quantitative Science Studies, 1(1):396–413, 2020.