Label-free Node Classification on Graphs with Large Language Models (LLMs)

Q1: Since LLMs generate annotations without access to ground truth labels, there is a risk of noisy annotations. It would be better to investigate the robustness of LLM-GNN to noisy annotations and potential strategies for mitigating their effects.

Response Thanks for your great comments. We'll first summarize our findings and show details in the next paragraph. (1) We find that normal cross-entropy loss has a smaller overfitting on LLMs' annotations compared to random noisy labels, which is one advantage of using LLMs as the annotators; (2) We find that (graph) noisy label learning methods do not show clear improvements compared to normal cross-entropy loss in our setting. Simple normal and weighted cross-entropy loss works better.

(1) To show the characteristics of LLMs' annotations, we train GNN models with ground truth labels, LLMs' annotations, and synthetic noisy labels with the same annotation quality (for example, suppose LLMs correctly label 80% of the points, we will randomly replace the labels of 20% of the points with incorrect labels based on the ground truth labels. ) We put the training curves in Appendix J and show the maximum test accuracies and final test accuracies in the following table. We observe that, unlike synthetic noisy labels, the gap between maximum accuracy and final accuracy for LLMs is very small. On one hand, it shows one good point of LLMs' annotations that we don't use a validation set to find the stopping point. On the other, it implies that it may be difficult to design a loss function to improve the performance.

(2) We further compare the differences among different loss functions. We compare the following four baselines: random selection and normal cross-entropy loss; random selection and NCERCE loss (design for noisy label learning) [1]; RIM with normal cross-entropy loss; random selection and weighted cross entropy loss; original RIM with weighted loss. It should be noted that RIM is an active learning method designed to process synthetic noisy labels. From the table, we find that loss designed for noisy label learning doesn't present an advantage over normal and weighted cross-entropy loss. One possible reason for this behavior is that the noisy pattern of LLMs' annotations is much more complex than synthetic noisy labels (see Appendix G.1), while noisy label learning methods are usually evaluated on synthetic noisy labels. Another reason is that the number of training samples is limited so it's hard to capture the pattern of noisy annotations. Compared to noisy label learning loss, we find that a simple weighted cross entropy loss ("WE" in the table) works well in most cases.

[1] Ma X, Huang H, Wang Y, et al. Normalized loss functions for deep learning with noisy labels[C]//International conference on machine learning. PMLR, 2020: 6543-6553.

Q2: LLM-GNN's performance is demonstrated on a specific dataset (OGBN-PRODUCTS), and while it achieves impressive results, its generalizability to other datasets or domains is not thoroughly explored in the paper. The effectiveness of the approach in different scenarios and with various types of graphs should be investigated to assess its broader applicability.

Response Thanks for your great comments. We are not quite sure whether we exactly understand some details of your remarks. We will first retell the key points and answer them accordingly.

We think you make the following key points: (1) Only the results of OGBN-Products are demonstrated. (2) Whether the proposed pipeline can be extended to more tasks like graph classification and more types of graphs beyond text-attributed graphs needs to be explored.

For the first point, we admit we focus on the performance of OGBN-Products in the abstract and introduction part, which may lead to confusion. We also test five other datasets in the experiment part. For the second question, we think our pipelines have the potential to be applied to more types of graphs and more types of tasks.

(1) Other than the performance on OGBN-PRODUCTS, we also show the performance of our methods on other datasets like Cora, CiteSeer, Pubmed, WikiCS, and Arxiv in the paper, and we will show them in the following table. Specifically, Cora, CiteSeer, and Arxiv are about papers from the computer science domain. Pubmed is about the papers from the medical domain. WikiCS is about the Wikipedia page. We have included multiple datasets from different domains to show the general applicability of our methods. Moreover, our pipeline shows consistent effectiveness across these datasets. In the following table, we demonstrate that our pipelines can get good performance with a much lower cost (both money and time) and scale to large graphs.

(2) We admit that our paper focuses on the node classification tasks on the text-attributed graphs. However, we think our pipeline has the potential to be extended to more types of tasks and graphs.

For different tasks like graph classification and knowledge graph QA, [1] does some exploration, and LLMs also show promising zero-shot capabilities. As a result, we may extend our pipelines to these tasks with some small modifications to the architectures.

For more types of graphs, in a recent paper [2], the authors introduced a prompt-based method that aligns all types of node features through natural language, allowing large language models to naturally process problems in the textual domain with notable success. For discrete features like bag of words, [3] also propose a potential way to let large language models handle them by converting the original features into clustering centers. These methods may be integrated with our work and extend our methods to more types of graphs. We think you make two valuable points for future directions.

[1] Guo J, Du L, Liu H. GPT4Graph: Can Large Language Models Understand Graph Structured Data? An Empirical Evaluation and Benchmarking[J]. arXiv preprint arXiv:2305.15066, 2023.

[2] Liu H, Feng J, Kong L, et al. One for All: Towards Training One Graph Model for All Classification Tasks[J]. arXiv preprint arXiv:2310.00149, 2023.

[3] Zhao J, Zhuo L, Shen Y, et al. Graphtext: Graph reasoning in text space[J]. arXiv preprint arXiv:2310.01089, 2023.

Q3: There could be better with a detailed comparison from the economic perspective.

Response Thanks for your great comments, which inspire us to further compare both the economy and performance of our pipelines. Compared to other baselines, our methods can achieve promising performance with very low costs.

We use the following data to validate our claims. The first table demonstrates the performance of each pipeline. For LLMGNN and GNN, we deliberately let them produce similar performances to conduct a fair cost comparison. For Pubmed*, we demonstrate the performance of LLMGNN with a larger budget since the performance of LLMs-as-Predictors is superior on this dataset. We can improve the performance of LLMGNN by slightly increasing the costs.

The second table compares the number of training samples needed to achieve the performance in the first table. Here, PL stands for pseudo labels generated by LLMs. GT stands for ground truth labels.

From these two tables, we can see that LLMGNN significantly reduces the costs compared to LLMs-as-Predictors. Comparing GNN with LLMGNN, the costs of ground truth labels are hard to estimate. From a recent study [1], it shows that the annotations generated by ChatGPT are more than 20 times cheaper than human annotators. If we use this ratio, we can see that although LLMGNN requires more pseudo labels, the overall costs are still much cheaper than traditional GNN-based pipelines.

[1] Gilardi F, Alizadeh M, Kubli M. Chatgpt outperforms crowd-workers for text-annotation tasks[J]. arXiv preprint arXiv:2303.15056, 2023.

Thanks for your response which solved all of my concerns. I have raised my score accordingly.

Q5: I also wonder how different LLMs affect the model (the reported results are all based on GPT-3.5-turbo).

Response Thanks for your great comments. We have added experiments using other large language models like GPT-4. Surprisingly, we find that GPT4 brings no clear improvement compared to the results of GPT-3.5-turbo.

We show the results in the following table. We can see that GPT4 only achieves clear improvements in DA-GraphPart on Cora. We appreciate your questions to help us have an exploration for this interesting phenomenon.

One possible reason for this phenomenon is that the primary area where GPT-4 surpasses GPT-3.5 is in complex reasoning capabilities. However, in tasks such as text classification, the difference between the two is not very significant. This phenomenon has also been observed in text classification tasks in other fields, such as financial text classification [1].

[1] Loukas L, Stogiannidis I, Malakasiotis P, et al. Breaking the bank with chatgpt: Few-shot text classification for finance[J]. arXiv preprint arXiv:2308.14634, 2023.

Q2: The authors should more explicitly mention that their proposed method only works for text-attributed graphs rather than any general graph, e.g., in the abstract and introduction. Otherwise, the paper may have overclaiming issues.

Response Thanks for your great comments. We agree that our pipelines focus more on text-attributed graphs, while our methods have the potential to be applied to more types of graphs. We add discussions on this problem in the revision.

Currently, for the task of node classification, most datasets feature text-type nodes, and thus our method is well-suited for the majority of datasets and tasks. Furthermore, our approach has the potential to be extended to more graphs with different types of node features. In a recent paper [1], the authors introduced a prompt-based method that aligns all types of node features through natural language, allowing large language models to naturally process problems in the textual domain with notable success. For discrete features like bag of words, [2] also propose a potential way to let large language models handle them by converting the original features into clustering centers. These methods may be integrated with our work and extend our methods to more types of graphs. Your point is incredibly insightful and valuable, and we consider it an excellent extension of our work. In our revised version, we have included a discussion about this issue in both the abstract, introduction, and preliminaries.

[1] Liu H, Feng J, Kong L, et al. One for All: Towards Training One Graph Model for All Classification Tasks[J]. arXiv preprint arXiv:2310.00149, 2023.

[2] Zhao J, Zhuo L, Shen Y, et al. Graphtext: Graph reasoning in text space[J]. arXiv preprint arXiv:2310.01089, 2023.

Q1: Though I acknowledge that the proposed method is a valid solution, the technical contribution of the paper is somewhat limited, especially considering that the three major components are largely based on heuristic observations, and the rest are based on existing LLMs and GNNs. It would make the paper stronger if some theoretical analyses could be provided for the proposed components.

Q1 (2) Besides LLMGNN we propose, we want to mention that as far as we know, we are the first to study how to improve the annotations generated by a real-world annotator. Previous works like [1] all use synthetic noisy labels, whose patterns are much simpler.

[1] Zhang W, Wang Y, You Z, et al. Rim: Reliable influence-based active learning on graphs[J]. Advances in Neural Information Processing Systems, 2021, 34: 27978-27990.

Q1: Though I acknowledge that the proposed method is a valid solution, the technical contribution of the paper is somewhat limited, especially considering that the three major components are largely based on heuristic observations, and the rest are based on existing LLMs and GNNs. It would make the paper stronger if some theoretical analyses could be provided for the proposed components.

Response Thanks for your great comments. We first summarize our responses and then provide more details. (1) We agree that theoretical analysis is important to support the effectiveness of our proposed methods. We use theoretical analysis to show why difficulty-aware selection can get high-quality annotations. We also add the following part into Appendix L. (2) We would like to strengthen our contributions: 1. We propose a flexible pipeline LLMGNN which demonstrates effectiveness with low costs; 2. We propose three effective strategies: difficulty-aware selection, confidence-aware annotations, and post-filtering to further enhance the effectiveness of LLMGNN.

Q1 (1) One important heuristic used in our pipeline is difficulty-aware selection. Here, we analyze why it can enhance the annotation quality of LLMs.

Given the parameter distribution of LLMs $\mathcal{Q}$, the parameter distribution of encoder $\mathcal{P}$ (SBERT), we assume ground truth $Y\_L\in \mathcal{R}^{N \times M}$, pseudo label $Y \in \mathcal{R}^{N \times M}$, and node features encoded by the encoder $X\in \mathcal{R}^{N \times d}$. Here, $M$ denotes the number of classes and $d$ denotes the hidden dimension. Our objective is to maximize the accuracy of annotations, which is thus to minimize the discrepancies between $Y$ and $Y\_{L}$.

where $(n\_{1},...,n\_{k})$ is the index of the $k$ selected nodes, and $f(Y)$ is defined as follows, where ${x}^{n\_{i}}$ represents the feature of node ${n\_{i}}$, while $x\_{L}^{n\_{i}}$ represents the unknown latent embedding of node ${n\_{i}}$.

Since we only have access to the node features $X$ generated by encoder $\theta$, we need to make a connection between $\mathcal{Q}$ and $\mathcal{P}$.

Lemma 1. For any annotation $y$ generated by LLMs, $\mathbb{E}\_{\theta\sim \mathcal{Q}}f(y)\leq \log \mathbb{E}\_{\theta\sim \mathcal{P}} \exp(f(y))+KL( \mathcal{Q} \| \mathcal{P})$

Proof:

Then, we transform the objective into

Assuming the label distribution $\\mathcal{H}$, we further have

Assuming $X$ and $X\_{L}$ follows gaussian distribution, where $X\sim N(\mu\_{i},\sigma\_{i});X\_L\sim N(\mu\_{j},\sigma\_{j})$, then

Since $(\mu\_{j},\sigma\_{j})$ is unknown, $\mu\_{i}$ is fixed, thus we want to minimize $\sigma\_{i}$. Given an arbitrary node, $\sigma\_{i}$ can be viewed as the distance to the clustering centers. The smaller $\sigma\_{i}$ is, the smaller the corresponding $\mathrm{E}\left(\|X-X\_{L}\|^{2}\right)$ also becomes, indicating that the minimum value of $\mathbb{E}\_{\theta\sim \mathcal{Q}}f(y)$ is attained when $\sigma\_{i}$ is at its smallest. This demonstrates why nodes closer to the clustering centers are "preferred" by LLMs and thus achieve better annotation quality.

Q4: There are some missing related works regarding zero-shot node classification such as [1-2], which should be added.

Response Thanks for your great comments which inspire us to further explore the effectiveness of our paper under the zero-shot node classification setting. We first summarize our responses and explain the details. (1) We've cited those related works and add a discussion section. However, there are some differences between the setting of label-free node classification and zero-shot node classification; (2) For zero-shot node classification, our methods can still achieve promising performance.

(1) The setting for label-free node classification involves training a GNN using annotations generated by LLMs, without any pre-existing labels. In contrast, the zero-shot node classification setting in these two papers involves training a model with training data for training classes (for example, A, B, C), which can then be generalized to new classes in the testing phase (for example, D, E, F). In practice, our label-free methods can be used to solve zero-shot node classification by directly annotating the new classes. However, those zero-shot node classification methods can not be directly applied to label-free node classification.

(2) In the following table, we compare the performance of LLMGNN to zero-shot node classification baselines DGPN [1], DBiGCN [2], and GraphCEN [3] in zero-shot node classification settings. For LLMGNN, we use the featprop + Post Selection with 20% pruning ratio and 140 annotation budget. We take the experimental settings from [3].

We can see that our methods can consistently beat all those baselines under the zero-shot node classification setting. We appreciate your comments which inspire us to demonstrate the strong capability of our methods in traditional zero-shot node classification problems.

[1] Wang Z, Wang J, Guo Y, et al. Zero-shot node classification with decomposed graph prototype network[C]//Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2021: 1769-1779.

[2] Yue Q, Liang J, Cui J, et al. Dual Bidirectional Graph Convolutional Networks for Zero-shot Node Classification[C]//Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2022: 2408-2417.

[3] Ju W, Qin Y, Yi S, et al. Zero-shot Node Classification with Graph Contrastive Embedding Network[J]. Transactions on Machine Learning Research, 2023.

Q3: In generating the initial node labels using LLMs, it seems that only the feature information is utilized and no structure is considered. Since it is well-known in the graph machine learning literature that both features and structures greatly affect the node labels, there exists a large room for improvement.

Response Thanks for your great comments, which inspire us to have a further look at structure-aware prompts. We find that incorporating structural information in the annotation prompt can not largely improve performance with the current prompt design.

We demonstrate both the accuracy of LLMs' annotations (the ratio of LLMs' annotations matching ground truth labels) and the accuracy of LLM-GNN in the following table. "No struct" means the "hybrid" (combining both "TopK" and "Most Voting") prompt strategy used in our papers. "Struct" is the new version that combines the "neighbor summarization" strategy in [1] and the "hybrid" prompt strategy.

We observe that:

1. Incorporating structural information in the annotation prompts can not improve the performance, and may affect the effectiveness of post-selection. For example, the annotation quality even drops after adding structural information.
2. Incorporating structural information will increase the costs of prompts since the length of prompts becomes longer.

Based on these points, we think that using text-only annotation prompts for LLMs is appropriate in the current stage. However, we also admit that how incorporating structural information in LLMs is a valuable future direction.

[1] Chen Z, Mao H, Li H, et al. Exploring the potential of large language models (llms) in learning on graphs[J]. arXiv preprint arXiv:2307.03393, 2023.

W1: Difficulty aware (DA) selection: According to the paper, LLMs annotation quality degrades when they have to annotate nodes that are away from the centers. It implies that the LLMs annotation quality would suffer in the case of the diverse nodes (away from the center). However, GNNs accuracy will only improve if the nodes are diverse. Hence, difficulty aware selection i.e. use of c-density might not always help and, in fact, it may hinder in some cases. This is also evident from the results shown in Table 2 : Active_Selection_Methods and the corresponding DA-Active_Selection_Methods show similar performance on average across different techniques (i.e., selection methods). Moreover, for at least 50% of the cases, the DA-method (row 2) performs poorly compared to the corresponding active learning method (row 1).

Response Thanks for your great comments. We want to first summarize our responses and then explain them in details. Applying the original DA may lead to clear performance degradation, which is due to the class imbalance problem by a small clustering center number $K$ on datasets like Pubmed. This problem can be solved by increasing $K$ to balance the diversity and annotation quality.

The reason for DA's bad performance on Pubmed is when performing K-means clustering with $C-Density$, the number of clustering centers $K$ is fixed as the number of classes. This approach can achieve good labeling quality in most cases. On datasets like Pubmed where the number of classes is small (only 3), the points selected by $C-Density$ tend to cause class imbalance (the annotation quality is still very good). To solve this problem, we find that by setting a larger value for the number of clustering centers $K$, the new $C-Density^{\prime}$ obtained can effectively solve this problem. We show the results in the following table. $k$ means the number of clustering centers to calculate $C-Density^{\prime}$, $C$ means the number of classes, and budgets means the number of selected annotated nodes. We use AGE as a demonstration to show the idea. From the results, we can see that setting a larger $K$ can improve the performance of DA on Pubmed.

In the following table, we demonstrate that DA can improve the performance of diversity-aware selection methods most of the time.

We can see for 10/12 cases, DA significantly improves the performance. In other cases, it also achieves comparable performance.

W3: The methods are heuristics and do not have theoretical evidence.

Response Thanks for your great comments. We agree that theoretical analysis is important to support the effectiveness of our proposed methods. We show why Difficulty-aware selection can improve the annotation quality of LLMs. We also add the following part into Appendix L.

Given the parameter distribution of LLMs $\mathcal{Q}$, the parameter distribution of encoder $\mathcal{P}$ (SBERT), we assume ground truth $Y_L\in \mathcal{R}^{N \times M}$, pseudo label $Y \in \mathcal{R}^{N \times M}$, and node features encoded by the encoder $X\in \mathcal{R}^{N \times d}$. Here, $M$ denotes the number of classes and $d$ denotes the hidden dimension. Our objective is to maximize the accuracy of annotations, which is thus to minimize the discrepancies between $Y$ and $Y_{L}$.

where $(n_{1},...,n_{k})$ is the index of the $k$ selected nodes, and $f(Y)$ is defined as follows, where ${x}^{n_{i}}$ represents the feature of node ${n_{i}}$, while $x_{L}^{n_{i}}$ represents the unknown latent embedding of node ${n_{i}}$.

Since we only have access to the node features $X$ generated by encoder $\theta$, we need to make a connection between $\mathcal{Q}$ and $\mathcal{P}$.

Lemma 1. For any annotation $y$ generated by LLMs, $\\mathbb{E}\_{\\theta\\sim \\mathcal{Q}}f(y)\\leq \\log \mathbb{E}_{\\theta\\sim \\mathcal{P}} \\exp(f(y))+KL( \\mathcal{Q} \\| \\mathcal{P})$

Proof:

Then, we transform the objective into

Assuming the label distribution $\\mathcal{H}$, we further have

Assuming $X$ and $X_{L}$ follows gaussian distribution, where $X\sim N(\mu_{i},\sigma_{i});X_L\sim N(\mu_{j},\sigma_{j})$, then

Since $(\mu_{j},\sigma_{j})$ is unknown, $\mu_{i}$ is fixed, thus we want to minimize $\sigma_{i}$. Given an arbitrary node, $\sigma_{i}$ can be viewed as the distance to the clustering centers. The smaller $\sigma_{i}$ is, the smaller the corresponding $\mathrm{E}\left(\|X-X_{L}\|^{2}\right)$ also becomes, indicating that the minimum value of $\\mathbb{E}\_{ \\theta \\sim \\mathcal{Q}} f(y)$ is attained when $\\sigma_{i}$ is at its smallest. This demonstrates why nodes closer to the clustering centers are "preferred" by LLMs and thus achieve better annotation quality.

Q4: Minor Questions: 1. Detail explanation on $f_{act}(vi)$ is missing; 2. It is mentioned in page 5 that the detailed descriptions and full prompt examples are shown in Appendix D. However, it is missing any detailed descriptions.

Response Thanks for your great comments. $f_{act}(vi)$ can be any traditional graph active learning score functions, and we add a detailed explanation in the revision. Furthermore, we add more examples of the prompt and detailed descriptions in the appendix.

Q5: Typos: Page 7: (4) Combing - supposed to be combining?; Page 3: GNN modelson- models on (space missing)

Response Thanks for your great comments. We've fixed these typos in the revision.

Q3: PS-DA-methods have not performed well in most cases compared to DA-methods (Table 2). Any insights on this can help in understanding these steps better.

Response Thanks for your great comments. For the effectiveness of applying PS and DA together, we agree that directly combining them without tuning can't outperform DA in most cases. If we tune the hyper-parameters, we can get good performance. However, PS-DA introduces numerous hyper-parameters which makes tuning complicated. We further show using LLMs' confidence as the weight for loss function is a better way to be combined with DA methods.

• The reason that directly applying PS together with DA could lead to a performance decrease is that we don't tune the parameters. If we do not tune the hyperparameters, in some cases where DA has already selected a good candidate set, further adopting PS to remove some nodes will degrade the performance.
• If we can tune the parameters, both DA and PS can be viewed as a special case of the more general PS-DA. However, in practice, PS-DA introduces many hyper-parameters, which makes it infeasible to tune the parameters without a validation set.
• Instead of combining PS and DA in a "hard" way by filtering nodes with LLMs' confidence, we find that a "soft" way which uses the confidence as the weight for loss function works better. In the following table, we show that compared to PS-DA, combining DA with weighted loss can effectively enhance the performance of DA. Compared to PS-DA, our new approach shares the same philosophy. PS-DA can be viewed as a hard selection process while weighted cross-entropy loss can be viewed as a soft selection. We find that such kind of soft selection can effectively solve the problem of hyper-parameter tuning. The full table can be viewed in revision. "-W" means the weighted loss. We find that using a weighted loss combination has two advantages: (1) no need to tune the hyper-parameters like PS-DA; and (2) better performance.

Q2: Providing the number of nodes selected in each step in the experiment will also help in understanding the effects of the steps. For instance, could you please provide the number of selected nodes during active selection and DA-active selection? Also, what is the pruning ratio when applying PS? All this information can help in understanding the efficacy of the steps.

Response Thanks for your great comments. For the number of nodes selected during active selection and DA-active selection, we adopt the same number of samples for each dataset as follows: the number of selected nodes is equal to the number of classes multiplied by $20$. For example, for Arxiv with $40$ classes, we select a total of $800$ nodes. For the pruning ratio, we drop 20% of the selected nodes for every dataset except Pubmed. For Pubmed, we drop 10% of the selected nodes because LLMs-as-predictors can achieve high accuracy on this dataset.

W2: Though the accuracy from the LLM-GNN model is good (and of course, the model is efficient), it couldn't outperform LLM as a predictor (Table 3).

Response Thanks for your great comments. Despite the superior performance of LLMs-as-Predictors in some datasets, it's not practical to be deployed in real-world scenarios because of efficiency. Our methods can scale to large-scale graphs with promising performance.

Although the original LLMs-as-predictors can achieve good performance, it's not practical to be applied to even medium-scale datasets like Arxiv. In the original paper [1], it's only applied to a small subset of the datasets to evaluate the performance. Taking Arxiv as an example, considering OpenAI's current limit of 60 accesses per minute, completing all predictions on Arxiv would take 44 hours, whereas LLMGNN only requires a few minutes. Our methods extend the original pipeline to real-world large-scale datasets.

Then, we compare the performance and the economic costs of these two pipelines to further demonstrate the effectiveness of our methods in the following two tables. The cost is estimated in units of dollars. On datasets like Citeseer, and Products, the gap between LLMGNN and LLMs-as-predictors is very small. On Cora, LLMGNN can even outperform LLMs-as-predictors. Only on PubMed, there's a relatively large gap between LLM-GNN and PubMed. It's because of LLMs' superior performance on this dataset, which is related to the shortcut prediction in attribute [1]. When LLMs' performance is superior, we can also add more budgets to further improve the performance of LLMGNN. For example, if we raise the budget in PubMed from 0.05 to 0.25, we can raise the accuracy of LLMGNN from 74.98 to 82.

In summary, the economic costs and efficiency of our methods are consistently much lower than LLMs-as-predictors. It's a valuable future direction to study how to further improve the performance of LLMGNN.

[1] Chen Z, Mao H, Li H, et al. Exploring the potential of large language models (llms) in learning on graphs[J]. arXiv preprint arXiv:2307.03393, 2023.

Q1: Post-filtering (PS): How useful are the confidence scores generated by the LLMs (Appendix F.1, table 6)? Showing the mean and variance of the confidence score for each dataset may help in understanding its impact on performance.

Response Thanks for your great comments. We show the mean and variance of confidence scores for each dataset in the following table and also put a figure for the Cora dataset in the revision (Appendix H). The confidence generated by LLMs is well calibrated and can be used to reflect the reliability of annotations.

The following table demonstrates the Mean, Variance, Mode, Maximum, and Minimum of the confidence for each dataset.

For the confidence calibration plot, we find that the hybrid prompt we use can generate accurate and diverse confidence scores, which demonstrates effectiveness.

Q2: In Table 6, is it realistic to use labels in 1-shot example when we are considering the "Label-free" setting? Furthermore, how are the confidence scores determined in that example? More detailed explanation is required.

Response Thanks for your great question. We agree that the original version may lead to confusion since we do not clarify that we only use "1-shot" prompts to demonstrate there's no clear gap between "zero-shot" and "1-shot" and we use the "zero-shot" prompt in all experiments of Section 4.

In the revision, we have updated the prompt example of zero-shot case in the appendix E. We agree that "1-shot" may be not appropriate for the "label-free" setting. However, we only use 1-shot prompts in Table 1 to demonstrate that there's no clear gap between the performance of zero-shot and 1-shot prompts. For the experiment part, we only focus on zero-shot prompts. For the generation of multiple labels and their corresponding confidence scores in the 1-shot case, we leverage LLMs to automatically generate both the topK predictions and confidence scores according to the philosophy of [1]. We've revised the corresponding part of the paper.

[1] Zhang Z, Zhang A, Li M, et al. Automatic chain of thought prompting in large language models[J]. arXiv preprint arXiv:2210.03493, 2022.

Q4: While there is an example in Figure 5, it would be beneficial to demonstrate on different datasets that LLM-GNN achieves competitive performance with GNNs trained on ground truth labeled data, while significantly reducing costs.

Response Thanks for your great comments which help us to refine the details of our paper. Compared to traditional GNN and LLMs-as-Predictors, our methods can achieve promising performance with much lower costs.

We demonstrate the economic comparison among LLM-GNN, LLMs-as-Predictors, and traditional pipelines to show the effectiveness. The first table demonstrates the performance of each pipeline. For LLMGNN and GNN, we deliberately let them produce a similar performance to conduct a fair cost comparison. For Pubmed*, we demonstrate the performance of LLMGNN with a larger budget since the performance of LLMs-as-Predictors is superior on this dataset, and increasing a little budget can improve the performance a lot.

The second table compares the number of training samples needed to achieve the performance in the first table. Here, PL stands for pseudo labels generated by LLMs. GT stands for ground truth labels.

From these two tables, we can see that LLMGNN significantly reduces the costs compared to LLMs-as-Predictors. A recent study [1], shows that the annotations generated by ChatGPT are more than 20 times cheaper than human annotators. If we use this ratio, we can see that although LLMGNN requires more pseudo labels, the overall costs are still much cheaper than traditional GNN-based pipelines. Using the OGBN-Products dataset as an example, if we estimate the cost, using LLMs-as-Predictors would incur an expense of 1572 dollars, while manual annotation would require at least 20 dollars (assuming each sample has 3 workers on a crowdsourcing platform), and it would also necessitate a considerable amount of waiting time. In contrast, using LLMGNN can complete annotation and prediction in a matter of minutes at a cost of less than one dollar.

[1] Gilardi F, Alizadeh M, Kubli M. Chatgpt outperforms crowd-workers for text-annotation tasks[J]. arXiv preprint arXiv:2303.15056, 2023.

Q3: In Figure 4, why do some methods show an increase in performance after a budget of 70, followed by a sharp decline? Could this be attributed to class imbalance issues during the selection process?

Q3 (3) The performance degradation is mainly due to the annotation quality drop with more budgets. For random selection, the drop is not obvious because of random sampling. For active selection like PS-Featprop, the annotation quality drops with more budgets since we fixed the dropping ratio to 20%. With more budgets, there will be correspondingly more wrong annotations.

In the following table, we demonstrate the annotation quality and entropy given by PS-Featprop. Entropy can be used to measure the imbalance of selected labels.

We can see that the performance drop is more correlated with the annotation quality drop. To mitigate this problem, one possible solution is to adaptively increase the dropping rate of PS-Featprop, and we may increase the performance when there's more noisy labels. This method can only mitigate the issue rather than resolve it completely. We acknowledge that exploring how to further enhance the performance of LLMGNN under high labeling rates is a very meaningful direction for future work.

Another potential solution is to further improve the zero-shot performance of LLMs, like using more complex prompts or agents-related techniques [2]. Using Pubmed as an example, since LLMs can achieve near 90% accuracy on this dataset, increasing the budget can effectively improve the performance of LLMGNN. This shows that if LLMs can get good annotation quality, directly using randomly selected annotations can already get good performance.

[2] Xi Z, Chen W, Guo X, et al. The rise and potential of large language model based agents: A survey[J]. arXiv preprint arXiv:2309.07864, 2023.

Q3: In Figure 4, why do some methods show an increase in performance after a budget of 70, followed by a sharp decline? Could this be attributed to class imbalance issues during the selection process?

Response Thanks for your great comment which points out an important phenomenon in our experiments. We first summarize our responses and then provide more details for the responses. (1) For ground truth labels, increasing the budget can usually improve the performance. However, for noisy labels, it's reasonable that increasing the budget will lead to a performance drop since labels are noisy [1]. (2) The original scales of the x-axis and y-axis are inappropriate, making the declining phenomenon appear very pronounced, when in fact the magnitude of the decline is relatively small. (3) The main reason for the performance degradation is attributed to the decline in annotation quality, rather than the class imbalance.

Q3 (1) Since we use LLMs as the annotators, two factors affect the final performance of LLMGNN: number of annotated samples, and annotation quality. For clean labels, usually, the performance will gradually increase with more budgets. However, for noisy labels, it's reasonable that performance drops with more labels since the overall annotation quality degrades [1].

Q3 (2) In the original figure, we use the x ticks [70, 280, 1120, 2240] with non-equal distance between every tick. This unreasonable design results in a visually pronounced appearance of the decline. We have redrawn the chart in the revision, and the corresponding results are presented in the table below, showing that the decline's magnitude is not particularly significant.

[1] Song H, Kim M, Park D, et al. Learning from noisy labels with deep neural networks: A survey[J]. IEEE Transactions on Neural Networks and Learning Systems, 2022.

R1-2 (2) For the effectiveness of applying PS and DA together, we agree that directly combining them without tuning can't outperform DA in most cases. While hyper-parameter tuning can help, we also explore using LLMs' confidence as the weight for weighted cross-entropy loss that can effectively combine DA with the information generated by LLMs. we will first explain why PS-DA doesn't work.

• The reason that directly applying PS together with DA could lead to a performance decrease is that we don't tune the parameters. If we do not tune the hyperparameters, in some cases where DA has already selected a good candidate set, further adopting PS to remove some nodes will degrade the performance.
• In our experiments, the purpose of demonstrating PS-DA is to compare it with the use of PS and DA individually and to explain that without hyperparameter tuning on a validation set, using just one of PS or DA is a better choice. Among them, PS often achieves better quality because it can utilize the confidence information from LLMs, but it may lose some sample points due to filtering. Since DA does not need to use the confidence information from LLMs, it can employ a regular zero-shot prompt instead of a consistency prompt, making it relatively lower in cost. Choosing between DA and PS can be seen as a trade-off in terms of cost and the number of samples. DA and PS can surpass the original baseline in most cases.
• In the following table, we show that compared to PS-DA, combining DA with weighted loss can effectively enhance the performance of DA. Compared to PS-DA, our new approach shares the same philosophy. PS-DA can be viewed as a hard selection process while weighted cross-entropy loss can be viewed as a soft selection. We find that such kind of soft selection can effectively solve the problem of hyper-parameter tuning. The full table can be viewed in revision. "-W" means the weighted loss. We find that using a weighted loss combination has two advantages: (1) no need to tune the hyper-parameters like PS-DA; and (2) better performance.

In the following table, we demonstrate that DA can improve the performance of diversity-aware selection methods most of the time.

We can see for 10/12 cases, DA leads to a significant gain. In other cases, it can also achieve comparable performance.

Q1-2: Even in Table 2, when DA is added to traditional graph active selection and when DA is added to the use of PS, there are many cases where performance actually decreases. I acknowledge that tuning was not performed, but there are still too many cases where performance declines. While the authors said that grid search would improve the performance in a specific case, it seems necessary to perform more tuning across a broader range of cases to clearly demonstrate the effectiveness of DA.

R1-2: Thanks for your great question. We first summarize our responses and then provide more details for the responses. (1) Applying the original DA may lead to clear performance degrade, which is due to the class imbalance problem by a small clustering center number $K$ on datasets like Pubmed. This problem can be solved by increasing $K$ to balance the diversity and annotation quality. (2) The original way to apply PS and DA together can not outperform DA. Although hyper-parameter tuning can definitely help, we agree that hyper-parameter tuning could not be flexible in practice. Therefore to further explore the potential of combining DA and PS, we introduce a simple way to utilize the confidence as weight for loss functions and we find that the new variants can outperform DA in most cases without any hyper-parameter tuning.

R1-2 (1) For the effectiveness of DA, we admit that sometimes DA will degrade the performance, especially on the Pubmed dataset. The reason for this phenomenon is that when performing K-means clustering with $C-Density$, the number of clustering centers $K$ is fixed as the number of classes for labels. This approach can achieve good labeling quality in most cases. On datasets like Pubmed where the number of classes is small (only 3), the points selected by $C-Density$ tend to cause class imbalance (the annotation quality is still very good). To solve this problem, we find that by setting a larger value for the number of clustering center $K$, the new $C-Density^{\prime}$ obtained can effectively solve this problem. We show the results in the following table. $k$ means the number of clustering centers to caluclate $C-Density^{\prime}$, $C$ means the number of classes, budgets means the number of selected annotated nodes. We use AGE as a demonstration to show the idea. From the results, we can see that setting a larger $K$ can improve the performance of DA on Pubmed.

Q1-1: In Figure 2 or Figure 12, the trend of decreasing accuracy as the distance between nodes and cluster centers increases seems somewhat weak in average accuracy.

R1-1: Thanks for your great question which inspires us to explain the details of our paper. There are line plots and bar plots in the figure. The changing trend is reflected by the line plot, which shows a clear declining trend while the bar plot only provides auxiliary information and may not affect the conclusion.

For Figure 2 and Figure 12, the bar shows the average values within each region, while the blue line represents the overall mean accuracy across different regions. Although the accuracy inside each bar doesn't show a clear trend, the line plot shows a clear decreasing trend across different regions. The nodes to be annotated are selected based on the accuracy across different groups (the blue line) rather than the accuracy inside each group (the bar). As a result, the declining trend is clear and difficulty-aware selection can effectively select those nodes with high-quality annotations.