When Do LLMs Help With Node Classification? A Comprehensive Analysis

We sincerely thank the reviewer for your valuable feedback.

---

Definition of Reasoning, LLM-as-Reasoner and LLM-as-Predictor, and Graph Foundation Models (GFM)

Sorry for any confusion on the terminologies. The categorization of LLM-as-Enhancer, LLM-as-Predictor, and GFMs is adopted from the survey paper [Ref 1]. Specifically, our use of LLM-as-Reasoner corresponds to the explanation-based LLM-as-Enhancer paradigm described in that survey. In this paradigm, LLMs generate predictions accompanied by explanations, requiring reasoning capabilities and general knowledge, as shown in TAPE [Ref 2].

The key difference between LLM-as-Reasoner and LLM-as-Predictor lies in whether we directly use the predictions generated by the LLM [Ref 1]. In the LLM-as-Predictor paradigm, the prediction is the label directly generated by the LLM. In contrast, in the LLM-as-Reasoner paradigm, although the LLM provides an initial prediction, the focus is on its reasoning process, i.e., the explanation text it generates. This reasoning is then used for augmentation, and the final prediction depends on other components like GNNs.

For GFMs, according to existing survey papers [Ref 1, 3], they are defined as models pre-trained on extensive graph corpora. These models utilize large-scale training data along with specifically designed techniques to achieve strong generalization capabilities across tasks and domains.

Takeaways 2, 3, 9

We thank the reviewer for the opportunity to further clarify our contributions. While Takeaways 2 and 3 may seem intuitive, they have not been explicitly discussed or established in prior research on LLMs for graphs. Our work provides empirical evidence and theoretical understanding to support these observations, contributing novel insights to the field.

In our paper, we do not have a Takeaway 9, so we assume the reviewer is referring to Takeaway 8. Takeaway 8 does not state that LLMs can perform better if the graph is heterophilic compared to a homophilic counterpart. Takeaway 8 states that in the LLM-as-Encoder paradigm, LLMs perform significantly better than smaller LMs on heterophilic graphs, while achieving only similar performance on highly homophilic graphs. To further support Takeaway 8, we have also added more explanations and four heterophilic datasets and they are given in Question 3 of Reviewer HQNY.

LLM API inference costs

We thank the reviewer for the valuable suggestion. Due to space limits, we use the Photo dataset as an example, reporting average token consumption and corresponding costs (in USD) for each prompt template across two LLMs. Completing four test datasets requires 59,259,894 tokens, costing ~ 19.5USD for GPT-4o-mini and 35.6USD for DeepSeek-V3. We will extend this discussion to all datasets in the revised manuscript.

Larger evaluation datasets

Thank you for raising the concern regarding the size of the evaluation datasets. We would like to clarify that the arXiv dataset consists of 169,343 nodes, exceeding the stated threshold of 100,000 nodes. Following standard practices like TAPE [Ref 2], we use the arXiv dataset primarily in the supervised setting.

We have conducted experiments on a larger dataset, ogbn-products, which contains 2,449,029 nodes and 123,718,152 edges. Due to space limitations, we invite you to refer to our response to Question 1 of Reviewer 4MPh for detailed results and discussion.

Biased label annotation on arXiv

While paper categorization could align with a multi-label classification task, existing research on node classification in Academic Networks consistently adopts a single-label classification approach. Specifically, the arXiv dataset [Ref 4] assigns each paper to its primary computer science sub-category, a setting widely accepted in prior works. Modeling this task as multi-label classification is promising but currently infeasible due to the lack of datasets with multi-label annotations. Therefore, our methodology and evaluation adhere to established practices and are appropriate and reasonable.

Lack of text classification baselines

We do include text classification baselines in Table 1, i.e., two fine-tuned LMs of different scales (SentenceBERT-66M and RoBERTa-355M).

---

[Ref 1] "A Survey of Graph Meets Large Language Model: Progress and Future Directions." In IJCAI, 2024.

[Ref 2] "Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learning." In ICLR, 2024.

[Ref 3] "Graph Foundation Models: Concepts, Opportunities and Challenges." In TPAMI, 2024.

[Ref 4] "Open Graph Benchmark: Datasets for Machine Learning on Graphs." In arXiv preprint, 2020.

We sincerely thank the reviewer for your insightful and constructive feedbacks.

---

E1: Experiments on KGs

We thank the reviewer for suggesting the inclusion of Knowledge Graphs. However, datasets like FB15k-237 and WN18RR primarily focus on the Link Prediction task, whereas most LLM-based algorithms are designed for node classification. Adapting these algorithms to link prediction lies beyond the scope of LLMNodeBed, as our benchmark specifically targets node classification, and the baselines we evaluated are tailored for this.

Weakness 1: GLBench

Thank you for the opportunity to clarify the differences between GLBench and our LLMNodeBed. We acknowledge GLBench’s pioneering contribution as the first comprehensive benchmark for evaluating GraphLLM methods, and it inspired aspects of our work.

While GLBench focuses on providing an overall performance leaderboard, LLMNodeBed extends this by addressing the key question: When do LLMs help with the node classification task? To answer this, we consider numerous variables (e.g., learning paradigm, LLM functional roles, LLM type and scale, dataset homophily, prompt templates, etc.) and perform extensive fine-grained analyses. These analyses aim to provide deeper insights and practical guidelines for the community. In addition, LLMNodeBed offers a systematic implementation of all baselines to ensure fair comparisons and fast deployment. For a detailed comparison, we invite the reviewer to refer to our response to Question 2 of Reviewer yVQW.

Weakness 2: Some insights are not novel

Thank you for the feedback. While some takeaways may seem intuitive, they are empirically validated through extensive experiments, providing concrete evidence rather than a vague conclusion. For example, in Takeaway 7, we show that the summary prompt is better than other prompts when integrating neighbor information.

Question 1: Guidelines for using LLMs instead of GNNs

We thank the reviewer for this insightful question. First, it's crucial to note that in industrial applications, even a marginal accuracy improvement (e.g., 2%) can yield disproportionately large profit gains given the substantial capital investments involved.

We agree that homophily alone is insufficient to fully characterize when LLMs should be used instead of GNNs. Based on our extensive studies, we provide the following guidelines:

• Zero-shot settings: Stronger LLMs can leverage direct inference to achieve satisfactory performance.
• Supervised settings: Beyond homophily, other factors should be considered like: ratio of supervision (LLM-based algorithms provide a larger gain in semi-supervised settings), and task-feature alignment (when the classification label relies heavily on textual attributes, LLM-as-Reasoner is the best).

We acknowledge that this question remains open and are committed to uncovering more practical guidelines in the future.

Question 2: Explanation for mutual information between structure and labels

In semi-supervised learning, the available labels are a subset of those in supervised learning. Thus, we have $\mathcal{Y}^{semi} = g(\mathcal{Y}^{full})$, where $g(\cdot)$ is a pre-processing function to filter out some labels. Due to the data processing inequality, we have $I(\mathcal{E}, \mathcal{Y}^{semi}) \leq I(\mathcal{E}, \mathcal{Y}^{full})$.

Question 3: LLM-as-Encoder compared with LM-as-Encoder

The strength of LLMs lies in their ability to better utilize textual information compared to LMs. In homophilic graphs, textual and structural information often overlap, as edges are typically formed between nodes with similar text. As a result, maximizing the use of textual information may yield limited performance gains, since this information is already captured and leveraged by GNNs through the graph structure. In contrast, heterophilic graphs exhibit less overlap between textual and structural information. Therefore, more effective utilization of textual features in these graphs can result in greater performance improvements. To support this claim, we present additional experiments on heterophilic graphs, which confirm Takeaway 8.

We include the Cornell, Texas, Wisconsin, and Washington datasets, which have homophily ratios of 11.55%, 6.69%, 16.27%, and 17.07%, respectively. Using the SAGE method, we evaluate node features derived from various LMs and LLMs. The semi-supervised performance (Accuracy in %) is summarized below, revealing a performance gap of 5–17% between LLM-based encoders and their LM counterparts.

We sincerely thank the reviewer for your insightful and constructive feedbacks.

---

Question 1: Model selection and determination

Thank you for the opportunity to clarify the selection process and total number of models evaluated.

For each baseline on a specific dataset, hyper-parameters were determined based on:

• Original papers and related benchmarks: We follow settings from original papers, official Github repos, and benchmark reproductions (if applicable) to ensure fair comparisons. Detailed configurations for LLMNodeBed are provided in Appendix C.2.
• Recent research on hyper-parameter tuning: Inspired by studies (e.g., [Ref 1]), we expanded the hyper-parameter search space with techniques like selectable normalization layers and jumping knowledge for GNNs.

After performing the hyper-parameter search, we finalize the configuration for each model and fix it during subsequent experiments to ensure consistency and reproducibility.

The total number of 2,200 models is derived from all experiments described in the paper. Below is a detailed breakdown:

• Main experiments (Table 1) Semi-supervised: 11 methods × 9 datasets × 4 runs = 396 models. Adding the supervised setting (10 datasets): 11×(9+10)×4 = 836 models.
• LLM-as-Encoder (Table 3) 4 LM/LLM encoders × 3 methods × 9 datasets × 4 runs, excluding overlaps (e.g., GCN with Mistral-7B): (4x3–1)×9×4 = 396 models. Adding supervised: 396 + 440 = 836 models.
• LLM-as-Reasoner (Tables 4&5) TAPE variant with GPT-4o: 9×4 (semi-supervised) + 10×4 (supervised) = 76 models.
• LLM-as-Predictor (Table 6) 5 Additional LLM backbones × 9 datasets × 4 runs (semi-supervised) + 5×10×4 (supervised) = 380 models.

Summing all parts mentioned above, we arrive at a total of ~ 2,200 models.

Question 2: Differences with existing benchmarks

Thank you for bringing our attention to clarify the differences between LLMNodeBed and existing benchmarks. We provide a preliminary comparison in Appendix A.4 and C.3, and we expand upon it for further clarification.

The most relevant benchmarks that evaluate LLM-based methods on node classification include GLBench [Ref 2], GraphLLM [Ref 3], and CS-TAG [Ref 4]. We first summarize the motivation behind each benchmark to show the overall differences:

• LLMNodeBed: Insights on when LLMs help node classification LLMNodeBed is designed to answer this research question. Therefore, we categorize existing LLM-based methods based on LLM's functional roles, incorporates a variety of variables (learning paradigms, LLM types and scales, dataset domain and homophily, prompt templates, etc.), and provides insights and guidelines for applying LLM-based methods in different contexts.
• GLBench: Overall performance comparison between classic and LLM-based methods GLBench focuses on comparing the overall performance of both classic and LLM-based methods under consistent learning configurations and data splits.
• GraphLLM: Fine-grained analysis of LLM-as-Encoder and Reasoner methods As an earlier benchmark in the LLM4Graph domain (released in July 2023), GraphLLM explores two specific LLM-based paradigms: LLM-as-Encoder (e.g., GNNs/MLPs with LLM-generated embeddings) and LLM-as-Reasoner (TAPE). It provides preliminary findings on the potential of these methods.
• CS-TAG: Benchmark for evaluating the performance of GNN, LM, and LM+GNN on text-attributed graphs CS-TAG enhances the development of graph learning by introducing diverse datasets with text attributes. It systematically investigates the performance of GNNs, LMs, and hybrid LM+GNN methods.

The table below further provides a detailed comparison of LLMNodeBed and the aforementioned benchmarks across key dimensions, including datasets, evaluation scenarios, baselines, variables considered, and system implementation:

---

[Ref 1] "Classic GNNs are Strong Baselines: Reassessing GNNs for Node Classification." In NeurIPS, 2024.

[Ref 2] "GLBench: A Comprehensive Benchmark for Graph with Large Language Models." In NeurIPS D&B, 2024.

[Ref 3] "Exploring the Potential of Large Language Models (LLMs) in Learning on Graphs." In KDD, 2024.

[Ref 4] "A Comprehensive Study on Text-attributed Graphs: Benchmarking and Rethinking." In NeurIPS D&B, 2023.

We sincerely thank the reviewer for your insightful and constructive feedbacks.

---

Weakness 1: Missed references

Thank you for pointing out these additional related works. We have incorporated GAugLLM [Ref 1] and GRENADE [Ref 2] as baselines. Due to time constraints, we have completed experiments on the Cora, WikiCS, Instagram, and Photo datasets, and will expand the discussion to all datasets in the revised manuscript.

For GAugLLM, we use Mistral-7B as the backbone LLM. For GRENADE, we consider two different LM backbones. Based on our categorization, GRENADE falls under the Classic Methods category, as it involves aligning GNNs and LMs for joint optimization.

As shown below, GAugLLM, categorized as an LLM-as-Reasoner method, achieves performance comparable to TAPE, further supporting Takeaway 3. For GRENADE, the performance surpasses that of single fine-tuned LMs, demonstrating the effectiveness of aligning LMs with GNNs. However, it still falls short of outperforming SOTA LLM-based methods.

Weakness 2: Memory costs

Thank you for highlighting the importance of including memory costs. We present the memory costs during both training and inference stages in the supervised setting for three datasets (Cora, WikiCS, and arXiv), scaling from 2,708 to 169,343 nodes. All memory usage was measured on a single H100-80GB GPU to ensure consistency and comparability.

Training Memory Costs (in GB)

Inference Memory Costs (in GB)

Based on the above statistics, we conclude the following: (i) LLM-as-Encoder methods have memory costs comparable to classic GNNs. (ii) Fine-tuning an LM is memory-intensive, especially for larger-scale LMs. (iii) LLM-as-Predictor methods are the most resource-intensive due to the high parameterization of LLMs.

These results highlight the importance of addressing efficiency challenges for deploying LLM-based methods in practice. We will include these findings in the revised manuscript.

Question 1: Experiments on larger datasets

Thank you for raising the scalability concern regarding the datasets in our benchmark. We have supplemented experiments on the ogbn-products dataset, which contains 2,449,029 nodes and 123,718,152 edges.

Experimental Setup To evaluate LLM-based methods within one week under limited GPU resources, we use the official data splits, which include 196,165 nodes (8.03%) for training, 39,323 nodes (1.61%) for validation, and a reduced test set of 221,309 nodes (9.04%), constituting a semi-supervised setting. The original dataset provides 100-dimensional node embeddings, which are directly adopted as shallow embeddings for classic GNNs.

Compared Methods Given resource and time constraints, we compare classic methods, LLM-as-Encoder method (GCN using Qwen-3B-derived embeddings), and LLM-as-Predictor method LLaGA. Other methods, such as TAPE, are excluded due to the high cost of generating reasoning texts for the entire dataset via APIs (e.g., ~ 500 USD for GPT-4o-mini). Similarly, $\text{LLM}_{\text{IT}}$ is omitted as it would require ~ 4 days for instruction tuning.

Results (%) LLM-based methods outperform classic methods by 5%.

---

[Ref 1] "GAugLLM: Improving Graph Contrastive Learning for Text-Attributed Graphs with Large Language Models." In KDD, 2024.

[Ref 2] "GRENADE: Graph-Centric Language Model for Self-Supervised Representation Learning on Text-Attributed Graphs." In EMNLP, 2023.