Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learning

We thank the reviewer for the insightful comments on our work. Please, find our answers to your comments and questions below.

Reviewer: The time analysis and money estimation are lacking. The paper utilizes the chatgpt API, nonetheless, there is a large restraint on the word limitation per day, and the cost of one dataset should also be taken into consideration.

Authors: We acknowledge the importance of a comprehensive analysis of time and cost, given the constraints of the ChatGPT API. We incorporated it in appendix D.10 (in blue) in our revised manuscript. Time and Cost Implications: Our primary dataset, ogbn-arxiv, with 169,343 nodes and 1,166,243 edges, serves as a representative case for our approach. On average, our input sequences consist of approximately 285 tokens, while the output sequences comprise around 164 tokens. For the ChatGPT-3.5 Turbo API, priced at $\textdollar$0.0015 per 1,000 input tokens and $\textdollar$0.002 per 1,000 output tokens, with a token per minute rate limit of 90,000, the monetary estimation for ogbn-arxiv is as follows:

Cost = ((285 * 0.0015) / 1000 + (164 * 0.002) / 1000) * 169,343 ≈ 128 USD

Considering the token rate limit, we estimate the deployment time as follows:

Time = 169,343/ (90,000/285) ≈ 536 min ≈ 9h

Cost-Effective Alternatives: Additionally, we have explored cost-effective alternatives, such as leveraging open-source LLMs like llama2. The use of llama2 is entirely free, and the querying process to llama2-13b-chat takes approximately 16 hours when utilizing 4 A5000 GPUs (24GB).

Efficiency through Single Query and Reuse: Our method requires only one query to the LLM, with predictions and explanations stored for subsequent use. This not only enhances efficiency but also minimizes the number of API calls, contributing to cost-effectiveness.

Dataset Sharing: We also make the GPT responses available for public use.

Reviewer: The robustness of the prompt is lacking. The paper proposed a single prompt for the node classification. Nonetheless, is another similar prompt can achieve a similar performance.

Authors: We have extensively explored the influence of various prompts, as outlined in Table 8 and Table 9 (in blue) in the appendix. Our findings reveal a positive correlation between the zero-shot accuracy of the LLM and the overall accuracy of our method, i.e. the higher the zero-shot prediction score, the better the accuracy of TAPE. Besides, despite employing different prompt designs, our approach consistently maintains similar accuracy, ranging from 0.7660 to 0.7750, demonstrating the robustness of the prompt.

Reviewer: There may lack of ablation study on the LM. Current LM focuses on BERT, while other LM are ignored. Recent paper demonstrates that they can achieve satisfying performance with only SentenceBert Embedding

Authors: We thank the referee for pointing out the necessity of a more comprehensive ablation study regarding the LM. In response, we ran additional experiments involving different LMs to assess their impact on TAPE.

Before revealing our findings, we would like to address a recent paper, SimTeG [1], which uses SentenceBert and might have been referenced by the reviewer. SimTeG indeed demonstrates satisfactory results with SentenceBert Embedding. However, it is crucial to acknowledge that, similar to our model, their approach entails fine-tuning the LM-SentenceBert, with raw text and node labels. Additionally, SimTeG highlights the seamless integration of our method with theirs. A claim substantiated through experimental testing, showing that SimTeG+TAPE with Ensembling establishes a new SOTA performance on the ogbn-arxiv dataset.

Moving forward, we extended our investigation beyond the deberta-base model by studying the effect of different LMs on ogbn-arxiv. In alignment with the approach taken in SimTeG [1], we incorporated two additional widely-used LMs from the MTEB leaderboard: all-roberta-large-v1 and e5-large. The detailed outcomes of our study are presented in Table 13 (in blue). Notably, our model demonstrates insensitivity to the choice of a specific LM, emphasizing its robustness to variations in LM selection.

We thank the reviewer for the examination of our work and the thoughtful comments provided. Kindly find our responses to the raised comments and questions below.

Reviewer: Lack of baselines. Many important baselines that conduct learning with language models on TAGs including GraphFormers [1] and Patton [2] are missing in the experimental section.

Authors: We appreciate the reviewer’s valuable feedback and acknowledge the concern regarding the absence of certain baselines in our experimental section, particularly GraphFormers [1] and Patton [2]. We would like to clarify the reason for this absence.

Our primary focus in this work is the node-prediction task in TAG. For this specific task, we concentrated on the most widely used datasets, i.e. ogbn-arxiv and ogbn-products. Additionally, recognizing the high popularity of Cora and Pubmed as node-classification datasets, we augmented these datasets by collecting text information, title and abstract, to create two additional TAGs (also made publicly available). We further introduced a new dataset, arxiv-2023, ensuring it is not part of the training set of GPT-3.5.

In the context of ogbn-arxiv and ogbn-products, we chose to compare our method against GLEM [3] and GIANT [5], recognized as the strongest baselines for these datasets. While we understand the concern about the absence of GraphFormers and Patton, we believe our study offers a robust comparison by evaluating against the strongest baselines within the constraints of the available datasets.

Specifically addressing GraphFormers, it was primarily designed for link prediction tasks, making a direct comparison less suitable due to differences in task objectives. Similarly, Patton involves pre-training on Microsoft Academic Graph (MAG) or Amazon e-commerce networks, followed by fine-tuning in a few-shot manner, making it also challenging for a direct comparison.

Moreover, it is important to note that the datasets used in the GraphFormers and Patton papers do not overlap with the datasets in our study. Consequently, we are unable to report from their papers the performance on our datasets.

Nevertheless, to address the referee’s concern, we conducted additional experiments within the rebuttal period. We extended our experiment to include MAG-Economics, a 40-way node classification task, to facilitate a comparison with Patton. The results, reported in Table 15, demonstrate that our technique TAPE outperforms GraphFormers and Patton by a significant margin of at least +13% accuracy on the MAG-Economics dataset.

In the revised version, we will commit to extending the comparative analysis to other datasets such as MAG-Mathematics, MAG-Geology, Amazon-Clothes, and Amazon-Sports. We believe this comprehensive evaluation across multiple datasets will address the gap mentioned by the reviewer and provide a more holistic comparison with existing baselines.

We thank the reviewer for the insightful comment, which contributes to the enhancement of the experimental section of our work.

Reviewer: The theorem is not specific to this problem. The theorem in 4.4 is not particular for LLM and LM for TAGs, and needs strong assumptions.

Authors: While it is true that Theorem 4.4 could be applied in other contexts, we found it valuable to provide a rigorous foundation for the intuitions guiding the proposed technique.

Our intention was to reinforce the understanding of why the technique is effective, beyond solely relying on experimental results. Specifically, the theorem underscores the importance of fidelity in explanations with LLM's reasoning, and emphasizes the non-redundancy between LLM and LM. These aspects constitute the key assumptions under which the technique was designed to operate successfully.

I would like to thank the authors for their detailed response and their added experiments. I'm still not fully convinced by the novelty of this work, but I'm happy to raise my score.

We thank the reviewer very much for the careful reading and comments regarding our work. Please, see below our answer to the raised comments/questions.

Reviewer: The main concern is that the method has little to do with graph. It seems like an application attempt of LLMs on natural language tasks, and TAG is just a scenario. Thus the contribution of the method is limited.

Authors: While it is true that the first part of our technique has no direct connection to graph topology, and focuses on node features, the overall methodology is strongly motivated towards addressing the challenges inherent in the TAG problem, an active research area propelled by the advent of language models.

As advocated in our global response, our approach is deeply rooted in addressing the challenges associated with the joint training of GNNs and LMs. The task of representation learning on TAGs involves integrating the raw text with graphs in a robust and scalable way. Existing literature can be broadly categorized into two paradigms: 1) End-to-end or joint training of LM and GNN, exemplified by approaches like GLEM [1] and DRAGON [2]; and 2) Two-stage training, where LM is initially trained followed by GNN training. Our approach belongs to the second paradigm, inheriting its efficiency and ease of plugging into existing GNN pipelines, but incorporates an LLM-to-LM interpreter to extract and interpret the LLM's useful prior knowledge.

The first approach is generally considered more attractive because the LM is trained to directly leverage graph structure. However, it often involves complex language and graph architectures, encounters training instabilities, and demands significant GPU resources, particularly challenging with LLMs. In this context, an important contribution of our work is the finding that with the help of our LLM-to-LM interpreter, the second paradigm of two-stage training can achieve equal or better performance than end-to-end or joint training, with much greater efficiency. In particular, our LLM-to-LM interpreter allows the valuable prior knowledge from LLMs to be embedded into GNN features, and our work shows that this does not sacrifice performance but greatly improves efficiency, compared to joint training.

Therefore, our work should be viewed as a strategic response aimed at overcoming these challenges. It goes beyond being a mere application of LLMs to TAG; rather, it represents a novel, efficient, and robust approach to addressing tasks that involve both text and graphs.

We acknowledge that we may not have sufficiently conveyed the important message that our work aims to resolve the intricate issues tied to LM+GNN training for TAG. We are committed to amending the paper to provide greater clarity on this aspect for the reader.

Reviewer: Since the Deberta need fully fine-tuning, training the method cost much more memory than pure GNN methods.

Authors: We appreciate the reviewer's observation regarding the increased memory usage in our approach compared to pure GNN methods, as illustrated in Figure 2. It is important to note that learning on TAG inherently demands the integration of LMs and GNNs. Established methods like GIANT [5], GLEM [1], GraphFormers [3], and Patton [4], among others, all involve LM in their processes. Therefore, a meaningful comparison of memory cost should be drawn against these baselines rather than pure GNNs.

Recognizing the inherent tradeoff between cost and accuracy, we highlight that despite the increased resource requirements, our method TAPE significantly boosts the accuracy of the pure GNN method from 70.83% to 77.50%. We firmly believe that the substantial performance improvement justifies the accompanying rise in memory costs. Notably, these elevated resource demands are manageable, even with relatively cost-effective academic GPU resources like our A5000 GPU (24GB). We greatly appreciate the reviewer's observation regarding memory usage, which prompted us to conduct a more in-depth analysis, as outlined in Table 14 (in blue), showing that TAPE provides both best performance, and comparable or better running time than other LM-based methods.

Reference

[1] Jianan Zhao, Meng Qu, Chaozhuo Li, Hao Yan, Qian Liu, Rui Li, Xing Xie, and Jian Tang. Learning on large-scale text-attributed graphs via variational inference. ICLR, 2023

[2] Michihiro Yasunaga, Antoine Bosselut, Hongyu Ren, Xikun Zhang, Christopher D Manning, Percy S Liang, and Jure Leskovec. Deep bidirectional language-knowledge graph pretraining. NeurIPS, 2022.

[3] J. Yang, et al. GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph. NeurIPS 2021.

[4] B. Jin, et al. Patton: Language Model Pretraining on Text-Rich Networks. ACL 2023.

[5] Chien, Eli, et al. Node feature extraction by self-supervised multi-scale neighborhood prediction.