LLM as GNN: Graph Vocabulary Learning for Graph Foundation Model

Thank you so much for your review. In the following, we clarify some of your misunderstanding.

W1. Thank you for your thoughtful feedback. Our work introduces a novel approach to implementing GNN functionality within the natural language space using LLMs. This approach is not intended to entirely replace embedding-based GNNs but rather to offer a complementary perspective that explores new possibilities for leveraging LLMs in graph-related tasks. Consistent with our motivations, we argue that in the era of LLMs, providing formal mathematical proofs may not always be strictly necessary. Instead, our focus on practical and conceptual alignment has already demonstrated meaningful and promising results. We will try to clarify our points to avoid misunderstanding.

W2. We would like to clarify that our work focuses specifically on building a graph foundation model for text-attributed graphs. In our approach, each node is represented by a textual description, referred to as a language-based ID, which ensures applicability and comparability across all text-attributed graphs. Unlike existing methods, we highlight that our graph vocabulary, composed of natural language tokens, is inherently sharable and transferable across different graphs, whereas approaches using out-of-vocabulary (OOV) tokens limit generalization to specific graph nodes. Furthermore, our work is fundamentally different from most existing methods and represents the first true implementation of "LLM as GNN," as outlined in the introduction. For instance, OFA defines functional nodes and uses an LLM to convert textual descriptions into embeddings but does not utilize the LLM for core GNN operations such as message passing. Instead, in OFA, message passing is performed in the vector space using traditional GNN models, leading to embeddings that lack interpretability and generalizability to other graphs or tasks. Our approach directly addresses these limitations by leveraging LLMs for the entire GNN workflow, enabling both interpretability and broader applicability.

W3. Our experimental design aligns with standard practices in the field, as demonstrated by studies such as [1][2][3], which follow widely accepted benchmark setups. Regarding resource consumption, we acknowledge that our model requires more computational resources compared to traditional GNNs. However, it delivers higher accuracy and focuses on foundation models with transferability and generalization capabilities, enabling cross-dataset and cross-task applications—advantages that traditional GNNs cannot achieve. Regarding the experimental details, please check our responses to the questions Q1-Q5.

Q1. In the graph understanding module, we only prompt the LLM to perform message passing without using labels. In fact, the performance of different downstream graph tasks depends on different aspects of the graph (node classification tasks focus on the smoothness of nodes, while link prediction tasks emphasize local structural information). Therefore, this approach ensures the outputs of the graph understanding module can generalize across both tasks, i.e. node classification and link prediction. In the graph inference module, when we need construct task-specific prompts, node labels are directly provided in the prompts for node classification tasks to avoid hallucination.

Q2. For node classification, we directly provide the candidate labels and ensure that LLMs can be select one of them. This design is aligned with traditional node classification settings.

Q3. As mentioned in Section 4.1, we rely on prompts "Note connected nodes should share similar semantics and vice versa" to intuitively optimize the textual representations because negative sampling is redundant in the situation. As for GNN baselines, the number of negative samples is set to 1 by default.

Q4. The size of the T5 model fine-tuned on the NC and LP tasks is 0.8B.

Q5. It was a full fine-tune. Since Flan-T5-large has relatively few parameters, we perform full-parameter fine-tuning.

Thanks for your review. We clarify your misunderstanding as follows.

W1. Apologies for the confusion. As discussed in the Introduction, PromptGFM is specifically designed for text-attributed graphs and can be applied to any type of text-attributed graph, regardless of the domain.

W2. Sorry for confusion. We would like to clarify our work aims to do graph foundation model for text-attributed graphs only. After multi-round message passing, each node can be represented by a finite sequence of language tokens and has embodied structural information, which is called language-based IDs. As language tokens are universal and transferable in NLP, the language-based IDs for nodes are also universal across different text-attributed graphs. With universal textual representations, we can construct a large number of task-specific instructions using pure language tokens, and put them in a single multi-instruction fine-tuning framework, adapting to various graphs and tasks. This is how our method works as a universal graph model.

W3. Sorry for your misunderstanding. As mentioned in the Introduction Section, our proposed PromptGFM is a graph foundation model for text-attributed graphs, which is the most studied setting in the domain. We will highlight this setting for better clarification.

W4. Thanks for your question. We highlight our contributions as follows. (1) Our work represents an attempt at LLM as GNN, where we use prompts to guide LLMs in replicating message passing for each node, aligning with the fundamental principles of existing GNN models. This approach demonstrates how LLMs can function as GNNs, fundamentally differing from existing LLM for GNN and GNN for LLM models, where GNNs and LLMs operate independently. Our approach represents an entirely new paradigm and an unexplored concept; it is by no means a simple direct combination of LLM and GNN. (2) By implementing LLM as GNN in the textual space, our approach addresses out-of-vocabulary issues present in previous works and enables the use of pure language prompt instructions for LLM inference. This resolves modality incompatibility and facilitates positive transfer, significantly advancing the development of GFMs.

Thank you so much for your review. In the following, we clarify the misunderstanding and address your concerns.

W1. (1) Sorry for your misunderstanding. As mentioned in the Introduction Section, our proposed PromptGFM is a graph foundation model for text-attributed graphs, which is the most studied setting in this research field. (2) Regarding dataset domains, most works in this field focus on how different domains incorporate varying semantics rather than the specifics of dataset construction. For instance, a recent comprehensive survey [1] adopts a similar approach, categorizing Cora/Citeseer/Arxiv as the CS citation domain and assigning PubMed to the MISC domain. We will consider including more datasets in future work.

[1] Chen, Zhikai, Haitao Mao, Jingzhe Liu, Yu Song, Bingheng Li, Wei Jin, Bahare Fatemi et al. "Text-space Graph Foundation Models: Comprehensive Benchmarks and New Insights." arXiv preprint arXiv:2406.10727 (2024).

W2. Thank you for your suggestion. The table presents node classification performance, showing comparable results. 'N/A' indicates that the results are not reported in the original paper. Please note that TAPE cannot handle link prediction tasks.

Q1. Thanks for your question. (1) Following a comprehensive analysis in Section 4.2, we directly use the final textual representations as language-based IDs, which is composed of a finite sequence of langugae tokens. (2) We do not specify a fixed number of tokens for the final textual representation. However, in practice, it typically ranges from 10 to 20 tokens, depending on the number of layers of the prompt-based GNN.

W4. Thank you for your suggestions. (1) Our work represents an attempt at LLM as GNN, where we use prompts to guide LLMs in replicating message passing for each node, aligning with the fundamental principles of GNN-based models. This approach demonstrates how LLMs can function as GNNs, fundamentally differing from existing LLM for GNN and GNN for LLM models, where GNNs and LLMs operate independently. (2) Apologies for any confusion. In Figure 2, we compare the prompt-based GNN (our method) with the embedding-based GNN (e.g., GraphSAGE) to highlight the faithful replication of the GNN workflow. It is important to note that PromptGFM is not a direct combination of a GNN and an LLM. (3) Regarding the second question, PromptGFM extends beyond pure information propagation. We argue that the essential elements of a GNN include input, neighbor sampling, aggregation-update mechanisms, multi-layer message passing, and optimization. In PromptGFM, we design specific prompts to empower LLMs to faithfully replicate each of these elements in the GNN workflow. Therefore, the term "prompt-based GNN" effectively summarizes our approach.

W5. Thanks for your comments. We will replace this term and thoroughly review the entire text for precise wording.

W6. Thanks. We will summarize these sections and move some detailed explanations to the Appendix.

Q1. Apologies for the confusion. The construction of the graph vocabulary is detailed in Section 4.2. Following a comprehensive analysis, we directly use the final textual representations as language-based IDs. These IDs are then indexed from the graph vocabulary and integrated into task-oriented prompt templates to create comprehensive instructions. Then, we fine-tune the LLM under a multi-instruction fine-tuning framework. We will consider reorganizing this content for better clarity.

Q2. Sorry for misunderstanding. We prompt LLM to iteratively refine textual representations, which goes beyond simple rephrasing. During the multi-round prompting process with LLMs, both semantic fusion and structural information capture occur simultaneously. Using language-based IDs, we can construct readable instructions for LLM inference, resolving modality incompatibility and facilitating positive transfer.

Q3. Thanks for your question. Both datasets do have text attributes from other sources, as indicated in the datasets provided in the code repository.

We truly appreciate your constructive feedback and suggestions. In the following, we clarify the misunderstanding and highlight our contributions.

W1. Thanks for you comments. (1) Our method differs significantly from [1, 3] in several key aspects. Specifically, in OFA [1], the authors use an LLM to transform each node into a fixed-length vector and perform message passing over a prompted subgraph in the embedding space. In contrast, PromptGFM introduces two key innovations: (a) Message passing is conducted in the textual space instead of the embedding space, an approach unexplored in prior work. (b) By stacking multi-layer LLM prompts, each node can theoretically capture high-order signals with rich semantics across the entire graph, rather than being limited to a subgraph as in OFA. Regarding "Talk Like a Graph" [3], while it establishes a benchmark by encoding graph data as text, it focuses solely on reasoning tasks and neglects graph understanding, such as message passing. Additionally, this work directly inputs all graph structural information (e.g., adjacency matrix, social networks) into a single prompt. This approach is clearly constrained by the LLM's allowable input length, rendering it impractical for real-world applications. In contrast, PromptGFM explicitly conveys neighboring information node by node via prompts, effectively replicating message passing—an approach not explored in prior works. (2) In each prompt, we include only the textual representations of the central node and its neighboring nodes from the previous layer to generate the textual representations for the current layer. By stacking multiple rounds of prompts to LLMs, we effectively capture long-range dependencies across the graph, similar to the workflow of GNNs. Additionally, inspired by GraphSAGE, we sample a limited number of 1-hop neighboring nodes for each node's prompt. This approach stays within ChatGPT's length constraints and avoids practical issues associated with long sequences. (3) We have included the templates of prompts for different functionalities in Appendix C. By feeding the title and abstract of each paper, we prompt LLM to construct the initial textual representation of each node. The second prompt in Appendix C illustrate how we replicate the message passing of each GNN layer, which is exactly the same as Figure 3. Additionally, we decompose this prompt to show how it align with the authentic GNN in Figure 2. For more clear demonstration, we will add a case study in Appendix F.1 to make it align with the example in the main content. It shows how we perform layer-by-layer message passing in the textual space and how information is aggregated from neighboring nodes.

W2. Thanks for your valuable suggestions. (1) For example, in the first prompt of Appendix C, we use text summarization to generate the initial textual representations for each node, analogous to initializing dense vector embeddings in embedding-based GNNs. Building on this, as shown in the second prompt, we employ aggregation and update processes to guide the LLM in performing GNN-style message passing, moving beyond simple summarization. (2) Theoretically, at each layer, we emulate the loss function in GraphSAGE by employing prompts such as "Note connected nodes should share similar semantics and vice versa." These cumulative, layer-by-layer prompts are analogous to layer-wise loss combinations, formally akin to mean pooling across all GNN layer embeddings, and mathematically extend beyond the textual representation in the final layer. As demonstrated in [4][5], such structural information can be effectively preserved. Experimentally, in Table 9 of Appendix F.2, we present the final textual representations alongside the original keywords from selected papers (the keywords are not included in the prompts and serve as ground truth). The observed similarity demonstrates that our prompt-based GNN effectively captures the core semantics of these nodes.

W3. Apologies for any misunderstanding. We would like to clarify that our work focuses exclusively on GFM for text-attributed graphs. In this framework, each node corresponds to a textual representation, referred to as a language-based ID, which is applicable and comparable across all text-attributed graphs and warrants the development of a universal vocabulary for such graphs. This vocabulary enables the construction of pure language instructions, which are crucial for the subsequent multi-instruction fine-tuning framework for LLM inference. Our approach differs from [1,2] and is not applicable to [3] (as indicated in the response above).