GL-Fusion: Rethinking the Combination of Graph Neural Network and Large Language model

Thank you for acknowledging our motivation. We address your concerns as follows.

• The paper is not well written.

I am sorry for our writting issues. We have made the notations consistent and add details for experiments in revision. Hope our revision can help you understand our work help and reevaluate our work.

• The use of notations presents a significant problem so that I could not understand the paper. In the preliminaries section, $L_n$ occurs only once. I think that $L$ describing the length of the node text in Line 129 should always be $L_n$.

We apologize for the inconsistency. we have fixed it and convert $L$ to $L_n$.

• In section 3.2, Equation 3 and Equation 4 are very unclear. None of $T_i,W_q,W_k,X_v$ is defined. It is not clear to me whether $T_i$ in this context refers to the T_i of Equation 2.

We agree that this was unclear. We have clarified and defined these notations at the beginning of Section 3 to ensure their proper usage throughout.

• In line 238, the authors also admits to abusing the notations, and I don't understand why the authors didn't make further corrections.

We appreciate your feedback and have addressed this issue. Previously, $X, T$ were used for both input token sequences and their representations in the model. We now use lowercase $x, t$ for representations to avoid confusion.

• I am suspicious of the time complexity of the cross attention layer because of the confusion of the notations.

The notations have been fixed. The time complexity of our cross attention mechanism is $O(nL_nL_t)$ as the query is input sequence of length $L_t$ and key and value is $n$ nodes' $L_n$-length text.

• The data format of the table is not aligned.

We have transposed Table 1 in the revised version and are happy to address any other formatting issues.

• There is a large amount of blank data in Table 3 with no further explanation as to why.

For the link prediction table, our baselines only report results for certain metrics or datasets. To provide a complete comparison, we include results for all metrics with our GL-Fusion model. Additionally, we have added baseline results for models such as GraiL and NBFNet. GL-Fusion consistently outperforms these baselines.

• Lack of important ablation studies.
• The paper proposes three modules, but in the experiments, only 'w/o cross-attention' is included in the graph property prediction task.

We have added detailed ablation studies on the ogbn-arxiv dataset, as shown below. Our GL-Fusion produces both GNN and LLM outputs simultaneously, so we report the test accuracy of both predictors and their ensemble accuracy (final results of the node classification task).

Here Low LoRA rank denotes using model with LoRA rank=8. w/o cross attention removes the graph-text cross attention. w/o gate removes the gate mechanism. w/o multi-aggr removes multiple aggregators in the message passing modules in transformer layers and uses mean aggregator only. w/o gnn predictor removes the gnn prediction and training loss on it. w/o text predictor removes the text prediction and training loss on it. Each module contributes to the overall performance, and removing any of them results in a performance drop.

Dear Reviewer yaRE,

Thank you for acknowledging our efforts to improve the paper and taking time to participate our discussion.

We believe that combining GNNs and LLMs is a promising direction, as demonstrated by previous work [1, 2, 3, 5]. This combination not only enhances the GNN’s ability to process text (such as node text and task descriptions), but also enables LLMs to better understand graph data. By processing graphs from various domains in a unified way—using LLMs to handle text features from these domains—combination of GNN and LLM serves as an important step toward developing graph foundation models [1, 2, 5].

Our architecture outperforms existing combinations of GNN and LLMs, as well as standalone GNNs, across node, link, graph, and graph-to-text generation tasks, confirming the effectiveness of our approach. Specifically, we achieve state-of-the-art (SOTA) performance on the ogbn-arxiv dataset, where multiple GNN-LLM combinations and ensemble methods have been explored. Additionally, on the ogbg-code2 dataset, our model achieves a 40% F1 score, surpassing the previous SOTA of 20%.

We now address your concerns point by point:

1. The merit of the paper is to combine the advantages of LLM and GNN to simultaneously perform text generation and node classification tasks.

Our contribution goes beyond simply producing two kinds of output for these tasks. The ability to generate both text and GNN outputs is one feature of our twin predictor, which is just one aspect of our architecture. Additionally, we propose a structure-aware architecture and node-text cross-attention, both of which are validated as effective in our ablation study (Table 8). The twin predictor not only produces two types of predictions, but the supervision on both predictions helps improve each other. Our ablation study shows that removing the loss on one prediction leads to a performance drop in the other.

Moreover, our model is not limited to node classification. The flexibility of text allows us to formulate a wide range of tasks, including node, link, and graph classification, question answering with graphs, and code summarization using code ASTs. Our model performs well across all these tasks.

2. When we only want to do node classification tasks, I think simply performing GNN-centered models is enough. The proposed model will lead to significant overhead compared to plain GNN-centered models, which may overshadow the performance margin.

On node tasks, our GL-Fusion model still significantly outperforms both existing GNNs and GNN-LLM combinations. Text information plays a crucial role in improving performance, and traditional GNN-centered models cannot fully utilize this. For example, in our experiments on ogbn-arxiv, extracting uncompressed text information with our cross-attention module leads to a 3% performance improvement, highlighting that previous embedding methods in GNN models are insufficient. Additionally, node classification tasks like ogbn-arxiv have been dominated by combinations of GNN and LLMs [4].

The cost of our method is minimal compared to the cost of the LLM backbone. Only 10% of additional parameters and a small amount of extra computation are added to the LLM backbone. Despite this, our method significantly enhances the LLM’s ability to process graph data, allowing the LLM to better understand graphs. When compared to GNN-centered methods, the training process for GNN-centered models typically involves precomputing LLM embeddings and running only the GNNs. In contrast, our method requires running both the GNN and the LLM, making our approach slower during training. However, during inference, when new node text features must also pass through the LLM, the cost of GNN-centered methods becomes comparable to that of the LLM backbone.

3. All the downstream graph-related tasks are not unified in the same template. In other words, when we train the model on node classification, we have to fine-tune the model for graph classification or link prediction tasks.

In our experiments, all downstream tasks are unified into the same input format. For each task, we combine the task description, input graph, and label into a mixed sequence of text and graph. The model is trained with standard LLM text cross-entropy loss to predict the label. Ideally, we can train our model for node, link, and graph tasks simultaneously and perform these tasks without additional fine-tuning.

In this case, the only task-specific component is the GNN predictor, which is a small MLP (approximately 10k parameters). Even without retraining for new tasks, our model can still generate text predictions and directly apply to unseen downstream tasks. While multi-task learning is our future goal, our current work already provides flexibility for adapting to multiple tasks without change to the architecture.

Dear Authors,

I appreciate all the efforts you have made to improve the paper. Meanwhile, I have carefully checked all the reviews. For me, the biggest issue of this paper remains still unsolved. The so-called merit of the paper is to combine the advantage of LLM and GNN to simultaneously perform text generation and node classification tasks. However, in my eyes, this is unnecessary in three reasons.

First, when we only want to do node classification task, I think simply performing GNN-center model is enough, because GNNs are good at classification tasks. The proposed model will lead to significant overhead than plain GNN-center models, which will shadow the performance margin. In this way, the merge of GNN and LLM as in the paper seems to be not attractive.

Second, all the downstream graph-related tasks are not unified in the same template. In other words, when we train the model on node classification, we have to fine-tune the model for graph classification task or link prediction task.

Further, a minor issue: given graphs that do not contain labels or texts, how can the model be trained? For example, when the graphs in the training data are non-textual, the model will only rely on labels to train. But the disaster is that we have a huge number of learnable parameters, which makes the model inapplicable.

Thank you for your feedback.

First, we acknowledge that our model has a larger training overhead when compared to GNN-centered models. It takes a similar cost to LLM-centered methods [1]. During inference on unseen data, the computational cost of ours and GNN-centered models will both be similar to that of LLM backbone. It's worth noting that existing GNN-centered models in our baseline also needs to finetune LLM to enhance performance [2, 3]. Freezing the LLM would not allow for a deep fusion of GNN and LLM features.

Second, one example from our degree prediction dataset (following [4]'s setting) is as follows:

Input Graph-Text Mixture Sequence: "What is the degree of node 8? Graph: <graph_start><graph_node>...<graph_node><graph_end>? 3"

Edge Index: [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 3, 3, 3, 4, 4, 4, 5, 5, 5, 6, 6, 6, 7, 7, 7, 8, 8, 8, 9, 9, 9, 10, 10, 10, 10, 10, 10, 11, 12, 12, 12, 12, 12, 13, 13, 13, 14, 14, 14, 14, 14, 14, 15, 15, 15, 15, 15, 15, 16, 17, 17, 17, 18, 19, 19, 19, 19, 19, 19, 19, 19], [16, 17, 8, 1, 12, 3, 4, 10, 14, 15, 19, 0, 9, 6, 5, 11, 7, 13, 12, 3, 4, 10, 14, 15, 19, 18, 14, 0, 1, 4, 0, 1, 3, 1, 9, 13, 1, 10, 19, 1, 9, 13, 0, 17, 19, 1, 5, 7, 0, 1, 6, 12, 15, 14, 1, 0, 1, 10, 15, 19, 1, 5, 7, 0, 1, 2, 10, 15, 19, 0, 1, 10, 12, 14, 19, 0, 0, 8, 19, 2, 0, 1, 6, 8, 12, 14, 15, 17]]

Node Text: Each node is represented by its ID, such as "id: 0", "id: 1", ..., "id: 19".

Edge Text: All edges are simply labeled as "edge".

Node label for GNN predictor: 0 for all nodes except node 8. node 8 has label 4 (0 is used to represent node not queried in problem, so 4 represents degree=3).

[1] Tang et, al. GraphGPT: Graph Instruction Tuning for Large Language Models. SIGIR 2024.

[2] Zhao et, al. Learning on Large-scale Text-attributed Graphs via Variational Inference. ICLR 2023.

[3] Chien et, al. Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Prediction. ICLR 2022.

[4] Guo et, al. GPT4Graph: Can Large Language Models Understand Graph Structured Data ? An Empirical Evaluation and Benchmarking. CoRR 2023.

Thank you for taking the time to read our clarification and reevaluate our work. However, there are still some misunderstandings regarding the facts of our method.

First, the statement that "we have to train all the parameters in the model that are numerous" is incorrect. In our GL-Fusion model, the LLM parameters are fixed, and only the newly added GNN layers (31% of all trainable parameters), the cross-attention block, and the parameter-efficient adapter for the LLM are optimized. As a result, the memory requirement for training our model is relatively modest, only requiring about 30 GB of GPU memory.

Second, the statement that "Given non-textual graphs, the model cannot be trained because we have a significant number of parameters and only labels can be used to train Transformer-based models" is also incorrect. In our synthetic experiments with node-degree data, our model, even with empty node text features and trained solely on node degree labels, can predict degree values accurately with 100% test accuracy. Unlike traditional statistical learning models, it is widely known that a large number of parameters does not necessarily lead to overfitting in deep learning models [1], due to regularization, implicit bias of stochastic gradient descent, and inherent properties of deep neural networks.

Third, regarding the claim that "graphs without text features are prevalent." While many datasets preprocess features and may not explicitly expose the raw text, the text data is often still available. For instance, in the Open Graph Benchmark, nearly all datasets—whether node, link, or graph-level—include text in some form. The only exception is ogbl-vessel, which provides 3D coordinates (although these can be converted to text). Current LLMs may struggle with numerical data like coordinates, but the fact remains that most datasets contain some form of textual data. Consequently, many state-of-the-art methods in this domain incorporate LLMs for processing such text.

We appreciate your feedback and hope these clarifications address your concerns.

[1] Zhang, et al. "Understanding deep learning requires rethinking generalization." ICLR, 2017.

Thank you for acknowledging our presentation, method design, and experiments on various graph tasks. We address your concerns as follows.

• Experiments are not comprehensive. Baselines and benchmarks are missing. For example, for the node classification tasks, there are only two datasets considered. In Table 3, many entries in the table are missing, which makes the results less convincing. I suggest the author refer to [1] for benchmarking models on text-attributed graphs.

We appreciate this feedback. The Arxiv dataset in [1] corresponds to ogbn-arxiv, where our model achieves state-of-the-art results. In addition, we have incorporated more node classification datasets from [1], and the updated results are as follows:

Our model, GL-Fusion, demonstrates significant improvements on the Children, Computers, and Sports datasets. For the Photo dataset, GL-Fusion outperforms most models except those based on GraphSage.

Regarding the link prediction table, our baselines conduct experiments on only some metrics or subsets of datasets. We have now added baseline results for models such as GraiL, NBFNet, and others. GL-Fusion still outperforms them across the majority of metrics and datasets.

• The manuscript proposes a twin predictor for node classification. However, edge-level and graph-level tasks should also be considered given their importance in graph-related tasks. Regression tasks should also be considered.

In our paper, we propose twin predictor as a technique to train & inference (predict) on both GNN-side and LLM-side for each sample in each task. For most GNN tasks, the output of GNN have been well designed and the output of LLM is translated from original labels in dataset. Specifically, in link prediction tasks (like FB15k237 in our paper), given a fixed head, the tail nodes are labeled as 1 and others are labeled as 0, and it can be predicted by GNN. At the same time, the LLM generate its prediction as usual. So twin predictor can work for link prediction tasks (though we only use gnn output in test as we need predicted probability to compute test metric). As for graph classification, the graph label is boardcasted to each node during training and the graph-level prediciton can be readout by voting on each node during inference. (though we only use text output in test as ogbg-code2 task is to generate text). Our twin predictor is also suitable for regression, we just need to change the CEloss to the MSEloss and other setting can be kept the same. Other tasks and predictors can be added to boost GNN, but our primary focus is to combine GNN and LLM better with a new architecture.

• In the Graph-text attention layer, the language tokens do not attend edges. Would the loss of the information affect performance?

Although language tokens do not directly attend to edges, the node tokens—containing language information—are further combined with edge information in the message-passing layers. Additionally, the edge text used in our model already encapsulates task-relevant information. Therefore, there is no loss of information in this setup. This design ensures that both node and edge attributes contribute effectively to the overall task performance.

We hope these clarifications address your concerns. Thank you again for your valuable feedback, which has helped us refine and improve our work. Let me know if any further refinement is required!

[1] Yan, Hao, et al. "A comprehensive study on text-attributed graphs: Benchmarking and rethinking." Advances in Neural Information Processing Systems 36 (2023): 17238-17264.

Thanks for the author's response. Although the authors have included more baselines. I still feel like there is a large amount of related work in the LLM + Graph fields, which has not been discussed in the manuscript or included in the performance table. For example

[1] UniGLM: Training One Unified Language Model for Text-Attributed Graphs

[2] LLaGA: Large Language and Graph Assistant

[4] Disentangled Representation Learning with Large Language Models for Text-Attributed Graphs

[5] Efficient Tuning and Inference for Large Language Models on Textual Graphs.

In addition, considering it an architecture innovation paper, I think the proposed method lacks novelty.

Thank you for acknowledging our presentation and method design. We address your concerns as follows.

• if there is a common understanding that self attention is a form of message passing, why is there a need to incorporate a separate MPNN in the structure aware Transformer layer? This aspect is unclear and has not been empirically studied as well.

We agree that self-attention can be viewed as a form of message passing, where messages are exchanged between token pairs based on their attention scores. However, encoding graph structures specifically requires message passing constrained by the graph's edges. In a vanilla attention layer, there is no inherent mechanism to enforce this constraint.

In our initial trials, we experimented with directly incorporating the adjacency matrix into the attention mechanism. However, we observed that this approach fails to capture even basic graph properties like node degrees. This limitation arises because simply adding graph edge information to the attention matrix cannot balance the dense, fixed-magnitude token attention with the sparse, degree-dependent edge aggregation. To address this, we introduced a separate MPNN layer, which enables effective graph-specific message passing.

This hybrid architecture is also consistent with recent graph transformer designs, such as those discussed in [1]. We believe this integration provides a principled way to incorporate graph-specific inductive biases while leveraging the power of transformers.

• in page 5, the positional encoding which would be ideal for both text and graph tokens is discussed. I am wondering if it is possible to have a unified structure based positional encoding which perceives both sequential structure and graph structure as 'arbitrary graph structure'? This is in context of recent works in graph transformer literature which show laplacian eigenvectors can generalize the text positional encoding of Transformers.

Thank you for pointing out this possible method. While eigenvector-based positional encodings (PEs) can theoretically preserve all graph structure information, incorporating an MPNN module remains critical for performance. As highlighted in [1], even when graph transformers inject structure information through PEs, the message-passing operator significantly enhances performance.

Moreover, [2] demonstrates that using Laplacian eigenvector-based PEs can lead to symmetry-breaking issues, resulting in different predictions for isomorphic graphs, which undermines generalization. Although [2] proposes modifications to mitigate this, these changes significantly alter the vanilla transformer architecture. In contrast, our structure-aware transformer layers retain seamless compatibility with vanilla transformer layers, ensuring broader applicability.

We also conducted an ablation study on the ogbn-arxiv dataset. Replacing MPNN layers with Laplacian eigenvector PEs resulted in a significant performance drop, with test accuracy decreasing from 78.2% to 72.78%. This highlights the importance of retaining the MPNN layer alongside other design choices in our framework.

• although the proposed aims to reduce complexity, the integration of graph structures into LLM layers may still encounter scalability issues with large graphs or highly interconnected datasets. How does GL-Fusion handle memory usage and processing time for very large graph structures?

We acknowledge the scalability challenges associated with using LLMs for large graphs. As discussed in Section 3.2, our cross-attention block extracts information from node text features in a more scalable manner compared to directly incorporating all node text as input to the transformer.

Nonetheless, running GL-Fusion on very large graphs remains a challenge due to the memory and computational demands of LLMs. In our experiments, when the input graph exceeds the memory limits, we adopt an ego-subgraph sampling approach, focusing on the subgraph centered around the node (or link) to predict. This approach is a widely accepted and standard technique for handling large graphs in both academia and industry [3].

Thank you again for your valuable feedback, which has helped us clarify and strengthen our methodology. Let me know if further refinements are needed!

[1] Ladislav Rampásek, Michael Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, Dominique Beaini. Recipe for a General, Powerful, Scalable Graph Transformer. NeurIPS 2022.

[2] Xiyuan Wang, Pan Li, Muhan Zhang. Graph As Point Set. ICML 2024.

[3] Haoteng Yin, Muhan Zhang, Yanbang Wang, Jianguo Wang, Pan Li, Algorithm and System Co-design for Efficient Subgraph-based Graph Representation Learning, VLDB, 2022.

Thank you for acknowledging our presentation and method soundness. We address your concerns as follows.

• The proposed method is complex which includes three components: text and graph token transformers, text to graph token attention module, and GNN prediction module. The optimization process on three components seem non-trivial. However, the authors did not discuss any details on it.

We appreciate your observation and provide more details on our optimization process below:

1. The entire model is trained end-to-end, with joint training of all components. However, not all parameters are trained from scratch.

2. For text and graph token transformers (referred to as Structure-aware Transformer Layer in our paper), we leverage the pretrained Llama-3-8B model as the backbone. As detailed in Section 3.1, we introduce new parameters through modifications to positional encoding and attention masks. Additionally, for the <graph_node> token, we simply add an embedding to the existing layer. The newly added MPNN component is initialized randomly. We fine-tune the transformer parameters using the Low-Rank Adaptation (LoRA) method from the peft library. This approach freezes the original transformer layers and optimizes only the newly introduced low-rank layers. The MPNN parameters, on the other hand, are trained entirely from scratch.

3. For the text-to-graph token attention module (referred to as Graph-Text Cross-Attention layers in our paper), all parameters are initialized randomly and trained from scratch.

4. For the GNN prediction modules, which are simple linear layers for generating outputs, parameters are initialized randomly and trained from scratch.

5 Overall, most parameters are initialized from pretrained models and frozen. Only about 10% of the parameters in the entire model are optimized.

• The method needs to train a new transformer architecture, which seems not easy to combine with existing pre-trained LLMs. Can authors describe if it is possible to include the power of pre-trained language models?

Thank you for raising this point. Our model is indeed designed to integrate seamlessly with existing pretrained LLMs. In fact, all our experiments were conducted using a modified version of the Llama-3-8B model.

As described in Section 3.1 of our paper, a standard transformer layer in the LLM is adapted to a structure-aware transformer layer by modifying the positional encoding, attention masks, and adding a separate MPNN layer. This modification ensures that, when the input consists of text only, the adapted transformer layer produces the same output as the original transformer layer. Thus, our method can fully harness the capabilities of pretrained LLMs.

• The included datasets for node classification is limited (only two datasets). The reported results in link prediction table lacks most of the comparison results.

We acknowledge the initial limitations and have expanded our evaluation.

For node classification, we added more datasets from [1]. The results are summarized in the table below:

Our model, GL-Fusion, achieves significant improvements on the Children, Computers, and Sports datasets. On the Photo dataset, GL-Fusion outperforms most models except those based on GraphSage.

For link prediction table, our baselines only conduct experiments on a part of metrics or a part of dataset. We have added full baseline results for GraiL, NBFNet, and other datasets. Our GL-Fusion still outperforms them.

4. Add Ablation study

We have conducted an ablation study on the ogbn-arxiv dataset, as shown below. Since GL-Fusion outputs both GNN and LLM predictions, we report the test accuracy of both predictors and their ensemble performance:

Here Low LoRA rank denotes using model with LoRA rank=8. w/o cross attention removes the graph-text cross attention. w/o gate removes the gate mechanism. w/o multi-aggr removes multiple aggregators in the message passing modules in transformer layers and uses mean aggregator only. w/o gnn predictor removes the gnn prediction and training loss on it. w/o text predictor removes the text prediction and training loss on it. Each module contributes to the overall performance, and removing any of them results in a performance drop.

5. Improved Readability and Notation Consistency

We have revised Section 3 to ensure consistent notation and improve readability.

6. Expanded Appendix

We have added details about the datasets and implementation in Appendices A and B.

Your comments help us improve readability and evaluate our method more compresensively. In addition to these revisions, we have addressed your specific questions in our individual responses. With the discussion period nearing its end, we sincerely appreciate your reevaluation of our work and hope our efforts address your concerns. Thank you once again for your valuable feedback and insightful suggestions.

We look forward to your response.

[1] Yan, Hao, et al. "A comprehensive study on text-attributed graphs: Benchmarking and rethinking." Advances in Neural Information Processing Systems 36 (2023): 17238-17264.

[2] Tan, Yanchao, et al. "MuseGraph: Graph-oriented Instruction Tuning of Large Language Models for Generic Graph Mining." arXiv preprint arXiv:2403.04780 (2024).

[3] Chien, Eli, et al. "Node feature extraction by self-supervised multi-scale neighborhood prediction." arXiv preprint arXiv:2111.00064 (2021).

[4] Huang, Xuanwen, et al. "Prompt-based node feature extractor for few-shot learning on text-attributed graphs." arXiv preprint arXiv:2309.02848 (2023).