Curriculum GNN-LLM Alignment for Text-Attributed Graphs

Q4: "In the main results, it seems that a simple GCN outperforms models like TAPE and ENGINE."

To ensure a fair comparison, we follow GLbench, a widely recognized benchmark for GraphLLM, to standardize the LLMs and GNNs used in both GraphLLM baselines and our method. Specifically, in GLbench, the GNNs in the GraphLLM baselines are replaced with GraphSAGE, the LLMs in the LLM-as-predictor methods are replaced with LLaMA2-7B, the LLMs used as text encoders in the GraphLLM baselines are replaced with Sentence-BERT, and BERT is employed to process raw text into embeddings for Vanilla GNNs (e.g., GCN, GAT, and GraphSAGE), which is also adopted in our method. Since we share the same goal of ensuring a fair comparison as GLbench, we follow their setting. However, our method is not constrained by this setting. As new evaluation methods emerge in future benchmarks, our approach can be seamlessly extended to those settings to maintain fair comparisons.

Our experimental results are similar to the findings of previous works, GLbench [1]. One possible reason is that GCN performs better than GraphSAGE, which causes some SOTA baselines to perform worse than GCN in this case.

We truly hope we’ve answered all your questions. Thank you for your support, and we’re happy to discuss further if there’s anything we missed. Thank you again!

Best regards,

The authors

Reference:

[1] Li, Yuhan, et al. "Glbench: A comprehensive benchmark for graph with large language models." NeurIPS 2024.

[2] Zhao, Jianan, et al. "Learning on large-scale text-attributed graphs via variational inference." ICLR 2023.

[3] Chen, Zhikai, et al. "Label-free node classification on graphs with large language models (llms)." arXiv preprint arXiv:2310.04668 (2023).

[4] Zhu, Yun, et al. "Efficient tuning and inference for large language models on textual graphs." CoRR 2024.

[5] He, Xiaoxin, et al. "Harnessing explanations: Llm-to-lm interpreter for enhanced text-attributed graph representation learning." ICLR 2024.

[6] Liu, Hao, et al. "One for all: Towards training one graph model for all classification tasks." ICLR 2024.

Q2: "In section 4.4, why does the accuracy of pseudo-labeled nodes selected by CurGL decrease as the number of steps increases?"

We sincerely apologize for not providing a clear explanation of this issue in the manuscript. We greatly appreciate your valuable suggestion. Below is our explanation, which has also been added to the manuscript along with relevant analysis.

Our method for calculating the accuracy of pseudo-labeled nodes is to divide the number of correctly labeled nodes among the selected $V_s$ nodes by the total number of selected $V_s$ nodes. However, as the number of selected pseudo-labeled nodes $V_s$ increases, the proportion (accuracy) tends to decrease, which is a reasonable phenomenon. For instance, if we select all pseudo-labeled nodes as $V_s$, the accuracy will approximate the overall accuracy of all pseudo-labels.

Definitions:

• $N$: Total number of pseudo-labeled nodes.
• $C$: Total number of correctly pseudo-labeled nodes. Overall accuracy: $A_{\text{overall}} = \frac{C}{N}$.
• $V_s$: Selected subset of pseudo-labeled nodes, with $k = |V_s|$.
• $C_s$: Number of correctly labeled nodes in $V_s$. Accuracy for $V_s$: $A_{V_s} = \frac{C_s}{k}$.

Step 1: Adding More Nodes

• Add a subset $\Delta V$, where $|\Delta V| = m$ and $\Delta C$ are correct labels.
• New accuracy after adding $\Delta V$: $$A_{V_s \cup \Delta V} = \frac{C_s + \Delta C}{k + m}.$$
• Since $\Delta C \leq m$, we have: $$A_{V_s \cup \Delta V} \leq \frac{C_s + m}{k + m}.$$

Step 2: Convergence

• As $k \to N$, $C_s \to C$: $$A_{V_s} \to \frac{C}{N} = A_{\text{overall}}.$$
• This shows that $A_{V_s}$ decreases as $k$ increases, converging to $A_{\text{overall}}$.

Conclusion:

The accuracy $A_{V_s}$ decreases with the size of $V_s$ and eventually converges to $A_{\text{overall}}$, as adding more nodes includes lower-confidence samples.

The iterative enhancement with confident pseudo-labels aims to progressively integrate valuable information between the LLM and GNN, enriching both components with a balanced combination of textual and structural insights. Unlike approaches that depend on random pseudo-label sampling, which can introduce substantial label noise, our curriculum strategy focuses on selecting confident pseudo-labels. This ensures more reliable mutual learning between the components and improves the overall effectiveness of the integration process.

Q3: "In Section 4.5, why do both of the two components, LLM and GNN, exhibit opposite trends across different datasets?"

Thank you for your comment. The LLM and GNN in Figure 5 (c) and (d) exhibit a trend of simultaneous improvement during the training process. This demonstrates that our LLM and GNN can collaboratively advance through iterative training.

Dear Reviewer s8oL

We sincerely appreciate your valuable comments! We believe the main questions may arise from a slight misunderstanding of our experiments. We address your concerns as follows.

W1: "There is a lack of explanation and mathematical proof as to why the EM algorithm is used instead of updating the model parameters of LLM and GNN simultaneously."

Thank you for your valuable suggestion. We have added explanations and mathematical proofs related to the EM algorithm in Appendix Section H.

W2: "The experiments in this work are insufficient, as they only include node classification experiments without covering graph classification or link prediction. Additionally, I have some concerns about the current experimental results, see below."

Many recent studies (e.g., [1–5]) focus on learning for text-attributed graph node classification. Our method emphasizes jointly learning both textual and structural features of nodes to learn effective node representations, which could be used for a variety of applications such as node classification and link prediction. The text embeddings generated by our method (LLM component) can serve as initial features for GNNs, and our fine-tuned LLM can be transferred to other methods that use LLMs as text encoders (e.g., OFA [6]), making them adaptable to various downstream tasks such as link prediction and graph classification.

Q1: "Could the authors provide more insights into the selection of the and values? For different datasets (Cora and Citeseer in Figure 4), they seem to require different values."

The hyperparameter $\lambda_0$ in Eq. 8 represents the initial selection proportion of candidate nodes, and its value is adjusted within the range $[0, 1]$. As shown in the experiments, $\lambda_0$ should be selected appropriately for each dataset. Setting $\lambda_0$ too high $\left(\lambda_0 \rightarrow 1\right)$ may result in the inclusion of too many samples initially, causing the training process to resemble standard training and neglecting the progressive nature of curriculum learning. On the other hand, setting $\lambda_0$ too low $\left(\lambda_0 \rightarrow 0\right)$ may result in the use of too few samples at the beginning, preventing the model from sufficiently learning basic features, which can adversely affect subsequent training stages.

The hyperparameter $\alpha$ in Algorithm 3 controls the number of selected pseudo-labeled nodes, and we adjust its value between 0 and 1. It is also essential to choose a moderate value to introduce an appropriate number of pseudo-labels. A value that is too high $\left(\alpha \rightarrow 1\right)$ would incorporate all pseudo-labels, causing the model to overly rely on them, while a value that is too low $\left(\alpha \rightarrow 0\right)$ would make the model overly depend on the original training labels, hindering the mutual learning between the GNN and the LLM.

Dear s8oL,

I hope this message finds you well. I am writing to kindly remind you about providing your response to the ongoing review discussion for our ICLR submission titled ”Curriculum GNN-LLM Alignment for Text-Attributed Graphs”.

We deeply value your insights and feedback, as they play a crucial role in shaping the quality and impact of our work. If you require any further clarification or additional information from our side to assist with your response, please do not hesitate to let us know.

We appreciate your time and effort in contributing to the peer-review process and look forward to your valuable input.

Best regards,

All authors

W4: "What is the computational efficiency of the proposed model? Calculating node difficulty involves computing shortest paths between nodes, which increases computational complexity and may limit the model’s applicability to large-scale datasets."

We appreciate your suggestion for computational efficiency. We have added efficiency analysis experiments in Appendix section F. Besides, we have extended our method to large scale datasets ogbn-products.

W5: "The integration of curriculum learning into the training stage is unclear. According to Algorithm 2, the training process appears closely similar to that of the existing method."

This algorithm utilizes a curriculum strategy to synergistically align the LLM and GNN. Initially, both models undergo pretraining on labeled data $\mathcal{V}_L$ to produce initial pseudo-labels $\hat{\mathbf{y}}$. Over the $T$ iterations, the models alternate in an EM process.

In each iteration, a candidate set of nodes $\mathcal{V}^t_C$ is selected, comprising both ground-truth and pseudo-labeled nodes. The difficulty of each node $D_{st}$ is then assessed, and a subset of nodes $\mathcal{V}^t_S$ is chosen based on their difficulty levels. This selection is tailored to each model’s strengths: the GNN prioritizes nodes with structural difficulty, while the LLM addresses nodes with textual difficulty. Following this, pseudo-labels $\hat{\mathbf{y}}$ are updated to reflect the refined insights gained during the step.

By leveraging the complementary capabilities of the LLM and GNN, this iterative process incrementally improves the accuracy of the pseudo-labels, ultimately enhancing the performance of the overall model.

W6: "The mathematical representation of equations and loss functions lacks clarity."

• a. "In Eq. (1), what does "MLP" represent? Is it a large language model or a multi-layer perceptron?"

MLP in Equation (1) refers to a multilayer perceptron used for classification immediately before the Softmax layer.

• b. "What is the relationship between $\hat{y}$ and $f_\theta(S)$ ? Are they equal?"

Yes, both $\hat{y}$ and $f_\theta(S)$ represents the predicted logits.

• c. "In Eq. (6), there is no $\hat{y}_v$. What does 1 represent? Is it a column vector or a scalar? Additionally, if $D_t(v)$ is larger, does this imply that the loss is also larger?"

$\hat{y}_v$ represents the output logits of node $v$ from either the LLM or GNN. The text difficult $D_t(v)$ is measured as the residual between the groud truth value(1) and $\hat{y}_v$.

Thanks! We have corrected the mentioned notations and rigorously checked other notations in the revised paper.

Q1: "Does selecting nodes with lower difficulty improve classification accuracy on nodes with higher difficulty?"

Previous studies have shown that an easy-to-hard learning strategy can significantly enhance a model’s generalization[2-4]. In our approach, we adopt a text-focused curriculum strategy for the LLM and a structure-focused curriculum strategy for the GNN. This dual curriculum enables both models to gradually develop a deeper understanding of textual and structural information, thereby reducing the risk of relying on shortcuts. Our experimental results verify the effectiveness of this approach.

We truly hope we’ve answered all your questions. Thank you for your support, and we’re happy to discuss further if there’s anything we missed. Thank you again!

Best regards,

The authors

Reference:

[1] Li, Yuhan, et al. "Glbench: A comprehensive benchmark for graph with large language models." NeurIPS 2024.

[2] Bengio, Yoshua, et al. "Curriculum learning." Proceedings of the 26th annual international conference on machine learning. 2009.

[3] Wang, Xin, Yudong Chen, and Wenwu Zhu. "A survey on curriculum learning." IEEE transactions on pattern analysis and machine intelligence 44.9 (2021): 4555-4576.

[4] Li, Haoyang, Xin Wang, and Wenwu Zhu. "Curriculum graph machine learning: A survey." arXiv preprint arXiv:2302.02926 (2023).

Dear Reviewer ZPUW

We highly appreciate your insightful feedback for refining our work. We address your concerns as follows.

W1: "The novelty of this paper is incremental. It is evident that the ideas of difficulty measurement and curriculum learning are borrowed from existing works, such as [1,2]. Additionally, using loss as a measure of learning difficulty is a well-adopted approach."

The previous GraphLLM paper employs the EM algorithm to jointly train LLM and GNN. However, our motivation is to address the issue of text-structure imbalance among nodes. While the EM algorithm is a classical approach, we adopt different curriculum strategies for GNN and LLM within the EM framework to tackle this imbalance. Specifically, the GNN focuses on structure-difficulty nodes, while the LLM targets text-difficulty nodes, ensuring that both models are fully trained and avoiding shortcuts.

In contrast, TSS uses a single curriculum to reduce the impact of noisy labels, whereas our approach measures the structural difficulty of nodes. We have relevant discussions in Appendix section A to highlight the differences between our method and TSS, as well as the advantages of our approach.

Using loss as a measure of learning difficulty is a widely adopted practice in previous research, as it is both simple and effective. We apply this established approach in our study. However, for our specific purpose of measuring the difficulty of nodes’ text, this method has not been explored before. Given that LLMs are specialized in text processing, their misjudgment of textual inputs can serve as a reliable indicator of the complexity of understanding the text. Our experiments confirm that using loss as a measure is effective in this context.

W2: "The authors emphasize the text-structure imbalance problem, i.e., the learning difficulties of different nodes vary depending on textual attributes and topological structures. However, they provide only an intuitive example in figure 1 to illustrate this issue, without presenting real experimental results to substantiate the existence of this problem."

We appreciate your suggestion for presenting real experimental results to substantiate text-structure imbalance problem. We have added an experiment in Appendix section G to illustrate the issue of text-structure imbalance. Specifically, we trained the LLM and GNN separately and evaluated their performance on the all nodes. The results show that some nodes are correctly predicted by the LLM but incorrectly predicted by the GNN, while others are correctly predicted by the GNN but incorrectly predicted by the LLM. This demonstrates the existence of the text-structure imbalance problem.

W3: "In Table 1, it is unusual that SOTA methods perform worse than the baseline GCN. An explanation for this unexpected phenomenon is needed."

To ensure a fair comparison, we follow GLbench, a widely recognized benchmark for GraphLLM, to standardize the LLMs and GNNs used in both GraphLLM baselines and our method. Specifically, in GLbench, the GNNs in the GraphLLM baselines are replaced with GraphSAGE, the LLMs in the LLM-as-predictor methods are replaced with LLaMA2-7B, the LLMs used as text encoders in the GraphLLM baselines are replaced with Sentence-BERT, and BERT is employed to process raw text into embeddings for Vanilla GNNs (e.g., GCN, GAT, and GraphSAGE), which is also adopted in our method. Since we share the same goal of ensuring a fair comparison as GLbench, we follow their setting. However, our method is not constrained by this setting. As new evaluation methods emerge in future benchmarks, our approach can be seamlessly extended to those settings to maintain fair comparisons.

Our experimental results are similar to the findings of previous works, GLbench [1]. One possible reason is that GCN performs better than GraphSAGE, which causes some SOTA baselines to perform worse than GCN in this case.

Dear ZPUW,

I hope this message finds you well. I am writing to kindly remind you about providing your response to the ongoing review discussion for our ICLR submission titled ”Curriculum GNN-LLM Alignment for Text-Attributed Graphs”.

We deeply value your insights and feedback, as they play a crucial role in shaping the quality and impact of our work. If you require any further clarification or additional information from our side to assist with your response, please do not hesitate to let us know.

We appreciate your time and effort in contributing to the peer-review process and look forward to your valuable input.

Best regards,

All authors

Q1: "How does the implementation achieve sampling of simpler nodes first?"

We rank all nodes in ascending order based on their text-structure difficulty within each class. Then, we select the top-rank nodes from each class according to a specified percentage, as detailed in Algorithm 1.

Q2: "Are the pretrained language models in the third block in Table 1 fine-tuned with labels?"

We use the same training set split and train the Pretrained language models on it as a text classification task.

We truly hope we’ve answered all your questions. Thank you for your support, and we’re happy to discuss further if there’s anything we missed. Thank you again!

Best regards,

The authors

Reference:

[1] Zeng, Liang, et al. "Imgcl: Revisiting graph contrastive learning on imbalanced node classification." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 9. 2023.

[2] Xue, Rui, et al. "Efficient End-to-end Language Model Fine-tuning on Graphs." arXiv preprint arXiv:2312.04737 (2023).

[3] Li, Yuhan, et al. "Glbench: A comprehensive benchmark for graph with large language models." NeurIPS 2024.

[4] Chen, Zhikai, et al. "Label-free node classification on graphs with large language models (llms)." arXiv preprint arXiv:2310.04668 (2023).

[5] Zhu, Yun, et al. "Efficient tuning and inference for large language models on textual graphs." CoRR 2024.

[6] He, Xiaoxin, et al. "Harnessing explanations: Llm-to-lm interpreter for enhanced text-attributed graph representation learning." ICLR 2024.

[7] Zhao, Jianan, et al. "Learning on large-scale text-attributed graphs via variational inference." ICLR 2023.

[8] Liu, Hao, et al. "One for all: Towards training one graph model for all classification tasks." ICLR 2024.

W6: "The paper’s claim of using LLMs seems overstated, as small language models are used. To solve this obvious mismatch and align claims with experiments , it would be more appropriate to use models like LLAMA or GPT."

Thank you for your valuable suggestion. In our experiments, to ensure a fair comparison, we standardize both the baselines and our method by using the same LLM and GNN, following GLBench. However, our method is inherently flexible and can be easily adapted to any LLM or GNN. Based on your insightful suggestion, we have utilized LLaMA2-7B as the LLM and obtained the following experimental results.

W7: "It is unclear if this approach generalizes to tasks beyond node classification, such as link prediction and graph classification. Focusing solely on node classification may limit the overall contribution."

Many recent studies (e.g., [3-7]) focus on learning for text-attributed graph node classification. Our method emphasizes jointly learning both textual and structural features of nodes to learn effective node representations, which could be used for a variety of applications such as node classification and link prediction. The text embeddings generated by our method (LLM component) can serve as initial features for GNNs, and our fine-tuned LLM can be transferred to other methods that use LLMs as text encoders (e.g., OFA [8]), making them adaptable to various downstream tasks such as link prediction and graph classification.

W8: "I am curious why GLEM achieves strong performance on Cora and Pubmed, which have very few labels. The results seem inconsistent with findings from existing works."

To ensure a fair comparison, we follow GLbench, a widely recognized benchmark for GraphLLM, to standardize the LLMs and GNNs used in both GraphLLM baselines and our method. Specifically, in GLbench, the GNNs in the GraphLLM baselines are replaced with GraphSAGE, the LLMs in the LLM-as-predictor methods are replaced with LLaMA2-7B, the LLMs used as text encoders in the GraphLLM baselines are replaced with Sentence-BERT, and BERT is employed to process raw text into embeddings for Vanilla GNNs (e.g., GCN, GAT, and GraphSAGE), which is also adopted in our method. Since we share the same goal of ensuring a fair comparison as GLbench, we follow their setting. However, our method is not constrained by this setting. As new evaluation methods emerge in future benchmarks, our approach can be seamlessly extended to those settings to maintain fair comparisons.

Many previous works may have overlooked these settings for a fair comparison. Our results are consistent with the GraphLLM benchmark, GLBench[3].

W9: "Similar to GLEM, this method relies heavily on the quality of pseudo-labels. Although an ablation study is provided, it does not fully address potential issues that could arise with low labeling ratios."

Unlike GLEM[7], it lies in addressing the issue of text-structure imbalance in node representation learning. While the EM algorithm serves as a classical approach, our Curriculum Co-play Alignment method introduces the following unique aspects:

1. Curriculum strategies tailored to text-structure imbalance: To tackle the imbalance issue, we design separate curriculum strategies for the GNN and LLM within the EM framework. Specifically, the GNN processes structure-difficulty nodes progressively, from easy to hard, while the LLM focuses on text-difficulty nodes. This ensures both components undergo comprehensive training and avoid shortcuts.

2. Iterative enhancement with confident pseudo-labels: A key objective is to iteratively integrate valuable information between the LLM and GNN, progressively enriching both components with a balance of textual and structural insights. Unlike methods that rely on random pseudo-label sampling—often introducing significant label noise, our curriculum strategy emphasizes the selection of confident pseudo-labels to facilitate mutual learning between the components. As shown in Figures 5(a) and 5(b), our method effectively selects high-confidence pseudo-labels.

Our method focuses on selecting confident pseudo-labeled samples rather than random pseudo-labeled samples, making it more robust to low-quality labels. Comparative experiments under low-quality label conditions, as detailed in Appendix Section E, show that our approach performs effectively even when the labels are of low quality.

Dear Reviewer S9p5

We highly appreciate your insightful feedback for refining our work. We address your concerns as follows. Please feel free to correct us if otherwise!

W1: "My primary concern is whether this method is effective for heterophilic graphs, as the proposed difficulty measurer seems tailored to homophilous graphs."

We appreciate your valuable insights. Currently, most text-attributed graph datasets are homophilous graphs, and our method is designed based on this assumption. The core idea of our structure difficulty measure is to evaluate whether a node lies near the class boundary or at the class center to measure its learning difficulty. For heterophilic graphs, such as sequential recommendation datasets, where edges are more likely to connect nodes from different classes, we can hypothesize that nodes near the class boundary are easier to learn due to their connections to different types of nodes, while nodes at the class center are more challenging to learn. Thus, our method can be naturally extended to heterophilic graphs by leveraging this assumption.

W2: "The class-based node selection assumes balanced class sampling within each subgraph. However, this assumption may be problematic for highly imbalanced datasets, potentially leading to biased or suboptimal node selection."

The purpose of class-based node selection is to incorporate difficulty considerations when selecting nodes based on their classes. For datasets with class imbalance, we can address this issue by modifying the original sampling strategy from selecting nodes in proportion to their class sizes to selecting an equal number of nodes from each class. This adjustment effectively mitigates the problem of class imbalance among nodes.

Besides, the class imbalance setting for node classification typically involves an imbalanced distribution of classes in the training set but an equal number of nodes in each class for the validation and testing set(e.g. [1]). However, our method can select nodes from the entire dataset, including the training, validation, and testing sets, which significantly mitigates the issue of class imbalance in the training set.

W3: "Related to the above, it’s unclear how the authors handle edges between selected nodes in the class-based node selection. In each subgraph, nodes from different classes are sampled proportionally, which does not guarantee connectivity among them."

Thank you for your valuable suggestions. We have added the explanations about edges into the revised version of our paper. First, we measure the text-structure difficulty of all nodes. Then, we select a certain proportion of nodes from each category. Next, we retain the edges connecting these nodes from various classes to form a subgraph. However, we cannot guarantee that these nodes will be connected, as this depends on whether there are existing edges between them in the original graph.

W4: "Another major concern is computational overhead. This method involves additional steps beyond GLEM, which already has considerable running time. Efficiency analysis is essential and appears missing from the experimental section."

We appreciate your suggestion for cmputational efficiency. Computational efficiency is indeed a critical aspect. We compare the time efficiency of our method with the baselines and included additional experimental analyses in the Appendix F. The experimental results show that our method achieves a significant improvement over the baselines, with an acceptable time consumption.

W5: "The performance appears insufficiently strong; for instance, the results on Arxiv are weaker than those of methods like LEADING. Larger datasets, such as ogbn-products and ogbn-papers100M should also be included."

LEADING[2] demonstrates powerful experimental results. We add its discussion to Related works. However, in our experimental setting, to ensure a fair comparison, we standardize the baselines by using Sentence-BERT as the text encoder for LLM-based methods and LLaMA2-7B for methods where the LLM is a predictor. Nonetheless, the LLM and GNN architectures used in LEADING differ from those in our framework, making a direct comparison inequitable. As we do not find publicly available code for LEADING, we replace the LLM and GNN in our method, achieving better results than LEADING. Additionally, we have expanded the experiment for larger datasets ogbn-products as LEADING. The detailed results are as follows.

Dear S9p5,

I hope this message finds you well. I am writing to kindly remind you about providing your response to the ongoing review discussion for our ICLR submission titled ”Curriculum GNN-LLM Alignment for Text-Attributed Graphs”.

We deeply value your insights and feedback, as they play a crucial role in shaping the quality and impact of our work. If you require any further clarification or additional information from our side to assist with your response, please do not hesitate to let us know.

We appreciate your time and effort in contributing to the peer-review process and look forward to your valuable input.

Best regards,

All authors

Dear Reviewer S9p5,

I hope this message finds you well. I wanted to kindly remind you that the discussion period for our ICLR submission titled “Curriculum GNN-LLM Alignment for Text-Attributed Graphs” is approaching its conclusion.

We truly value your insights and feedback, and your response would greatly contribute to refining and strengthening our work. If there are any remaining questions or clarifications we can address to assist with your feedback, please don’t hesitate to reach out.

We understand how busy this time can be and sincerely appreciate the effort you invest in the review process. We look forward to your input and hope to hear from you soon.

Best regards, The authors

W4. "The proposed CurGL relies heavily on the availability of training data, and shows poor potential in transferring learning and zero-shot generalisation. Although it leverages an LLM for textual-based learning, the design of node-wise difficulty measurer and class-based selection strategy requires high-quality labels for the model to learn and converge. While it is quite reasonable for a graph learning model, the trained CurGL cannot be adapted to unseen data (unlike LLaGA, OFA and others), making its potential application quite narrow."

1. "transferring learning, zero-shot generalisation and unseen dataunseen data"

   Our framework is designed to enable LLMs and GNNs to effectively learn both textual and structural representations of nodes. Following the GLBench setup, we use node classification as the primary task to demonstrate the effectiveness of our method. In the future, we plan to extend our approach to other tasks. But now, many existing GraphLLM methods leverage LLMs as text encoders (e.g., OFA[6]). Our fine-tuned LLM can be effectively integrated into these methods, enhancing their performance in transfer learning, zero-shot tasks, and generalization to unseen data.

2. "the design of node-wise difficulty measurer and class-based selection strategy requires high-quality labels for the model to learn and converge"

   We have conducted comparative experiments under low-quality label conditions in Appendix section E. The results show that our method performs well even with low-quality labels. This is because our approach samples high-quality pseudo-labels as Figure 5 (a) and (b), thereby mitigating the impact of noise in the training set.

Dear Reviewer t915

We appreciate your efforts and insightful feedback, which have been instrumental in refining our work. We provide detailed responses below to address your concerns.

W1. "Novelty. The E-M training between LLMs and GNNs has been explored by a handful of pioneer works, e.g., SimTEG and GLEM. The proposed CurGL enhances such framework by adding additional strategies for balancing the LLM and the GNN."

The previous GraphLLM(e.g. GLEM[11]) paper employs the EM algorithm for jointly training LLM and GNN. However, our motivation lies in addressing the issue of text-structure imbalance in node representation learning. While the EM algorithm serves as a classical approach, our Curriculum Co-play Alignment method introduces the following unique aspects:

1. Curriculum strategies tailored to text-structure imbalance: To tackle the imbalance issue, we design different curriculum strategies for the GNN and LLM within the EM framework. Specifically, the GNN processes structure-difficulty nodes progressively, from easy to hard, while the LLM focuses on text-difficulty nodes. This ensures both components undergo comprehensive training and avoid shortcuts.

2. Iterative enhancement with confident pseudo-labels: A key objective is to iteratively integrate valuable information between the LLM and GNN, progressively enriching both components with a balance of textual and structural insights. Unlike methods that rely on random pseudo-label sampling often introducing significant label noise, our curriculum strategy emphasizes the selection of confident pseudo-labels to facilitate mutual learning between the components. As shown in Figures 5(a) and 5(b), our method effectively selects high-confidence pseudo-labels.

W2. "Unclear and potentially unfair experimental setting for baseline llm-as-predictor models. Is is unclear that how are the llm-as-predictor models testified, especially in Table 1. Is is based on a zero-shot manner as in Glbench, or are they also trained with the same training set? In terms of training, is it training-from-scratch or fine-tuning?"

1. "Unclear and potentially unfair experimental setting for baseline llm-as-predictor models.Is is unclear that how are the llm-as-predictor models testified, especially in Table 1. Is is based on a zero-shot manner as in Glbench, or are they also trained with the same training set?"

   To ensure a fair comparison, all our baselines strictly follow the supervised learning settings defined in GLBench[7]. Specifically, for the LLM-as-predictor methods, we train them using the same training dataset as the baselines and our method. We replace the LLMs in the LLM-as-predictor methods with Llama2-7B. This is because the different LLMs used in the original papers might influence the results. All models are trained on the same training set as the other baselines. For the LLM training part, we strictly follow the guidelines in the original papers, respecting whether LLM training is included or not.

2. "In terms of training, is it training-from-scratch or fine-tuning?"

   We fine-tune all models using the publicly available parameters of LLaMA2-7B, ensuring consistency and reproducibility across experiments.

W3. "Lack of discussion of a very relevant work, GraphGPT: Graph Instruction Tuning for Large Language Models."

Thank you for your suggestion. In response, we have added GraphGPT [1] in Appendix Section B under Related Work. GraphGPT leverages instruction tuning to adapt LLMs for downstream graph tasks, utilizing natural language alongside a graph-text aligner to effectively encode and communicate the structural information of graphs. This approach enables LLMs to understand complex graph structures and enhances their adaptability across diverse datasets and tasks. Additionally, we have included several other relevant studies [2–5].

W5. "Can CurGL generalise beyond node classification, to link prediction or graph classification?"

Many recent studies (e.g., [7-11]) focus on learning for text-attributed graph node classification. Our method emphasizes jointly learning both textual and structural features of nodes to learn effective node representations, which could be used for a variety of applications such as node classification and link prediction. The text embeddings generated by our method (LLM component) can serve as initial features for GNNs, and our fine-tuned LLM can be transferred to other methods that use LLMs as text encoders (e.g., OFA [6]), making them adaptable to various downstream tasks such as link prediction and graph classification.

We truly hope we’ve answered all your questions. Thank you for your support, and we’re happy to discuss further if there’s anything we missed. Thank you again!

Best regards,

The authors

Reference:

[1] Tang, Jiabin, et al. "Graphgpt: Graph instruction tuning for large language models." Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2024.

[2] Huang, Chao, et al. "Large Language Models for Graphs: Progresses and Directions." Companion Proceedings of the ACM on Web Conference 2024. 2024.

[3] Tang, Jiabin, et al. "Higpt: Heterogeneous graph language model." Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2024.

[4] Wei, Wei, et al. "Llmrec: Large language models with graph augmentation for recommendation." Proceedings of the 17th ACM International Conference on Web Search and Data Mining. 2024.

[5] Guo, Zirui, et al. "Graphedit: Large language models for graph structure learning." arXiv preprint arXiv:2402.15183 (2024).

[6] Liu, Hao, et al. "One for all: Towards training one graph model for all classification tasks." ICLR 2024.

[7] Li, Yuhan, et al. "Glbench: A comprehensive benchmark for graph with large language models." NeurIPS 2024.

[8] Chen, Zhikai, et al. "Label-free node classification on graphs with large language models (llms)." arXiv preprint arXiv:2310.04668 (2023).

[9] Zhu, Yun, et al. "Efficient tuning and inference for large language models on textual graphs." CoRR 2024.

[10] He, Xiaoxin, et al. "Harnessing explanations: Llm-to-lm interpreter for enhanced text-attributed graph representation learning." ICLR 2024.

[11] Zhao, Jianan, et al. "Learning on large-scale text-attributed graphs via variational inference." ICLR 2023.

Thanks for the authors' detailed response. I appreciate the efforts the authors have made to clarify on some of the concerns. The response addressed most of my questions. However, based on the description that "We replace the LLMs in the LLM-as-predictor methods with Llama2-7B. We fine-tune all models using the publicly available parameters of LLaMA2-7B, ensuring consistency and reproducibility across experiments.", I still think such a setting should be reconsidered. Directly re-initialize the parameters with Llama2-7B and train these Graph LLMs from scratch seems to diminish the advantage of large-scale graph-text SFT as a pre-training staged considered for Graph LLMs, and which is also fundamental for Graph LLMs. We need to reconsider whether further fine-tuning on the released checkpoints of these models would serve as a better comparison method. I understand under the current settings, it is convincing that CurGL derives more expressive embeddings for graph tasks with exactly the same amount of data. In that case, plugging CurGL to SOTA Graph LLMs as featurizers (like CLIP in MLLMs) seems to be more reasonable and easier to scale up. In all, I'll keep my vote for the current version, and I truly feel that this version has much space for improvement.

Dear Reviewer t915

Thank you very much for your valuable feedback, which has deepened my understanding of Graph LLMs. I fully agree that “the advantage of large-scale graph-text SFT as a pre-training staged considered for Graph LLMs”. However, I believe this represents only one part of the broader scope of Graph LLMs. In our paper, we follow the definition of GraphLLM as outlined in GLBench[1], where GraphLLM are defined as models that integrate LLMs and graphs to learn TAGs.

I believe current Graph LLM research can be broadly categorized into two parts. The first focuses on using LLMs for large-scale graph-text SFT, achieving strong performance across various domains/tasks, as you have aptly pointed out. The second focuses on processing graph-text data by leveraging LLMs/LMs to understand the textual modality rather than emphasizing their cross-domain and cross-task applications after large-scale pretraining. Both directions undoubtedly have their merits. Our work emphasizes the latter, where LLMs/LMs serve primarily as encoders for processing textual modality.

I deeply appreciate your suggestion, which has inspired us to explore whether graph-text SFT as a pre-training stage could enhance the performance of “LLM as a predictor,” potentially surpassing the “LLM as an enhancer” approach trained on a single dataset. Notably, we observe that some “LLM as predictor” baselines do not fine-tune their LLMs. Therefore, we select LLAGA[2] as a representative work that involves fine-tuning. Moreover, due to differences in the training/testing splits between the released pre-training parameters and our setup, there is a possibility that our test set overlaps with the training set used in the original paper. To address this, we pretrain LLAGA on our datasets and finetune it on individual datasets. The results are shown as below, indicating that LLAGA still falls significantly short of the performance achieved in our experiments. Due to time constraints, we are unable to conduct a more extensive comparison. We will further explore this in future work.

We sincerely appreciate your recognition of our work and your thoughtful suggestions. Your proposal to integrate CurGL into SOTA Graph LLMs as featurizers is exceptionally insightful. However, the current lack of a unified framework for Graph LLMs presents significant challenges in implementing it. We are eager to explore this direction further in our future work.

[1] Zhao, Jianan, et al. "Learning on large-scale text-attributed graphs via variational inference." ICLR 2023.

[2] Chen, Runjin, et al. "Llaga: Large language and graph assistant." ICML 2024.

Best regards,

The authors

R2: The Experiemental settings.

Baseline settings

To ensure a fair comparison, we follow GLbench[2] (published in NeurIPS 2024), a widely recognized benchmark for GraphLLM to standardize the LLMs and GNNs used in both GraphLLM baselines and our method. Specifically, in GLbench, the GNNs in the GraphLLM baselines are replaced with GraphSAGE, the LLMs in the LLM-as-predictor methods are replaced with LLaMA2-7B, the LLMs used as text encoders in the GraphLLM baselines are replaced with Sentence-BERT, and BERT is employed to process raw text into embeddings for Vanilla GNNs (e.g., GCN, GAT, and GraphSAGE), which is also adopted in our method. Since we share the same goal of ensuring a fair comparison as GLbench, we follow their setting.

GCN outperforms some GraphLLM methods, especially in Cora dataset.

Reviewing our table, GCN consistently outperforms GraphSAGE by 1–3% across all datasets. To ensure a fair comparison of GraphLLM methods, we replaced all GNNs in the GraphLLM baselines with GraphSAGE. This might explain why GCN performs particularly well on the Cora dataset, surpassing almost all GraphLLM baselines. However, it is evident that GCN still falls short compared to the best GraphLLM baselines on other datasets. Another possible reason for GCN’s relatively strong performance is that the initial textual input of the Cora dataset is already well-understood. As a result, adding LLM/LM enhancements may not yield significant improvements. Our experimental results are consistent with observations in GLbench[2].

More Experiements beyond node classification.

Our purpose is to learn a text-structure balance node representation, as node classification is the most relevant task for evaluating node representations. we have followed a well-known GraphLLM benchmark GLBench[2] that uses node classification as the evaluation task, We believe that our method’s performance surpassing this benchmark have effectively demonstrated its capability in learning node representations, aligning well with our purpose. Due to the limited time available during the rebuttal period, we may not be able to provide results for other tasks in a timely manner, as the GraphLLM baseline requires significant computational resources. We plan to extend our method beyond node classification in future work.

Compared to the initial version, we have made the following improvements based on the reviewers’ suggestions:

1. Thanks to Reviewer t915, We have added comparative experiments under low-quality label conditions in Appendix E. The results show that our method performs well even with low-quality labels. This can be attributed to our approach samples high-quality pseudo-labels as Figure 5 (a) and (b), thereby mitigating the impact of noise in the training set.
2. We have added the time efficiency of our method with the baselines and included additional experimental analyses in the Appendix F. The experimental results show that our method achieves a significant improvement over the baselines, with an acceptable time consumption.
3. Thank you to Reviewer ZPUW for the suggestion to present real experimental results to support the text-structure imbalance problem. We have added an experiment in Appendix G to illustrate the issue of text-structure imbalance. Specifically, we trained the LLM and GNN separately and evaluated their performance on the all nodes. The results show that some nodes are correctly predicted by the LLM but incorrectly predicted by the GNN, while others are correctly predicted by the GNN but incorrectly predicted by the LLM. This demonstrates the existence of the text-structure imbalance problem.
4. Thanks to Reviewer s8oL for the suggestion. We have included detailed explanations and mathematical proofs related to the EM algorithm in Appendix H.

We sincerely thank the reviewers for their valuable suggestions, which have strengthened our work. We hope our responses address any confusion and help alleviate your concerns.

Best regards,

The authors

Reference:

[1] Zhao, Jianan, et al. "Learning on large-scale text-attributed graphs via variational inference." ICLR 2023.

[2] Li, Yuhan, et al. "Glbench: A comprehensive benchmark for graph with large language models." NeurIPS 2024.