Combining Structure and Text: Learning Representations for Reasoning on Graphs

Thank you for your positive comments on our work! Below are our responses to your concerns.

More details about experiments and evaluation metrics

Thanks for your valuable suggestions. We provide details about datasets, hyperparameters, and implementation details in the Appendix, and we will add more necessary details in the later version. For the evaluation metrics, we choose the widely used metrics as described in Line 366.

Impact of hyperparameters

Our approach is based on existing LM and GNN methods, the effects of hyperparameters have been explored in their original papers, and we simply choose the optimal combination of parameters. In addition, in Appendix D we discuss the effect of the newly key hyperparameter introduced by our method, i.e. the number of pseudo-targets, and we can observe that our method is relatively stable for the performance of this parameter.

Finally, thank you again for your efforts and we will refer to your comments based on improving our paper.

Dear reviewer jTir,

We appreciate your reviews. As the discussion period ends very soon, may we ask if our response to your concerns is helpful? We are always happy to answer any questions and improve our work.

Sincerely,

Authors of paper3161

Thank you for your constructive review and suggestions! We are pleased that you recognize our contribution to the work, experimentation, and writing. Below, we will respond to your concerns.

In the proof of Theorem 2.1 (Equation 13), the derivation of the last equality lacks detail. It requires a detailed elaboration on how the prefix $\mathcal{E}$ is added to the first term and why the second term lacks a $\mathcal{E}$ compared to the description in Theorem 2.1. Could you provide a step-by-step derivation of the last equality, specifically explaining the addition of the expectation operator to the first term and the absence of it in the second term? This would help clarify the mathematical reasoning behind Theorem 2.1.

1.how the prefix $\mathbb{E}$ is added to the first term

From the previous step of the last step, we have:

And then we integrate $q_{\theta}\left( {H_{(h,r)}} | h, r, \mathcal{E}, \mathcal{T} \right)$ on both sides of the equation. The formula on the left is independent of $q _ {\theta}\left( {H_{(h,r)}} | h, r, \mathcal{E}, \mathcal{T} \right)$, so it remains the same, the first term $\int _ {q _ {\theta}\left( {H _ {(h,r)}} | h, r, \mathcal{E}, \mathcal{T} \right)}\log \frac{p _ {\phi} \left( {O _ {(h,r)}}, {H _ {(h,r)}} | h, r, \mathcal{E} _ {/\{(h,r,\hat{t})\} _ {\hat{t}\in O_{(h,r)}}}, \mathcal{T} \right)}{q _ {\theta}\left( {H_{(h,r)}} | h, r, \mathcal{E}, \mathcal{T} \right)}$ in the formula on the right can be regarded as the expectation of $\log \frac{p _ {\phi} \left( {O _ {(h,r)}}, {H _ {(h,r)}} | h, r, \mathcal{E} _ {/\{(h,r,\hat{t})\} _ {\hat{t}\in O _ {(h,r)}}}, \mathcal{T} \right)}{q _ {\theta}\left( {H _ {(h,r)}} | h, r, \mathcal{E}, \mathcal{T} \right)}$ with probability $q _ {\theta}\left( {H _ {(h,r)}} | h, r, \mathcal{E}, \mathcal{T} \right)$, and the second term $-\int _ {q _ {\theta}\left( {H _ {(h,r)}} | h, r, \mathcal{E}, \mathcal{T} \right)} \log \frac{p _ {\phi} \left( {H _ {(h,r)}} | h, r, \mathcal{E}, \mathcal{T} \right)}{ q _ {\theta}\left( {H _ {(h,r)}} | h, r, \mathcal{E}, \mathcal{T} \right)}$ is the divergence of KL between $p _ {\phi} \left( {H _ {(h,r)}} | h, r, \mathcal{E}, \mathcal{T} \right)$ and $q _ {\theta}\left( {H _ {(h,r)}} | h, r, \mathcal{E}, \mathcal{T} \right)$. So we can draw conclusions to the last step.

2.why the second term lacks a $\mathcal{E}$

As mentioned in the paper (Line 153), $q_{\theta}\left( {H_{(h,r)}} | h, r, \mathcal{E}, \mathcal{T} \right)$is modeled by LM model and the input to LM model is only related to $h,r$and $\mathcal{T}$, so actually we have$q_{\theta}\left( {H_{(h,r)}} | h, r, \mathcal{E}, \mathcal{T} \right) = q_{\theta}\left( {H_{(h,r)}} | h, r, \mathcal{E}, \right)$. Sorry for the confusion, we will modify it in the later version to keep the description consistent.

Could you explain the rationale behind choosing these specific metrics for each dataset, and whether this choice affects the comparability of results across datasets?

We chose the metrics following the works that present certain datasets. In our view, different metrics are employed for different datasets due to their distinct characteristics and purposes. A smaller value of $n$ (e.g., $H@1$) indicates a greater emphasis on the precision of the model's output, which is more relevant for datasets like question-answering datasets. Conversely, a larger value of $n$ (e.g., $H@50, H@100$) indicates a greater focus on the recall performance of the model, which is more pertinent for datasets like recommendation datasets. We believe that this approach does not compromise the evaluation of different models' performance, as the overall performance of the same model generally exhibits a linear relationship with $n$.

Supplementary figures

Thanks for your suggestion, we have tried to add more necessary figures in the latest version.

Finally, thank you again for your review. Your comments are very important for the improvement of our work. If you have other concerns, please let us know.

Thanks for your time and valuable reviews. We are pleased with your recognition of the methods and effectiveness of our work, and we will address your main concerns below.

For various LM backbone and why not use LLM backbone

• In our work, the role of the Language Model (LM) is to transform the textual information of nodes and edges into high-quality text embeddings. Therefore, we selected the MiniLM model, which is widely recognized for its effectiveness in text representation tasks and is one of the most downloaded models in the sentence-transformers library. Additionally, in Appendix D, we present the performance of our methods under different LM architectures, including Tiny-Bert, Bert, and MiniLM. Our results demonstrate consistent performance improvements across these architectures. So we believe that CoST can migrate well to other models.
• The primary reason we did not conduct experiments with Large Language Models (LLMs) like LLaMA is that LLMs are typically designed with a decoder-only architecture, which excels in natural language generation but is not well-suited for generating text embeddings. In contrast, most contemporary text embedding models are based on encoder architectures, which are better suited for this task. However, we also provide some results based on llama3.2-1b for reference, although it has more parameters, it does not bring much performance improvement.

Complexity of Training Paradigm

• Our method can be seamlessly integrated into existing Language Model (LM) and Graph Neural Network (GNN) model codebases with minimal modifications.
• As demonstrated in the experiments presented in Section 3.3 and Appendix D, our method exhibits rapid convergence when applied to pre-trained LM and GNN models, requiring fewer training iterations.
• Furthermore, our method demonstrates robustness to the introduction of primarily additional hyperparameters, such as the number of pseudo examples.

Details about Pseudo Targets

1.Selection of pseudo-targets

As mentioned in Lines 232 and 266, we obtain pseudo targets by sampling from the distribution generated by the LM/GNN from the previous iteration, which can be viewed as a Monte Carlo estimation. In our method, this approach allows us to optimize the correlation between distributions, thereby minimizing the variational loss. Instead of using pseudo targets as direct input signals for teacher forcing training, the quality of the pseudo targets is less critical. In contrast, the number of pseudo targets is more important. This is because the number of pseudo targets influences the precision of the estimation.

2.Impact of the number of pseudo-targets

In Appendix D, we provide the experimental results regarding different numbers of pseudo-targets. It can be observed that as the number of pseudo-targets increases, the performance also improves, which is consistent with our previous hypothesis. However, this improvement is limited, and our model exhibits notable performance enhancements even with a relatively small number of pseudo-targets.

Please see Part II

Experiments of GNN+LLM in Table 3 and Table 4

Initially, we try to provide the performance of GNN+LLM baselines in all datasets. However, since the scale of datasets including CitationV8 and GoodReads is very large, the time overhead and GPU memory overhead of LLM-engaging methods are hardly acceptable. That's why many LLM-engaging methods will sample a small dataset for experiments. So we include the small-scale datasets including AmazonSports, AmazonClothing, MAGGeology, and MAGMath to compare with LLM-engaging methods. For the heterogeneous datasets FB15k237 and WN18RR, there lack baselines that combine the GNN and LLM, which mainly dues to the large scale of these datasets and the additional complexity of modeling the multi-relation. And most existing works [1][2] only work on link-level prediction, which is not scalable in our settings, since it needs $n$ forward inference for reasoning over all $n$ nodes. For example, on FB15k237 dataset, it will need 1w+ forward inferences for single query $(h,r,?)$. Based on this, we think our approach can provide insights on these datasets. There are indeed works that only use textual information to reason by LLM generation, however, because the structural information is essential in the graph reasoning task, this paradigm usually performs poorly, please refer to the following table for instance.

[1] LLaGA: Large Language and Graph Assistant. ICML 2024.

[2] One for All: Towards Training One Graph Model for All Classification Tasks. ICLR 2024.

[3] ExploringLargeLanguageModelsforKnowledgeGraphCompletion. Arxiv 2024.

Lastly, thank you again for your careful review. We hope that our response will address your problems and if you still have additional problems, please don't hesitate to let us know.

Thank you for addressing my concerns! I will update my rating accordingly.

Thank you for your efforts and detailed review. We are glad that you recognize the motivation and effectiveness of our work. In response to some of your concerns and questions, we make the following responses.

Clarification of position and the contribution

1. Clarification of task

Due to the unique characteristics of text-attributed graphs, several studies have explored the inference of representation learning and downstream tasks by integrating text attributes and structural information. However, most of this work (as works [1][2][5] you have given) has primarily focused on:

• Node-level and graph-level classification tasks.
• Homogeneous datasets.

In contrast, this paper works on the graph reasoning task, leading to the following key differences:

• Graph reasoning task requires inferencing a set of answering nodes based on graph data for a given query node (and possible query relation). Unlike node-level and graph-level classification tasks, where the classification label is fixed, the graph reasoning task requires considering different query conditions and predicting dynamic answer node sets.
• Compared with the link-level classification task, the definition of graph reasoning task is more general, which requires the reasoning result of any candidate node, while the link-level classification problem always makes inference between two given nodes. On a graph with $n$ nodes, The graph reasoning model usually obtains the reasoning result of any candidate node through one-step forward, while the link-level classification model requires $n$ times forward inference.
• Additionally, we consider heterogeneous data, which requires additional modeling for relational dimensions.

2. Clarification of CoST

Although our paper and some other work (as works [3][4][5] you have given) also use the idea of variational EM, the specific challenges and differences of our work differentiate it from the aforementioned work. For example:

• Due to the dynamic set of answer nodes, the estimation of the posterior distribution is not as straightforward as the classification problem of fixed label sets;
• Due to the existence of query conditions and the requirement of pairwise representation, there are specific challenges in the construction of LM and GNN models and the interaction between them.

Also, we have also discussed similar points in Appendix A of the paper, which you can refer to.

About comparisons with GNN-based and LLM-based baselines

First, we want to clarify that the naming of some baselines as GNN-based or LM-based does not imply that they solely utilize structural information or text information. For example, GNN-based baselines (e.g., those listed in Table 2 and Table 3) use fixed text embeddings as features, while LM-based baselines (e.g., Topological Contrastive Learning based Method) incorporate structural information through structural training methods. For scalability reasons, these baselines often fix certain models, which limits their performance. This limitation is a primary motivation for our work. Therefore, we believe it is reasonable to compare our approach with these baselines.

Second, we have made efforts to include more GNN+LLM baselines. However, these methods either: 1) Are not suitable for graph reasoning tasks [1][5], or 2) Require very high computational overhead when run on graph inference tasks [2]. Nevertheless, we have done our best to incorporate some GNN+LLM baselines (e.g., LinkGPT, LLaGA) on small-scale benchmarks.

About the writing of abstract and introduction

Thank you for your suggestions. Previous work in graph reasoning has focused less on the text modality, which is why we have emphasized this part extensively. However, your suggestions are very valuable, and we will modify our manuscript accordingly to highlight the specific challenges and contributions of our work. We appreciate your feedback.

Please see Part II

Q1: In Equation 3, how do you define the probability over an answer set $O_{(h,r)}$? Equation 1 & 2 only define the probability over a single answer.

Similar to Equation 5, the distribution is defined by $p _ {\phi}\left( O_{(h,r)} \vert h, r, \mathcal{E} _ {/\{(h,r,\hat{t})\} _ {\hat{t} \in O _ {(h,r)}}}, \mathcal{T} \right) = \prod _ {\hat{t}\in O _ {(h,r)}} p _ {\phi}\left( \hat{t} \vert h, r, \mathcal{E} _ {/\{(h,r,\hat{t})\} _ {\hat{t} \in O _ {(h,r)}}}, \mathcal{T} \right)$.

Q2: How to understand the meaning of the latent variable $H _ {(h,r)}$? It looks like $\mathcal{V} _ {/O _ {(h,r)}}$excludes the ground truth answers from all entities. How can you call it the candidate target node set for the query $(h,r,?)$?

Most real-world graph datasets are incomplete, so the $O _ {(h,r)}$is only the observed answer node set. $H _ {(h,r)} = \mathcal{V} _ {/O _ {(h,r)}}$could contain some unobserved answer nodes, so we call it candidate target node set.

Q3: Equation 10 should be log probabilities, not probabilities.

Thank you for pointing out the problem, we will fix it.

Q4: Why do you omit $H_{(h,r)}$in $p_{\phi}$ in the second line of Equation 10?

We have $O_{(h,r)} \cup H_{(h,r)} = \mathcal{V}$, so we can use the summation over $t \in \mathcal{V}$.

Q5: Line 461 - 463: I understand CoST pretrained GNN only uses structural information. How do language models come up here?

We leverage the pretrained-language model to generate text embeddings and the generated embeddings is the input to the GNN model.

Q6: It's a little bit complicated for CoST to use two optimization stages. Is there any reason for not using a mixed objective of pseudo labels and ground truth.

As we mentioned in Line 143-145, to jointly optimizing LM and GNN is not tractable, especially for large-scale datasets. So in this work, we leverage an alternative optimization process. [5] also includes two-stage optimization as described in Section 4.3 and 4.4 of the original paper.

Finally, thank you again for your careful review, your comments are very helpful to improve our paper. I hope our response has solved your questions, and if you have any questions, please don't hesitate to let us know.

Thanks the authors for their proper response.

I agree with the authors' clarification of the task. With that being said, given the well-known nature of combining two modalities with variational inference, I can hardly see any new insight or lesson from this paper. Hence, the contribution of this paper seems limited to applying an existing methodology to a new task.

For the experiments, the authors have convinced me that the baselines for homogeneous datasets are strong enough. However, my concern remains regarding the baselines for heterogeneous datasets, as none of them consider both structure and text.

Based on these points, I decide to keep my rating.

Dear reviewer LMJF,

We appreciate your reviews. As the discussion period ends very soon, may we ask if our response to your follow-up concerns is helpful? If the concerns have been fully addressed, we would be grateful if you could consider updating your score accordingly. Thank you again for your thoughtful consideration and hard work.

Sincerely,

Authors of paper3161