Investigating Instruction Tuning Large Language Models on Graphs

Reason to reject

1. Thank you for your suggestion. In our final version, we will update this part by adding more details about the graph instruction tuning.
2. Thank you for suggesting a broader comparative analysis. We have recently conducted some experiments with another LLM, Google’s Gemma-7b. The current results on the Amazon Metadata network also support our following observation in the paper: JSON format gives the LLMs the best performance after tuning. The model presents an excellent generalization on the unseen sub-tasks of Node, Pair, Bool, and Graph tasks, with a small drop in the performance, but fails to generalize to the unseen sub-tasks of Count, Path, and Link prediction tasks. More models and results will be added to the final version of the paper.

Questions To Authors:

1. One of our research questions is “To what extent can LLMs fine-tuned on limited graph-related tasks generalize to unseen tasks?”, the graph-related tasks can be “limited” from several perspectives: limited queried information (only a few node types and edge types), limited graph structure distribution (a few distinct domains) and limited trained graph algorithm. Therefore, we would like to choose graph datasets that have only a few node and edge types and the datasets should be from distinct domains. Also, to study the LLM generalization capability with its general world knowledge, we would like the graph datasets to contain some general types, like paper and products, rather than some specialized types like diseases and pharmacology. Therefore, we choose the Amazon Metadata and MAPLE, where both contain 3 to 4 common node and edge types and have different graph structure distributions against each other.

2. We have collected some statistics of the graphs in both datasets. Below are the statistics of the Amazon Metadata training dataset using JSON format: max node #: 176, max edge #: 276, 80% have between 29 - 107 nodes, 80% have between 30 - 140 edges. We also observe that the instruction-tuned LLM’s performance on different tasks decreases differently as the number of nodes or edges increases. We will include more detailed data and discussion in our final version.

3. In most of the failure cases, the outputs are some meaningless text like “node1, node2, …”. We hypothesize that this is because LLM didn't understand the graph structure from our prompt. We will try different prompts with more detailed explanations and report any influence in the final version.

We have recently conducted some experiments with another LLM, Google’s Gemma-7b. The current results on the Amazon Metadata network also support our following observation in the paper: JSON format gives the LLMs the best performance after tuning. The model presents an excellent generalization on the unseen sub-tasks of Node, Pair, Bool, and Graph tasks, with a small drop in the performance, but fails to generalize to the unseen sub-tasks of Count, Path, and Link prediction tasks. More models and results will be added to the final version of the paper.

You only test three small-scale LLMs less than 10B. From my opinion, which data format for graphs is better is more likely model-independent and a such conclusion may be misleading. You can also recommended to test larger LLMs or APIs to convince the readers.

In most of the failure cases, the outputs are some meaningless text like “node1, node2, …”. We hypothesize that this is because LLM didn't understand the graph structure from our prompt. We will try different prompts with more detailed explanations and report any influence in the final version.

From my experience with LLMs for graph problems, this behavior seems unusual. It likes the behavior of LLaMA 2 7b without the chat version. The chat version can follow your insturctions and make a prediction. Please double check it, and also consider providing some examples in prompts.

Overall, I think the paper has some merits and interesting findings that worth an accept. So I raised my score.

Reasons To Reject:

1. Thank you for your suggestion! We agree that there are many networks from domains like chemistry and biology, which can make the benchmark more diverse and raise more general interests. Since one of our research questions is “To what extent can LLMs fine-tuned on limited graph-related tasks generalize to unseen tasks?”, the graph-related tasks can be “limited” for limited queried information (only a few node types and edge types) or limited graph structure distribution (a few distinct domains). This is why we choose the Amazon Metadata and MAPLE, where both contain 3 to 4 common node and edge types and have different graph structure distributions against each other. We will try to include more networks in our dataset if they satisfy these requirements.

2. We may have different ideas at this point. Our benchmark is built for two purposes: first, to find the optimal graph representation that helps LLMs learn graph structures effectively; second: to analyze the generalization capability of LLMs fine-tuned on limited graph-related tasks to unseen tasks. These are the novelty of our work.

Questions To Authors:

1. Yes, we have recently conducted some experiments with another LLM, Google’s Gemma-7b. The current results on the Amazon Metadata network also support our following observation in the paper: JSON format gives the LLMs the best performance after tuning. The model presents an excellent generalization on the unseen sub-tasks of Node, Pair, Bool, and Graph tasks, with a small drop in the performance, but fails to generalize to the unseen sub-tasks of Count, Path, and Link prediction tasks. More models and results will be added to the final version of the paper.

2. Our technique contributions are:

• We create a benchmark for investigating the influence of different graph representations in graph instruction tuning and the generalization capability of the LLM instruction tuned on limited graph-related tasks.
• Our result shows the JSON format gives the LLMs the best performance after tuning.
• Our experiments on generalization reveal tasks where LLM could be overfitted or hard to generalize. Our experiments also show LLM can derive algorithms from the learned algorithms.

3. We think the small models could be fine-tuned to each specific task with a specific network but is hard to generalize to a different task. On the other hand, LLM with strong capability in reasoning and understanding, can generalize better in unseen tasks.

Add to Table 3:

The results show JSON format provides the best performance after graph instruction tuning.

Add to Figure 3:

The results show the graph instruction tuned LLM generalizes well on unseen sub-tasks of Node, Pair, Bool, and Graph tasks, with a small drop in the performance, but fails to generalize to the unseen sub-tasks of Count, Path, and Link prediction tasks.

Thank you for your detailed response. Most of my concerns have been resolved. I will maintain the score.

Reasons To Reject:

1. Using a graph encoder to generate the graph representation is an interesting and novel approach. However, currently, there are many works still using the rule-based linearization approach, which can directly make use of the knowledge in the popular decoder-only LLM. Therefore, one of our goals in the paper is to provide insights into how different rule-based graph representations will affect the performance under the setting of graph instruction tuning so that it may help other researchers when choosing their graph representation.
2. Maybe this is not a drawback? Since LLM can be easily adapted to specific tasks with a few thousand training examples, a small and diverse dataset may be appropriate to study the generalization of LLM in graph-related problems.

Questions To Authors:

1. Yes, we do have a few queries written in natural language. For example, when asking for the products of a brand, the query is formulated as “What are the products of brandxxx?”. We mostly use the edge types because we can use some templates to generate different sub-tasks that share the same graph algorithm but focus on different node types or edge types.
2. This is an interesting direction to explore! However, we didn’t allow this. We test the LLM’s understanding of graph and graph algorithm by evaluating how well it can generate correct answers for any graph-related query so that we can decide whether LLM can be trusted as an independent graph problem solver. Therefore, we didn’t consider the approach of scoring over the possible outcomes. Since this may involve changing the dataset, we may not be able to test this during the rebuttal, but we may consider this as a future work.