GraphAgent: Exploiting Large Language Models for Interpretable Learning on Text-attributed Graphs

1. The authors claim to utilize LLMs to parameterize the graph agent, but they do not explicitly demonstrate the process of how to parametrize. It appears that LLM is inherently a pre-parametrized agent.

2. The framework seems to be an implementation of the chain-of-thought paradigm applied to LLMs in the context of graphs. Therefore, the novelty is limited. Also another existing work explored the graph agent [1], it is helpful to make comparisons and discussions about the differences and connections with it.

3. Can the proposed paradigm easily applied to other graph-related work, e.g., link prediction and graph classification. Can you provide some experimental results on these tasks.

4. Unclear notations: $s_v$ in section 3.1 is not defined and explained; the multiple uses of $i$ in $y_i$ and $c_i$ may lead to confusion.

5. Comparison with graph models that leveraged LLMs, e.g., [2-3], is necessary.

[1] Qinyong Wang, Zhenxiang Gao, and Rong Xu. Graph Agent: Explicit Reasoning Agent for Graphs. arXiv preprint arXiv:2310.16421 (2023).

[2] Ruosong Ye, Caiqi Zhang, Runhui Wang, Shuyuan Xu, and Yongfeng Zhang. Natural language is all a graph needs. arXiv preprint arXiv:2308.07134, 2023.

[3] Jianan Zhao, Le Zhuo, Yikang Shen, Meng Qu, Kai Liu, Michael Bronstein, Zhaocheng Zhu, and Jian Tang. Graphtext: Graph reasoning in text space. arXiv preprint arXiv:2310.01089, 2023.

• The description of the agent prompt can only be found in Appedix, making it difficult to understand the author's approach without referring to the supplementary materials.
• There is a lack of efficiency comparison. Intuitively, actions allow LLMs to see more node information, but processing this information also incurs additional costs. However, the paper's experiments do not discuss this issue.
• There is a lack of discussion on stability. I am concerned about whether small perturbations can make the agent unstable, which would greatly affect the reliability and reproducibility of the experimental work.

1. The case shown in Figure 2 is not clear. Which part is the output of the LLM and which part the given by the environment? Besides, Section 4.1 designs three types of actions, but their formats are not consistent. I can find "get features" and "get neighbors" as actions but the answer is given by natural language. Please clarify your method.
2. According to [1], GPT3.5 can achieve 67% accuracy on Cora, 50% on arxiv, 70% on Product. But table 2 shows it can only achieve very low performance (gpt-3.5-turbo + ICL). I guess it is because of the prompt used. I suggest to use a better prompt for the baseline.
3. The performance of the method cannot outperform GNN based methods.

[1] Chen, Z., Mao, H., Li, H., Jin, W., Wen, H., Wei, X., ... & Tang, J. (2023). Exploring the potential of large language models (llms) in learning on graphs. arXiv preprint arXiv:2307.03393.

1. The experimental results are insufficient to substantiate claimed contributions. Specifically, the test is conducted on only 50 samples, which significantly undermines the credibility of the experimental outcomes. Concurrently, the interpretability mentioned by the authors lacks further elucidation and quantitative evaluation.

2. The performance gap between GraphAgent and conventional methods is still large. In fact, 9.7% in ogbn-arxiv, and a 6.5% margin in ogbn-products indicate a large gap instead of comparable performance. Not to mention the computational efficiency.

3. Possible test data leakage. The training corpus of GPT includes papers from arXiv, implying that the GPT model may already be aware of the test node labels, that is, the paper's field. This can be tested by directly asking GPT about a paper’s field, given its text-attributes (title and abstraction). The authors are encouraged to conduct this experiment to discuss the possible test data leakage problem.

4. Notations in Section 3.1 are really confusing. For example, what is f_v? y_L denotes “a few labeled node” while y_U are “labels of unlabeled nodes”. “K is the number of labeled nodes of each category” Does each category necessarily have the same number of labeled nodes K?

5. The writing needs to be polished and the authors need to be more careful with their claims. For instance, “For this goal, we introduce GraphAgent as our solution to leverage Large Language Models (LLMs) for graph machine learning tasks, with a specific focus on node classification due to its wide application.” Based on my understanding, GraphAgent might be a solution for the node-classification task in text-attributed graphs, instead of a solution for graph machine-learning tasks with a focus on node classification.

6. It might be better to place the tables and figures on the top of each page.