A Hierarchical Language Model Design For Interpretable Graph Reasoning

1. The design of natural language descriptions of graphs is very rigid. Personally, I don’t buy in this kind of language descriptions of the graphs according to my own experience.
2. As the authors mention that “the node feature annotation for a node is a natural language sequence that describes the attributes over a predefined vocabulary”, which indicates that the model can only be applied to text-attributed graphs, but no other kinds of the graph, which narrows the use of the model.
3. Lack key ablation studies on the blocks. The ablation experiments in the appendix actually don’t answer the questions of how each of the component in your framework contribute to the final performance. I’d like to see the experimental results with local block or global block only to see if these designs are really helpful.
4. To be honest, the synthetic graph datasets with up to 40 nodes and 700+ edges in graph reasoning are too small, and the experimental results are not persuasive. Although it is mentioned in the appendix that currently available datasets (e.g., NLGraph and GraphQA Benchmark) suffer from category imbalance and low number of nodes (around 10-20), these random synthetic graphs are still unconvincing. It is not possible to adequately demonstrate the performance of the model on real-world graph data.
5. Incomplete experimental comparison. In the experiments on real world datasets, the model is mainly compared with the zero/few samples setting of traditional GNN and LLM, and the coverage of the experiments is not comprehensive enough. For example, lacking LLaGA as mentioned in the existing work, OFA is only demonstrated in the link prediction task, lacking a broader comparison. This makes it difficult to fully assess the competitiveness of the model against state-of-the-art techniques.
6. I am doubt about the interpretability, and the approach of reflecting the interpretability of the model through the distribution of attentional weights is somewhat controversial. Whether the attentional weights can truly reflect the model's understanding of the graph structure is still an open question, as high weights do not always imply the importance of the features to the final decision.
7. When dealing with graphs with larger size, nodes are divided into different batches, each of which contains only 40 nodes in the paper, which make information difficult to flow over different batches, especially for the global block. (Actually there is even no interaction between different batches!)

Disadvantages

1. The motivation behind the paper is not clearly explained.
2. Section 3.1 presents the reformulation of graph-level task, describing the graph structure associated with each node. However, this approach may contain redundant information.
3. The ablation studies does not seem to explore the effect of local and global blocks on model performance, respectively.
4. Section 4.2 employs ground truth ranking to evaluate the model's interpretability; however, this approach seems imprecise. The ranking more accurately reflects the model's effectiveness rather than its interpretability.
5. The paper is not rigorously expressed. For example, the variable subscripts are inconsistent, and the variables “N” and “n” are used confusingly in Section 3.

1. The lines 040-058 of the article introduce two important challenges: the first is the challenge of obtaining structural information, and the second is the challenge of scalability. However, regarding challenge two, the strategy proposed in lines 060-072 does not provide me with a clear understanding of how this challenge is addressed.
2. From the design of the article, it is not clear how the ability of LLMs to understand graph structures is enhanced, as referenced in challenge 1. Moreover, according to Figure 1, I cannot even identify the role played by LLMs within it.
3. In the comparative analysis of interpretability performance, the article uses tasks such as shortest distance, reachability, edge existence, and node degree. However, the main downstream tasks used for comparison are graph classification and node classification tasks.
4. On page 20, in the Task 1 section for Shortest Distance, the range is only 0-5, and experiments with path graphs are missing. Clearly, the paths in path graphs are longer.
5. On page 19, the first line states, "In comparison, our method is inherently task-agnostic and demonstrates high interpretability." Honestly, I do not find this model to be particularly model-agnostic, and there is a lack of stronger experimental evidence to support its interpretability claims.
6. In the last sentence of the final paragraph on page 30, it states, "This indicates that HLM-G can effectively encode 1-hop neighborhood information, assigning higher similarity to nodes that are similar in position or structure." However, there is no explanation of the impact of 2-hop encoding on the nodes.
7. On page 30, the first line states, "This discrepancy arises because node 0, consistently presented at the beginning during training, is permuted during testing, causing BERT to misidentify its position." This suggests that node 0's position in the training set is always first, leading to poor performance when its order is changed in the test set. However, Figure 6 does not reflect this issue in BERT's test results.

1. Relation of HLM-G to LLMs. The paper associates HLM-G with large language models (LLMs). However, based on the text, it appears that HLM-G is trained separately and from scratch on each task without any apparent connection to pre-trained LLMs (unlike, for instance, GraphToken). This association could be misleading, as there is also no indication that HLM-G inherits the capabilities typically attributed to LLMs. Given HLM-G’s merits as a graph encoding method, it stands as a valuable contribution on its own independent of the LLM association, and this should be reflected in the manuscript too.

2. Performance on Baseline Comparisons. While HLM-G demonstrates impressive performance on synthetic datasets (Table 1), this superiority does not extend as clearly to real-world datasets (Tables 3-5), where its performance closely aligns with simpler models like GCN. Insights from the authors on this discrepancy could shed light on the strengths and limitations of HLM-G in practical settings.