GITA: Graph to Visual and Textual Integration for Vision-Language Graph Reasoning

Thanks for your thorough comments and insightful suggestions, we here address your concerns and adopt your suggestions as follows:

W1: It is not clear that LLM-based methods like GITA are better than GNNs in terms of generalizability, flexibility, and user-friendliness, and the motivation for using LLM for graph reasoning is not justified.

In Appendix A, we have detailed GITA's flexibility, user-friendliness, and generalizability. Unlike GNNs, whose task-specific feature engineering and architecture adjustments require professional techniques on model architectures and coding, GITA employs a consistent architecture that simplifies adaptation to new tasks using language-based templates. This makes it accessible even to non-experts, significantly enhancing flexibility. Moreover, GITA utilizes natural language for input and output, enabling intuitive graph reasoning through simple queries like 'Is there a cycle in this graph?' and providing straightforward 'Yes' or 'No' answers, thereby improving user-friendliness compared to the human-unreadable vector representations/embeddings in GNNs.

Besides, we further demonstrate that GITA-ZS (zero-shot) also has promising performance. According to the results shown in the Table 3 in the rebuttal supplement PDF, where 'PT' represents advanced prompting techniques including in-context-learning (ICL), chain-of-thought (CoT), and self-consistency (SC), its performance can be further improved with both the evolution of the VLM reasoner (i.e., from GPT-4v to GPT-4o) and advanced prompting techniques including ICL, CoT, and SC. Based on this result and the results reported in the submission, our GITA method exhibits promising zero-shot capabilities (generalizability on tasks).

W2: While the paper compares GITA to various language models, it does not provide a comparison to dedicated GNN models that are designed specifically for graph reasoning tasks. A comparison to GNNs would help demonstate the performance gains of GITA and its advantages or disadvantages.

According to your suggestion, we compare with dedicated GNNs, including GCN and SAGE, and present the results in Table 4 in rebuttal supplement PDF. Compared to dedicated GNNs, zero-shot GITA (with prompt techniques) and fine-tuned GITA-7B model have similar average graph reasoning performance. The larger GATA-13B model performs slightly better.

In particular, compared to GNNs, the GITA model shows a stronger ability to recognize local structures in graphs (Connect and Cycle) and to accomplish tasks with obvious layout heuristics (BGM). We believe that this advantage comes from GITA's visual perception. For SP and MaxFlow, GITA performs inferior to GNNs. This may be because GNNs process edge weight more effectively through its message passing mechanism. For HP and TS, GITA-ZS performs best. Those results will be put in the revision.

W3: The paper mentions k-hop subgraph sampling for handling large graphs but does not provide a detailed analysis of GITA's scalability or performance as graph sizes increase. A more comprehensive evaluation of scalability, potentially including comparisons to dedicated graph methods, would be valuable for assessing the practical applicability of GITA in real-world scenarios.

According to your suggestion, we conducted experiment to evaluate the scalability and performance for GITA and dedicated GNNs while increasing the number of hops $k$ in the ca-HepTh dataset.

The results shown in Table 5 in supplement PDF demonstrate the scalability of GITA is kept stable with the sampled graph size (i.e., $k$) increasing. According to the accuracy reported in Table 6 in supplement PDF, GITA, GCN, and SAGE achieve their respective peak performance when $k$ equals 2, demonstrating that a small sampled graph size is enough for the performance. Dedicated GNNs show higher peak performance than GITA, but also show worse performance when $k$ becomes larger (e.g., 3 or 4), which demonstrates the performance of GITA is more stable than dedicated GNNs w.r.t. $k$. We will include those results in the revision.

W4: The alignment between visual and textual modalities could be explored. The paper notes performance degradation in some tasks for larger models due to potential alignment issues, but it does not provide a detailed analysis or solution. Further investigation into improving modality alignment is needed.

Because existing Multilingual Large Language Models (MLLMs) are not inherently attuned to graph data, they require fine-tuning to effectively align vision and text inputs in a graph context. The effectiveness of this alignment during fine-tuning is influenced by the proportion of tunable parameters. Following LLaVA, for the larger GITA-13B, the trainable parameter ratio is only 0.78%, which is much smaller than the 1.46% for GITA-7B. This limitation in tunable parameters may result in less effective alignment for GITA-13B compared to GITA-7B, potentially leading to poorer performance of GITA-13B on some tasks.

To address this issue, two solutions are proposed: one is to employ full-finetuning, which directly tunes all trainable parameters to align the modalities. Another approach involves applying the proposed GITA to more diverse graph data, thereby obtaining a richer source of vision-language data for alignment ([r1] illustrates diverse and rich data can hugely improve alignment). These additional vision-language data can be fine-tuned together with the current task data, or used as pretraining (as demonstrated in Section 5.3, using GVLQA checkpoints for real-world datasets).

[r1] Visual Instruction Tuning. NeurIPS, 2023

W5: Some places need double-check, including the inconsistent full name of GITA and a typo in line 120.

Thank you for pointing out the inconsistent terms and notations. We will fix them in the revision.

We acknowledge and appreciate your insightful review. Below, you can find our responses addressing your concerns point by point. If you have any additional questions or require further clarification, please feel free to let us know.

W1: This paper appears to be quite technical or engineering-oriented, with the models largely based on existing tools, and lacks strong novelty in terms of methodology.

We acknowledge that our work builds upon existing backbone models and tools. Incorporating visual information into general graph reasoning, however, is an interesting idea that has not been explored before. As a result, we have encountered several challenging issues during our work. For instance,

1. how to maintain the usability of vision graphs while managing context length in large-scale graphs;

2. how to balance consistency and variability in vision graphs, and the impact of specialized augmentations for vision graphs, etc.

To handle those issues that have never been explored before, we proposed the GITA framework and provided valuable empirical findings.

W2: This paper primarily focuses on integrating visual graphs into large models, however, it fails to provide any specific examples of visual graphs throughout the paper, making it difficult for readers to comprehend how the visual image improve the performance.

The illustrations of the graphs have been provided in Figure 2 as part of the case study for readers to understand how vision plays a role in graph reasoning. We have also included examples of visual graph generated by different graph visualizer tools implemented by us in Appendix C, as well as illustrations of these visual graphs for each subset of GVLQA in Figures 6-9 in Appendix G. We will highlight them in the revision.

W3: In page3 line 120, the 'G' in the end actually refers to 'D'? Some explaination should be provided.

Thank you for pointing out this typo. The 'G' in the end should be 'D'. We will correct it in the revision.

Q1: Will the dataset be released in the future? If so, I appreciate it.

Of course! The dataset will been released soon. Due to the submission policy, we do not provide the link in the submission.

We acknowledge and appreciate your insightful review. Below, you can find our responses addressing your concerns point by point. If you have any additional question or require further clarification, please feel free to let us know.

W1: The applications are limited. The proposed method is only applicable to tiny graphs or large graphs with k-hop sampling, which hurts its practical application value.

In fact, k-hop subgraph sampling is a common practice in graph learning. Typically, a k-hop sampling may not lead to performance degradation. On the contrary, k-hop subgraphs show more dedicated awareness of the local structure details. For example, ShaDow-GNN [1] indicates from both theoretical and empirical analysis, that in graph data, one can ignore distant neighbors ($k >= 4$), and the most effective $k$ is 2 or 3.

To further illustrate, We conducted experimented to show the performance of both GITA and Vicuna by varying $k$ on large graph datasets ca-GrQc and ca-HepTh and show the results in the following table. It is evident that both GITA and Vicuna reach their good performance at $k = 2$. Thus, increasing $k$ does not necessarily enhance the performance, and k-hop sampling does not lead to a decline in performance. Based on this observation, we think that the proposed GITA method is applicable to general graph reasoning tasks.

[1] Decoupling the Depth and Scope of Graph Neural Networks. NeurIPS, 2021.

W2: The design of key components, including the Graph Describer, Graph Visualizer, and Task-specific Query, requires human efforts to adjust to fit different datasets.

We think that the requirements of human efforts in GITA are negligible based on the following aspects.

1. For a new dataset, the Graph Visualizer and Graph Describer in GITA can be directly used in a function-invoking manner. We provide the default setting. As a result, users do not need to design them.
2. The task-specific template inside the Questioner is the only component requiring human efforts. It only requires users to describe the task definition and the concrete meanings of the graph elements in user-friendly natural language.
3. We also offer an automated approach that allows users to generate task-specific queries by prompting an agent like ChatGPT. An example of the query generation for a custom gaming scenario has been provided in Appendix E. Though this automated approach still necessitates human input to describe the task initially, it is a minimum and unavoidable requirement for any language-based method.

W3: Lack of comparison with SOTA methods. The experiments only compared with LLMs and VLMs; some graph LLMs should also be compared, such as Graph Chain-of-Thought, GraphLLM, and GraphToken.

Graph Chain-of-Thought is not applicable to the general graph reasoning tasks, because it is built on knowledge graph (KG), but general graph reasoning tasks usually do not have the corresponding KG. GraphToken does not provide its data and code. Hence, in the following table, we here provide the performance comparison with GraphLLM on a randomly chosen half-size GVLQA-Base subset.

Based on the results, for substructure-awareness tasks such as Connect and Cycle and tasks with beneficial visual heuristics such as BGM, the visual modality introduced by GITA is more advantageous than the graph modality introduced by GraphLLM. GraphLLM shows its superiority in MaxFlow and SP, which may be due to their graph transformer encoder being more effective in processing edge weights. Finally, GITA and GraphLLM show similar abilities on sequential ordering tasks TS and HP, and comparable average performance. We will include this comparison in the revision.

Q1: Can you provide some statistics about the graph size (average number of nodes and edges) used in your dataset?

The average numbers of nodes and edges for each task in GVLQA are shown in the following table.

Q2: In Table 1, is GITA-7B (VO) equivalent to LLaVa-7B?

GITA-7B (VO) represents a variant of GITA using LLaVa-7B reasoning on the vision graph generated by GITA Graph Visualizer and a direct question like "Is there a cycle in this undirected graph", but without the textual descriptions of the graph generated by GITA Graph Describer.

Q3: In Table 1, in the fine-tuning setting, how are the LLMs fine-tuned? Is GITA-7B fine-tuned on only the alignment projector or the whole model?

We have introduced the detailed fine-tuning settings in both Section 5 and Appendix F in our submission. For the fine-tuning setting in Table 1, we fine-tune the LoRA adapters for all weight matrices within the text decoder and the alignment projector, while keeping the vision encoder in the VLM reasoner frozen.

We acknowledge and appreciate your insightful review. Below, you can find our responses addressing your concerns point by point. If you have any additional questions or require further clarification, please feel free to let us know.

Note: Some tables mentioned in this rebuttal are contained in our rebuttal supplement PDF.

W1: While I think it is an interesting idea to introduce vision into graph reasoning tasks, I am skeptical about its value. This is because vision is a perceptual ability, which is a fast intelligence (i.e., system one). While graph reasoning is a task that requires rigorous inference modeling and step-by-step execution to get the final precise answer, e.g., the graph reasoning tasks covered in this paper have their corresponding graph algorithms to get the precise answer. In my opinion, compared with visual ability, the ability to rigorously execute traditional graph data structures and algorithms is the key for LLM/MLLM to have the ability to solve graph reasoning tasks. The experiments in Table 1 also show that, except for Connect and Cycle, which can be answered quickly and visually (provided the graph layout is concise and clear), the other graph reasoning tasks, GITA (VO) do not perform well.

We agree with you that only the visual modality may be not so good for graph reasoning tasks. Please note that our work is to show that integrating the visual modality with the textual modality could achieve better performance than a single modality, as evidenced by our submitted paper Tables 1 and 3.

While "VO" demonstrates limited performance in Table 1 for most tasks, we have successfully enhanced its capabilities through layout augmentation in GITA, as evidenced in Table 2.

The visual modality could complement the textual modality in graph reasoning tasks. Here we provide a case study. Typically, the visual modality is more excellent in recognizing beneficial substructures/local patterns compared to the textual modality, and some of them are crucial in graph reasoning. For instance, "hop number" serves as a heuristic in shortest path calculations, "leaf nodes" are critical in topological sorting, and "cycles" must be prevented in Hamiltonian path construction. We extracted these substructures inside the GVLQA-Base and manually labeled them. Employing the frozen ViT in LLaVA with a trainable MLP decoder, we achieved identification accuracies of 89.92%, 95.16%, and 92.39%, respectively for hop number counting, leaf nodes identification, and cycle identification. In contrast, using a pre-trained Bert with the same trainable MLP decoder, the accuracies are significantly lower (i.e., 55.47%, 26.33%, and 60.32%). Therefore, the effectiveness of the integration of the visual and textual modalities may be due to that the visual modality provides extra beneficial structural information such as those substructures.

Besides, The benefits of vision do not conflict with the benefits that come from the ability to step-by-step reasoning (e.g. Chain-of-thought, CoT). Table 1 in our rebuttal supplement PDF file shows the model benefits from them both simultaneously.

W2: I think this paper is like a data track paper, i.e., proposing novel datasets, testing the capabilities of existing LLM/MLLM, and finally proposing viable solutions for testing. Instead, this paper is counterproductive in wrapping it up as a methodological framework paper. The biggest problem is that the graph visualizer, the graph describer, and the questioner are both part of the methodology and the means of constructing the dataset, which is very confusing. I don't think they fit as part of the methodology because they are just tools for engineering the dataset, and there is no innovation at the level of ideas, modeling, or anything else.

We would like to point out

1. GITA is the first to enable the use of MLLMs in graph reasoning. This is a significant advance because nearly all existing graph reasoning works do not utilize the visual modality, which could help improve the performance.
2. GITA is general for almost all existing graph reasoning data because it only requires graph information in bare structures $G=\{V, E\}$.
3. GITA solves technical problems including: how to use MLLM for graph reasoning based on bare graph structure, how to maintain vision graph usability and keep context length in large-scale graphs, how to trade-off the consistency and variability of vision graphs, special augmentations for vision graphs and their impacts, etc.
4. GVLQA is just a by-product of our core contribution, GITA. Therefore, the methodologies inside GITA are not limited to any individual benchmark or scenario but are significant and valuable to graph reasoning. As a result, we think our work fits the main track instead of the dataset track.

W3. Table 1 compares only text-based LLMs and lacks a comparison with visually aware MLLMs such as GPT-4V/GPT-4O/LLaVa/Gemini.

We would like to point out in our experiments, LLaVa has been utilized as the VLM Reasoner for both the GITA and GITA (VO) configurations. As one of the components of GITA, LLaVa receives the visual and textual inputs from the other components of GITA (though we store them as GVLQA) and the results are shown in the Table as GITA and GITA (VO) terms. Therefore, LLaVa is not necessary to be compared again as an individual baseline because it is a special case of GITA. Similarly, GPT-4V serves as the VLM Reasoner for the GITA-ZS and GITA-ZS (VO) configurations in Table 1 and is not considered a baseline. According to your suggestion, we utilized GPT-4o (not released since the submission of our work) and Gemini, as the MLLM reasoner and recorded the results in Table 2 in our rebuttal supplement PDF file , which shows that the benefits of incorporating vision with GITA are consistent across various VLM reasoner choices.