Spatial Reasoning with MLLMs: A New Path to Graph-Structured Optimization

1. Missing Key References and Overstated Contributions: The paper lacks critical references and may exaggerate its contributions. For instance, GITA [1] introduced a framework that systematically converts graphs to images and feeds them into MLLMs for seven fundamental graph reasoning tasks and two downstream tasks. It provides detailed comparisons with LLMs and GNNs to showcase the advantages of using MLLMs with image representations of graphs with its visual intelligence. Similarly, VisionGraph [2] explores leveraging large multimodal models for graph theory problems in a visual context, establishing a toolchain for eight complex graph problem tasks. Therefore, the claimed core contribution of the paper, "introducing an original and novel approach by feeding graphs as images into MLLMs," has already been addressed in previous work. Furthermore, a significant portion of the findings in the paper overlap with those in GITA [1] and VisionGraph[2]. For example, the intuitive perception and understanding of graph structures by MLLMs; the zero-shot graph reasoning capability of MLLMs; MLLMs being better at tasks requiring global and visual perception; the significant impact of layout on performance in the image representation of graphs; MLLMs outperforming LLMs in graph-related tasks; and the challenges posed by the increasing complexity of visual representations of graphs (including the number of nodes and edges) as the graph scale increases, are also analyzed and mentioned in GITA [1] or VisionGraph [2]. Therefore, the paper seems to exaggerate its contributions and lacks distinct novelty and constructive findings. Future versions should clearly differentiate and highlight their unique contributions, perhaps by focusing on specific scenarios in combinatorial optimization problems.

2. Poor Writing Structure: The structure of the paper appears disorganized, with the introduction followed immediately by the experimental section. Discussions of related work and descriptions of the methodology are embedded within the introduction and experimental sections. This makes the paper resemble a brief experimental report rather than a polished academic paper. Additionally, some concepts and terms such as BA, ER, and WS are not adequately introduced or explained in the main text, making it difficult for readers to follow their divergences and properties.

3. Lack of Methodological and Experimental Details: The paper lacks crucial details about the methodology and experiments, such as how graph structures are converted into images and which specific MLLM and LLM are used in the experiments. These should be clearly explained in the main text. Furthermore, the model selection in the experimental setup is too narrow, and the baselines are insufficient, raising questions about the generality and robustness of the conclusions.

[1] GITA: Graph to Visual and Textual Integration for Vision-Language Graph Reasoning. https://arxiv.org/abs/2402.02130. NeurIPS 2024.

[2] VisionGraph: Leveraging Large Multimodal Models for Graph Theory Problems in Visual Context. https://arxiv.org/abs/2405.04950. ICML 2024.

There are several confusions should be addressed:

1. Given the importance of trasnformation from graph to image, the transforming manner is also important. Therefore, the transforming/imaging manner should also be discussed, in case there is any prior knowledge from human are invloved, which may leads to unfair comparison.

2. There is typos in the caption of Fig(5), only three agents are presented.

3. For the graph visualization. How to deal with complex graphs, such as the hetergeneous ones? These complex graphs are more common in the real-world problems compared with the simple ones presented in the paper.

4. Fig(7) and (9) are confusing and lacking of explanation. What is the LS? What do the lines of degree, betweeness, and closeness mean? It is hard to understand why it shows the intelligence of the MLLM for structural understanding.

5. For the large-scale dataset, the color and chosed degrees for construting the image already involves prior knowledge from human, which could lead to unfair comparison with conventional methods.

• The paper relies on specific layout methods (e.g., force-directed layouts) to structure graphs in ways that facilitate MLLM processing. However, this dependency may limit generalization to more complex graphs with less intuitive visual layouts, potentially affecting the model's versatility.

• The current approach lacks interactive visualization tools, such as zooming capabilities, that could enhance MLLMs’ performance in dense networks. This limitation affects MLLMs' ability to accurately analyze highly detailed or node-dense regions.

• While the paper explores common graph-related tasks, the evaluation is limited to specific problem types and graph structures. Further testing on diverse graph types, such as dynamic or weighted graphs, would provide a more comprehensive assessment of MLLMs' effectiveness.

1.Experimental Setup Issues: The experiments in “4. MLLM on Basic Graph-Related Tasks" have significant flaws. The specific MLLM and LLM models used are not even explicitly mentioned. Even if model types were specified, inconsistencies in model selection and prompt settings prevent a fair assessment of MLLM performance, as variables are not properly controlled. This lack of experimental rigor weakens the claim that MLLMs outperform LMMs. 2.Simplistic Conclusions with Limited Analysis: Some conclusions are overly idealistic and lack detailed analysis. For example, the statement that “regardless of how large or complex the network is, the input remains a fixed-size image, allowing the MLLM to interpret and process it efficiently” is an oversimplification. In reality, MLLMs often struggle to accurately recognize nodes and edges, leading to misinterpretation of even basic structural information. Based on this, high-level insights tend to be guesses rather than grounded outputs. So the results provided by MLLMs often lack interpretability and practical feasibility. And based on this, the claim that “MLLM is able to process global relationships in graphs efficiently” needs more convincing experiments. 3.Lack of Novelty and Credible Proof: The paper attempts to showcase a well-known conclusion that the potential of MLLMs in graph-structured optimization. However, the experimental approach is not rigorous, and the absence of in-depth case studies undermines the persuasiveness of the arguments. Overall, the paper resembles an incomplete experimental report, lacking the necessary theoretical explanations, methodological robustness, and compelling case studies to support its conclusions.

• The writing of the manuscript can be improved. For example, it is suggested to provide the full name of an abbreviated term as it's first used (HD, HCI)

• This manuscript, while addressing an intriguing problem, reads more like a report than a research paper. It examines the capability of MLLMs to interpret images containing structured data. I believe this manuscript might be suitable for another conference that focuses more on the application.

• The comparisons should contain some stronger heuristic methods to set the upper bound of the non-learning-based method.

• As the MLLM relies on the image of the graph, different visualization methods should be examined. Meanwhile, for larger networks and fine-grained tasks (like recommendations on social networks), visualization might be an inefficient or even impossible way to represent the graph.