Mind's Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models

Thank you for your feedback, and we appreciate your support!

Spatial reasoning in LLMs is a less explored research topic, we only scratch the surface of it. Despite this testbed is relatively simple to humans and real-world spatial reasoning challenges, it's still challenging for LLMs. It covers various aspects of spatial reasoning, and offers various difficulty levels to appropriately evaluate LLMs.

Question: 3D spatial reasoning tasks and other modalities

As LLMs' inherent ability to generate text-form visualizations develops, VoT can be applied to more and more tasks, including 3D tasks. For example, folding a net into a 3D cube, and identifying different perspectives of a 3D cube. Moreover, making VoT work with multi-modal LLMs is one of our primary future works. Besides text-form visualizations, VoT can be adapted to generate visual modality by code generation (indirect) or image generation (direct). The code generation approach might be applicable for proprietary advanced MLLMs at current stage. We'll explore the approach to directly generate images as visualization to guide subsequent reasoning. For open-source MLLMs, they can be adapted to multimodal generation to gain the visualization ability. This will also benefit proprietary MLLMs for two reason. First, they can leverage external visualizers to ground their reasoning and build more powerful applications. Second, the generated multimodal reasoning trajectory is appropriate for data augmentation, which will benefit interested researchers in the community.

We sincerely thank you for your careful and constructive review of this paper. Your insights are invaluable, and we have carefully considered each of your comments.

Weakness 1: Unclear details

We'll improve clarity of visuospatial sketchpad, caption of figure 4 and 10 in the camera-ready version. The data construction and evaluation code will be released after paper acceptance to facilitate replication. Below are explanations per question:

1. We recognize that "augment" here is misleading. The concept "visuospatial sketchpad" is borrowed from the "Working Memory Model"[1]. We use it to refer to LLMs' ability of retaining visuospatial information within its context, rather than an external tool added to LLMs.
2. All the input for LLMs are text only, the map/rectangle input in the tasks is 2D grid comprising of special characters (line 48). For example, 🏠, 🚧 and 🟦 are emoji icons in the input that LLMs can understand, and 🚶is generated by LLMs in the output. For GPT-4V, we use Pilmoji library to draw special characters into an image. We discussed this implementation in appendix A.
3. Visualizations are also text-form (line 130).
4. We apologize for any confusion caused by Figure 4. It's indended to provides the overview of three tasks without covering all details. It illustrates the input format, VoT prompt and the visualizations generated by LLMs. We'll add descriptions to the caption of Figure 4. Regarding comprehensive details about inputs and responses, they are listed in appendix B.1.
5. Figure 8 provides side-by-side comparisons to show the how response is influenced by prompt, where bold texts on the top are prompts to compare.
6. We regret the brevity of Figure 10's caption. Figure 10 is to illustrate diverse visualization formats in LLMs to track the navigation progress, including marking the path, marking the current location, and directional/temporal steps. We will include these descriptions in the figure caption and note that detailed descriptions are available in Appendix E.1.
7. As specified in section 4.1 (line 152), model version of GPT-4 is 1106-preview, and GPT-4V is vision-preview.
8. In the natural language navigation task, LLMs often output additional words in the extracted answer besides the expected object name. For example, "Answer: You will find ...". We adopted the substring matching method as previous work did. For visual navigation and visual tiling task, we adopted exact matching. Since sub-string matching can encompass exact matching, we refer to f_correct implementation using this term for simplicity.

Weakness 2: Scaling

We appreciate the insightful question regarding the scalability of the tasks. Our testbed indeed provide flexible difficulty control across different tasks. For visual tiling task, the difficulty is controlled by the number of empty squares. As the number increases, the more spatial arrangements LLMs need to consider. Regarding the visual navigation task, as illustrated in figure 2, we use the number of roads to control difficulty, which is directly correlated to map size. Data distribution is provided in table 4 and 5. Specifically, in the next-step-prediction task, the difficulty is also controlled by the number of navigation instructions. For example, a 7 * 7 map with 6 instructions is significantly more complex than a 3 * 3 map with 1 instruction. The distribution of map size is as follows:

The performance of LLMs across these varying difficulty levels is shown in Figure 9. We observe that for advanced LLMs, VoT is more robust to various difficulties compared to CoT. To clearly illustrate the performance gap between VoT and CoT in advanced LLMs across varying difficulty levels, we provide the following comparative table:

Bold numbers in this table demonstrate the significant advantage of VoT particularly at higher difficulty levels. It indicates VoT's superior robustness in handling more complex spatial reasoning tasks.

Weakness 3: Anthropomorphism

We apologize for some unnecessary anthropomorphism. We used mental image and mind's eye as metaphors to help readers intuitively understand the described mechanisms, without introducing unfamiliar terminology. We'll revise unnecessary usage which hinders scientific rigors.

Question

Both the moving house and person icon are among the 30 symbols LLMs automatically generate to track the navigation progress, as described in the appendix E. In figure 8b, since the direction of each instruction is correct, the left example is a correct solution with imperfect intermediate visualization. Due to the limitation of this inherent visualization ability, even for the most advanced models, it's still challenging to generate a perfect visualization. We provide analysis about the accuracy of the final visualization in Table 2, which are lower than 30% in both visual tasks.

We hope these clarifications will address the concerns raised and provide a clearer understanding of our work.

References

[1] Alan Baddeley. Working memory. Science, 255(5044):556–559, 1992

Thank you for your thorough evaluation and constructive comments of our work. We'd like to clarify:

Weakness 1: Limited Task Diversity

We fully appreciate the commonly adopted benchmark such as bAbI[1], StepGame[2], SpartQA[3], SPARTUN[4] etc., which lay solid foundations for spatial reasoning based on text understanding. As explained in section 2 (line 73-77), our main concern is that as the development of LLMs and their exceling in linguistic and logic reasoning, it's challenging to provide an accurate measure of their spatial awareness with these tasks. So we carefully select tasks to cover various aspects of spatial reasoning and offer flexible difficulty controls. These visual tasks with grid input avoid LLM's linguistic and logic "short-cut", and the dynamic nature of navigation tasks requires LLMs to track the changing state in the environment. Despite these tasks are relatively simple to humans, they are challenging to current stage LLMs.

Weakness 2: Generalizability

We evaluated VoT and other baselines in GPT-family models (GPT-4/4V/3.5 turbo) and LLama3 family (LLama3-8B-instruct/70B-instruct). The resuluts and corresponding analysis are provided in section 5.3. Although the overall performance improvement of VoT in less advanced models is not as significant as in advanced models (least significant for LLama3-8B), the improvement in easier levels are distinctive (discussed in appendix D). As difficulty level increases, less advanced models exhibit irregular performance trajectory for both CoT and VoT. This observation indicates their inherent weakness, i.e., reliance on random guessing for increased task difficulty.

Weakness 3: Prompt Sensitivity

We adopted task-agnostic 0-shot prompting for VoT as it's the simplest approach to mimic corresponding human cognition in LLMs, and assure a fair comparison with other baselines. We only scratched the surface of this ability in LLMs and hope this initiative could possibly inspire others. For practical usage, users can add task-aware specifications about what to visualize and how to visualize, to force LLMs appropriately generate visualizations to guide subsequent reasoning. More fundamentally, this limitation could be addressed by instruction tuning with visual state tracking data. Due to the space limitation, we discuss the possibility that VoT might benefit from code pre-training in appendix C (line 600-627).

We hope that our clarifications address your concerns and provide a better understanding of our work.

References

[1] Weston, Jason, Antoine Bordes, Sumit Chopra, Alexander M. Rush, Bart Van Merriënboer, Armand Joulin, and Tomas Mikolov. "Towards ai-complete question answering: A set of prerequisite toy tasks." arXiv preprint arXiv:1502.05698 (2015).

[2] Shi, Zhengxiang, Qiang Zhang, and Aldo Lipani. "Stepgame: A new benchmark for robust multi-hop spatial reasoning in texts." In Proceedings of the AAAI conference on artificial intelligence, vol. 36, no. 10, pp. 11321-11329. 2022.

[3] Mirzaee, Roshanak, Hossein Rajaby Faghihi, Qiang Ning, and Parisa Kordjmashidi. "Spartqa:: A textual question answering benchmark for spatial reasoning." arXiv preprint arXiv:2104.05832 (2021).

[4] Mirzaee, Roshanak, and Parisa Kordjamshidi. "Transfer learning with synthetic corpora for spatial role labeling and reasoning." arXiv preprint arXiv:2210.16952 (2022).

Thank you for your insightful comments, which are invaluable in helping us refine and clarify our work. We are grateful for the opportunity to address the points raised and provide further clarification on the aspects that may have been misunderstood.

Weakness 1: Unclear descriptions

There might be some misunderstanding about the modality of mental images in LLMs during spatial reasoning. As explained in line 42-44, VoT leverages LLMs' potential ability of ascii-art for visualization, rather than depending on external visualization tools. We also provide explanations about ascii-art contained in code comments and some examples in appendix C. Regarding the input format, we specify the use of a 2D grid with special characters in line 48. Due to space limitations, detailed implementation information for the textual 2D grid dataset is presented in Appendix A, along with the rendering method used to generate the corresponding image input for GPT-4V.

In summary, during spatial reasoning, LLMs generate ascii-art as text-form visualizations. The image input used by GPT-4V is produced by rendering the special characters from the textual 2D grid, rather than the reverse process.

Weakness 2: Experiment

W2-1: Concept of state s

We introduce the concept of state s in line 139 to explain the visual state tracking nature of VoT. We discuss the visual state tracking behaviors among different baselines in section 5.1. As could be seen in Figure 5, there is a notable disparity in state tracking rate between GPT-4 VoT and GPT-4 w/o Viz across all tasks. We hope this comparison could address your concern.

W2-2: Poor performance of GPT-4V

Your observation is indeed insightful. We were also suprised by the poor performance of GPT-4V, particularly in visual navigation task (being the poorest). After conducting a qualitative analysis of the responses, we hypothesize that visual hallucination may be hindering GPT-4V's performance.

To further investigate, we conducted an additional experiment using 496 maps from the route planning task, focusing on a simple task: identifying the direction of the roads connected to the home icon. Despite the task's simplicity, where only one road begins at the home icon, both GPT-4 and GPT-4V occasionally provided multiple answers. Accuracy is measured by the ratio when the exact answer is outputed.

As could be observed in the table, GPT-4V with image input performs poorest in this easiest task. It indicates that GPT-4V suffers from hallucination in visual navigation task.

W2-3: Unanswered question of section 5.2

We appreciate your observation regarding the clarity of Section 5.2. We did analyze the correlation between accurate visualization and final answers. We provided the metric explanation in line 230-231, which corresponds to the last column in table 2. And we concluded in line 235 that accurate visualizations ensure a correct final answer with high probability (above 65%). Similarly, other metrics demonstrated in table 2 are also explained in the first paragraph of section 5.2. They measure the visualization capability of LLMs, e.g., to what degree LLMs can generate accurate visualizations.

We recognize that the connection between the table and the narrative may not have been as explicit as intended. If these explanations address your questions, we'd like to adjust highlighting of numbers and observations to achieve a better understanding of table 2.

W2-4 Less powerful LMs

We acknowledge that the overall performance of Llama3-8B is less impressive when using VoT, as noted in our discussion of limitations in line 314-315. We believe this can be attributed to the increased difficulty in handling complex tasks, leading to outcomes akin to random guessing in less powerful models. To further explore this issue, we conducted an analysis of performance trajectory in next-step-prediction task across various difficulty levels, as presented in line 640-643 in appendix D. However, as highlighted in figure 9, there's a modest improvement when the task complexity is lower (K ≤ 3). This suggests that VoT can potentially benefit less powerful models in less complex scenarios.

We look forward to your further feedback and thank you for your valuable contributions to improving our manuscript.