Imagine While Reasoning in Space: Multimodal Visualization-of-Thought

Thank you for your recognition of our work and your valuable suggestions. We would like to address your comments as follows to get more support:

Toy grid-world based experiments

Our use of grid-based benchmarks offers better controllability and systematic investigation across various aspects of spatial reasoning—from pattern complexity to action complexity. Grid-based benchmarks also enable easier evaluation towards whether the generated visualization is correct in terms of the spatial transition between steps rather than focusing on the image pixel details.

We agree that it would be interesting to see how MVoT performs on real-world reasoning tasks. However, due to the current lack of interleaved text-image reasoning datasets, it’s hard for us to adapt MVoT to real-world scenarios such as robotics or 3D reasoning in this work. But we hope our paper as the first exploration of natively generating multimodal reasoning traces offers foundational insights and inspires further studies in this direction.

Zero-shot / out-of-distribution evaluations

We appreciate your suggestion to evaluate MVoT in cross-task generalization settings. The inter-task OOD performance of MVoT relies on image generation ability. Currently, due to computational constraints, we fine-tune the model using LoRA on a limited dataset, which restricts its ability and makes it more challenging to generalize across tasks and scenarios.

However, we conducted preliminary experiments to test MVoT’s ability to generalize to OOD grid sizes. We found that the model can successfully generate up to 7 consecutive visualizations on larger grids, adapting its stepwise spatial transitions accordingly. That said, we also observed occasional redundancies and inaccuracies, indicating room for improvement.

We plan to explore MVoT’s OOD behavior more extensively as future work, and will include further discussions on this topic in the camera-ready version, should the paper be accepted.

Failure cases of the proposed method

Most of the failure cases for MVoT are caused by generating incorrect visualizations during inference, including generating wrong or redundant visualizations with perturbed or blurred image details, as we illustrated in Section 6 and Appendix D.2. We will include more analysis and discussion on failure cases in our camera-ready version upon acceptance.

Model choices

Chameleon supports interleaved text-image generation, which meets the requirements of MVoT. In principle, MVoT can be extended to other multimodal models that support interleaved text-image generation. However, many current MLLMs are restricted to producing textual outputs only. We look forward to the development of architectures that support richer, interleaved multimodal outputs, which would further broaden the applicability of MVoT.

We thank the reviewer for appreciating the novelty and the constructive feedback on our work. We will reflect these considerations in our camera-ready submission upon acceptance.

Thank you for your recognition of our work and your valuable suggestions. We would like to address your comments as follows to get more support:

Visualization Consistency and Vulnerability

We acknowledge the concern that unconstrained visualization could introduce inconsistencies, particularly if visual outputs do not precisely match the underlying reasoning steps. However, in our framework, visualization is not a standalone prediction but is tightly coupled with the reasoning chain. The generated visualization is conditioned on both the current reasoning step and the accumulated context. This is similar to textual rationales, which also lack hard constraints on internal consistency but are still useful in guiding reasoning. Importantly, MVoT does not generate full images from scratch; instead, it predicts spatial transitions between steps, which is more structured and less generative-intensive. This design significantly reduces the likelihood of arbitrary or unaligned visual outputs.

From a performance perspective, while pure text-based reasoning can be effective in general tasks, prior work [1, 2] and our own experiments on FrozenLake show that it falls short in complex multimodal spatial reasoning. In contrast, MVoT demonstrates more robust performance by leveraging visual signals, suggesting that visualization acts as a regularizer rather than a vulnerability.

Lastly, we would like to clarify that concerns around the generalization ability are derived from the image generation model, which does not necessarily represent the drawback of MVoT as a reasoning method.

Error Accumulation in Visual Prediction

We agree that cascading errors are a valid concern in any multi-step reasoning framework, especially in modalities like vision. However, it’s important to note that text-only reasoning is equally susceptible to error accumulation, especially when dealing with ambiguous or spatially grounded tasks. For example, our FrozenLake results show that when errors occur in describing environment layouts, Chain-of-Thought can even underperform the Direct baseline due to compounding misrepresentations.

In contrast, our results show that MVoT performs comparably or better than both Direct and CoT approaches, suggesting that the visual modality does not necessarily amplify errors, and may in fact help mitigate them by providing an interpretable, stepwise grounding of spatial transitions.

Out-of-Distribution (OOD) Generalization

We appreciate your suggestion to evaluate MVoT in cross-task generalization settings. This is an important direction. The inter-task OOD performance of MVoT relies on image generation ability across scenarios. Currently, due to computational constraints, we fine-tune the model using LoRA on a limited dataset, which restricts its ability and makes it more challenging to generalize across tasks such as MAZE to MINIBEHAVIOR.

However, we conducted preliminary experiments to test MVoT’s ability to generalize to OOD grid sizes. We found that the model can successfully generate up to 7 consecutive visualizations on larger grids, adapting its stepwise spatial transitions accordingly. That said, we also observed occasional redundancies and inaccuracies, indicating room for improvement.

We plan to explore MVoT’s OOD behavior more extensively as future work, and will include further discussions on this topic in the camera-ready version, should the paper be accepted.

More realistic spatial reasoning tasks

We agree that it would be interesting to see how MVoT performs on realistic spatial reasoning tasks. However, due to the current lack of interleaved text-image reasoning datasets, it’s hard for us to adapt MVoT to these scenarios in this work. We believe our paper—being one of the first to explore native multimodal reasoning trace generation—lays important groundwork for future research, and we hope it inspires further development of models and datasets in this space.

We thank the reviewer for providing us with more recent references, which we would include in our discussion together with the comments above in our camera-ready version upon acceptance.

Reference
[1] Ramakrishnan, Santhosh Kumar, et al. "Does Spatial Cognition Emerge in Frontier Models?." arXiv preprint arXiv:2410.06468 (2024).
[2] Wang, Jiayu et al. “Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models.” The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 

Thank you for your recognition of our work and your valuable suggestions. We would like to address your comments as follows to get more support:

Experiment and task scope

Our use of grid-based benchmarks offers better controllability and systematic investigation across various aspects of spatial reasoning—from pattern complexity to action complexity. Grid-based benchmarks also enable easier evaluation towards whether the generated visualization is correct in terms of the spatial transition between steps rather than focusing on the image pixel details.

We agree that it would be interesting to see how MVoT performs on real-world reasoning tasks. However, due to the current lack of interleaved text-image reasoning datasets, it’s hard for us to adapt MVoT to real-world scenarios in this work. But we hope our paper as the first exploration of natively generating multimodal reasoning traces offers foundational insights and inspires further studies in this direction.

Visual tokens

We generate 1024 visual tokens per visualization. Bounded by a 4096-token context limit, to manage this efficiently, when implementation, we employ a recursive generation strategy with a Markovian assumption: the visualization of the next step $v_{i+1}$ is derived based on the visualizations of the previous step $v_{i}$ and the initial step $v_{0}$. This assumption holds across all the benchmarks we used in this work since at each step the information of the visualization (with textual description for MiniBehavior) is complete.

While this design is well-suited to current benchmarks, we acknowledge the opportunity to improve scalability—for instance, through more compact visual representations, as we stated in Appendix E Limitation. As the first work to explore generating native multimodal reasoning traces, MVoT opens up new possibilities, and we hope it will serve as a foundation for future advancements in more complex multimodal tasks.

Intuition of token discrepancy loss

Token discrepancy loss discourages the model to output visual tokens that deviate too much from the corresponding golden visual tokens. By penalising deviations from ground-truth visual tokens, it effectively guides the model toward more faithful and visually semantically aligned generated visualizations, thereby improving reasoning performance, as our ablation results confirm.

We thank the reviewer for acknowledging the potential broader impact of our work to a diverse set of newly released benchmarks. We will include relevant discussion and corresponding modifications in our camera-ready version based on the valuable suggestions from the reviewer upon acceptance.

Thank you for your recognition of our work and your valuable suggestions. We would like to address your comments as follows:

Benchmark selection

All three benchmarks used in this paper is synthetic and under limited scale.

Our use of grid-based benchmarks was intentional to ensure better controllability and to systematically evaluate various aspects of spatial reasoning—from pattern complexity to action complexity. In addition, grid-based benchmarks enable easier evaluation towards whether the generated visualization is correct in terms of the spatial transition between steps rather than focusing on the image pixel details.

With simple fine-tuning, the models’ performance on all the three benchmarks can be boosted to over 90%, which may implies the oversimplification of these benchmarks.

We emphasize that these benchmarks are not oversimplified. In fact, larger models consistently fail across all tasks, and even with fine-tuning, models only achieve up to 70% accuracy on FROZENLAKE. In contrast, our proposed MVoT surpasses 80% accuracy, highlighting the non-triviality of the benchmarks and the effectiveness of our approach.

Token discrepancy loss

The proposed token discrepancy loss seems more like a trick to improve model’s image generation capability, but does not closely related to the reasoning process.

We believe that there may be some misunderstanding. Our design is grounded in the hypothesis that visualization quality is closely related to reasoning performance. Experimental results across all three tasks show that more accurate visualizations consistently improve reasoning performance. Furthermore, our ablation studies indicate that integrating the token discrepancy loss yields higher-quality visualizations with fewer redundancies. By penalising deviations from ground-truth visual tokens, it effectively guides the model toward more faithful and visually semantically aligned generated visualizations, thereby improving reasoning performance. Notably, without the token discrepancy loss, the generated visualizations struggle to enhance reasoning, underscoring that it is not a mere “trick” but an essential component for our framework. Therefore, we emphasize the relevance and contribution of the token discrepancy loss to MVoT’s overall reasoning process.

Why LoRA instead of fully fine-tuning?

We chose LoRA due to computational constraints as an approximation of full model fine-tuning. Full fine-tuning requires recording and updating gradients for all the model parameters, which is resource-intensive.

Can MVoT be built on top of other multimodal models with continuous visual encoder?

We appreciate the insightful question regarding broader applicability. In principle, MVoT can be extended to other multimodal models that support interleaved text-image generation. However, many current MLLMs with continuous visual encoders are restricted to producing textual outputs only. We look forward to the development of architectures that support richer, interleaved multimodal outputs, which would further broaden the applicability of MVoT.

Language-only input output?

We agree that it would be interesting to see how RL helps in textual spatial reasoning. However, our current work specifically targets multimodal spatial reasoning, and extending to language-only reasoning lies out of the present scope. We consider it a promising avenue for future exploration.

Evaluation of MVoT on real-world visual reasoning tasks and other abstract reasoning tasks.

We agree that it would be interesting to see how MVoT performs on these tasks. However, due to the current lack of interleaved text-image reasoning datasets in those domains, we focused on tuning Anole with LoRA on three grid-based datasets. This constraint limits generalizability, but we hope our work offers foundational insights and inspires further studies in this direction.

Once again, we thank the reviewers for their constructive comments and for pointing us toward relevant recent work. We will reflect these considerations and include the suggested references in our camera-ready submission upon acceptance.