Hypo3D: Exploring Hypothetical Reasoning in 3D

Interesting.

Q1: About Novelty.

Our Hypo3D benchmark is a novel, methodological contribution in itself, as noted by other reviewers. It focuses on structured evaluation and it is the first 3D reasoning benchmark with explicit question-type annotations, enabling fine-grained evaluation. Many influential multimodal reasoning benchmarks [1–3] focus on evaluation rather than proposing new models, yet remain highly impactful.

Current vision models primarily handle basic spatial and semantic relationships but lack robust, holistic scene understanding. At the same time, reasoning in dynamic 3D environments remains a long-standing real-world challenge. We argue this challenge should not be deferred until all vision problems are solved. Instead, hypothetical reasoning offers a parallel research path, enabling progress on complex 3D reasoning alongside ongoing advances in foundational vision tasks. Furthermore, hypothetical reasoning is central to human cognition. As humans, we often reason with incomplete perceptual information, updating our mental models as new data emerges. This iterative process—forming, testing, and refining hypotheses—is fundamental to intelligent behavior. Integrating this capability into VLM is a step toward more human-like reasoning and a key milestone on the path to AGI.

Our problem fits into the prediction-and-reasoning domain only when robust, real-time 3D scene updates are possible. However, current limitations in 3D reconstruction and editing, as noted in the paper, make accurate future scene prediction challenging. As a result, the task naturally shifts to an imagination (inference)-and-reasoning problem.

Even widely used benchmarks like SQA3D could conceptually be constructed by posing situational descriptions and revising answers. Both our work and SQA3D deliberately avoided this. SQA3D emphasizes situational diversity, whereas Hypo3D focuses on diverse context changes. Our pipeline first collects a wide range of valid context changes and then generates corresponding questions and answers. Starting with fixed questions would have greatly limited feasible changes, especially given the strict constraints Hypo3D enforces on both changes and question design.

Text descriptions may not always be available, but they remain a cost-effective and widely accessible alternative to reconstructing the full 3D scene after each change. Our future work will explore multimodal representation of changes (e.g., images, depth) for more comprehensive hypothetical reasoning. Notably, in other reasoning tasks, early works like SQA3D relied on text to define the observer's situation, while later studies such as MSQA expanded to multimodal representations. This progression highlights the importance of text as a foundational step in developing new reasoning tasks.

Q2: Evaluation Criteria.

Most methods we evaluate are vision-language models (VLMs), not standalone LLMs. While VLMs include an LLM backbone, it is instruction-tuned with large-scale visual data (e.g., image captioning, VQA), endowing it with visual perceptual abilities beyond standard language understanding.

We conducted additional experiments to examine the impact of visual encoders on VLMs' hypothetical reasoning. Specifically, we compared Cambrian-1 [4](using SigLIP, CLIP, DINOv2, and ConvNeXt) and LLaVA-Next (using only CLIP) [5], both sharing the same LLM backbone. Partial match results on semantic top-view maps below show that while leveraging more visual features can improve reasoning performance, a significant gap remains compared to human performance.

Q3: Difference to Existing Problem Formulation.

Unlike traditional VQA tasks (2D or 3D), where answers can be directly inferred from the visual input, our hypothetical reasoning task introduces a key distinction: the visual input provides only prior context and is insufficient on its own. Hypo3D shifts the focus from “see and answer” to the more complex cognitive process of “see, imagine, and answer”.

[1] TopViewRS: Vision-Language Models as Top-View Spatial Reasoners, EMNLP, 2024.

[2] SQA3D: Situated Question Answering in 3D Scenes, ICLR, 2023.

[3] HourVideo: 1-Hour Video-Language Understanding, NeurIPS, 2024.

[4] cambrian-1: a fully open, vision-centric exploration of multimodal llms, NeurIPS, 2024.

[5] LLaVA-NeXT: Improved reasoning, OCR, and world knowledge, Arxiv, 2024

Thank you for your valuable feedback. We have added further explanations and additional experimental results to address your concern regarding the input data type issue.

Q1: 2D VLM Input

We adopted top-view images for 2D VLMs to maintain consistency with established 3D reasoning benchmarks like SQA3D. While top-view images cannot fully capture 3D geometry, they provide the most comprehensive spatial information in a single image.

Besides, modern 2D VLMs have been able to extract depth cues from single images and reason about 3D positioning. This has been validated by our results, where 2D VLMs perform adequately on questions involving vertical reasoning (e.g., height). Also, Human evaluators also have reported that top-view images were generally sufficient for them to answer questions in our dataset.

Additionally, we evaluated 2D VLMs (semantic maps) using multi-view inputs (top, front, back, left, and right) compared to using top-view only. The results on 50 randomly sampled scenes below shows that performance remains comparable to using only the top view. This suggests that while multi-view inputs offer richer visual information, integrating visual features from different views presents another challenge for the models.

Q2: LLM Input

To assess how caption detail affects LLM performance, we tested LLaMA-3.2 with varying numbers of sampled captions. As shown in the table below, more detailed inputs do not consistently improve performance—possibly due to the increased challenge of long-text reasoning. Following the SQA3D protocol, we use 30 randomly sampled object captions for the final scene description.

Q3: Dataset Comparison

The table below compares Hypo3D and existing 3D visual grounding (VG) and captioning datasets. Hypo3D is the first to annotate all question types and world frames, requiring models to understand hypothetical scenes.

Q4: Impact of Recognition on Reasoning

To mitigate the impact of object recognition on hypothetical reasoning, most experiments in the paper utilized semantic top-view maps with explicit text labels. We also evaluated VLMs on unchanged scenes in Table 3, where the performance was significantly higher. This confirms that the main challenge primarily lies in reasoning about hypothetical changes. All results above will be included in the final version of the paper.

We sincerely appreciate your insightful feedback. We’re pleased to hear that you found our dataset to be a great contribution. All 2D VLM results reported below use the semantic map by default.

Q1: Answer Annotation

Similar to the answer types in Fig. 12 of SQA3D, our annotations include object names, attributes (e.g., shape, color, size, functionality, state, etc.), directions, numbers, and so on. The complete answer distribution will be included in the final version.

Q2: World Frame

We randomly rotated the top-view images before inputting them to VLMs, ensuring the world frame definition aligns with the person's orientation in 25% of scenes.

Q3: Anchor

4.41% of context changes involve changes to the anchor object. E.g.,

Orientation: The shower is on the front side of the scene.

Change: The shower has been moved to the left of the toilet.

Q4: PM

Following TopviewRS, we used PM to measure word-level overlap, particularly for answers like "front" and "front left". We also employed SBERT scores to reduce evaluation bias in phrasing (Fig. 14). We further computed GPT-based score following MSQA, with results detailed in our response to kxrB. Both metrics show consistent ranking with PM.

Q5: Baseline

We conducted a baseline experiment with GPT-4o, which showed significantly lower EM accuracy than models in Table 1. This confirms that LLMs/VLMs are not merely recognizing objects or guessing, but possess limited capacity for hypothetical reasoning. Also, using semantic maps improved performance over non-semantic inputs, as VLMs benefit from explicit hover-text labels for more accurate recognition.

Q6: In-context learning (ICL)

We applied three-shot ICL to 2D VLMs and LLMs. ICL generally reduced EM performance, potentially because the limited example set cannot adequately represent the diversity in our context changes and questions. Moreover, we observed models directly copying answers from examples, indicating their inability to process long context effectively.

Q7: Chain-of-Thought (CoT)

We have utilized CoT to explicitly decompose the task into: (1) imagine how the change affects the scene, and (2) answer the question based on the changed scene.

To further investigate CoT, we tested models with a simplified prompt:

Scene orientation: {}

Context Change: {}

Question: {}

Answer:

Removing CoT prompting reduces performance in most models except Qwen2-VL 72B, suggesting that step-by-step reasoning aids hypothetical reasoning to some extent. Still, results lag behind human levels, and we plan to explore more advanced CoT methods in the future.

Q8: Hallucination

The hallucination in line 359 refers to the model incorrectly adjusting its answer in response to an irrelevant change, as noted in Insight 5. More examples of such hallucinations are provided in the response to Q10.

Q9: Anchor Object as Answer

The table displays the probability at which models predict anchor objects as answers. LLaVA-3D exhibits a much higher rate compared to both other models and ground truth frequencies, suggesting it tends to copy anchor objects as answers rather than engaging in reasoning.

Q10: No Overlap

We considered questions where the queried and changed object do not overlap. E.g.,

C: The lamp has been put onto the bath cabinet.

Q: Where is the toilet paper relative to the trash can?

Q11: World Frame Definition

The main difference between the w. camera and w/o frame settings is the inclusion of an additional camera view image. While models may not accurately interpret scene orientation from it, the image still provides extra visual context, so a significant accuracy drop is not expected.

Q12: Non-semantic vs. Text-only

GPT4o (non-sem.) underperforms due to difficulty recognizing objects in noisy scenes. In contrast, GPT (text-only) directly receives explicit object names and attributes from the caption.

Q13: Human Performance

Human performance falls short of 100% due to typos, formatting mismatches, and occasional misinterpretation of noisy scenes. Notably, less-than-perfect human performance is common in previous open-ended VQA benchmarks such as ScanQA and SQA3D. Below are human-failed examples:

More examples can be found in our response to kxrB.

We appreciate your insightful feedback. We’re glad to hear that you found our work to be novel and interesting.

Q1: Scene Orientation

For 3D VLMs using point clouds (e.g., LEO), inputs have been explicitly aligned to a top-view perspective with the floor on the XY-plane and vertical structures along the Z-axis. We acknowledge that the 3D VLMs taking ego-view images (e.g., LLaVA-3D) are expected to align 3D scenes internally for orientation understanding. Yet, they are provided with comprehensive information (i.e, multi-view RGB, depth, camera poses, and axis-alignment matrices) for implicit scene alignment. Again, while orientation may be more challenging to interpret for 3D VLMs compared to 2D VLMs, all models are provided with sufficient information to support effective orientation comprehension.

Q2: GPT-4 as Judger

Beyond EM and PM, we used SBERT (Fig. 14) for text similarity scoring to reduce evaluation bias in phrasing. Here, we explored GPT-based evaluation for open-ended responses following MSQA [1]. Each GPT score $C$ is formulated as:

$C = \frac{1}{N} \sum_{i=1}^{N} \frac{s_i - 1}{4} \times 100\%$

where $N$ is the number of questions, $s_i \in [1, 5]$ (higher is better) is the discrete score assigned by GPT-4o-mini taking the question, ground truth, and model response as input. The scores for all models are shown below:

Rankings of GPT scores closely align with PM rankings in Table 1, validating the reliability of PM.

Q3: Human Performance

Human performance rarely reaches 100% in open-ended VQA datasets. For example, the best EM@1 on ScanQA is 51.6%, and SQA3D reports 85–95% accuracy. Our dataset achieves even higher human scores, suggesting fewer ambiguities. Most errors are due to typos, vague phrasing, formatting mismatches, and inherent noise in 3D scenes. E.g.,

Q4: InternVL-2.5

In response, we have evaluated InternVL-2.5 [2] (8B and 34B) on the Hypo3D benchmark using both semantic and non-semantic top-view maps. EM results show that InternVL-2.5 (8B version) consistently underperforms LLaVa-OV 7B and Qwen2-VL 7B across all settings on Hypo3D task.

Q5: Instruction-tuning

We agree that exploring instruction-tuned 2D and 3D VLMs on our task would be valuable. Given the limited availability of computing resources, we intend to carry out a systematic study as part of our future work. We currently focus on zero-shot evaluation to ensure a fair comparison of all models. Notably, LLaVa-3D (7B), the SoTA 3D VLM, performs comparably to similar-sized 2D VLMs, suggesting that 3D VLMs' relatively weaker performance may stem from smaller model sizes.

[1] msqa: multi-modal situated reasoning in 3d scenes, NeurIPS, 2024.

[2] Expanding performance boundaries of open-source multimodal models with model, arxiv, 2024.