On Inherent 3D Reasoning of VLMs in Indoor Scene Layout Design

Q: Motivation. It would be helpful to clarify why VLMs are really necessary for these layout tasks.

Response: We believe VLMs are absolutely necessary for visual layout generation! We limit the scope of tasks here to simple rectangular bedrooms, where we are still able to demonstrate visual deficiencies in models. We include tasks where humans can trivially solve a task visually (S2 all-in) and find that VLMs still prefer symbolic computation over reasoning using the image. Our S3 task requires reasoning about affordances and functionality, which can change depending on the specific 3D assets under consideration. In the pursuit of creating a visual design assistant, many additional challenges will emerge that require vision, such as dealing with irregular room or object shapes – we currently only describe objects as 3D bounding boxes in the literature but we hope it’s easy to appreciate that they are much more than 3D boxes and offer complex affordances based on their shape. The shape of the arm of a sofa decides whether a human could sit on it. There exist objects that do not have a specific orientation defined by functionality such as plants, carpets/rugs and certain kinds of lights being a few examples. Of course, one could argue that detailed descriptions of these could suffice, which would be equivalent to arguing that a blind human could easily approach design with detailed descriptions. We attribute the higher performance of VLMs with text and simplified sketches to the following, which we believe uncovers deficiencies in the data and methods for training VLMs:

• Bias towards symbolic computation when clear visual evidence is available
• Reduction in symbolic reasoning performance in the presence of image input

Q:* Quantity and quality of the dataset.

While we use 25 QA pairs per sub-task (per question and rendering type), they overall constitute 3400 QA pairs over all our tasks which is comparable or larger than datasets used for evaluating VLMs [1,2,3]. The cost of evaluating VLMs is a concern towards increasing the size of evaluation datasets and the community has evolved to prefer many small eval datasets for holistic evaluation of diverse skills in VLMs. Moreover, we ensure that for every question, information relevant to the question is always available either in text or visible on the image. For example, in S3, where most occlusion is possible, the target marker when a new object is to be placed is always visible from the rendered viewpoint.

[1] Eyes wide shut? exploring the visual shortcomings of multimodal llms. Tong et al. [2] Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs. Tong et al. [3] WildVision: Evaluating Vision-Language Models in the Wild with Human Preferences, Lu et al.

Q: Finally, as already noted in the limitations, the tasks mainly focus on 2D reasoning rather than 3D, which makes the title somewhat misleading.

Response: We inherit this 2D reasoning for the 3D indoor scene synthesis task from the indoor scene synthesis literature which generally follows this abstraction [1,2,3,4]. Since all reviewers mention this confusion, we will remove 3D from our title to prevent further misunderstanding. We believe our findings are directly applicable to current indoor scene synthesis methods and hope that this satisfies the reviewer’s concern.

[1] Holodeck: Language Guided Generation of 3D Embodied AI Environments, Yang et al. (CVPR 2024) [2] DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesis, Tang et al. (CVPR 2024) [3] ATISS: Autoregressive Transformers for Indoor Scene Synthesis, Paschalidou et al. (NeurIPS 2021) [4] Fast and Flexible Indoor Scene Synthesis via Deep Convolutional Generative Models, Ritchie et al. (CVPR 2019)

Q: Would including axis and units as visual prompts be beneficial?

Response: Indeed, including an axis and unit as visual prompt could be beneficial. There is a submission to ICLR which explores this exact question (https://openreview.net/forum?id=IXOoltTofP) It would be great to include human performance on each of the setups. We agree with the reviewer on this! We note that our S1 and S2 task are trivially solvable by humans. Our S3 task is more difficult and we will study human performance on this task. Note that the dataset is created from human generated designs from the 3D-Front dataset and human created negatives, and all VLMs show chance performance on S3.

Q: It would be great to include human performance on each of the setups.

Response We agree with the reviewer on this! We note that our S1 and S2 task are trivially solvable by humans. Our S3 task is more difficult and we will study human performance on this task. Note that the dataset is created from human generated designs from the 3D-Front dataset and human created negatives, and all VLMs show chance performance on S3.

We thank the reviewer for their time and look forward to continued discussion!

Q: It seems like the paper is solely studying VLM's performance in a limited 2D domain, it would be beneficial if the authors could provide a clarification on how the task is different from or harder than assessing VLM on unlimited domains of images.

Response: While the indoor scene synthesis task is indeed abstracted into 2D floorplan placement in current literature, real world applications would necessitate complete 3D understanding from renderings. Our finding that simplified 2D sketches are preferred to 3D rendering in our tasks shows that current gen VLMs are not ready to be used as general purpose visual design assistants due to their failure on simplistic tasks. Since our evaluation is on controlled rendered data, we can test models with the exact same data in different configurations, such as our variation of rendering viewpoints, or whether different objects in question are rendered or not. For example, for a VLM to understand whether a target object can be placed at a specified location, if that target object needs to be rendered (S2 all-in), this implies an O(N) rendering steps which is expensive, vs. if the VLM can make the judgement while only rendering existing objects or none at all (S2 existing-in, S2 Empty) which is O(1) in rendering steps. Understanding these are important to building a visual design assistant and we believe our evaluation has simplicity while being difficult for current-gen VLMs and being applicable to the design of a LLM and VLM based visual design assistant. Generating such data and insights from natural images would be difficult, but we do agree that if possible, that would enable evaluation at a much large scale and diversity. We will modify the paper to highlight this better and we thank the reviewer for the suggestion.

Q: Tab. 2 missing the final Avg. score. The caption is not sufficient for readers to understand the score marked as red and bolded.

Response: We apologize for this oversight. We will remove the red text as it was an error on our part. We use bold text to show the best performance, we will improve the caption to reflect this. We omit the final Avg. score since that would average over all models and all question types, which we do not believe is valuable information. The averages per model reflect the strength of each model, whereas the averages over All-in, Empty and Existing-in show the strength of question types, which shows us that the “Empty” question type shows superior performance across models and corroborates our finding that VLMs do better with lesser (best with none) image information in these tasks, which we believe is due to:

• Over-reliance on symbolic processing in the presence of visual information, and
• Degradation in symbolic reasoning abilities in the presence of visual information

Q: "an With, ..." in Ln. 297, I assume "an " should not be there.

Response Thank you for noticing this. We will fix this.

Q: Inconsistent formatting of Table. Tab. 5 does not have bolded text and red text, and Tab. 3 does not have red text.

Response Indeed, including red text in previous tables was an oversight and we will correct them. In Table 5, we do not show any bold text since all models have close to chance performance and are hence not worth highlighting. Once again, we will improve the text to reflect this reasoning. Please let us know if the reviewer agrees with these proposed changes.

We thank the reviewer for their time and valuable comments and look forward to continued discussion!

Q: The major issue with this work is the overclaiming of evaluating VLM for 3D reasoning while the efforts are all made on 2D.

Response: We agree with the reviewer and also discuss in our limitations section that our tasks could be considered 2D reasoning. This generally reflects the state of the 3D indoor scene synthesis field [1,2,3,4], which most commonly abstracts the problem into the placement of furniture on a floorplan. We did intend to extend this to layered furniture, but found that the floorplan placement task is already challenging for VLMs and hence limited our scope. We believe our findings apply directly to the 3D indoor scene synthesis literature. To avoid any further confusion, we will remove 3D from our title.

[1] Holodeck: Language Guided Generation of 3D Embodied AI Environments, Yang et al. (CVPR 2024)

[2] DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesis, Tang et al. (CVPR 2024)

[3] ATISS: Autoregressive Transformers for Indoor Scene Synthesis, Paschalidou et al. (NeurIPS 2021)

[4] Fast and Flexible Indoor Scene Synthesis via Deep Convolutional Generative Models, Ritchie et al. (CVPR 2019)

Q: Since the efforts are all made on 2D, it is worth noticing that the findings in the paper are mostly well-studied in the previous literatures

Response: Thank you for pointing out papers on visual information grounding. These are indeed relevant and valuable techniques to improve decoding with better visual grounding. We will include these in our discussion as potential methods to improve performance on our tasks. We believe our findings supplement the findings of concurrent work of Pooyan et al. (Vision language models are blind) since we test models with text-only, simplified 2D sketches as well as 3D rendering of indoor rooms, from multiple view points in a unified way. Moreover, we test methods to communicate coordinates with VLMs whereas prior work has used different methods without standardization. Furthermore, our tasks are clear decompositions of primitive skills required for indoor scene synthesis, with our last experiment also including understanding affordances and object functionality. Since VLMs already struggle at these tasks, we limited our scope and did not go beyond to include layered furniture (such as stuff on a table, or a chair pushed under a table which is not discernable from 2D alone) which would require additional 3D understanding skills.

Q: Since the paper claims to study the problem in the 3D domain, it is suggested that the evaluation of the model should focus more on how to provide the right 3D representations or how to formulate the questions. For example, different angles of the rendered scene with no pre-defined coordinate systems to ask questions about spatial relations like left and right.

Response: We note that while qualitative relationships such as left or right are studied in prior work such as Visual Spatial Reasoning, Liu et al. ACL 2023. Since the task of 3D scene synthesis also requires precise placement and assessment of furniture, we focus on dealing with coordinates over qualitative relationships. We would also like to restate that the field of 3D indoor scene synthesis currently focuses on layout synthesis on a floor-plan. As discussed in the paper, prior state of the art work on LLM-based indoor scene synthesis such as Holodeck (Yang et al.) has focused on generating “object relationship graphs” in a scene which are converted to precise placements using a heuristic algorithm. We ask whether a VLM could be used to replace the heuristic algorithm to make an end-to-end indoor scene synthesis method with LLMs and VLMs. We will be removing 3D from the title of our paper to avoid further confusion. Our tasks are all solvable from a single image alone by humans and hence we do not test 3D representations as input to models specifically trained to work with 3D inputs such as camera information with multi-view images or point clouds. However, we agree that these would be interesting future work and since our data is synthetically generated, future users could modify our codebase to perform such research.

Q: Different prompting techniques can impact VLM performance. Have you tried using chain-of-thought prompting or applying visual prompts to the input images?

Response: Indeed. We tried multiple phrasings and started from prompts used by Layout-GPT (Feng et al.) and Holodeck (Yang et al.). We also tried CoT prompting and do test with Set-of-marks visual prompting in the paper. Using CoT resulted in no performance change, and we report a subset of the performance here on our S2 task with gpt-4o-2024-05-13 (used in the paper) and the newer gpt-4o-2024-08-06. We report the results below, and we will add a discussion regarding this to our supplementary material and refer to it in the main paper.

Q: How should people leverage these findings from this paper in training a VLM with better spatial reasoning ability?

Response: Along with providing a benchmark to measure spatial performance in our task, we believe there are three actionable findings that can be studied independently. While these are not techniques to readily improve training, we believe they can be used to inform the kind of data utilized in pre-training or post-training / visual instruction tuning.

• Bias towards symbolic computation when clear visual evidence is available
• Reduction in symbolic reasoning performance in the presence of image input
• The availability of functions to compute free-space over perceiving it can readily improve performance on S2, which can be used for new methods using current-gen LLMs and VLMs for spatial tasks

Q: other minor issues:

Response: Thank you for bringing these to our attention. We will fix these in our revision.

We thank the reviewer for their time and valuable comments and look forward to continued discussion!

Q: Lack of in-depth analysis.

Response: We agree with the reviewer that understanding possible reasons for sub-par performance on our evaluation is necessary to improve the next generation of VLMs. Simultaneously, to measure progress on a skill in VLMs, we require evaluation data. While LLM-based indoor scene synthesis methods seem to produce excellent results (such as Holodeck, Yang et al. CVPR 2024), they might also signal a false sense of spatial understanding in LLMs. In their specific case, object placement is done externally from the LLM, through an iterative backtracking solver that exhaustively searches for potential solutions to a generated constraint graph. Our evaluation is intentionally simple and breaks down the indoor scene synthesis task into easily evaluated core components and shows limitations in problems that could be trivially solved by humans (S1 and S2). We believe this raises valuable awareness of the true spatial capabilities of current VLMs for the indoor scene synthesis task (and LLMs since we evaluate text-only performance as well). Moreover, we believe three findings merit targeted exploration beyond the indoor scene synthesis task:

• Bias towards symbolic computation when clear visual evidence is available
• Reduction in symbolic reasoning performance in the presence of image input
• The availability of functions to compute free-space over perceiving it can readily improve performance on S2, which can be used for new methods using current-gen LLMs and VLMs for spatial tasks

Q: Inclusion of 3D LLMs.

Response: While 3D LLMs would be a good candidate for such tasks, our tasks are completely solvable by a human from a single image. 3D LLMs that we are aware of (and are available to test) require additional inputs such as camera info, depth, multi-view images, point clouds. We could produce this data and agree with the reviewer that a future version of the benchmark could include such information to allow testing 3D LLMs as well. Our evaluation is intentionally simple and can be solved from a single image by humans, sometimes trivially, such as our S2 all-in task. We believe this builds valuable awareness of lack of spatial reasoning in current image-only VLMs. We believe it will be valuable to add this discussion to our manuscript and thank the reviewer for the suggestion. Moreover, if the reviewer has a particular 3D LLM in mind that is compatible with our scope i.e. single images and is available for testing, we would appreciate a concrete reference and we will be happy to experiment and add results.

Q: Diversity in room appearances.

Response: Our findings in-fact show that simple visual rendering (sketches) induce better reasoning performance in VLMs over 3D rendering. Hence we believe that improved and diverse rendering in the dataset would add complexity to the task that is currently unnecessary. As models evolve and saturate this benchmark, such diversity as proposed by the reviewer could become useful. Additionally, we intentionally re-render the data ourselves instead of using prior 3D-front rendering methods (such as that from the original dataset or from ATISS, Paschalidou et al. which is commonly used) to ensure the data is not seen in training. We will discuss this in our updated manuscript.

Q: Embodied view is skipped for task type 3

Response: This is intentionally done since a single embodied view of the room might not see the whole room context. To simplify the tasks to be a single image task (not all VLMs support multiple images), we skip the embodied view. We agree that this does limit the applicability of S3 to broader spatial reasoning tasks in embodied AI. We will release our rendering code, which future users could use to render multiple views or other 3D information to test on the same task.

Q: Evaluation Strategy for Task Complexity.

Response: This is a great suggestion and we could extend evaluating S1 and S2 without providing the necessary text input to be able to perform symbolic reasoning. Our current evaluation helps us evaluate fairly across visual modalities and without any vision input by providing the same prompt to the model. When an image is provided, we only add the following prompt – “Please use the attached image for reference. Both existing and target objects are shown in the image”. Our experiments with additional prompting, including prompting to first describe the image and then answer the question failed. We did not include them in the paper to keep the scope and evaluation contained, but this could indeed be a future strategy to improve performance on such data.

We thank the reviewer for their time and valuable comments and look forward to continued discussion!

Q: Unclear Positioning.

Response: The reviewer is right and we use this abstraction following representative indoor scene layout synthesis methods in the literature [1, 2, 3, 4]. We agree that this does not reflect in-the-wild 3D reasoning which will require further work in both 3D layout synthesis methods and in evaluation, where the latter is our focus. We will update our manuscript to clarify this distinction. Moreover, to avoid confusion, we will remove 3D from the paper title.

[1] Holodeck: Language Guided Generation of 3D Embodied AI Environments, Yang et al. (CVPR 2024)

[2] DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesis, Tang et al. (CVPR 2024)

[3] ATISS: Autoregressive Transformers for Indoor Scene Synthesis, Paschalidou et al. (NeurIPS 2021)

[4] Fast and Flexible Indoor Scene Synthesis via Deep Convolutional Generative Models, Ritchie et al. (CVPR 2019)

Q: Limited Dataset and Task Scope.

Response: This is a valid criticism, and again reflects indoor scene synthesis literature as a whole, where most methods work on similar abstractions. Our original intention was to extend to stacked furniture (stuff on a table, for example.). However, we find that VLMs already struggle at understanding furniture placement on a 2D floorplan, which is why we limited our scope of evaluation to make it concise. Moreover, large public datasets with stacked furniture / layered decor are yet to be made available to enable such research.

Q: Over-Reliance on Symbolic Reasoning.

Response:

• We indeed observe this. Our final evaluation experiment (S3) specifically addresses this question. In S3, we do not provide text inputs that could enable symbolic reasoning, thus forcing the model to visual reasoning alone. We find that all models we test have close to chance performance. In our S2 all-in task, where both the target and existing furniture are rendered, the task is trivially solvable visually. Yet, the VLMs we evaluate chose to reason symbolically. This is indeed a clear limitation of VLMs on a simple visual task, and our attempts to alleviate this with prompting to focus on the image did not help. Our intention is thus to leave this as a problem for the community to solve through better prompting, or through better training for visual reasoning, where symbolic reasoning is not the preferred mode.
• The other two evaluations, S1 and S2, provide both symbolic and visual input, where the VLM is free to choose any or both. This enables us to fairly evaluate models in text-only mode i.e. without any visual input. This enabled us to find that the addition of visual inputs decreases symbolic reasoning performance in current VLMs.

Q: Single Data Source.

Response: The 3D-FRONT dataset does contain additional room types and furniture that we could add with our data generation script, which we intend to release. We focus on bedrooms with rectangular layouts following layout-gpt (Feng et al.), since it allows us to simply define room boundaries in text to evaluate in text-only mode. We intentionally re-render 3D-Front in a novel style with Blender to ensure the data is not seen during training and hence evaluates generalization to some degree. We agree that a future version of such an evaluation dataset (once this eval is saturated) should include diverse room types, layered furniture and diverse styles to capture more of the real world. For now, we intentionally scope our data to be as simple as possible while still showing stark limitations in current models.