Struct2D: A Perception-Guided Framework for Spatial Reasoning in MLLMs

We greatly appreciate your careful review and insightful comments. Thank you for recognizing the contributions of Struct2D and Struct2D-Set. Below, we provide detailed responses to your concerns.

1. Comparison with GPT4Scene: Novelty and Prompting Design

We respectfully clarify that although GPT4Scene and Struct2D both use BEV representations and object markers, our method differs significantly in motivation, design, and technical execution:

• Targeted Spatial Reasoning Focus: Our primary goal is to enable explicit spatial reasoning, particularly global spatial tasks such as relative directions and route planning, as proposed in VSI-Bench. In contrast, GPT4Scene focuses more on scene understanding and QA in relatively constrained settings. The task scope and required reasoning complexity are therefore different.

• Selective Object Prompting: While GPT4Scene includes all objects in a scene (via STO-Markers), Struct2D only includes objects relevant to the question, significantly reducing visual clutter in the BEV image. This makes the image more interpretable and improves reasoning performance.

• Guide Prompts & Reasoning Chains: Our prompting strategy integrates guide prompts and intermediate reasoning steps during both training and inference (akin to chain-of-thought prompting). This improves model interpretability and performance and offers a more structured approach to 3D scene reasoning.

• Efficiency and Generalizability: Our prompting method is more concise and efficient, as it avoids exhaustive object rendering. In addition, we introduce a generalizable data generation pipeline, enabling scalable tuning of MLLMs for spatial reasoning, which goes beyond the static pipeline of GPT4Scene.

• Key Insight: Our zero-shot results suggest that the performance bottleneck in modern MLLMs lies in 3D perception, not reasoning. We believe this is an important insight that can shape future work on 3D vision-language models.

2. Fair Comparison with GPT4Scene: Resolution and Training Cost

We appreciate your concern regarding fair comparisons! We have carefully matched the experimental settings:

• Resolution Matching: GPT4Scene stitches 16 frames into a 2×8 grid. Each frame has a resolution of 128×123, resulting in a composite keyframe image of 512×246. Their BEV images are also resized to 512×512. We adopted the same resolution setup to ensure a fair comparison.

• Data Volume Differences: GPT4Scene uses all 16 video frames, whereas our Struct2D approach averages only ~2.5 keyframes per sample, with selection driven by object relevance. Despite this much lower visual input volume, our method matches or outperforms GPT4Scene, and can do well at reasoning-heavy benchmarks like VSI-Bench.

• Updated Quantitative Comparison (Qwen2-VL-7B):

• Training Cost Clarification: Using the same hyperparameters as GPT4Scene (batch size, learning rate, dataset), our training takes 6.5 hours on 8×H200 GPUs. The 8-hour figure mentioned in the paper includes additional training with extra curated QA types. We will clarify this in the revised manuscript.

3. Struct2D-Set: Dataset Design and Advantages

We designed Struct2D-Set to specifically address spatial reasoning gaps in existing 3D instruction-tuning datasets:

• Reasoning-Centric QA Pairs: Most prior datasets (ScanQA, SQA3D, Scan2Cap) emphasize object attributes, descriptions, or basic viewpoint understanding. In contrast, Struct2D-Set focuses on global spatial reasoning, such as route planning, orientation, and relative positioning.

• Instruction-Tuning Compatibility: We enrich both QA types and formats to better support instruction tuning of MLLMs, incorporating tasks like step-by-step spatial deduction, instruction following, and spatial transformations.

• 2D BEV Representation: All samples include BEV images with minimal but relevant object annotations, allowing efficient and scalable instruction tuning without the need for complete 3D scene representations as used in existing 3D tuning datasets.

• Diversity and Coverage: Struct2D-Set includes both complex reasoning questions and standard QA formats such as from ScanQA and SQA3D, ensuring broader coverage for generalization while staying focused on our spatial reasoning objective.

We hope this clarifies our contributions and positioning relative to GPT4Scene. Thank you again for the valuable feedback! We will revise the manuscript accordingly to better communicate these points.

We are grateful to Reviewer QoNX for the helpful and encouraging feedback. We appreciate the recognition of our paper’s organization and the breadth of our experimental analysis. Please find our detailed responses to the reviewer’s questions and suggestions below.

1. Metadata Extraction and Filtering

We extract metadata by detecting 3D object bounding boxes and categories from the reconstructed point cloud using off-the-shelf 3D detectors such as Mask3D and UniDet3D. Each object’s center coordinates are taken as its location. This structured metadata is used to provide grounding context in prompts. To ensure relevance and reduce noise, we filter metadata to only include objects referenced in the question.

2. Evaluation Subset Construction

We do not randomly sample the subset. Instead, we first verify that all objects referenced in each question are clearly annotated in the 3D point cloud. Then, we select a subset that preserves the distribution of question types from the full dataset. This ensures a balanced and representative evaluation set.

3. Text Query Length and Ablation

The text query length includes image tokens, structured metadata (converted to text), and the input question. We set the maximum input length to 4096 tokens, which empirically accommodates all required information without truncation. Reducing the length to 2048 degrades performance (accuracy drops from 41.3 to 33.5) due to incomplete input encoding. Increasing it to 6000 brings no additional gains, indicating 4096 is sufficient.

The $T^{out}$ refers to model output. We use max_new_tokens=2048 to allow complete answer generation.

4. VLM Usage Settings

For zero-shot prompting, we use a temperature of 1.0 and generate five responses per input. The final answer is chosen by majority vote. For QA pair enrichment, we also set the temperature to 1.0 and generate each sample with a single forward pass.

5. Comparison with Qwen2-VL

Thank you for the suggestion. We added a comparison using Qwen2-VL-7B to evaluate the impact of newer models. As shown below, Qwen2.5-VL provides slight performance gains over Qwen2-VL. Importantly, our pipeline improves performance in both cases and consistently outperforms GPT4Scene.

6. Limited Improvements on Some Benchmarks

The improvements on benchmarks such as ScanQA and Scan2Cap are relatively modest (44.9 Vs 43.4, 34.8 Vs 36.3), as these tasks largely depend on localized or viewpoint-level understanding and can often be addressed effectively with just a single or dual-frame input.

Our method is designed for complex, global spatial reasoning, which is better reflected in VSI-Bench tasks (e.g., multi-object relationships, navigation). Limitations in keyframe selection, missing object detections, and occlusion may hinder performance on traditional benchmarks that lack comprehensive spatial annotations.

Additionally, standard benchmarks often use rule-based metrics (e.g., BLEU, METEOR, ROUGE scores) that may bias models toward templated or memorized responses. We instead emphasize accuracy-based evaluation to more reliably measure spatial reasoning capabilities, as in VSI-Bench.

We sincerely thank Reviewer a2sz for the thoughtful and constructive feedback. We appreciate the recognition of Struct2D as practical, technically sound, and experimentally thorough. Below, we address the reviewer’s questions and concerns point by point.

1. 3D Preprocessing Requirement

Thank you for raising this point. As we noted in our supplementary, Struct2D does rely on 3D perception in preprocessing, but this step is lightweight (under 5 seconds per scene on a single GPU) and entirely offline. Importantly, this design choice decouples perception from inference, enabling flexible and efficient deployment. Accurate 3D perception is a standard practice in recent 3D-LLM research and plays a critical role in enabling robust spatial reasoning. Continued progress in this area will further amplify the effectiveness of models like Struct2D:)

2. Failure Cases of Struct2D

Thank you for the valuable suggestion. We conducted a qualitative analysis on 30 representative questions across different QA types in VSI-Bench. Among the 16 failure cases, we identified two primary causes: (1) in 11 cases, noisy or incomplete 3D reconstructions led to degraded BEV inputs, and (2) in 5 cases, missing detections, particularly for small or occluded objects, resulted in incomplete or incorrect structured inputs. These factors hinder Struct2D’s ability to provide accurate spatial cues, which are essential for effective reasoning.

While the NeurIPS rebuttal policy prohibits image inclusion, we will present representative visual examples and a more detailed breakdown of these failure modes in the revised manuscript.

3. Case Study on Outdoor / Open-world Domains

Thank you for the suggestion. We conducted a case study using scenes from DL3DV-10K, which includes various outdoor environments such as tourist sites, educational institutions, and natural settings. We reconstructed point clouds from RGB frames using VGGT, manually labeled object marks, and constructed 15 spatial reasoning QA pairs (e.g., “What is in front of the door of the public restroom?” → “Three trash bins”). Using Struct2D prompting with GPT-O3 in a zero-shot setting, we achieved 72% accuracy. Most failures were due to noisy or incomplete reconstructions, which are more common in outdoor scenes.

While our current focus aligns with existing indoor benchmarks, we agree that broader generalization is important (as in supplementary) and plan to incorporate outdoor scenes into future training and evaluation.

4. Dataset Generation a bit Unclear

Thank you for the helpful suggestion. Our dataset generation process involves: (1) sampling scenes from existing 3D indoor datasets; (2) leveraging object annotations to generate diverse QA types using templated codes; (3) manually balancing the distribution of QA types; (4) enriching questions and answers with explicit reasoning steps; and (5) conducting quality control to ensure clarity and correctness.

We agree that a flowchart would significantly clarify this pipeline. Although we cannot include figures in the rebuttal due to PDF restrictions, we will incorporate a clear flowchart in the revised paper to enhance presentation:)

We sincerely appreciate reviewer 54eJ’s thoughtful and constructive feedback. We are grateful for the positive remarks on the clarity of our writing, the depth of our evaluation, and the value of our insights. Below, we address the reviewer’s comments and questions in detail.

1. Guided Prompt used in Instruction Tuning

Yes, we incorporate guided prompts during instruction tuning, particularly for "relative direction" questions. This design is inspired by our findings from zero-shot prompting, where structured guidance proved especially beneficial for spatial reasoning tasks:)

2. Details on Keyframe Selection

Thank you for the suggestion. We will include more details and a pseudocode block in our revision. Our keyframe selection aims to enhance the BEV representation by providing better object detail, particularly for questions involving object attributes and viewpoint-based spatial relations.

To build the keyframe set $I_{\text{keys}}$, we sample $N=100$ RGB-D frames and iteratively select those that improve coverage of objects not yet captured in the BEV. For each candidate frame $I_i$, we project the remaining uncovered objects onto the image and its depth map. If the projection lies within the image bounds and has valid depth, the object is marked as visible and added to the BEV. The corresponding frame is then included in $I_{\text{keys}}$. This process repeats until all target objects are covered.

3. Limited Novelty

Thank you for the feedback. While GPT4Scene also leverages 2D inputs for 3D reasoning with MLLMs, our work tackles a different and more challenging problem scope -- complex spatial reasoning tasks such as relative direction and route planning, which require multi-step geometric inference rather than simple object-level understanding.

Moreover, our contributions go beyond incremental improvements. We introduce a data generation pipeline and a tunable dataset specifically designed for spatial reasoning instruction tuning. Our structured reasoning framework integrates object-centric BEV representations, guided prompts, and keyframe selection to support fine-grained geometric reasoning.

Lastly, in contrast to GPT4Scene’s primarily rule-based evaluation, our work adopts accuracy-based metrics that more directly reflect reasoning performance.