3D Question Answering via only 2D Vision-Language Models

1. "... finetuning LLaVA-OV ..."

Due to space limitation, please kindly refer to the response of PgGq 2

---

2. "... analysis of the types of questions" & "a discussion on the limitations ..."

We conduct a detailed analysis of question types on the SQA dataset, with per-type performance reported in Table S2 (Supplement). The results show that our zero-shot 2D LVLM-based method achieves strong performance across most question types, particularly excelling in “what”-, “which”-, and “other”-type questions.

These categories primarily involve identifying objects and reasoning about their spatial configuration within the 3D scene. Our method, powered by multi-view aggregation, can aggregate multi-view observations to infer spatial layouts and object relations in 3D environments, even when key objects are never observed within a single view. As shown in Figure R3 (1st and 2nd sample), the model correctly infers relationships between spatially separated entities across the room.

Limitation: In contrast, we observe relatively lower performance on “can”-type questions, which often involve agent-centric reasoning and require understanding how the scene would change with respect to the agent’s actions or viewpoint shifts (e.g., “Can I see the faucet if I turn around?”). These cases demand the modeling of dynamic viewpoints and agent-scene interaction, which are not fully captured by current 2D LVLMs in a frozen, zero-shot setting.

Overall, our analysis confirms that frozen 2D LVLMs, when equipped with informed view selection, can already support a broad range of 3D spatial reasoning tasks.

---

3. "... broader spatial relations ..."

Our method is capable of reasoning about spatial relationships between objects that are never observed in the same view. As shown in the second example of Figure R3, the printer, trash can, and paper cutter are distributed across different views with no co-observability. Despite this, the model accurately infers their spatial arrangement and correctly answers the question.

This demonstrates that, even without explicit 3D reconstruction, our approach can integrate multi-view information to support global spatial reasoning.

---

4. "Strong claims"

• [L153-156]: change from "3D understanding tasks" to "3D-QA tasks"
• [L155-157]: correct to "... as a critical factor in zero-shot 3D-QA, for which there is a lack of an efficient solution in prior works."
• [L125-127, right column]: correct to "... but rely on explicit 3D reconstruction, needing additional models and causing more processing steps. In contrast, our method uses 2D views and feeds them into a unified 2D LVLM, which makes a simpler pipeline".
• [L142-144]: correct to "They focus more on evaluating pretrained 2D LVLMs on 3D-QA tasks, rather than developing approaches to adapt and improve their performance for spatial reasoning."

---

5. "... cdView component contributes to the overall performance ..."

We agree that LLaVA-OV provides a strong foundation, and even the uniform view sampling yields competitive results. This actually motivated us to explore more in the direction of leveraging well-trained 2D LVLMs for tackling 3D-QA tasks, and then we proposed the light-weight learnable module cdViews.

As we can see in Table 1 (manuscript), using cdViews brings consistent and clear performance improvements over uniform sampling across multiple benchmarks. Specifically, it yields +2.0% / +2.1% EM@1 and +7.0% / +7.1% CIDEr gains on the ScanQA test set (w/o objects), and +3.4% EM@1 improvement on SQA. These gains can be regarded as "substantial", especially considering that prior work typically improve by only 1% EM@1 or 3–4% CIDEr (see Table 1).

---

6. "claims zero-shot ..."

Due to space limitation, please kindly refer to the response of rVhy 1

---

7. “Problem formulation”

[Lines 162–164]: “..., each associated with a camera matrix containing the position and orientation.”

---

8. "L323, L300, L359"

We thank the reviewer for pointing these typos out. We will fix them in the revised version.

---

9. "Differ from video-VQA"

Traditional video-based VQA methods, while also using 2D information, are designed to process sequential frames to understand temporal dynamics. In contrast, cdViews focuses on selecting the most informative static 2D views from a 3D scene to answer questions about the 3D space.

---

10. "Computation cost and memory cost"

Due to space limitation, please kindly refer to the response of PgGq 3

1. "... the unfair comparison ..."

We agree that LLaVA-OV is a strong 2D LVLM. However, we do not consider the comparison unfair, as our core motivation and contribution lie in exploring how to leverage powerful 2D LVLMs in a zero-shot manner for 3D-QA.

• Compared to 3D-based methods, current 3D-LLMs are limited by the scarcity of large-scale 3D-language data. This data bottleneck is exactly what we aim to address. By contrast, our method directly utilizes pretrained 2D LVLMs, which benefit from tons of 2D vision-language data.

• Compared to 2D-3D hybrid methods, it may seem fairer to replace the 2D module with LLaVA-OV. However, these hybrid pipelines typically require additional training to align 2D and 3D features. For large 2D LVLMs like LLaVA-OV, this alignment process incurs substantial computational cost. This introduces a significant resource bottleneck.

• Our method is designed as a practical solution under both data and resource constraints, showing that strong off-the-shelf 2D LVLMs can be effectively used for 3D-QA. Moreover, the framework is model-agnostic and generalizable to future 2D LVLMs with improved capabilities.

---

2. "... finetuning LLaVA-OV ..."

The reviewer suggests fine-tuning LLaVA-OV within a 2D–3D hybrid pipeline, which could potentially benefit from both strong language–vision alignment and 3D spatial context. However, this setting introduces fundamental limitations: combining 2D LVLMs with 3D inputs requires additional model components, training supervision, and modality alignment, making the pipeline heavier and less scalable. As discussed in Lines 68–74 of our Introduction, "2D features extracted from LVLMs are already well-aligned with language, but further alignment with 3D features requires careful model design and advanced training techniques. "

As requested, we implement a hybrid variant based on BridgeQA, the strongest hybrid baseline in our comparisons. Specifically, we follow the BridgeQA architecture and replace its 2D LVLM with LLaVA-OV. The input views are combined with question and 3D features. For fair comparsion, we use 9 views selected by cdViews as the input. Results are reported in Table R1.

Table R1: Evaluating fine-tuned LLaVA-OV in a 2D–3D hybrid pipeline.

---

3. "... computation efficiency comparison ..."

• Number of candidate views: Each 3D scene contains between 6 and 382 RGB views, with an average of 79 views per question.

• Computation during view selection: All candidate views are processed by the view selection module. However, this stage contributes only a small fraction of the total cost—less than 10% of overall FLOPs—as shown in Table R2.

• Comparison with image retrieval: In the view selection stage, our method reduces FLOPs by 54.8% (26.5T vs. 58.6T) and runtime by 90% (0.04s vs. 0.40s). In the QA stage, FLOPs are reduced by 49.9% (268T vs. 535T), and inference time by 51.3% (1.16s vs. 2.38s).

• GPU memory usage: As shown in Table R2, cdViews achieves a lower peak GPU memory cost, making it more suitable for deployment on devices with limited memory capacity.

Table R2: Efficiency comparison between baseline method and our cdViews..

---

4. Ablation on the number of selected views and computational cost

We provide the requested ablation on computational cost with respect to the number of selected views. As shown below, FLOPs grow approximately linearly from 268 TFLOPs (9 views) to 534.98 TFLOPs (17 views). For the performance impact of view number, please kindly refer to Figure 4 of the main paper.

1. “... ensure the quality for the training of viewSelector.”

We agree that using LVLMs alone for annotation may lead to unreliable views.

To address this, viewAnnotator is designed to capture informative views beyond simply image matching. Specifically, we incorporate a step-by-step system prompt (shown in Figure R1) in the View Matching process. This prompt ensures that all key objects, attributes, and spatial relationships in the caption align with the image, reducing ambiguity and improving consistency. Uncertain views are explicitly excluded, enhancing the robustness of the annotation process. Additional examples of positive and negative views are shown in Figure R2.

To further validate the reliability of the positive views, we conducted a human evaluation:

• We randomly selected 50 QA pairs with their associated positive views.
• Three human evaluators assessed whether each view could answer the question.

Their accuracy rates were 96.72%, 94.28%, and 97.56%, confirming that the quality of positive views is sufficient for training.

These details and results will be included in the supplementary document of the revised version.

---

2. “Equation (3) is confusing ...”

We appreciate the reviewer's detailed comments. In the original Equation (3), the number of sampled views $k$ was not clearly represented on the right-hand side. We will modify the Equation (3) as follows:

$$F_{uniform}(M, k) = \{V_{i_j}\}_{j=1}^k, i_j \sim {Uniform}(1, N).$$

---

3. “ ... Duplicated definition of $S_i$...”

Both instances of $S_i$ on Line 289 (left column) should be updated to $\hat{S}_i$ for consistency with Equation (7). In our notation, $S_i$ denotes the ground-truth label, and $\hat{S}_i$ represents the predicted score from the model. The expressions on Line 289 correspond to model outputs, so the appropriate notation is $\hat{S}_i$.

---

4. “... directly add these two distances ...”

Yes, the two distances are on comparable scales. Based on statistics from the ScanQA and SQA datasets, position distance ranges from $[0, 14.72)$, and the orientation distance is within $[0, \pi)$. Given their similar scales, they can be combined.

We tested different combination weights on the validation set (Table R1). The results show that viewNMS is fairly robust to weighting variations, with equal weighting (i.e., direct summation) yielding the best performance.

Table R1: Ablation study on distance weighting in viewNMS. We vary the relative weights of position distance ($D_{pos}$) and orientation distance ($D_{ori}$) while keeping the threshold fixed at 0.5. Equal weighting (1:1) achieves the best EM@1 performance.

---

5. “... viewNMS Thresholds part ...”

We clarify that viewNMS does not require searching over parameter pairs. As noted in our response to the previous question (regarding Equation (12)), the position and orientation distances are directly combined with equal weights, so only a single threshold needs to be selected.

This threshold is chosen based on the validation performance. As shown in the line chart in Figure 7 of the manuscript, a threshold of 0.5 yields the best result.

---

6. “... the backbone matters a lot ...”

Our 3D-QA pipeline consists of three main steps: view selection, feeding the selected view into a 2D LVLM, and generating an answer from the LVLM. Our main contribution is the whole pipeline of using 2D LVLM for 3D-QA. Our add-on contribution is proposing cdViews for more efficient view selection in the first step.

Since most network parameters (in our pipeline) reside in the 2D LVLM, the final performance heavily depends on its pre-trained knowledge. Compared to LLaVA-Next and Bridge-3D, LLaVA-OV performs better due to its improved vision-language alignment—enhancing image understanding and generating more accurate, detailed descriptions—achieved through stronger language modeling and high-quality instruction tuning on open-vocabulary tasks.

1. “The claim of zero-shot is questionable”

Sorry for the confusion caused. For our best method LLAVA-OV + $F_{cdViews}$, the term zero-shot could be more precisely scoped: we will revise Lines 153–156 to state it as zero-shot 2D LVLM inference (rather than fully zero-shot).

We would like to clarify that our goal in this paper is to make use of 2D LVLMs (which are much better trained than 3D ones) for free, i.e., to adapt them for 3D tasks without any large-scale 3D pre-training or fine-tuning. To this end, we proposed two pure no-training methods denoted as LLAVA-OV + $F_{uniform}$ and + $F_{retrieval}$,
and one with small-scale training (on the viewSelector) denoted as + $F_{cdViews}$. During inference, 3D-QA is conducted by feeding the uniformly-sampled / retrieved / selected 2D views and the question into LLAVA-OV, and LLAVA-OV outputs the answer based solely on its knowledge learned from large-scale 2D pre-training.

In the paper, we will revise the motivation paragraph of the Introduction section to make these clearer.

---

2. “... which may limit its generalization ... ”

We address this question from two perspectives: the generalization of our 3D-QA pipeline and the trainable module viewSelector.

First, our pipeline leverages 2D LVLMs for 3D tasks without large-scale 3D-language pretraining. It supports three view selection methods: viewSelector, random sampling, and image retrieval. The latter two require no training (assuming minimal domain gap between retrieval models' training data and 3D-QA data). This ensures flexibility and broad generalization of the whole pipeline.

Second, viewSelector is a lightweight trainable module that easily adapts to another similar task with small-scale 3D training data. Compared to large-scale 3D-language pretraining, fine-tuning viewSelector offers a practical way to enable the generalization of 2D LVLMs to 3D tasks.

---

3. “... the accuracy of the annotated positive views ...”

We agree that using LVLMs alone for annotation may lead to unreliable views.

To address this, viewAnnotator is designed to capture informative views beyond simply image matching. Specifically, we incorporate a step-by-step system prompt (shown in Figure R1) in the View Matching process. This prompt ensures that all key objects, attributes, and spatial relationships in the caption align with the image, reducing ambiguity and improving consistency. Uncertain views are explicitly excluded, enhancing the robustness of the annotation process. Additional examples of positive and negative views are shown in Figure R2.

To further validate the reliability of the positive views, we conducted a human evaluation:

• We randomly selected 50 QA pairs with their associated positive views.
• Three human evaluators assessed whether each view could answer the question.

Their accuracy rates were 96.72%, 94.28%, and 97.56%, confirming that the quality of positive views is sufficient for training.

These details and results will be included in the supplementary document of the revised version.

---

4. “An ablation study is missing...”

As suggested, we performed ablation comparing the retrieval + viewNMS and cdViews and show the results in Table R1. Combining image retrieval with viewNMS reduces the number of input views due to redundancy removal, but it brings very marginal performance improvement (+0.1%).

The reason is that viewNMS prioritizes diversity, but the overall performance still depends on whether the selected views contain the necessary visual evidence for the question. This highlights a key difference between image retrieval and viewSelector--while image retrieval focuses on matching text and images, viewSelector is explicitly trained to find views that are most important for reasoning.

Table R1: Comparison of image retrieval and cdViews with and without viewNMS on the ScanQA validation set. The best EM@1 scores are reported with the corresponding optimal $k$.

---

5. “Typos: ...”

We thank the reviewer for pointing this out. We will revise it.