Q1: Tabs 1-2-3 mention a version of the proposed method that actually uses labels (MLLM-based Model (w/ label)), as opposed to the original one that does not. I could not find details about this implementation in the paper.

R: Thank you for catching this omission.

• The “MLLM-based Model (w/ label)” variant reported in Tabs 1–3 is not part of our proposed label-free pipeline, but serves as a reference baseline to assess how close our pseudo-label-based training approaches full 3D supervision.
• It adopts a hybrid training strategy, where both 2D unprojected pseudo-labels (from multi-view reasoning) and available 3D ground-truth annotations are jointly used to supervise the 3D segmentation network.

Implementation Details:

• We use the same 3D backbone (MinkowskiNet14) as in our main pipeline.
• During training:
  • For samples where 3D labels are available, we supervise the 3D prediction using ground-truth 3D masks via standard cross-entropy loss.
  • For the remaining samples, we apply our proposed pseudo-labeling pipeline: multi-view masks are unprojected into 3D and aggregated using our token attention mechanism to serve as weak supervision.

---

Q2: Line [144–157]: How are token attention weights used?

R: We appreciate the request for clarity.

• The token attention weights are used to reweight each view’s contribution during 3D mask fusion.
• Specifically, each view’s unprojected 3D mask is scaled by its corresponding attention score, and the final mask is a weighted average across all views.

We will revise Section 3.2 to describe this aggregation strategy.

---

Q3: Inference Procedure — How is the final mask produced?

R: We clarify as follows:

• At inference:
  1. The 3D model produces per-point features.
  2. The transformed query embedding \mathbf{t} is computed from the [SEG] token.
  3. Cosine similarity between \mathbf{f}_p^{3D} and \mathbf{t} is computed for each point.
  4. Points above a similarity threshold are included in the predicted mask.

This will be added to Section 3.3 for completeness.

---

Q4: Why Use Both LISA and SAM?

R: Thank you for this perceptive observation. We do not use an external SAM model.

• LISA already integrates SAM, and we simply reuse the SAM decoder inside LISA to extract 2D binary masks.
• Meanwhile, we separately leverage LISA’s MLLM reasoning module to obtain the [SEG] token embeddings.

This modular design allows us to decouple:

• Semantic reasoning ([SEG] tokens from LISA)
• Geometric supervision (masks from SAM decoder) And makes our framework flexible, allowing future substitution of LISA with other MLLMs or mask generators.

We’ll clarify this architectural detail to avoid confusion.

---

Q5: Variance from Random 4-View Selection?

Thank you for raising this robustness concern.

To assess the sensitivity of our model to stochastic view selection, we conducted a robustness study under the label-free setting, using our default LISA-7B (MLLM-For3D) configuration. Specifically, we repeated the 4-view setup 5 times with different random seeds on the Instruct3D (ScanNet++) validation set.

We report the mean and standard deviation across runs:

• Acc@0.25: 39.10 ± 0.61
• Acc@0.50: 29.70 ± 0.48
• mIoU: 23.90 ± 0.52

These low standard deviations demonstrate that our token attention mechanism effectively mitigates view-level noise, ensuring that the final 3D pseudo labels are consistently aligned with the textual query despite variability in input views.

We will include this robustness analysis in Appendix.

---

Q6: Line [179] Formula: Should be \mathbf{t} Instead of \mathbf{q}?

Thank you!

• The logit score should be: $$s_p = \mathbf{f}_p^{\text{3D}^\top} \cdot \mathbf{t}$$

Dear Reviewer QEj4,

We sincerely thank you for the thoughtful and detailed feedback. We acknowledge the reviewer’s concern about architectural novelty and would like to state what specific problem we aim to solve, why it is important, and how our approach differs from existing open-vocabulary 3D segmentation methods.

1. Problem Definition: 3D Reasoning Segmentation, a challenging task where the goal is to segment an object in 3D that satisfies a free-form spatial or intention-driven query.

2. Why This Task Matters: While prior open-vocabulary 3D segmentation works (e.g., OpenScene, Open3DIS) focus on object recognition under closed-set or loosely aligned textual categories, reasoning segmentation requires inference grounded in spatial semantics, not just class prediction.

3. What’s New Compared to Existing Works:

• Task Level: We go beyond open-vocabulary recognition to spatial reasoning-based segmentation, which introduces ambiguity (e.g., multiple objects match partial semantics).
• Method Level: While we leverage the idea of unprojecting 2D masks (as Open3DIS and OV3D do), our key novelty lies in:
  1. Query-token binding: Aligning LISA-generated [SEG] tokens (query-conditioned) with 3D features in a weakly supervised manner.
  2. Cross-view token-weighted fusion: Not all views are helpful; we introduce attention-based selection to mitigate hallucination and disagreement.

We appreciate the reviewer’s pointer to Open3DIS (CVPR 2024), and we will cite and discuss it in the related work section to better position our approach within the landscape of 2D-to-3D segmentation frameworks.

Dear Reviewer ujXK,

We sincerely thank you for the constructive and thoughtful feedback. We appreciate your recognition of our motivation and results, and we address your comments below.

---

Q1. Computational Cost of MLLMs + SAM at Inference

R: Thank you for this important concern. We acknowledge that repeated inference over 2D views using MLLMs and SAM incurs computational cost, especially when using large backbones like LISA-13B.

• 2D mask and token generation is conducted offline before 3D training begins, which decouples the MLLM from the training pipeline.

• Inference Efficiency
  | Method | Modalities | Components | Inference Time (per scene) | |------------------------|-------------------|------------------------------------|-----------------------------| | Reason3D | 3D only | 3D-LLM encoder + retriever | ~2.1 sec | | Intent3D | 3D only | 3D Transformer-based architecture | ~1.8 sec | | Ours (MLLM-For3D) | 2D + 3D (6–8 views) | LISA (7B) + SAM + 3D decoder | ~3.5–4.0 sec |

Note: With fewer views (e.g., 2–4), our inference time drops to ~2.5 sec per scene.

---

Q2. Generalization to Outdoor or Dynamic Environments

R: We agree this is a compelling direction. Our current evaluation focuses on indoor environments due to the availability of 3D reasoning datasets (e.g., Instruct3D, Intent3D, Reason3D).

That said, our framework is modular and transferable to outdoor and dynamic settings, but poses new challenges:

• Outdoor scenes (e.g., nuScenes, SemanticKITTI) often rely on LiDAR and involve larger-scale environments, leading to increased computational and memory overhead during 2D/3D projection and fusion.

• Dynamic scenes (e.g., autonomous driving, embodied agents) require real-time processing. The current inference speed of 2D MLLMs like LISA makes it challenging to meet these latency constraints in practice.

---

Q3. Sensitivity to 2D Model Choices (LISA vs. Others)

R: We thank the reviewer for this thoughtful suggestion. Currently, reasoning segmentation in 2D/Video/3D is still an emerging research area, and we have made our best effort to benchmark against the most representative models and datasets in this space.

We will include a discussion in Sec. 4.4 to clarify that:

• SAM can be replaced with other segmentation backbones, yielding similar 2D pseudo-label quality.
• The [SEG] token from LISA plays a unique role in handling compositional, referential, and relational queries. For simpler settings, lighter models may suffice.

In future work, we plan to systematically study the impact of model choices across reasoning types, and investigate trade-offs between backbone size, query complexity, and mask quality.

---

Q4. Confidence-based Filtering for Pseudo-Labels

R: Our current token attention and spatial consistency modules already suppress noisy or hallucinated views to some extent.

However, we agree that adding confidence calibration or uncertainty filtering would improve pseudo-label quality, particularly for:

• Views with occlusions or ambiguous perspectives.
• Multi-instance scenes where view disagreement is high.

We plan to integrate this in follow-up work by leveraging mask entropy, token confidence, or Bayesian fusion strategies.

---

Q5. Broader Impacts and Ethical Considerations

R: Thank you for raising this. We will include a broader impact statement in the final version, covering both benefits and risks:

• Positive impacts:

  • Assistive robotics (e.g., household tasks via natural language),
  • Education and accessibility (e.g., describing 3D content via language),
  • AR/VR applications grounded in user intention.

• Potential risks:

  • Misuse in safety-critical systems if hallucinated masks are trusted blindly,
  • Privacy concerns when applying multi-view capture in real-world environments,
  • Dataset bias leading to reasoning shortcuts.

Dear Reviewer XtxR,

We appreciate your insightful feedback. Below, we address your concerns point by point.

---

Q1: Limited Novelty in Multi-view Aggregation; Relation to Prior Works (e.g., [a], [b], [c])

We appreciate this important question. While prior works such as VointCloud [a], MVTN [b], and View-GCN [c] have extensively explored multi-view aggregation for shape classification and part segmentation, our setting diverges in two critical dimensions:

1. Input Modality and Supervision

   • Prior works assume closed-set labels and operate under fully supervised settings, typically aggregating viewpoint-consistent geometric features for tasks like shape classification.
   • In contrast, our method handles open-vocabulary free-form queries using natural language, without any 3D supervision. This requires reasoning over semantically-aligned pseudo masks and [SEG] tokens across views — a far more unconstrained and ambiguous setup.

2. Aggregation Objective

   • Prior methods aim to learn view-invariant geometric embeddings. Our goal, however, is to align language-grounded semantics across views — using cross-view token alignment and geometric consistency, not just geometric feature fusion.
   • Our token-for-query binding strategy addresses semantic misalignment and view-level hallucinations — challenges largely absent in prior work.

We thank the reviewer for highlighting these important works, and we will include explicit citations and comparisons to [a], [b], and [c] in our related work section to clarify the distinction.

---

Q2: Memory and Computational Cost from Multi-view MLLM Processing

We acknowledge this limitation and appreciate the reviewer raising this point. Our method indeed trades off higher computation in exchange for annotation efficiency and generalization capability.

Clarifications:

• No End-to-End MLLM Training
  All view-wise outputs (binary masks + [SEG] tokens) are precomputed and cached, and the 3D network is trained separately. Thus, the memory bottleneck occurs only once during offline preprocessing, not during training or inference.

• Inference Efficiency
  | Method | Modalities | Components | Inference Time (per scene) | |------------------------|-------------------|------------------------------------|-----------------------------| | Reason3D | 3D only | 3D-LLM encoder + retriever | ~2.1 sec | | Intent3D | 3D only | 3D Transformer-based architecture | ~1.8 sec | | Ours (MLLM-For3D) | 2D + 3D (6–8 views) | LISA (7B) + SAM + 3D decoder | ~3.5–4.0 sec |

Note: With fewer views (e.g., 2–4), our inference time drops to ~2.5 sec per scene.

• Scalability and Trade-off Justification
  Our framework delivers zero-shot or label-free 3D reasoning segmentation, even outperforming supervised baselines. In real-world settings where 3D annotation is expensive or unavailable, this trade-off offers practical value.

We will make the resource requirements and caching strategy clearer in the final version.

---

Q3: Lack of Multi-view Reasoning Visualizations (e.g., Fig.7 in MVTN [b])

We agree that richer visualizations would enhance transparency and interpretability. Due to space limits in the initial version, these were omitted, but we plan to include the following in the camera-ready version:

• View-wise token attention heatmaps to show how relevant views are selected;
• Cross-view mask disagreement maps, visualizing alignment before and after pseudo-label fusion;
• Before/after spatial consistency correction in the unprojected 3D space.

---

We thank the reviewer again for the insightful comments that have helped us improve the clarity and positioning of our work.

W1: visual comparisons

R: We appreciate the reviewer’s question.

---

1. Evaluation: Answering Before Masking

As in SegPoint [1] and Reason3D [2], we adopt a two-step evaluation protocol:

• Accuracy@kIoU (k = 0.25 / 0.5): Measures whether the model identifies the correct target object.
• mIoU: Assesses segmentation quality after the correct target is identified.

Thus, for this task, it is more important to correctly ground the object before evaluating mask quality.

---

2. Why Reason3D Fails?

Take Figure 3(a) as an example:

“The container designated for holding waste and it is the one closest to the door.”

There are three trash bins in the scene (see Fig. 1). All satisfy "holding waste", but only one is closest to the door.

• Reason3D retrieves an incorrect bin due to limited spatial understanding. This illustrates that the failure is not due to model power, but to a fundamental task-method mismatch.

---

References

[1] He et al., SegPoint: Segment Any Point Cloud via Large Language Model. ECCV, 2024.
[2] Huang et al., Reason3D: Searching and Reasoning 3D Segmentation via Large Language Model. 3DV, 2025.

---

• W2: Why Does Performance Drop with 8 Input Views? Shouldn't More Views Always Help?
• Q3: How will the performance change if we keep increasing the number of input views?

R: We thank the reviewer for raising this important question.

1. Empirical Observation

• Performance peaks at 4 views, but drops at 8 views.
• Without view filtering, extra views introduce occlusions, redundant angles, or low-quality reasoning results.
• Our token attention module selectively filters these noisy views, regaining performance.

---

2. Why More Views Hurt

As discussed in Lines 39–51 of the paper, we identify two key limitations when using 2D reasoning outputs to supervise 3D:

1. Each 2D frame is processed independently by the MLLM (e.g., LISA) and SAM. If the object is partially visible or occluded, hallucinated [SEG] tokens or masks can be misleading.
2. The [SEG] token must not only match the query, but also align semantically with the correct target object in the specific view.

---

3. What Makes a "Good" View?

We define a valid view as one that:

• Contains the target object.
• Produces a [SEG] token and binary mask that semantically align with the textual query.

This explains why naively adding more views increases noise instead of improving pseudo-label quality.

---

Q1: Why Not Use 3D-MLLMs like ChatScene or GPT4Scene + 3D-SAM?

R: We appreciate this insightful suggestion.

1. Limitations of Current 3D-MLLMs require fine-tuning on labeled 3D datasets, such as Intent3D or Reason3D. Such datasets are limited and expensive to scale.

---

2. Quantitative Comparison

As shown in Table 2, even fine-tuned 3D-MLLMs underperform our zero-shot 2D MLLM-based method:

• Our zero-shot pipeline matches or surpasses fine-tuned 3D MLLMs, without requiring 3D labels.

---

3. We acknowledge that GPT4Scene [Qi et al., arXiv 2025], which builds 3D representations from video, is promising. However:

• It is not designed for 3D reasoning segmentation.
• It requires dense multi-view inputs and video data, which limits applicability in static indoor scenes like ScanNet++.
• Due to time and computational constraints, we could not include it in this version, but will discuss it in the camera-ready paper.

---

Q2: Efficiency*

R:

Note: With fewer views (e.g., 2–4), our inference time drops to ~2.5 sec per scene.

• While our method introduces moderate overhead, it achieves notably higher accuracy in Acc@IoU and mIoU metrics (see Table 2 and Table 4).
• Our use of frozen 2D models allows for easy parallelism and batching, making the pipeline scalable for practical deployment.