COS3D: Collaborative Open-Vocabulary 3D Segmentation

We thank the reviewer for appreciating our work (e.g., original contribution, proposing a technically solid and well-justified method, conducting extensive experiments, and maintaining a clear structure) and for providing insightful comments (e.g., non-category queries and the trade-off between kernel regression and the MLP counterpart). We are pleased to address the issues raised in the review.

Q1: Non-category queries.
A1: Thank you for the insightful suggestion. Note that the standard benchmarks (e.g., LeRF, ScanNet) commonly used in existing open-vocabulary baselines do not include evaluations for non-category queries. But we have provided some qualitative examples with non-category queries (e.g., materials: fleece, affordance: drinkable, abstract: music) in Supp. Fig. 1 and Supp. Fig. 4. These results demonstrate the capability of our approach to understand non-category queries to some extent. Both the quantitative comparisons on standard evaluations and the qualitative examples with non-category queries consistently highlight the effectiveness of our proposed COS3D. We agree that building a benchmark for non-category queries and designing comprehensive quantitative evaluations is an interesting and valuable direction, which we leave for future work.

Q2: 3D Gaussians Splatting (3DGS) representations.
A2: Thanks. Our COS3D adopts 3DGS as the 3D representation with the following considerations. First, 3DGS has become a widely used and popular representation for real 3D scenes to support efficient rendering. Moreover, it has also been adopted by recent baselines [1, 2, 3, 4, 5] for the open-vocabulary 3D segmentation task.

• Generalization to other 3D representations: We believe our methods have the potential to generalize to other 3D representations, such as point clouds. For instance, recent advancements (e.g., VGGT [6]) can connect the original 2D image with 3D representations (e.g., point (cloud) maps), enabling the use of 2D foundation models to provide open-vocabulary segmentation for 3D point clouds. We leave further explorations in future work.

We will include these discussions regarding further generalization in the revised version.

Q3: Hyperparameter analysis in adaptive refinement.
A3: Thank you for your questions. There are two hyperparameters in the adaptive refinement algorithm: the region expansion threshold $\mathcal{T}$ and the widely used relevance threshold $\tau$.

• The region expansion threshold $\mathcal{T}$: Instead of using a fixed value for this hyperparameter, we implement a hyperparameter generation strategy as described in Algorithm 1 (see Supp Page 2). In this approach, the final expansion threshold is determined based on the calculated mean segmentation feature distances. Additionally, we conducted experiments with different expansion thresholds by applying permutations (ranging from -20% to 20%) to the mean feature distances. The results, shown in the table below, demonstrate that COS3D achieves stable performance within a reasonable range of region threshold values and consistently outperforms the latest SOTA method (InstanceGaussian[5]).

Table 1. Quantitative comparisons of different region expansion threshold $\mathcal{T}$ values. We conduct the experiments under different expansion thresholds by adding a moderate permutation (from -20% to 20%) to the mean feature distances.

• The relevance threshold $\tau$: This hyperparameter has been widely used and proven to be relatively robust in existing methods [1, 2, 3, 4, 5]. Therefore, we simply set it to a fixed value of 0.5 throughout our experiments.

Q4: Practical analysis between kernel regression and its MLP counterpart.
A4: This is a great question. First, both kernel regression and MLP are approaches used to implement our instance-to-language mapping function. The improved performance achieved by these two strategies demonstrates the effectiveness of the mapping design.

We selected these two implementations primarily because of their distinct theoretical properties. Specifically, kernel regression is a traditional fitting technique that provides strong interpretability (e.g., transparent kernel functions) and intuitive locality. In contrast, the MLP counterpart can handle flexible data patterns (both local and global) and requires only the storage of model parameters after training, making it more compact.

In practice, we observed a trade-off between accuracy and memory efficiency. On the one hand, kernel regression achieves better accuracy compared to the MLP counterpart. As discussed in the ablation section of the main paper, we attribute this to the discriminative instance features chosen as the source domain, which make the mapping process inherently an easy regression task. In such cases, the traditional kernel regression method is well-suited.

On the other hand, the MLP counterpart, consisting of small layers, offers better GPU memory efficiency. Specifically, the computational complexity of kernel regression depends on the size of the training data, resulting in higher memory requirements. To illustrate this, we report the following table comparing the accuracy and GPU memory usage of kernel regression and MLP, using an identical input batch size of 100K.

Table 2. Trade-off between accuracy and memory efficiency on LeRF.

[1] LangSplat: 3D Language Gaussian Splatting. CVPR2024.
[2] LEGaussians: Language Embedded 3D Gaussians for Open-Vocabulary Scene Understanding. CVPR2024.
[3] OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understanding. NeurIPS2024.
[4] Dr. Splat: Directly Referring 3D Gaussian Splatting via Direct Language Embedding Registration. CVPR2025.
[5] InstanceGaussian: Appearance-Semantic Joint Gaussian Representation for 3D Instance-Level Perception. CVPR2025.
[6] VGGT: Visual Geometry Grounded Transformer. CVPR2025.

We thank the reviewer for their helpful comments and suggestions (e.g., ablation on different VLMs), which have helped enhance the quality of our paper. We are pleased to address the issues raised in the review.

Q1: Ablation on different VLMs.
A1: Thank you for your suggestions. We utilize CLIP and SAM as the default 2D language and segmentation models in our experiments, following common practice in recent baselines [1, 2, 3, 4, 5], to ensure that the improvements are attributed to the proposed collaborative fields design.

However, our methods can adopt other foundation models. Based on your suggested alternatives, we conducted an ablation study comparing different 2D foundation models (e.g., CLIP [6] vs. SigLIP [7], SAM [8] vs. SAM2 [9] & Semantic SAM [10]). The results, shown in the following table, demonstrate that our framework is compatible with different 2D foundation models. Furthermore, we empirically observed that using more advanced models (e.g., SAM2 for segmentation, SigLIP for language) can lead to performance improvements. We sincerely appreciate your insightful suggestions, which contributed to these valuable observations!

We will include ablation results on different 2D foundation models in the revised version of our paper.

Table 1. Ablation on 2D foundation models on LeRF.

Q2: Procedure for SAM instance mask generation.
A2: Thanks. To generate the 2D instance masks from SAM, we follow the automatic mask generation strategies commonly used in prior works [1, 2, 3, 4, 5]. Specifically, we create a grid of point prompts across the image and obtain the final segmentation results using the same hyperparameters as in OpenGaussian[3]. Further details can be found in the supplementary materials (Supp. L26–L28). We will clarify this procedure in Sec. 3.2 paragraph (“Stage 1: instance field learning.”) for the revised version.

Q3: Typos.
A3: Thank you for pointing out the typos in Fig. 2 and Fig. 4. We will update them in the revised version.

[1] LangSplat: 3D Language Gaussian Splatting. CVPR 2024.
[2] LEGaussians: Language Embedded 3D Gaussians for Open-Vocabulary Scene Understanding. CVPR 2024.
[3] OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understanding. NeurIPS 2024.
[4] Dr. Splat: Directly Referring 3D Gaussian Splatting via Direct Language Embedding Registration. CVPR 2025.
[5] InstanceGaussian: Appearance-Semantic Joint Gaussian Representation for 3D Instance-Level Perception. CVPR2025.
[6] Learning transferable visual models from natural language supervision. ICML 2021.
[7] Sigmoid Loss for Language Image Pre-Training. ICCV 2023.
[8] Segment Anything. ICCV 2023.
[9] SAM 2: Segment Anything in Images and Videos. ICLR 2025.
[10] Semantic-SAM: Segment and Recognize Anything at Any Granularity. ECCV 2024.

We are encouraged to see that the reviewer finds the proposed method novel, effective in addressing the limitations, achieving strong performance improvements, and well-organized. We sincerely thank the reviewer for their constructive comments (e.g., discussions on extensions and additional analysis), which help enhance the quality of our paper. We are pleased to address the issues raised in the review.

Q1: Potential extension to reasoning ability.
A1: Thanks for your question. This work currently focuses on the open-vocabulary 3D segmentation task, and we plan to explore reasoning ability in our future work, as noted in our limitations. Moreover, incorporating reasoning abilities using advanced LLM models is a promising direction. Specifically, it is technically feasible to replace the CLIP[1] feature with the vision encoder feature from a V-LLM model, such as Qwen-vl2.5 [2]. However, how to effectively integrate the learned 3D Qwen feature fields with the LLM backbone to handle the reasoning queries and enhance segmentation quality remains an open question, as the relevance between the 3D Qwen feature fields and the text queries cannot be directly modeled using feature distance (e.g., cosine similarity) as in CLIP-style models (e.g., CLIP[1], SigLIP[3] ). One possible solution is to render the 3D Qwen feature field into a 2D feature map based on a reference view, allowing it to be fed into the LLM backbone along with reasoning queries. It is an interesting research problem to design comprehension solutions that enable interaction between 3D LLM-style feature fields and reasoning queries. We will provide more discussion on the future work in the revised version.

In our supplementary robotic demo, we incorporate reasoning ability to some extent using a two-stage strategy: first, the Qwen-VL model parses complex commands into short queries; then, our COS3D model predicts the 3D segmentation results based on the queries.

Q2: Analysis between kernel regression and the MLP counterpart.
A2: Thanks for your interesting question. Both kernel regression and the MLP counterpart are means to achieve our instance-to-language feature mapping function, and the improved performance achieved by these two strategies consistently demonstrates the effectiveness of our feature mapping design.

Moreover, we provide a discussion from the perspectives of accuracy and memory efficiency:

• Accuracy: We observe that the kernel regression version tends to achieve better performance than the MLP counterpart in our experiments. We attribute this to the selection of discriminative instance features as the source domain, which makes the mapping learning process inherently an easy regression task, where the traditional kernel regression method is extremely suitable. Moreover, kernel regression benefits from its training-free property, bypassing the unstable optimization process required by the MLP counterpart.

• Memory efficiency: On the other hand, the MLP counterpart, consisting of small layers, offers better GPU memory efficiency. Specifically, the computational complexity of kernel regression depends on the size of the training data, resulting in higher memory requirements.

To illustrate this, we report the following table comparing the accuracy and GPU memory usage of kernel regression and MLP, using an identical input batch size of 100K.

Table 1. Trade-off between accuracy and memory efficiency on LeRF.

We will incorporate more discussion in the revised version.

Q3: Hyperparameter analysis in kernel regression.
A3: Thanks. For kernel regression, we set this hyperparameter to 0.1 across all experiments, and it performs well without requiring additional tuning. Additionally, we conducted an ablation study on $\sigma$, with the results presented in the following table. Overall, the results remain stable under moderate changes to $\sigma$, and our method consistently outperforms the latest state-of-the-art approach, InstanceGaussian [4].

Table 2. Quantitative comparisons for different $\sigma$ values in kernel regression.

Q4: Training times.
A4: Thank you. We originally plotted the training time of most baselines in the main paper Fig. 1(d) and included partial information in the main paper Tab. 5 due to space constraints. For clarity, we have organized the complete information into the table below. We will update the full table in the supplementary materials.

Table 3. Training efficiency analysis on LeRF. We report mIoU.

Q5: Typo.
A5: Thank you for pointing out the typo in L318. We will correct this in the revised version.

[1] Learning transferable visual models from natural language supervision. ICML 2021.
[2] Qwen2.5-VL Technical Report. ArXiv.
[3] Sigmoid Loss for Language Image Pre-Training. ICCV 2023.
[4] InstanceGaussian: Appearance-Semantic Joint Gaussian Representation for 3D Instance-Level Perception. CVPR 2025.
[5] LangSplat: 3D Language Gaussian Splatting. CVPR 2024.
[6] LEGaussians: Language Embedded 3D Gaussians for Open-Vocabulary Scene Understanding. CVPR 2024.
[7] Dr. Splat: Directly Referring 3D Gaussian Splatting via Direct Language Embedding Registration. CVPR 2025.
[8] OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understanding. NeurIPS 2024.

We thank the reviewer for appreciating our work (e.g., achieving state-of-the-art results, being fast, making COS3D practical for scalable deployment, supporting flexible queries and applications) and for their inspiring and thoughtful comments, which have helped us improve our paper. We are pleased to address the issues raised in the review.

Q1: …Gaussian splatting…not directly applicable to raw point clouds, online reconstruction ...
A1:

• Raw point clouds: Thanks. Our COS3D adopts the 3D Gaussian splatting (3DGS) representation reconstructed from multi-view image inputs, with the following considerations: First, Our 3DGS model can be derived from multi-view images, which are easy to acquire, widely available, and offer higher resolution along with rich textural information. In contrast, raw point clouds typically require specialized equipment for collection and are more expensive. Second, compared to raw point clouds, the 3D Gaussian representation enables the utilization of 2D foundation models like CLIP and SAM which contain rich world-level knowledge; thus, the 3DGS representation improves our model’s capability on the open-vocabulary segmentation. Besides, the 3D Gaussian splatting is a commonly-used 3D representation of most recent open-vocabulary 3D segmentation (OV3DS) approaches [1, 2, 3, 4, 5].

• Extension to online reconstruction: Thank you. Following recent approaches [1, 2, 3, 4, 5], we evaluate our model on standard offline benchmarks. It is an interesting direction to incorporate an online setting. To do so, one possible solution is to integrate our COS3D with existing online Gaussian reconstruction methods [6, 7,8, 9] to update our collaborative fields incrementally. We acknowledge that efficient online OV3DS requires additional research efforts, and we leave this for future work.

Q2: Discussion on potential extension to relational or multi-object queries.
A2: Thanks. This work currently focuses on the OV3DS task, and the proposed collaborative framework (COS3D) significantly surpasses previous state-of-the-art methods. We plan to explore relational reasoning in future work, as noted in our limitations. While extending our framework to support relational reasoning remains an open question, COS3D provides a solid foundation. For instance, the recent study [10] demonstrates that reasoning 3D segmentation (e.g., querying "the red chair next to the table") can be achieved to some extent by employing an LLM as an agent in combination with an OV3DS approach as a tool. We will provide more discussion on the potential extension in the revised version.

Q3: More comparisons with InstanceGaussian and Dr. Splat on ScanNetv2.
A3: Thank you for your reminder. Since InstanceGaussian and Dr. Splat were not open-sourced at the submission deadline, we did not include their results on ScanNetv2. Recently, we observed that InstanceGaussian and Dr. Splat were open-sourced thus we include more comparisons here. Considering the inconsistent evaluation protocols (as mentioned in the main paper, L282–L284) and input (i.e., segmentation mask) in these baselines, we ran the official codes of InstanceGaussian and Dr. Splat on ScanNetv2 using the same input and evaluation protocol as ours, following OpenGaussian, to ensure fair comparisons. The results presented in the following table demonstrate that our methods outperforms InstanceGaussian and Dr.Splat in all metrics.

Table 1. Comparison with the latest baselines on ScanNetv2. Our * are sourced from Table 2 in our main paper.

[1] LangSplat: 3D Language Gaussian Splatting. CVPR2024.
[2] LEGaussians: Language Embedded 3D Gaussians for Open-Vocabulary Scene Understanding. CVPR2024.
[3] OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understanding. NeurIPS2024.
[4] Dr. Splat: Directly Referring 3D Gaussian Splatting via Direct Language Embedding Registration. CVPR2025.
[5] InstanceGaussian: Appearance-Semantic Joint Gaussian Representation for 3D Instance-Level Perception. CVPR2025.
[6] Gaussian Splatting SLAM. CVPR2024.
[7] Gaussian-SLAM: Photo-realistic Dense SLAM with Gaussian Splatting. ArXiv.
[8] RGB-Only Gaussian Splatting SLAM for Unbounded Outdoor Scenes. ICRA 2025.
[9] Outdoor Monocular SLAM with Global Scale-Consistent 3D Gaussian Pointmaps. ICCV 2025.
[10] LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agent. ICRA2024.