We thank the reviewer for recognizing the effectiveness of our method and providing feedbacks. We would like to take this opportunity to clarify the following points.

$W1, Q1$ Clarification of Motivation and Contribution

Thank you for your feedback. While we appreciate the reviewer’s perspective, we would like to respectfully clarify our motivation and contributions. Our primary focus is on open-vocabulary segmentation, and our core motivation is to assign each Gaussian an accurate and discriminative language embedding to enable precise text-driven segmentation. We achieve this by eliminating semantic ambiguities and establishing correspondence between Gaussian groups and objects (e.g., Gaussians belonging to the same object have identical embeddings and each Gaussian only represents one object).

Specifically, we design a robust tracking module and develop Gaussian-specific optimization strategy, all these are served for establishing object-Gaussian correspondence and eliminating ambiguity. These components are central to our method and serve the overall goal of improving open-vocabulary segmentation performance.

Therefore, we respectfully disagree with the assessment that our method primarily focuses on class-agnostic segmentation. While class-agnostic grouping is used as an intermediate step, our full pipeline is thematically and technically aligned with the goal of open-vocabulary 3D segmentation.

$W2, W5, Q2$ Novelty

We would like to emphasize the novelty and contributions of our proposed pipeline, and clarify how it fundamentally differs from previous works.

All prior methods (e.g., OpenGaussian, LangSplat, Gaussian Grouping, etc.) follow a “Splat-then-Segment” paradigm. Taking OpenGaussian (mentioned by the reviewer) as an example, it first reconstructs the scene using 3D Gaussian Splatting while simultaneously learning a feature field via contrastive loss. It then clusters the Gaussians based on the learned features, and finally assigns CLIP embeddings to each cluster to enable open-vocabulary segmentation.

“Splat-then-Segment” will cause several issues.

1. Geometric and Semantic Ambiguities: Without object-level constraints during reconstruction, Gaussians near object boundaries may represent multiple objects and carry mixed semantic information, leading to ambiguity.
2. Object-Gaussian Misalignment in Dynamic Scenes: In dynamic scenes, a single Gaussian may represent different objects across time, requiring impractical time-varying embeddings for open-vocabulary segmentation. Thus, direct adaptation of “Splat-then-Segment” methods to dynamic scenes is fundamentally limited.

We propose Segment-then-Splat, we first “segment” Gaussians into object-specific groups then during reconstruction, each group of Gaussians only contributes to its own object, thus eliminating the ambiguity and misalignment issue inherent in “Splat-then-Segment” approaches.
Our method achieves superior segmentation performance compared to baselines, particularly around object boundaries (see Fig. 4 in the main paper), and can be directly applied to dynamic scenes without additional modification.

$W3$ Higher mIoU of OpenGaussian compared to the original paper

Thank you for your observation. The original OpenGaussian implementation holds out every eighth image (i.e., llffhold=8) during training to follow the standard evaluation protocol for reconstruction. However, this results in some segmentation-labeled images being inadvertently included in the training set, which is inconsistent with the evaluation protocols followed by other baseline methods.
To ensure a fairer evaluation and to align with the settings used by other baseline methods, in segmentation experiments, we exclude all labeled images from the training set and use only the unlabeled images for training. This adjustment allows more images to be used for reconstruction compared to the llffhold=8 setting, thus leading to higher reconstruction quality and, consequently, improved segmentation performance.

$W4$ Missing Citations

Thank you for pointing this out. As some of the referenced works are not directly focused on open-vocabulary segmentation, we did not include them primarily in the current version. However, we acknowledge their relevance and will carefully review these works and consider including appropriate citations and discussions in the final version of the paper.

$W5$ Reliability of the Robust Tracking Module

We have conducted comprehensive experiments on 4 datasets including LERF-OVS, 3DOVS, Neu3D, and HyperNeRF ensuring its robustness. Additionally, we conduct new experiment on the more challenging MipNeRF360 dataset, including scene with hundreds of cluttered objects (e.g., Counter and Bonsai) to showcase the reliability and effectiveness of our method, the results are provided below:

Dear AC and Reviewers,

We thank AC for initiating this discussion, and also appreciate reviewer cG1h for his/her valuable feedback.

1. First we want to justify why OpenGaussian is not “Segment-then-Splat”.
   a. The following are the training steps stated in the official github repository of OpenGaussian:

[Stage 0] Start 3dgs pre-train ... (step 0-30k)  
[Stage 1] Start continuous instance feature learning ... (step 30-50k)  
[Stage 2.1] Start coarse-level codebook discretization ... (step 50-70k)  
[Stage 2.2] Start fine-level codebook discretization ... (step 70-90k)  
[Stage 3] Start 2D language feature - 3D cluster association ... (1 min)

OpenGaussian first trains 3DGS for 30k steps in stage 0, which is “Splat”, then in the stage1 it utilizes contrastive learning to learn a feature field. After that in stage 2.1 and 2.2 it conducts codebook discretization, which clusters Gaussians into different groups, this is “Segment”. Thus, OpenGaussian is “Splat-then-Segment”, not “Segment-then-Splat”.

b. Similar evidence can also be found in the appendix A.1(1) section of the OpenGaussian paper. They claim their training strategy is consistent with LangSplat (also Splat-then-Segment), we copy and paste the paragraph here:

Training Strategy. Consistent with LangSplat, we first pre-train the standard 3DGS for 30,000
steps. Subsequently, we freeze the Gaussian coordinates, scale, and opacity parameters, and train the
instance features for 10,000 steps (ScanNet is 20,000 steps) and the two-layer codebook for 30,000
steps (ScanNet is 40,000 steps). The 2D-3D feature association step is training-free. The extraction
methods for SAM masks and CLIP features also align with LangSplat. While LangSplat extracts
three layers of SAM masks (small, middle, and large), our implementation uses only one layer (large).

c. This distinction also explains why OpenGaussian still suffers from geometric and semantic ambiguities. Please refer to the supplementary video material for a direct segmentation comparison between our method and OpenGaussian. As demonstrated, our method produces notably more accurate and precise object boundaries than OpenGaussian.

2. Regarding the perceived similarity between our Fig. 1 and OpenGaussian's Fig. 1. The differences are as follows:

• We are comparing the difference between Splat-then-Segment and Segment-then-Splat. While OpenGaussian is comparing 2D segmentation and 3D segmentation, which is totally different.
• We include both training and inference stage in the figure, while they only include the inference stage.

The reviewer thinks they are similar, maybe because in Fig.1(a) we are demonstrating the splat-then-segment pipeline, which OpenGaussian follows.

3.Furthermore, we would like to clarify the novelty of our proposed tracking module, which fundamentally differs from that employed by GaussianGrouping. GaussianGrouping independently performs per-frame segmentation first and then employs a temporal propagation model to associate masks across frames. This approach inherently risks breaking multiview consistency, as it assumes that segmentation of the same object remains identical across frames. For instance, an object segmented as a single instance in one frame (e.g., a man wearing a hat) might be segmented as separate instances (e.g., a man and a hat) in a subsequent frame. Indeed, if one examines the project page or GitHub repository of GaussianGrouping, the provided GIFs or videos clearly illustrate frequent abrupt mask changes, directly evidencing compromised multiview consistency.

In contrast, our tracking module leverages the mask from the first frame as an initial prompt and employs SAM2 to propagate this mask consistently across subsequent frames, thus explicitly ensuring temporal consistency. Simultaneously, our robust tracking strategy effectively detects newly appearing objects and manages issues such as tracking loss. This methodology fundamentally differs from the tracking approach employed by GaussianGrouping.

Moreover, GaussianGrouping does not explicitly utilize the tracked masks to group Gaussians. Instead, it learns an implicit identity encoding through differentiable rendering, which further distinguishes their method from ours.

4. We thank the reviewer for recognizing the new techniques we introduced, however, as we stated in rebuttal $W1, Q1$ , all these designs are primarily focusing on how to eliminate the geometry and semantic ambiguity of the Gaussian, thus enabling a better performance in open-vocabulary segmentation. We respect the perspective of the reviewer, but don’t agree that our work is focusing on class-agnostic segmentation.

Thank you for your review, we sincerely appreciate your comments and would like to clarify the following points.

Robustness and Strong Reliance on SAM2

While we use SAM2 to obtain the initial tracking results, our proposed robust tracking module is designed to handle potential failure cases of SAM2. Our pipeline is evaluated across four datasets (e.g., LERF-OVS, 3DOVS, Neu3D, and HyperNeRF), ensuring its robustness. Additionally, we conduct further experiments on the more challenging MipNeRF360 dataset, including scene with hundreds of cluttered objects (e.g., Counter and Bonsai) to showcase the reliability and effectiveness of our method, the results are provided below:

Multiview Consistency

We are not entirely sure what the reviewer means by “multiview consistency.” In our pipeline, mask consistency is maintained by SAM2 along with our robust tracking module. Additionally, our proposed object-centric reconstruction ensures that the 3D object geometry is consistent across multiple views, and the CLIP embedding association enforces embedding consistency for the same object across different viewpoints.

Three scales of SAM

Sorry for the confusion. The 3 scales are semantic levels pre-defined in the Segment Anything (SAM) $1$ as “whole”, “part” and “subpart”. These levels are used to capture varying degrees of semantic granularity within an object. Please kindly refer to the SAM paper for more details. As these scales are semantic-wise, they are not related to metric (geometric) scales.

$1$ Kirillov, Alexander, et al. "Segment anything." Proceedings of the IEEE/CVF international conference on computer vision. 2023.

Scenes with camera moving close-by and far-away

Thank you for the question. All scenes in the LERF-OVS dataset, on which we conducted our experiments, feature camera trajectories that observe objects both from close-up and from a distance, and our approach performs robustly under these conditions. Please check the LERF $2$ paper and website for the visualization of their dataset.

$2$ Kerr J, Kim C M, Goldberg K, et al. Lerf: Language embedded radiance fields $C$ //Proceedings of the IEEE/CVF international conference on computer vision. 2023: 19729-19739.

Missing comparison to point cloud-based related works

Thank you for mentioning these related works. However, OpenScene, OpenMask3D, and Segment3D operate under a different problem setting compared to ours. All of these methods assume access to existing 3D geometry (e.g., point clouds) as input, whereas our method takes only multi-view 2D images as input and performs open-vocabulary segmentation after reconstructing the scene geometry.
As a result, the segmentation performance in our method and the baseline ones inherently influenced by the quality of the reconstruction, introducing a distinct set of challenges not present in the above-mentioned works. Therefore, a direct comparison would not be fair or meaningful, as the input assumptions and task formulations differ significantly. We will go through these works and consider discussing them in the related work section.

The method is not fully open-vocabulary

Though our method can only return the reconstructed objects at 3 different semantic levels, namely, “whole objects”, “parts”, and “sub-parts”, these levels are designed to comprehensively cover the semantic granularity of most objects and their components in the scene.
As a result, arbitrary text queries can still be matched to the most relevant reconstructed object or part through CLIP-based embedding comparison. While the system does not generate new segmentations on-the-fly, it remains practically open-vocabulary in that it supports flexible and diverse text queries without being constrained to a fixed category set.

We thank the reviewer for recognizing the reasonableness, practicality and performance of our proposed method. Your support is greatly appreciated and we would like to take this opportunity to clarify several points.

$W1$ Evaluation on 2D Segmentation Mask

Thank you for pointing this out, and we apologize for the confusion. The evaluation results presented in Tab.1 are computed based on 2D segmentation masks, in order to align with the evaluation protocols used by all baseline methods.
For the 3D part in Tab.1, we modify 2D-based baselines (e.g., LEGaussian and LangSplat) to perform segmentation directly on 3D Gaussians instead of pixels. However, to maintain a fair and consistent comparison, all methods are ultimately evaluated on their rendered 2D segmentation maps. This ensures comparability across different approaches while preserving the integrity of 3D-aware segmentation modeling.

$W2$ Experiments on Mipnerf360

Thank you for your suggestion. We have conducted additional experiments on the Mipnerf360 dataset. And the results are shown in the table below, where our method performs well on this more challenging dataset and surpasses all baseline methods:

$W3$ Text query referring to multiple objects

Yes, this is indeed a limitation of our current method. Since object embeddings are derived solely from objects themselves, our model does not explicitly capture spatial or relational context between multiple objects. However, this limitation is shared by all baseline methods $1$

$2$

$3$

$4$

$5$ , as their embeddings are constructed in a similar object-isolated manner.
Incorporating a global scene-level representation, such as a scene graph that models relationships between objects (e.g., proximity, co-occurrence, or functional interaction), could potentially address this issue. We consider this a promising direction for future work and appreciate the reviewer for highlighting this important point.

$1$ Shi J C, Wang M, Duan H B, et al. Language embedded 3d gaussians for open-vocabulary scene understanding $C$ //Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 5333-5343.

$2$ Qin M, Li W, Zhou J, et al. Langsplat: 3d language gaussian splatting $C$ //Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 20051-20060.

$3$ Ye M, Danelljan M, Yu F, et al. Gaussian grouping: Segment and edit anything in 3d scenes $C$ //European conference on computer vision. Cham: Springer Nature Switzerland, 2024: 162-179.

$4$ Wu Y, Meng J, Li H, et al. Opengaussian: Towards point-level 3d gaussian-based open vocabulary understanding $J$ . Advances in Neural Information Processing Systems, 2024, 37: 19114-19138.

$5$ Labe I, Issachar N, Lang I, et al. Dgd: Dynamic 3d gaussians distillation $C$ //European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024: 361-378.

Thank you for your thoughtful and encouraging review. We sincerely appreciate your valuable feedback and your recognition of our method's novelty and strong performance. We are delighted that you highlighted the innovation of the "Segment then Splat" concept and acknowledged its advantages for dynamic scenes. Your support is very motivating.

$W1$ More fragmented model pipeline.

Although our method introduces a more complex preprocessing step, it eliminates the need to train an additional feature field. Moreover, the CLIP embedding association is performed during training, not at inference time. At inference, we simply compare the input text embedding with the precomputed embeddings of each Gaussian group to retrieve the most relevant object, ensuring an efficient and lightweight retrieval process.

$W2$ Complex scenes may be challenging.

In complex scenes where semantic information is dense and noisy, language field-based methods tend to struggle with entangled representations. In contrast, our approach associates language embedding with object-specific Gaussian groups, leading to more robust and disentangled representation.

The tracking logic has shown reliable performance on LERF-OVS, 3DOVS, Neu3D and HyperNeRF dataset. We conduct additional experiment on MipNeRF360 dataset, which contains scenes with hundreds of cluttered objects (i.e. Counter and Bonsai), to demonstrate the effectiveness and reliability of our approach:

$W3, Q2$ Object-level ground truth is not fair for those pixel-level methods.

Sorry for the confusion. During the experiments the ground truth used is actually pixel-level, not object-level. This aligns with the original setting of all the baselines. For 3D comparison with LangSplat and LEGaussian, we modify their inference code to extract text-related Gaussians instead of pixels and render the extracted Gaussians to image plan to evaluate on pixel-level ground truth. Based on this evaluation strategy, the performance of the original baselines will not degrade, thus the comparison is fair for baseline methods. While our approach may suffer from small performance degradation due to the pixel-level ground truth.

$Q1$ Performance analysis on different object granularities.

Thank you for pointing this out. We separate the ground truth objects in LERF_OVS dataset into two categories: 1) large/whole objects (e.g. ramen, bear, sheep), represented by (L), 2) small/sub-parts (e.g. onion segment (in the ramen), bear nose, hooves (of the sheep)), represented by (S). And provide the quantitative results in the following table:

$Q3$ Supervision may be insufficient due to object random sampling.

Thank you for the excellent point. You are correct that random sampling can dilute the supervision signal. In our settings, we chose to supervise 3 objects to balance the performance and optimization efficiency. As shown in Tab.2(a), while performance does not fully saturate, we observe a clear convergence trend: the improvement from supervising 1 to 3 objects is substantial, while gains from 3 to 5 (and beyond) become marginal. In scenes with a large number of objects (e.g., Ramen), achieving full saturation may require supervising more objects or running additional optimization iterations.

$W4, Q4$ Limitations and potential improvements.

One limitation is that sometimes the associated embeddings with each Gaussian set cannot be successfully queried by text-input. This is likely due to our current use of a simple averaging strategy when assigning CLIP embeddings. In cases where an object is heavily occluded in certain viewpoints, the embedding from that view may ruin the entire object embedding and result in query failure. This is also a future direction, developing a more robust and adaptive embedding assignment strategy could further enhance retrieval accuracy and overall performance.

Thank you for your response. In Table 1(a), our evaluation strategy aligns with the original settings of the following baselines: LangSplat (2D), LEGaussian (2D), G-Grouping (2D), and OpenGaussian (3D), ensuring their reported performance remains unaffected. However, for LEGaussian (3D) and LangSplat (3D), which rely on differentiable rendering to learn language embeddings for each Gaussian, our evaluation setup may lead to a degradation in performance. Due to the lack of other suitable 3D-based baselines for comparison, we adopt the experimental setting used in OpenGaussian[1] to compare with LEGaussian (3D) and LangSplat (3D). For details, please kindly refer to Section 4.1(2) Baseline in the OpenGaussian paper.

[1] Wu, Yanmin, et al. "Opengaussian: Towards point-level 3d gaussian-based open vocabulary understanding." Advances in Neural Information Processing Systems 37 (2024): 19114-19138.

We thank the reviewer for the thoughtful feedback and for recognizing our proposed “Segment-then-Splat” approach as a novel reversal of the traditional pipeline, the comprehensiveness of our evaluation, and our method’s ability to support dynamic scenes. We would like to take this opportunity to clarify the following points.

$W1$ Novelty

We appreciate the reviewer’s perspective. While our “Segment then Splat” pipeline may appear conceptually simple, we would like to argue in the following aspects:

1. We reverse the long established “Splat then Segment” approach in Gaussian Splatting based open-vocabulary segmentation tasks.
2. Based on (1) we establish an object-centric reconstruction approach, achieving a better object boundary localization as shown in Fig.4 compared to baseline “Splat then Segment” methods. Unlike prior methods, our approach does not require training an additional feature field, making it more efficient and easier to optimize.
3. Based on (1), we eliminate the Gaussian-object misalignment issue in dynamic modeling, thus can be directly applied to dynamic scenes without additional modification to form a unified 3D open-vocabulary segmentation approach.

$W2, W3, Q1$ Efficiency, Scalability and Performance on Larger Scenes

Thank you for pointing this out. In the current implementation we proposed a random sampling strategy to compensate for the overhead introduced by object-specific supervision during training. To evaluate the scalability of our method, we conducted additional experiments on the MipNeRF360 $2$ dataset, which includes highly cluttered scenes with hundreds of objects (e.g., Bonsai and Counter). The performance and efficiency metrics are summarized in the following table:

As shown, our method outperforms baseline approaches without any modification to the training pipeline, while maintaining superior efficiency. This suggests that our random sampling strategy effectively balances supervision and training cost, even in more complex scenes.

However, such overhead on larger scenes can still increase linearly if we need either more objects to be sampled per iteration or extend the training iteration when facing extreme cases.
To further improve the training efficiency, the rasterization backend can be modified to support batch rendering on the objects (e.g. one pass rasterization for all the objects). Notably, the GSplat $1$ backend already supports batch rasterization for a single set of Gaussians. By modifying the rasterizer to add Gaussian masks during rasterization, we can render different objects in batch, thus enabling a more efficient and scalable training process. Although we have not implemented this yet, we consider it a promising future refinement, and we believe the efficiency issue can be fully resolved through such engineering strategies.

$1$ Ye V, Li R, Kerr J, et al. gsplat: An open-source library for Gaussian splatting $J$ . Journal of Machine Learning Research, 2025, 26(34): 1-17.

$Q2$ Robustness of Object Tracking

SAM2 itself may easily fail to track objects accurately in complex scenarios (e.g., Figurines in Tab.2(c)), due to issues like lost tracking, duplicate tracking etc.. And that’s the reason why we designed a robust tracking module to handle these issues, ensuring reliable supervision for the following object-specific reconstruction. And we also conduct further experiment on MipNeRF360 dataset to demonstrate the reliability of our tracking module on complex scenes with cluttered objects, please refer to the table in the above section.

$Q3$ Highly Dynamic and Occluded Scenes

The ability to handle significant object deformation primarily depends on the underlying 4D Gaussian Splatting (4DGS) method used. Our approach is compatible with any 4DGS pipeline, thus, the capacity to model complex dynamics is inherently limited by the dynamic modeling capabilities of the chosen 4DGS method.
For highly occluded scenarios, the Figurines in the LERF-OVS dataset and Kitchen in MipNeRF360 dataset can serve as representative examples. In these challenging settings, our proposed pipeline and tracking method demonstrate better performance over baseline methods.

$Q4$ Partial Mask Filtering

Sorry for the confusion. Partial mask filtering is not a correction for mask inaccuracies. The mask provided by our tracking module is accurate enough. The motivation for incorporating this strategy is that the rendered object Gaussians may reveal the full object structure, even in views where the object is heavily occluded, as it can learn from other views. A demonstration is shown in Fig. 5 of the main paper. In such cases, direct supervision using the SAM mask from the occluded view would impose an incorrect constraint, potentially distorting the learned object structure. Partial mask filtering helps avoid this by preventing such misleading supervision signals. The ablation study for partial mask filtering has already been done in main paper Tab. 2(b). In highly occluded scenes such as Ramen and Waldo-Kitchen, partial mask filtering significantly improves both segmentation and reconstruction performance.

$W4$ Discussion on Open-Vocabulary 3D Detection

Thank you for pointing this out. Open-vocabulary 3D detection and open-vocabulary 3D segmentation are indeed closely related tasks. The key distinction lies in the level of spatial precision required: 3D segmentation aims to recover accurate object boundaries, whereas 3D detection only requires coarse localization via bounding boxes. Our approach can naturally be extended to open-vocabulary 3D detection by simply computing a 3D bounding box that encloses the extracted object-specific Gaussians.