OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understanding

1. More Detailed Ablation of Two-Stage Codebooks

Thank you for your constructive comments. Following your suggestions, we have incorporated two additional experiments (refer to cases #3 and #4 in the table below) to evaluate the performance when considering xyz coordinate information at the coarse level alone. Our analysis yielded the following insights:

• Importance of xyz Information: Comparing cases #1 and #3, we observe a significant improvement in mIoU (3.36%) and mAcc (2.55%) when incorporating xyz information at the coarse level. This highlights the crucial role of spatial context in our approach.
• Codebook Size: Comparing cases #2 and #4, we find that simply increasing the number of codewords (k) does not necessarily lead to performance gains.
• Fine-Level Codebook Contribution: Comparing cases #3 and #6, we see a substantial performance boost of 6.25% in mIoU and 5.37% in mAcc when incorporating the fine-level codebook. This demonstrates the significant contribution of the fine-level codebook in capturing detailed semantic information and enhancing the overall performance.

The additional experiments reinforce our claim that incorporating xyz coordinate information at the coarse level is essential and that the fine-level codebook plays a critical role in boosting performance. We appreciate your suggestions, which have significantly strengthened the persuasiveness of our method.

---

2. Sensitivity Analysis of k1 and k2

Thank you for your question. In the ScanNet dataset, our default settings are k1=64 and k2=5. To address your concerns, we conducted a comprehensive analysis with k1 values of 48, 64, and 80, and k2 values of 3, 5, and 7, as illustrated in the table below. Our findings reveal that the optimal values for k1 and k2 should not be too high. Specifically, performance remains comparable when k1 is set to 64 or 48, and when k2 is set to 5 or 3. However, a performance degradation is observed when k1 and k2 are increased to 80 and 7.

We sincerely appreciate your insightful suggestions, which led to these valuable observations. While our initially chosen values are empirically derived, these experiments highlight that setting fixed values may not always be optimal. In the future, we plan to investigate scene-adaptive k values, which could potentially enhance the performance across diverse scenarios.

We commit to incorporating the aforementioned experiments and this limitations analysis in the revised version

1. Ablation of Inter/Intra Mask Loss

We truly appreciate your constructive feedback and apologize for any overlook regarding the ablation of the inter/intra mask loss. In response, we have conducted targeted ablation experiments, and the results are presented in the table below. Our analysis is summarized as follows:

• The inter-mask contrastive loss proves to be more crucial. Employing only this loss achieves respectable performance. Adding the intra-mask loss further enhances results, leading to a 3.05% improvement in mIoU and a 2.76% increase in mAcc.
• The intra-mask smooth loss exhibits comparatively lower importance: This can be attributed to the inherent characteristics of 3DGS, where a single Gaussian point represents multiple pixels. Consequently, features of neighboring pixels tend to be similar, indicating that 3DGS naturally induces a smoothing effect on adjacent pixels. This intrinsic smoothing mechanism partially mitigates the contribution of the intra-mask smooth loss.

These additional experiments substantiate the contribution of our proposed inter/intra mask loss. We commit to incorporating these results and analysis in the revised version.

---

2. Efficiency and Real-Time Performance Analysis

We appreciate the reviewer’s question. Our experimental setup aligns with LangSplat and LEGaussian, focusing on semantic understanding of well-reconstructed 3D scenes, which is not a real-time task. However, we believe our method has potential for real-time applications for the following reasons.

• Incremental Input Support: Unlike LangSplat[30] and LEGaussian[34], which necessitate acquiring all scene objects prior to training the autoencoder-decoder or features distillation, our method allows for the incremental input of new images. Each new frame can be processed immediately without prior preprocessing. This aligns well with the incremental needs of real-time tasks such as SLAM and robotics.
• Training Efficiency: Our statistics on the ScanNet indoor dataset show that 3D point feature learning on a 640x480 image takes approximately 50ms per iteration for a scene with around 100,000 points, achieving a 20fps frame rate. On the Waymo outdoor dataset, processing a 960x640 image for a scene with approximately 700,000 points requires about 80ms per iteration, resulting in a 13fps frame rate. Furthermore, techniques like keyframes, sliding windows, and multithreading can be employed to further enhance processing speed in applications such as SLAM.

The analysis suggests our method possesses the potential for real-time implementation. We also hope to inspire the community to utilize the proposed method across various downstream tasks by releasing our code publicly.

---

3. Discussion of Limitations

We apologize for not analyzing the limitations of the proposed method. In the revised version, we will incorporate the following discussion:

• The geometric properties of the Gaussian (position, opacity, scale) are fixed. This may lead to inconsistencies between geometric representation and semantic content. We will consider joint optimization of instance features and geometric properties in future work.
• The values of k for the two-level codebooks are currently determined empirically. It is necessary to study scenario-specific adaptive values to optimize performance across diverse contexts.
• Currently, we have not considered dynamic factors, which are common challenges in real-world applications. Integrating the proposed method with 4DGS would be meaningful.

By acknowledging these limitations, we aim to provide a more balanced perspective on our method and suggest areas for future improvements.

---

4. Improvements in Paper Writing

We appreciate the reviewer pointing out the shortcomings in our writing. Indeed, our contribution 1 (inter/intra-mask loss for instance feature learning) and contribution 2 (two-level codebooks for discrete feature learning) are closely linked, with contribution 1 essentially serving as the initialization for contribution 2. As for contribution 3, we do realize that this part is somewhat independent, leading to difficulty in understanding. We apologize for any confusion caused. In the revised version, we will enhance the readability of the paper in the following ways:

• Reorganizing the introduction to better connect the three methods we proposed.
• Adding contextual transitions between sections.

---

5. Other Question

Thank you for pointing out the typo in Eq(2). We will correct this in the revised version.

Dear Reviewer sSBZ,

Thank you once again for your insightful review, which has greatly enhanced the quality and clarity of our paper. We sincerely hope that our rebuttal has effectively addressed your questions and concerns. Should you require any additional clarifications or further information, please do not hesitate to reach out. We greatly value your insightful suggestions.

Thank you very much for your time and consideration.

Best regards,

Authors of Submission 1591

1. Scalability and Generalization

We appreciate the reviewer’s insightful question, which encouraged us to explore the scalability and generalizability of OpenGaussian in other 3D datasets and scenarios.

• We selected 6 scenes from the Waymo outdoor dataset captured by vehicle-mounted cameras to demonstrate the effectiveness of the proposed method in large-scale complex scenarios.
• We captured images of a real-world office scene using a mobile phone to demonstrate the method’s generalization.

Please refer to the attached PDF in the General Rebuttal section on OpenReview for detailed results. We kindly request the reviewer to check the file:

• Fig.1, Fig.2: Visualization of 3D point features in 6 scenes from the Waymo dataset and the real-world office scene;
• Fig.3: Results of rendering 3D point features into 2D feature maps;
• Fig.4: Comparison of 3D point features after coarse-level and fine-level codebook discretization.

We hope the added results can address your concerns about scalability and generalizability.

---

2. More Detailed Ablation

Thank you for your question. Due to the interdependent nature of the three proposed methods, we cannot fully isolate them for ablation; the mask-based instance feature learning is essential for the two-level codebook discretization, and the discretized features are necessary for the 2D-3D association. However, we conducted a detailed ablation on their subcomponents, as shown in the table below. Cases #2, #3, and #4 were added during the rebuttal period.

• Cases #2(w/o intra-mask) & #3(w/o inter-mask): Ablation of inter-mask and intra-mask losses, highlighting the importance of inter-mask loss.
• Cases #1(two-level) & #4(coarse-level): Ablation of two-level codebooks, emphasizing the significance of fine-grained codebooks.
• Cases #5(w/o feat. dis.) & #6(w/o IoU): Testing the 2D-3D feature association strategy, demonstrating the superiority of combined strategies. | Case|Inter/Intra-mask loss || Two-level codebook || 2D-3D association || mIoU | mAcc. | |:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:| ||Inter|Intra|Coarse|Fine|IoU|Feat.Dis.||| |#1|✓|✓|✓|✓|✓|✓|38.29|55.19| |#2|✓||✓|✓|✓|✓|35.24|52.43| |#3||✓|✓|✓|✓|✓|25.89|42.76| |#4|✓|✓|✓||✓|✓|32.04|49.82| |#5|✓|✓|✓|✓|✓||35.28|53.19| |#6|✓|✓|✓|✓||✓|34.01|51.35|

---

3. Efficiency and Real-Time Performance Analysis

We appreciate the reviewer’s question. Our experimental setup aligns with LangSplat[30] and LEGaussians[34], focusing on semantic understanding of well-reconstructed 3D scenes, which is not a real-time task. However, we believe our method has the potential for real-time applications for the following reasons.

• Incremental Input Support: Unlike LangSplat and LEGaussian, which necessitate acquiring all scene objects prior to training the autoencoder-decoder or features distillation, our method allows for the incremental input of new images. Each new frame can be processed immediately without preprocessing. This aligns well with the incremental needs of real-time tasks such as SLAM and robotics tasks.
• Training Efficiency: Our statistics on the ScanNet indoor dataset show that 3D point feature learning on a 640x480 image takes approximately 50ms per iteration for a scene with around 100,000 points, achieving a 20fps frame rate. On the Waymo outdoor dataset, processing a 960x640 image for a scene with approximately 700,000 points requires about 80ms per iteration, resulting in a 13fps frame rate. Furthermore, techniques like keyframes, sliding windows, and multithreading can be employed to further enhance processing speed in applications such as SLAM.

The analysis suggests our method possesses the potential for real-time implementation. We also hope to inspire the community to utilize the proposed method across various downstream tasks by releasing our code publicly.

---

4. Comparison with 3D Open Vocabulary Detection Methods

Thank you for raising this interesting point. In fact, these two types of methods differ significantly in their setups.

The two papers mentioned by the reviewer require training a 3D Encoder-Decoder Backbone and a task-specific head, meaning they are data-driven and need to be trained on large-scale point cloud datasets. In contrast, our method follows the 3DGS setup, involving no networks (MLP, convolution, or Transformer) and only training point features. Besides, our method is scene-specific and not data-driven, thus avoiding domain gaps. The differences in model, training approach, and data make fair comparison challenging.

Following your suggestion, we will include a discussion on these data-driven 3D open vocabulary understanding methods in the revised version.

---

5. Discussion of Limitations

We apologize for not analyzing the limitations of the proposed method. In the revised version, we will incorporate the following discussions:

• The geometric properties of the Gaussian (position, opacity, scale) are fixed. This may lead to inconsistencies between geometric representation and semantic content. We will consider joint optimization of instance features and geometric properties in future work.
• The values of k for the two-level codebooks are currently determined empirically. It is necessary to study scenario-specific adaptive values to optimize performance across diverse contexts.
• Currently, we have not considered dynamic factors, which are common challenges in real-world applications. Integrating the proposed method with 4DGS would be meaningful.

By acknowledging these limitations, we aim to provide a more balanced perspective on our method and suggest areas for future improvements.

---

6. Implementation Complexity

We apologize for any confusion caused. We commit to releasing the code for reproducibility and to contribute to the community. To enhance the clarity of the paper, we will provide more detailed implementation details in the revised version and simplify any non-essential components that may cause confusion.

Dear Reviewer ANGY,

Thank you once again for your insightful review, which has greatly enhanced the quality and clarity of our paper. We sincerely hope that our rebuttal has effectively addressed your questions and concerns. Should you require any additional clarifications or further information, please do not hesitate to reach out. We greatly value your insightful suggestions.

Thank you very much for your time and consideration.

Best regards,

Authors of Submission 1591