Gaussian-Det: Learning Closed-Surface Gaussians for 3D Object Detection

Q4: Verification on the quality of 3D-FRONT dataset.

A: Thank you for pointing this out. In [1], the authors mentioned that ''effort has been spent on splitting complex scenes into individual rooms and cleaning up bounding boxes. A total of 159 usable rooms are manually selected, cleaned, and rendered in 3D-FRONT. '' Moreover, extensive manual cleaning is conducted on the bounding boxes in each room. We have also checked the image data of this dataset and illustrated the example scenes in the Appendix of the revised manuscript (see Figure 8).

Q5: Results on extra real-scene dataset.

A: During the discussion phase, we have further experimented on the task of open-vocabulary instance segmentation in 3D scenes, which detects the target objects in real-world 3D datasets in a much finer granularity. On the Gaussian-Grouping baseline [2], we have implemented the estimation of surface closure as a prior on the reliability of the object instance, thus supporting or suppressing the segmented results.

From the quantitative results in Table D&E (Table 6&11 in the revised manuscript), we can see that leveraging the surface closure provides an informative prior on objectness deduction and largely enhances the performance of instance segmentation in 3D scenes. Moreover, it substantially speeds up the training period and saves the GPU memory footprint after ruling out potential outliers and cutting down the total number of 3D Gaussians. The rendered results of qualitative examples containing multiple object instances are shown in Figure 7 of the revised manuscript. It can be seen that the incorporation of the prior of surface closure contributes to fewer noisy predictions.

We are sorry that during the discussion phase, we are provided with limited time and computational resources to perform the 3D Gaussian Splatting on the large-scale dataset for autonomous driving. This will be our future work where we investigate how Gaussian-Det performs on the outdoor datasets.

Table D: Open-vocabulary 3D instance segmentation on the real-world LERF-Mask/figurines dataset

Table E: Comparisons on model efficiency in the task of open-vocabulary 3D instance segmentation.

[1] Hu, Benran, et al. "Nerf-rpn: A general framework for object detection in nerfs." CVPR. 2023.

[2] Ye, Mingqiao, et al. "Gaussian grouping: Segment and edit anything in 3d scenes." ECCV, 2024.

We truly appreciate your valuable comments. In the following, we provide responses to the concerns.

Q1: The goal of using the backbone in the framework figure.

A: The backbone aims at downsampling the large amount of (usually >100k) 3D Gaussians into a smaller and fixed number (<10k) of candidate Gaussians, while extracting their corresponding features. In the framework figure (Figure 3) of the revised manuscript, we have (1) used an isosceles trapezoid and (2) noted the number of input&output 3D Gaussians to highlight the downsampling process.

Q2: How the CIM module works in the framework figure.

A: In the framework figure (Figure 3) of the revised manuscript, we have made the following polishments to better convey how CIM works in Gaussian-Det:

• We have included the text of ''CIM'' into two subheads in the bottom to highlight that they belong to the CIM module.
• We have followed the commonly-used illustration of the residual connection by adding a shortcut in the part of ''CIM: Partial Surface Inference''.
• In the part of ''CIM: Holistic Surface Closure Coalescence'', we have incorporated illustrative examples of how CIM uses the surface closure (measured by $|\Phi|$) as the quantitative prior on the object proposals. In the illustrative examples, we have established correspondences between the framework figure (Figure 3) and Figure 2 to better motivate our design choice.

Q3: Fitting Theorem 1 into the elaboration of CIM module.

A: Thank you for your insightful suggestion. Theorem 1 mathematically defines our measurement of surface closure. We strongly agree that it is crucial to conceptually introduce such a measurement, and properly convey how the measurement is leveraged by the CIM module in Gaussian-Det. In the revised manuscript, we have re-organized the content in Section 3.3. Specifically, we have made the following polishments to better fit Theorem 1 into the elaboration of CIM module conceptually, mathematically and practically:

• Before Theorem 1: We have moved the illustrative example of the surface closure prior ahead, thus bringing up earlier the concept that the surface closure is utilized as a measurement of the quality of the predicted objectness.
• Before Theorem 1: We have highlighted that CIM leverages a key geometric property that the partial surfaces corresponding to an object can approximate a closed surface (L259-L262 in the revised manuscript), thereby naturally educing the details of Theorem 1 that mathematically describes this property.
• After Theorem 1: In the following narration, we have noted how to utilize such a property as a measurement of the objectness quality in practice.
• The proof of Theorem 1 has been moved to the Appendix for brevity.

We truly appreciate your valuable comments. In the following, we provide responses to the concerns.

Q1: Qualitative comparisons with ablated components.

A: Thank you for your valuable suggestion. In the revision, we have accordingly illustrated the qualitative comparisons of the ablation studies on different components (see Figure 11 in the Appendix of the revised manuscript). It can be seen that either discarding the holistic surface coalescence or using the deterministic partial surface representation leads to sub-optimal visual results, such as inaccurate sizes and locations.

Q2: How the underlying Gaussian representation affects the results.

A: Thank you for your insightful suggestion. To investigate the effect of underlying Gaussian representation, in the discusion phase, we have experimented on the ablation study you mentioned by detecting from 3D Gaussians that were reconstructed via the original 3DGS[1] algorithm. The quantitative results are displayed in Table C. While the detection accuracies drop with the input of lower quality 3D Gaussians, they still outperform all the compared methods in Table 1 which consume higher-quality 3D Gaussians that were reconstructed via 3DGS+SuGaR. In the revised manuscript, we have included the results in Table C and the analysis (see Appendix D.1).

Table C: Quantitative results on the Gaussian representation.

Q3 part 1: Clarifications on the writing of the proposed method

A: We would like to thank you for your suggestions on improving the overall clarity of the manuscript. In the revision, we have thoroughly gone through the elaboration of our method and made the following improvements:

• On denotations: We have checked the math denotations that you have mentioned and corrected the usages.
• On network modules: We have differentiated between the two $M$s and highlighted that they are independent but sharing the same architecture. Moreover, we have used an isosceles trapezoid to represent the backbone in the framework figure (Figure 3 in the revised manuscript), to highlight its downsampling effect.

Q3 part 2: Clarification on relationship between open and closed surfaces:

A: Ideally, a set of closed surfaces can fully enclose a volume. However, when there exists obvious discontinuity in this enclosure due to incompleteness or the introduction of outlier surfaces, the set of surfaces does not fully enclose a volume and are regarded as open. Such enclosure is quantitatively measured by the flux value in Theorem 1. The more closed a set of surfaces is, the closer the $|\Phi|$ value computed on them is to zero.

We agree that Figure 3 in our original manuscript is crucial for motivating our method and any confusion should be avoided. In the revised manuscript, we have moved it ahead as Figure 2 and clarified that the open or closed surfaces are measured by the $|\Phi|$ value. In the framework figure (Figure 3) of the revised manuscript, we have also added the necessary information that $|\Phi|$ measures how well the surfaces enclose a volume, which is used to discriminate the degree of surface closure.

[1] Kerbl, Bernhard, et al. "3D Gaussian Splatting for Real-Time Radiance Field Rendering." ACM Transactions on Graphics 42.4 (2023): 1-14.

We truly appreciate your valuable comments. In the following, we provide responses to the concerns.

Q1: Camera poses exploit a type of ground truth (point cloud) to be inferred. In my opinion, this is clearly a strong limitation of the paper. How could a noisy estimate of the cameras affect overall performance?

A: Thank you for pointing this out. We agree that since the officially-released dataset of [1] uses ground truth point cloud to infer camera poses, the effectiveness of our proposed method can be better reflected under the setting of noised camera parameters. To this end, we have accordingly conducted benchmark experiments with noised poses as input. Specifically, we manually added normally distributed perturbations with mean of 0 and variance of 0.07 to the camera extrinsics estimated by [1]. The quantitative results are shown in Table B. We can see that with noised poses, Gaussian-Det showcases less decline on both AP and AR, which verifies the effectiveness of Gaussian-Det against either inherent outlier 3D Gaussians or low-quality input poses. In the revision, we have included the quantitative comparison shown in Table B into the Experiment section and demonstrated the corresponding qualitative results in the Appendix.

Table B: Comparison under Noised Poses

Q2: Lack of challenging cases. In my opinion, some qualitative instances could be included in the paper, showing the effectiveness of the method in complex cases, especially, in comparison with other approaches. In figure 6, maybe the authors could use a different color per object.

A: Thank you for your valuable suggestion. In the original submission (Figure 9 in Appendix), we have included several qualitative instances under challenging scenarios including occlusions and jointedness. In Figure 13 of the revised manuscript, we have included more qualitative results on occlusions, where the compared methods predict inaccurate sizes or locations. Moreover, to more comprehensively demonstrate the effectiveness of the proposed method, we have also used a unique color for each bounding box in Figure 4 of the revised manuscript.

Furthermore, we have experimented on the challenging open-vocabulary 3D instance segmentation, where the incorporation of the prior of surface closure substantially enhances the segmentation performance of the Gaussian-Grouping baseline [2] (see Table 6 in the revised manuscript). The rendering results of qualitative examples containing multiple object instances are shown in Figure 7 of the revised manuscript. It can be seen that incorporating the prior of surface closure leads to fewer noise-like erroneous predictions.

[1] Hu, Benran, et al. "Nerf-rpn: A general framework for object detection in nerfs." CVPR. 2023.

[2] Ye, Mingqiao, et al. "Gaussian grouping: Segment and edit anything in 3d scenes." ECCV, 2024.

We truly appreciate for your valuable comments. In the following, we provide responses to the concerns.

Q1: Novelty of comibining existing Gaussian Splatting and PointNet++.

A: Thank you for your insightful question. We would like to clarify that our core contribution aims at tackling the drawback of simply combining Gaussian Splatting and PointNet++ (or other point-based architectures).

While our proposed method consumes the output of Gaussian Splatting and uses a PointNet++ backbone, such a simple combination followed by a detection head cannot overcome the numerous outlier Gaussians which can distract the point-based detectors, thus leading to sub-optimal results prefixed by ''G-'' in Table 1.

To address the aforementioned challenge, we seek to leverage the surface-based representation of 3D Gaussians and design the Closure Inferring Module (CIM) which (1) probabilistically models the underdetermined Gaussians for partial surfaces and (2) coalesces them into a quantitative measurement of the quality of object proposals via calculating the overall surface closure, thereby providing an informative prior on refining the sub-optimal objectness deduction. This contributes to a substantially better detection accuracy than methods that simply combine 3D Gaussians with point-based detectors.

Moreover, we have further validated the generality of such measurement on the task of challenging open-vocabulary 3D instance segmentation (shown in Table A). The incorporation of the prior on surface closure sizably enhances the segmentation of the object instances.

Table A: Open-vocabulary 3D instance segmentation on the real-world LERF-Mask/figurines dataset

Q2: Definition of the overall framework.

A: Thank you for your valuable suggestion. Apart from the overall framework in Figure 3 of our revised manuscript where we have supplemented more details, we have added more network details in Appendix B.1, including the backbone network, the querying-and-pooling function, the stacked MLP for proposal refinement.

Q3: Necessity of Theorem 1.

A: Thank you for your feedback on Theorem 1. Theorem 1 is aimed at establishing the theoretical foundation for our Closure Inferring Module (CIM). The flux value $\Phi$ in Theorem 1 quantifies how well these surfaces approximate a closed shape, thus informing the closure weight $S_k$ in Eqn. (10) which influences the holistic feature representation used for proposal refinement. In implementation, we use the quadrature form (Eqn. (9)) of Theorem 1 by computing the flux over the partial surfaces within each object proposal to assess the surface closure.

We are sorry that the previous version of narration regarding Theorem 1 causes your concern. We have carefully re-organized the related elaboration which we hope will address your concern. Moreover, for the reader's convenience, we have moved the proof of Theorem 1 to the Appendix for brevity.

Q4: Definitions of $F^{cand}$ and $\mathbf{f}^{part}_k$.

A: In the original manuscript, $F^{cand}$ is initially defined as the feature of candidate Gaussians (see L207 in the original manuscript, L217 in the revised manuscript). $\mathbf{f}^{part}_k$ is defined as the partial-aware features of each proposal (see L211-L213 in the original manuscript, L220-L223 in the revised manuscript).

Q5: Motivation and clarification of Eqn. (6)(7)(10) in the revised manuscript.

A: For Eqn. (6), in the revised manuscript, we have noted that as stated in [1] inferring local surface features is underdetermined due to the supervision from 2D signals, leading to unstructured outliers with unreliable local surface descriptors. Therefore, we model such randomness in CIM by incorporating a variational term into the candidate features, establishing a probabilistic feature residual.

For Eqn. (7), we follow [2] to optimize the evidence lower bound (ELBO) of the variational auto-encoder. In the revised manuscript, we have included the reference to [2] for clarity.

For Eqn. (10), in the revised manuscript, we have noted that $S_k$ serves as the closure weight parameter which controls the support ($|\hat{\mathbf{\Phi}}_k|\downarrow, S_k\uparrow$) or suppression ($|\hat{\mathbf{\Phi}}_k|\uparrow, S_k\downarrow$) of $\mathbf{f}^{hol}_k$. Moreover, we have added illustration on such support or suppresion in Figure 3 of the revised manuscript.

Q6: How the point cloud based methods are used in the experiment.

A: For the point cloud based methods, the original 3D Gaussians $\mathbb{G}=${$\mathbf{g}_p=(\mathbf{\mu}_p,\mathbf{\Sigma}_p,\alpha_p, \mathbf{c}_p)$}$^P_p$ (generated by 3DGS+SuGAR) were used as the input, where the Gaussian center $\mathbf{\mu}_p$ was regarded as the input point-wise coordinate. $\mathbf{\Sigma}_p,\alpha_p, \mathbf{c}_p$ were concatenated and regarded as the input point-wise features. Except for the first layer whose input dimension was modified to fit the input Gaussians, we faithfully followed the network and training configurations of the point cloud based methods we compared with. In the revised manuscript, we have added the aforementioned details to Appendix B.1.

Q7: On qualitative examples.

A: Quality of the 3D boxes: Thank you for pointing this out. In the revised manuscript, we have removed the first row with lower quality in Figure 5, and used a unique color for each bounding box for intuitive presentation.

NMS: We clarify that in the qualitative results, we have faithfully used the official implementation of FCAF3D and NeRF-RPN, both using NMS as post-processing. We also clarify that the usage of Non-Maximum Suppresion (NMS) only removes the boxes that are overlapped over a pre-set threshold. Therefore, the overlapping observed in the competing methods is due to this threshold, which may have caused your concern.

[1] Antoine, Guédon, et al. "Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and high-quality mesh rendering." CVPR, 2024.

[2] Kingma, Diederik P. "Auto-encoding variational bayes." ICLR, 2013.