EmbodiedSAM: Online Segment Any 3D Thing in Real Time

Thank you for your valuable comments and kind words to our work. Below we address specific questions.

Q1: About the novelty of Geo-aware Pooling

We agree with the reviewer that PointNet-like architecture of Geo-aware Pooling is not new. But we think this is a new technique for superpoint pooling. Previous method like SPFormer and Oneformer3D directly use average pooling condense point features to superpoints without considering the shape of each superpoint. We believe this operation is sub-optimal and propose an efficient module to predict the weights of each point for better pooling. We think weighted average is a better way to obtain superpoint features, which may be helpful for future works.

Q2: Time cost of each module

We have reported the time cost of each module in Appendix A.3. The 3D part of EmbodiedSAM takes 80ms per frame, while the 2D part (VFM for superpoint building) takes 20ms per frame.

Q3: More detailed explanations on results in Table 6

Thanks for the kind suggestion. We notice that removing the box representation results in the most significant performance degradation. This is because most cases of merging can be decided based on geometry. In fact, if two 3D masks are very close to each other, or the whole geometry of them are likely to be overlapped, they are likely to belong to the same object. We also observe that the influence of removing contrastive representation is greater than removing semantic representation. This is because contrastive similarity is designed for instance merging, while semantic similarity is originally designed for mask classification.

Q4: Ablation on a single dataset

Since query lifting and query decoder are not related to inter-frame merging, we conduct ablation on them on ScanNet-25k dataset in Table 5. The performance is evaluated on each single RGB-D image, rather than a whole RGB-D sequence. Following your advice, we further conduct ablation studies of query lifting and query decoder on ScanNet sequences, as shown below:

It is shown that these two methods can significantly improve the performance of EmbodiedSAM with negligible computational overhead.

If you have any further question, please let us know. Thank you very much!

Thank you for your valuable comments and kind words to our work. Below we address specific questions.

Q1: Visualization on other datasets

We further visualize different methods on SceneNN and 3RScan, as shown in Appendix A.6. Consistent with Figure 4, ESAM predicts accurate and clean 3D masks. On the contrary, the prediction of SAM3D is noisy or incomplete while the prediction of SAI3D tends to be over-segmented.

Q2: Experiments on real-world captures

EmbodiedSAM requires posed RGB-D stream as input, where RGB-D images can be captured by a RGB-D camera and camera pose can be online estimated by SLAM or visual odometry system. Therefore, to conduct real-world experiments, we should first build an online 3D reconstruction system and then apply EmbodiedSAM simultaneously with camera tracking. Since it will take us some time to buy accurate RGB-D camera, set up the whole hardware and software system, calibrate the camera and select proper hyperparamters to filter noisy depth, we may not be able to finish these tasks before the rebuttal due.

Nevertheless, since the RGB-D sequences used in our paper are all captured with real-world sensors, we can evaluate the real-world performance of EmbodiedSAM by adding noise to the process of camera tracking. We add different scales of noise (random noise, with a maximum ratio) on the camera extrinsic parameters and run EmbodiedSAM on ScanNet200-class-agnostic following the setting of Table 1. The results are shown below:

It is shown that our method is robust to the noise on camera pose, which is a huge advantage when applied in real-world applications.

Q3: Why $z^{global}$ is added instead of concatenated to the pooled point features

We empirically find adding the features can achieve slightly better performance than concatenating them. Moreover, adding is also more computationally efficient than concatenating since it does not require a linear layer for channel projection.

Q4: Why using the original point features instead of $z^{local}$

The nature of geometric-aware pooling is a weighted average of point features inside each superpoint. $z^{local}$ is only used to predict the weights of each point for superpoint pooling. It is not a better geometric representation than the original point features. We empirically find that the overall performance decreases by about 0.7% if we use $z^{local}$ instead of the original point features.

Q5: Any class-aware experiments

We need to clarify that the experiments in Table 3 are class-aware. There are 18 classes in ScanNet and 15 classes in SceneNN. We report the per-category performance of TD3D-MA and our method on ScanNet as below:

AP25:

AP50:

If you have any further question, please let us know. Thank you very much!

Thank you for your valuable comments and kind words to our work. Below we address specific questions.

Q1: An ablation of the geometric-aware query lifting and dual-level query decoder

Since query lifting and query decoder are not related to inter-frame merging, we conduct ablation on them on ScanNet-25k dataset in Table 5. The performance is evaluated on each single RGB-D image, rather than a whole RGB-D sequence. Following your advice, we further conduct ablation studies of query lifting and query decoder on ScanNet sequences, as shown below:

It is shown that these two methods can significantly improve the performance of EmbodiedSAM with negligible computational overhead.

Q2: Ablation study on the weights of the different terms in Eq.(7). Adding or multiplying.

An ablation study on the weights for geometric, contrastive and semantic similarity is shown below (following the setting of Table 3):

We empirically find adding these terms is more robust than multiplying them. The latter is more sensitive to the hyperparameters and may get worse performance when we apply EmbodiedSAM to new environments.

If you have any further question, please let us know. Thank you very much!

Dear reviewer:

Since the discussion stage is nearing its end, we would appreciate your feedback and are happy to address any concerns you may have.

Thank you for your valuable comments and kind words to our work. Below we address specific questions.

Q1: How to acquire accurate and stable intrinsic and extrinsic parameters in embodied environments

The input to EmbodiedSAM is streaming RGB-D video, so an online camera pose tracking method should be applied to obtain extrinsic parameters. Many SLAM methods can solve this problem, such as DROID-SLAM [1], DPVO [2] and SplaTAM [3]. The intrinsic parameters can be obtained by camera calibration. After calibration, we treat them as a constant matrix, as done in our experiments.

Q2: Robustness to noise

We add different scales of noise (random noise, with a maximum ratio) on the camera extrinsic parameters and run EmbodiedSAM on ScanNet200-class-agnostic following the setting of Table 1. The results are shown below:

It is shown that our method is robust to the noise on camera pose. As for intrinsic parameters, since the camera calibration technology is very mature, the error is negligible. All our experiments on the same dataset use a constant matrix obtained by camera calibration as the camera intrinsic.

Q3: How to align point clouds

Since the camera extrinsic parameters can be estimated as described in Q1, we do not need to align the point clouds of different frames. We can use camera intrinsic and extrinsic parameters to convert depth image into point clouds in world coordinate system and directly concatenate them.

Q4: Visualization of weights in Geometric-aware Pooling

We have provided the visualization of point-wise weights predicted in Geometric-aware Pooling in Appendix A.5. It is shown that noisy boundary with inaccurate shape information will be assigned low weight, while regions contain objects (especially small objects) are assigned high weight for better segmentation.

Q5: Comparison with Zero-shot Methods

We appreciate your constructive suggestion. We agree that EmbodiedSAM uses semantic and instance labels for supervision while some baselines do not. However, the dataset transferring experiments in Table 2 validate the generalization ability of EmbodiedSAM. Following your advice, we further implement TD3D-MA and compare with it following the setting of Table 1, as shown below:

It is shown that EmbodiedSAM significantly outperforms fully-supervised online 3D instance segmentation method. This is because our VFM-assisted prediction pipeline can better segment small objects, which is a bottleneck for other fully-supervised online 3D instance segmentation methods.

[1] DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras. NeurIPS 2021.

[2] Deep Patch Visual Odometry. NeurIPS 2023.

[3] SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM. CVPR 2024.

If you have any further question, please let us know. Thank you very much!

Thank you for your valuable comments and kind words to our work. Below we address specific questions.

Q1: How to obtain category? Why Table 1 for class-agnostic 3D instance segmentation

We need to clarify that our method is a segment anything model, whose output is class-agnostic 3D masks. We follow the same setting with other segment anything models, like SAM3D and SAI3D. But our method is also able to predict category for each mask, which can be easily done by adding a MLP upon the 3D queries of each mask to predict the corresponding category. The class-aware instance segmentation results (reported in Table 3) are just predicted in this way.

Q2: Revision on Figure 4

We appreciate your constructive suggestion. However, since the prediction of 3D instance is not well-ordered, the color for each mask cannot be easily set to the same as the ground-truth. Other segment anything methods like SAM3D and SAI3D also suffer from this problem. We will try to design a more suitable visualization paradigm for 3D instance segmentation for easier comparison between prediction and ground-truth. Due to time constraints, we aim to revise our figure in the final version of this paper.

Q3: Compare with OpenMask3D and OpenIns3D

OpenMask3D and OpenIns3D are two representative open-vocabulary 3D instance segmentation methods, which focus on how to assign open-vocabulary labels on each 3D mask. These two methods adopt off-the-shelf 3D segmentation model and study how to classify each mask. Therefore, these two methods are classification approach, rather than segmentation approach.

However, it is very valuable to combine these two kinds of approaches. We have conducted experiments to adopt EmbodiedSAM for 3D mask generation and adopt OpenMask3D for mask classification. In this way, we can achieve open-vocabulary and online 3D instance segmentation. We compare the open-vocabulary results with SAI3D as below:

We have also cited and discussed with these two papers in Appendix A.4.

If you have any further question, please let us know. Thank you very much!

Dear reviewer:

Since the discussion stage is nearing its end, we would appreciate your feedback and are happy to address any concerns you may have.

Thank you for your response. I trust that the authors will revise Figure 4, and I am willing to accept this work. Additionally, I appreciate your efforts in producing such excellent work.

Thank you for your valuable comments and kind words to our work. Below we address specific questions.

Q1: An overview of the proposed method

We thanks for your suggestion on the introduction of our approach. The main idea of our architecture is to lift the 2D segmentation results generated by SAM to 3D queries for fast and robust merging. Previous methods like SAM3D directly project 2D mask to 3D point clouds, followed by mask-wise merging operation to unify masks of the same instance across frames. This merging operation is very slow and not robust due to the handcrafted criterion. On the contrary, our EmbodiedSAM adopts a learnable projection to convert 2D mask to 3D queries, which converts the mask merging problem into query merging. In this way, we can utilize matrix operation to compute the similarity between masks and merge them in parallel.

Specifically, 3D-aware is achieved by our 3D backbone and geometric-aware pooling. The role of query is to represent 3D mask in a fixed-length vector for efficient mask merging. The iterative refinement is designed to extract better representation for each 3D mask. The dual-level decoder is designed to balance the computational cost and granularity of mask generation, as described in Line 223-Line 233.

Q2: Discussion with Gaussian Grouping and LangSplat

Gaussian Grouping and LangSplat are two representative offline segmentation methods for 3D gaussian. They focus on how to accurately segment the 3D scene represented by 3D gaussian. This is a different setting from EmbodiedSAM, since we focus on how to online segment anything for streaming RGB-D video. We have cited these two papers and discussed with them in our related work.

If you have any further question, please let us know. Thank you very much!

Dear reviewer:

Since the discussion stage is nearing its end, we would appreciate your feedback and are happy to address any concerns you may have.