View-Independent 3D Feature Distillation with Object-Centric Priors

We thank the reviewer for their suggestions and feedback. We reply to individual comments below with references to new uploaded material in parts.

A broader set of baselines should be included in experiments, such as LERF [1], D3Field [2], and Semantic Gaussians [3]. The results from real-world scenes (Fig. 7) are not particularly impressive; comparisons with recent methods, such as Semantic Gaussians [3] and GaussianGrasper [6], would strengthen the evaluation

We agree that quantitative comparisons with such works in multi-view experiments would benefit the paper in terms of demonstrating the practical advantages of DROP-CLIP vs. Structure-from-Motion (SfM) works such as the ones mentioned by the reviewer. We refer the reader to Appendix A.6.3 of our updated submission for a detailed discussion and comparison with such works.

The MV-TOD dataset, while useful, is synthetic and introduces a sim-to-real gap when using modified CLIP/DINO for dense feature extraction, which limits its impact.

We agree that distilling from a synthetic dataset introduces a sim-to-real domain gap that potentially influences the impact of the approach. In order to explore such an effect, we conducted the zero-shot generalization experiments in the real-world dataset OCID (Sec 4.3). As we report in our paper, our method still provides improvements over baselines trained on real-data, even though distilled from our synthetic MV-TOD.

Further, we included a supplementary video (iclr_drop_reb_supp.mp4) where we demonstate real-time performance of DROP in real-world scenes vs. a vanilla DFF baseline based on back-projecting MaskCLIP features.

For open-vocabulary grasping, comparing the proposed 3D representation to baselines like F3RM [4], LERF-TOGO [5], and GaussianGrasper [6] would help validate its advantages.

Since our approach focuses on CLIP feature distillation and not grasping, we mainly consider segmentation tasks and included our robot grasping application of Sec.4.4 as a demonstration of DROP-CLIP’s efficiency advantages over NeRF/GSplat based works (such as the ones mentioned by the reviewer), and not advantages in terms of grasping success rate

In particular, as we extensively motivate in our Introduction and Related work sections, compared to Structure-from-Motion (SfM) works that rely on training NeRFs / GSplats [1-5], our method:

• Does not need to collect multiple camera images, but works from single-view RGB-D. Works such as F3RM [4] make the robot grab a selfie stick and move it around in order to collect images before being able to make a representation. Other works require setups with multiple cameras fixed around the scene. DROP-CLIP assumes a single camera view mounted on the robot.
• NeRF / GSplat works require multiple minutes to fit the representation in the specific scene. DROP-CLIP requires 0 minutes since it has already been distilled from MV-TOD, and does not require training.
• NeRF works require significant time to do inference. DROP-CLIP inference is just a forward pass through the distilled encoder, so it is real-time. GSplat works with efficient implementations also provides close to real-time inference, although whether it can be real-time for feature rendering (and not just RGB rendering) is unclear.
• NeRF-based works are training representations that are scene-specific, and thus changing the scene (e.g. due to robot action) would require re-training the NeRF from scratch. GSplat works partially address this through deformable Gaussians, although to the best of our knowledge, deformable feature splatting has not been demonstrated in literature. DROP-CLIP works real-time, so it updates as the scene changes and doesn’t require special treatment.

We hope the above clarifies the practical advantages of our method compared to the aforementioned works. We also refer the reader to our included supplementary video (iclr_drop_reb_supp.mp4), where we demonstrate the real-time performance of DROP in more real-world scenes.

The novelty of the method may fall short for an ICLR submission, as it relies on projecting 2D CLIP or DINO features into 3D points via simple averaging, which is not a new approach

We believe that the above comment under-represents the contributions of our work. As we extensively motivate in our Introduction and Related work sections, unlike previous works, our work does not rely on simple averaging. Instead:

• it proposes a semantic informativeness metric to weight the contributions of views based on text annotations, and
• exploits segmentation masks to do object-wise instead of point-wise fusion.

See Tab.1 for quantitative results regarding the contributions of our proposed approach compared to simple averaging.

Further, the reviewer’s acknowledged novelty refers to merely the first stage of our work, as the object-centric fusion strategy is a method for improving the quality of the 3D features to-be-distilled. The second stage is actually distilling the target features with a point-cloud encoder such that during test-time, you can without training obtain object-centric 3d features from single-view RGB-D, with real-time performance and without needing segmentors to extract masks. To get there, we need a multi-view indoor cluttered dataset with broad object catalog, which is currently missing. In our work, we generate such resource and release it (MV-TOD). We hope the above clarify the contributions of our work and would appreciate suggestions for make it more clear in our main paper.

[1] Kerr, Justin et al. “LERF: Language Embedded Radiance Fields.” 2023 IEEE/CVF International Conference on Computer Vision (ICCV) (2023): 19672-19682.

[2] Qin, Minghan, et al. "Langsplat: 3d language gaussian splatting." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[3] Guo, Jun et al. “Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian Splatting.” ArXiv abs/2403.15624 (2024): n. pag.

[4] Shen, Bokui (William) et al. “Distilled Feature Fields Enable Few-Shot Language-Guided Manipulation.” Conference on Robot Learning (2023).

[5] Qiu, Ri-Zhao et al. “Feature Splatting: Language-Driven Physics-Based Scene Synthesis and Editing.” ArXiv abs/2404.01223 (2024): n. pag.

It is beneficial to provide some comparisons with more baselines like sparsedff[1] or d3field[2].

We thank the reviewer for bringing up such works, which are conceptually close to our paper, since they do not rely on NeRFs / GSplats to do feature distillation, but use depth to project the 2D feature and then improve consistency with point-pruning (SparseDFF). Such works maintain the real-time requirement that our work also strives for, although still having several differences. In particular:

• Neither SparseDFF nor D3Field use CLIP features and evaluate for semantic / referring segmentation. SparseDFF uses DINOv2 features to obtain pose descriptors for imitation learning. D3Fields directly distills SAM masks, so there are no CLIP feature to do similarity computation with text queries, but just instance segmentation.
• Both works rely on sparse reconstruction, i.e. needing 4 views to work, compared to 1 of DROP.

Adjusting these works to our setup (MaskCLIP features instead of SAM / DINO for referring segmentation tasks and single-view reconstruction), then they become equivalent to our MaskCLIP->3D baseline used in our paper, albeit the point-pruning step (introduced in SparseDFF).

We include a new supplementary video where we illustrate advantages of DROP vs. vanilla DFFs in terms of feature consistency and grounding performance, incl. real-world scenes (see iclr_drop_reb_supp.mp4).

The symbols in the paper are poorly represented. I suggest the author to simplify these symbols and remove some meaningless ones.

We used standard mathematical notation used in literature (e.g. OpenScene [6], OpenMask3D [7]). We would appreciate some further feedback regarding which symbols are poorly represented / meaningless, so we can improve our methodology section.

Some of the fonts in the tables and figures are too small to recognize, like Tab.1, Fig.2, Fig.3.

In our new submission, we increased the font of text in Fig.2 and Fig.3 and increased the overall size of Tab.1 to address the reviewer’s point. We kindly ask the reviewer to let us know whether the fonts are comprehensive in this draft (can also use zoom in the pdf viewer).

[1] Kerr, Justin et al. “LERF: Language Embedded Radiance Fields.” 2023 IEEE/CVF International Conference on Computer Vision (ICCV) (2023): 19672-19682.

[2] Qin, Minghan, et al. "Langsplat: 3d language gaussian splatting." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[3] Guo, Jun et al. “Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian Splatting.” ArXiv abs/2403.15624 (2024): n. pag.

[4] Shen, Bokui (William) et al. “Distilled Feature Fields Enable Few-Shot Language-Guided Manipulation.” Conference on Robot Learning (2023).

[5] Qiu, Ri-Zhao et al. “Feature Splatting: Language-Driven Physics-Based Scene Synthesis and Editing.” ArXiv abs/2404.01223 (2024): n. pag.

[6] Peng, Songyou et al. “OpenScene: 3D Scene Understanding with Open Vocabularies.” 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022): 815-824.

[7] Takmaz, Ayca et al. “OpenMask3D: Open-Vocabulary 3D Instance Segmentation.” ArXiv abs/2306.13631 (2023): n. pag.

We thank the reviewer for their suggestions and feedback. We reply to individual comments below with references to new uploaded material in parts.

Summary: This paper proposes a multi-view feature fusion strategy that employs object-centric priors to eliminate uninformative views based on semantic information, and fuse features at object-level via instance segmentation masks.

We believe that the reviewer’s summary significantly under-represents our paper’s contributions. The quoted sentence describes only our contribution regarding the multi-view feature fusion strategy (i.e. object-centric priors), which was done to improve the quality of target features to-be-distilled, but fails to acknowledge the contributions of the distillation itself. As we highlight throughout our Introduction and Conclusion sections, our work provides:

• MV-TOD, a synthetic dataset with dense multi-view images, cluttered tabletop scenes from 3.3k objects and mask / grasp / semantic annotations. As we motivate in Sec.2, there is no other dataset in literature that provides all such utilities in tandem. The dense multi-view asset is essential for 2D->3D feature distillation frameworks in environments other than rooms with furniture (e.g. ScanRefer, ReferIt-3D – see Sec.2) and was previously missing.
• After we extract 3D features with our object-centric priors (as the reviewer summarizes), we use our dataset to distill them with a point-cloud encoder that works from single-view, without needing test-time segmentors and offering real-time performance (i.e. DROP-CLIP).

My main concern is the novelty of the proposed method. As far as the reviewer knows, the authors just replace the pixel-level features by object-level features through cropping the images. It seems that the contributions are mostly on the dataset and downstream applications, which is not enough for this venue

Similar to above, the reviewer just mentions 1 / 3 techniques used during generating the MV-TOD distillation targets as the sole contribution of our work (object-level 2D features through cropping). Next to that, we also: a) replace vanilla average pooling of previous works with weighted average pooling that considers semantic informativeness metric to rank view contributions and eliminate uninformative ones, and b) fuse the 2D features object-wise instead of point-wise. We refer the reviewer to Tab.1 to inspect quantitative advantages of the above contributions.

Further, the reviewer’s acknowledged novelty refers to merely the first stage of our work, as the object-centric fusion strategy is a method for improving the quality of target features during train-time. The second stage is distilling the target features with a point-cloud encoder such that during test-time, you can obtain without training object-centric 3d features from single-view RGB-D, without segmentors and with real-time performance.

Compare that to online feature distillation works that rely on Structure-from-Motion (SfM) datasets, obtained with NeRF / GSplats [1-5]. These works:

• Need multiple camera images to train the NeRF / GSplat (e.g. F3RM [4] makes the robot grab a selfie stick and move it around the environment to collect images).
• Require multiple minutes to do training (in order to fit the NeRF / GSplat to the specific scene) and considerable time for inference.
• NeRF-based methods are not adaptable to scene changes. Once the scene changes (e.g. due to the robot performing an action), you would have to retrain a new field. This limitation is partially addressed with deformable GSplat works, although inference time for feature splatting is unclear whether it can be real-time.
• Need external segmentors (e.g. SAM) in order to obtain the masks that will improve the quality of features with object-centric priors (LangSplat [2], F-Splatting [5], SemanticGaussians [3]).

In order to make the above points more compact and also add qualitative comparison, we included a new chapter in our Appendix (A.6.3) where we extensively discuss and compare DROP with SfM approaches.

The dataset was constructed in a simulated environment (Blender). Its impact on real-world scenarios is unknown. I suggest the authors provide clarification or experiments to demonstrate its generalization ability in real-world scenarios

We kindly ask the reviewer to read Sec.4.3 of our paper. In general, we agree that evaluation on real-world images is essential, and that is why we conducted the zero-shot generalization experiments in the OCID dataset, which contains real-world cluttered scenes, unseen object instances and unseen vocabulary (see Sec.4.3 and App.A.6.1). As you can observe from the results of these sections (Tab.4, Tab.11), our work improves zero-shot semantic / referring segmentation performance compared to other zero-shot single-view approaches. Check also Appendix A.6.3 of our updated submission for further comparisons in the LERF real-world dataset.

We thank the reviewer for their suggestions and feedback. We reply to individual comments below with references to new uploaded material in parts.

Unclear performance in real-world data Unfortunately the OCID dataset, for which we conducted real-world evaluations in Tab.4, provides only single-view images, so there is no means of comparison with approaches that require multiple views. That is why we only conducted comparisons with 2D CLIP-based approaches LSeg, OpenSeg and MaskCLIP, which are the de-facto choices for 2D zero-shot semantic segmentation.

However, in order to address the reviewer’s concern, we also conducted comparative experiments with modern Structure-from-Motion (SfM) methods LERF [1], LangSplat [2] and SemanticGaussians [3] in the localization and semantic segmentation tasks in a scene of the LERF dataset. We refer the reader to the A.6.3 chapter of our Appendix in the updated submission.

Unreported efficiency We mention throughout the paper (lines 67, 95, 299, 520) that DROP-CLIP achieves real-time performance, since inference is a simple forward pass from the distilled encoder. In particular, we can achieve ~25 fps on a RTX-3090 GPU for obtaining the feature-clouds illustrated in our paper (e.g. Fig.8, Fig.18), without considering PCA visualizations.

Further, we include more qualitative results, where we wish to show the real-time efficiency and view-independence of CLIP vs. a vanilla DFF baseline that relies on MaskCLIP feature back-projection (see included supplementary video iclr_drop_reb_supp.mp4).

Unclear pipeline The segmentation masks are provided as part of the training dataset MV-TOD, which generates them automatically from the Blender engine. We highlight however that segmentation masks are needed only at train-time, as a means to improve the quality of the fused 3D features that will be distilled. Masks are not needed at test-time. As we mention in Sec.3,3: "Importantly,no labels,prompts,or segmentation masks are needed at test-time to reproduce the fused feature-cloud [...]~

So even in real-data, you don’t need segmentation masks. DROP-CLIP has learned to reproduce features that were constructed with the aid of segmentation masks, but doesn’t need segmentation masks itself. We include this point in the updated submission (line 241) to address the reviewer’s point.

Justification of the dataset contribution The main limitation of previous room scan data, as we motivate in Sec.2, is that:

• They only contain category-level annotations from a small fixed set (e.g. 51k expressions for 18 unique objects in ScanRefer), while MV-TOD has open-vocabulary language annotations generated with VLMs (671k expressions for 149 categories).
• They do not contain cluttered tabletop scenarios, but mostly furniture objects in room layouts. MV-TOD contains cluttered tabletop scenes from over 3300 unique object CAD models. We refer the reader to Tab.1 for a more comprehensive comparison with existing datasets.

Is it possible to do joint training on both MV-TOD and some other datasets to show possible improvement? If not, why?

The focus of this work is on robotic manipulation applications, where we mostly care about household-like objects that could appear in tabletops. Previous datasets such as ScanRefer or ReferIt-3D focus on room layouts, so objects are mostly furniture (chairs, sofas, counters etc.). It is considerable that one could filter out scenes from such datasets that feature tabletops and clutter in order to complement the MV-TOD dataset. However, just from a pure numbers perspective, we believe the contribution of such datasets would be minimal considering the extremely narrow distribution of household objects that appear.

Discussion of related work. We thank the reviewer for providing additional references. We already had a comparison with LangSplat [2] in our extended related work section (App.A-8), which we re-write in our updated submission, in order to include more related works, including the ones mentioned by the reviewer.

[1] Kerr, Justin et al. “LERF: Language Embedded Radiance Fields.” 2023 IEEE/CVF International Conference on Computer Vision (ICCV) (2023): 19672-19682.

[2] Qin, Minghan, et al. "Langsplat: 3d language gaussian splatting." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[3] Guo, Jun et al. “Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian Splatting.” ArXiv abs/2403.15624 (2024): n. pag.

We thank the reviewer for their suggestions and feedback. We reply to individual comments below with references to new uploaded material in parts.

My primary concern lies [...] paper's key contributions in this context.

We have tried to extensively highlight the contributions of our work compared to present works throughout the Introduction and Related Work section. We briefly summarize here for the consideration of the reviewer.

Compared to online feature distillation works that rely on Structure-from-Motion (SfM) datasets, obtained with NeRF / GSplats [1-5] these works:

• Need multiple camera images to train the NeRF / GSplat (e.g. F3RM [4] makes the robot grab a selfie stick and move it around the environment to collect images).
• Require multiple minutes to do training (in order to fit the NeRF / GSplat to the specific scene) and considerable time for inference.#
• NeRF-based methods are not adaptable to scene changes. Once the scene changes (e.g. due to the robot performing an action), you would have to retrain a new field. This limitation is partially addressed with deformable GSplat works, although inference time for feature splatting is not yet real-time.
• Need external segmentors (e.g. SAM) in order to obtain the masks that will improve the quality of features with object-centric priors (LangSplat [2], F-Splatting [5], SemanticGaussians [3]).

In contrast, DROP-CLIP doesn’t need test-time segmentors, works from a single RGB-D image and runs real-time (so can also capture scene changes without special treatment). We also highlight the above points during Sec.3.3 of our paper. We would appreciate further feedback by the reviewer regarding how to better clarify the above point in our paper.

Compared to offline feature distillation works that, similar to DROP-CLIP, use point-cloud encoders to distill multi-view fused features from a dataset (e.g. OpenScene [6] ), our work:

• generates and trains in MV-TOD, a multi-view cluttered tabletop scene dataset, instead of room scan datasets, which contains dense multi-view images, broad variety of objects and semantic concept annotations (see Sec.2 and App.A for details).
• introduces object-centric priors to improve the quality of the target features that will be distilled (see Sec.4.1 for qualitative comparisons with previous works).
• demonstrates with ablations (Sec.4.1), in-distribution (Sec.4.2) and out-of-distribution (Sec.4.3) experiments that the distilled features significantly improve performance for single-view semantic / referring segmentation tasks compared to previous works.

We also refer the reader to Appendix A.6.3 of our updated submission for extended discussion and comparison with modern NeRF / GSplat approaches in a scene of the LERF dataset

How can we ensure that the fused 3D features are consistent across multiple views?

If the reviewer refers to the fused 3D feature during train time (the result of the object-centric feature fusion) we can ensure that the features are view-independent since we explicitly construct them by fusing across views.

If the reviewer refers to the reconstructed features from DROP-CLIP at test-time, we agree that our approach does not guarantee that the features will be consistent across views, but it encourages it during training. In particular, by training on a dataset such as MV-TOD which contains 73 views for each scene, and forcing the same target features for each one of the views during training, we encourage DROP to produce the same features regardless of the input view.

From an architectural perspective, since the input to our encoder is a point-cloud that has been transformed to world frame and mean-shifted (using the camera extrinsics), the method can guarantee translation-invariance (so camera position). Regarding rotation-invariance, it is true that our MinkUNet architecture we're using cannot guarantee it. However, we apply extensive rotation augmentations to the 73 scene views during training to make the network robust to camera orientation.

To demonstrate view independence qualitatively, we include a supplementary video where we move the camera or the object-of-interest in real-time and observe the changes in the feature space between CLIP and a standard DFF that relies on vanilla MaskCLIP back-projection (see iclr_drop_reb_supp.mp4).

The robotics experiments [...] a two-stage open-loop demonstration

The focus in our paper was on distilling CLIP features and therefore we wish to mostly demonstrate advancements in 3D segmentation tasks, and not robotic grasping. We do not claim our robotic pipeline is optimal in terms of grasping success rates, and was added as a demonstration of the practical advantages of our work compared to modern NeRF / GSplat works for CLIP feature distillation (as mentioned above and discussed in Appendix A.6.3 of our updated pdf, no need for multiple views, real-time, immune to scene changes, no need for segmentors at test-time).