Object2Scene: Putting Objects in Context for Open-Vocabulary 3D Detection

1. The main idea of this paper is essentially an object cut-paste augmentation, which has been exploited as a common practice in most 3D object detectors. For example, starting from Point R-CNN, those 3D object detectors leverage object cut-paste to augment the 3D driving scenes. The proposed object anchor selection, normalization, and placement share very similar spirits with Point R-CNN [1] and Back-to-Reality [2].

Thanks for your reminder, we clarify the differences with the existing data augmentation methods in the overall rebuttal section and have updated the related discussion in the revised paper.

2. The contrastive learning strategy has also been studied by many relevant papers. For example, CLIP^2 [3] leverages contrastive learning to align the text and point cloud representations for open-vocabulary 3D object detection.

Contrastive learning itself is a very versatile learning strategy, widely used in a variety of tasks. However, how to design suitable contrastive learning objectives and construct appropriate positive and negative learning samples for different scenarios is an important issue. In our paper, we address the domain gap problem introduced by multi-object dataset training through a carefully designed category-level contrastive learning loss. However, in CLIP^2, contrastive learning is employed to align text and point cloud representations.

3. Text prompts are generated by pre-defined rules and lack generalization ability.

Our generated prompts are indeed rule-based and to some extent lack generative ability. However, the purpose of generating prompts is to align the point clouds and texts of objects in scenes, thereby achieving open-vocabulary. Experimental results validate the effectiveness of our prompt generation. Of course, we believe that more complex and natural language descriptions will have better generalization capabilities and bring improved point cloud-text alignment results. We will consider introducing LLM or some other tools in the future to generate more natural prompts for experiments.

4. The proposed detection framework didn't change much compared to the existing detectors. The detector is trained with supervised learning and doesn't rely on open-vocabulary foundation models such as CLIP. Hence it is hard to say whether the proposed method has the open-vocabulary detection ability, as the detection vocabulary is actually constrained by the 3D object datasets.

We offer the related statement in the overall rebuttal section. Thanks for your question.

5. Evaluation is also questionable: The authors split the datasets into seen and unseen categories, but those unseen categories are not really ``unseen", since the 3D object datasets they used can have those categories. Hence it is an unfair comparison with the baseline methods.

We offer the related statement in the overall rebuttal section.

6. Essentially this paper is not an open-vocabulary 3D detection paper. It is 3D detection with categories augmentation by inserting new objects from 3D object datasets. The detected categories are bounded by the used object datasets.

We restate the definition of open-vocabulary 3D detection in the overall rebuttal section.

1. Compared to the last version, the results in Table 1 are slightly dropped, why?

Since in the former version, there exist the parf of dataset overlap between the test set ScanNet and the training dataset ScanObjectN, so in the last version we change the ScanObjectNN to OmniObject3D, a new large-scale real-scanned 3D object dataetst, which has no overlap with ScanNet.

2. May I know the scale of inserted 3D objects and generated prompts? For example, the number of them? Since contrastive training like GLIP usually requires large-scale objects and prompts.

For OV-ScanNet20 and OV-SUN RGBD20 benchmark , the inserted object datasets cover 6w 3D objects with 190 categories. As for the generated prompts, since they are generated randomly and dynamically, based on our statistics, the actual number of prompts generated during the training process is approximately 500,000.

3. As mentioned in the last rebuttal, "we will add and discuss the cut and mix augmentation methods, such PointCutMix, in our final version." I could not find the discussion.

Thanks for your reminder, we have added PointCutMix in the related work section in our updated version.

4. Similarly, it is mentioned that "we will consider using 'spatial proximity' instead of 'affordance' and update in our final paper." Do you have any comment about this?

Actually we have updated most of our descriptions in this ICLR version, thanks for your suggestions in last review. In this version, after double check, the statement about 'affordance' only lies in the " Specifically, we choose seen objects in the scene as reference objects to guide the physically reasonable 3D object insertion according to their physical affordance. ", and we have revised it in our updated version. Besides, in this version, we clarify the generated prompts into three categories: Vertical Proximity, Horizontal Proximity and Allocentric in Section 3.2, and more details can be found in our supplementary.

1. Do the inserted object classes fully contain the unseen classes in the test set? If the unseen classes are fully covered during training, does this still counts as open vocabulary 3D object detection. Moreover, if these unseen classes are not included during object insertion, how well would the trained model perform on the unseen classes?

In our OV-ScanNet20 and OV-RGB-D20 benchmarks, inserted object classes fully contain the unseen classes in the test set, but not in OV-ScanNet200 benchmark. If the unseen classes are fully covered during training, this still counts as open vocabulary 3D object detection according to the definition of open-vocabulary object detection stated in overall rebuttal statement, and here we offer the part of classes which are not included in training datasets:

2.While using existing 3D object datasets can provide annotated objects across more categories for model training, the training process may still be limited by the vocabulary of these datasets. Learn semantics beyond these datasets’ object categories pose a challenge for the model.

Yes, it's true that learning semantics beyond these datasets’ object categories poses a challenge for the model during the training. However, once the number of object categories included in the training data is large enough, like Detic using ImageNet21K datasets to broaden the object concept, then the model's open-vocabulary capability can cover most objects in daily scenes.

3.The “Object Placement” process is central to Object2Scene but needs further clarification. How are injection regions determined? How is physical reasonability checked during placement? A pseudo code on this process could enhance understanding.

Thanks for your advice, we have added this pseudo code in the appendix of our updated paper for reference.

4.Lack of comparison on the larger OV-ScanNet200 benchmark.

1.This paper never brings up the highly overlapping work "RandomRooms: Unsupervised Pre-training from Synthetic Shapes and Randomized Layouts for 3D Object Detection" from ICCV 2021. That paper covers the exact same idea of placing random objects into random locations to create diverse training rooms. To me it seems critical to discuss this work, and differentiate from it in some concrete ways.

Thanks for your reminder! We have added RandomRooms to our related work in our updated version for discussion. Here we want to clarify that the only common point between Object2Scene and RandomRooms may be that they both use 3D object datasets to enhance 3D scene understanding. However, they have significant differences:

1. Completely different implementation methods: RandomRooms proposes to generate random layouts of a scene and construct the virtual scene using objects from synthetic CAD datasets, learning the 3D scene representation. On the other hand, Object2Scene inserts 3D objects into real-scanned 3D scenes to enhance diversity in existing scenes and further guides detectors with open-vocabulary detection capability through appropriate prompts.
2. Different problems addressed: RandomRooms aims to pretrain models using generated virtual scenes, which can serve as better initialization for later fine-tuning on 3D object detection tasks. Object2Scene aims to train a model that can be directly used in real-scanned scenes and conduct 3D understanding.
3. In initial experiments, we attempted training by directly composing virtual scenes using RandomRooms approach. However, due to the significant domain gap between constructed virtual scenes and real-scanned scenes, the detector trained on virtual scenes only achieved poor performance in open-vocabulary detection in real-scanned scenarios. By adopting our proposed Object2Scene, which is a less invasive method, we can avoid the scene-level domain gap and make it more practical.

2.The paper says "With the proposed cross-domain category-level contrastive loss, the model is forced to learn generalizable feature representations based on their class labels irrespective of the source dataset of the objects." It is unclear to me why the contrastive loss will force generalizable features. I think the high-level idea here is that the contrastive loss will make the features more similar within a class, but this effect is already achieved by the classification loss. Although empirically the contrastive loss may help further still, I don't think the given explanation really works."

Thanks for your question, here we show more experiment results to support our claim. Table 4 in our paper demonstrates the effectiveness of our proposed cross-domain category-level contrastive loss. It is shown that cross-domain object-level contrastive learning boosts the mAP25 of open-vocabulary 3D detection from 7.34% to 11.21%. Besides, we also offer TSNE visualization to support our interpretation. The figure can be found in our updated version.

3. Section 3.2 is very difficult to follow. I am lost from the first sentence: "The 3D object insertion approach composes new 3D scenes with original scenes annotated by seen categories and new objects from large vocabulary categories covering seen and unseen categories." I think a comma will be good somewhere, or maybe this sentence can be broken into multiple sentences. The notation <target - class><spatial - relationship><anchor - class> is confusing for me. If these are 2-tuples, what is a "spatial" and what is a "relationship"? Finally this section states that the "Absolute Location Prompt" is proposed as a mechanism to validate the "Relative Location Prompt", but it is unclear how this mechanism works. The end of that paragraph is confusing as well, as it seems to say (via double negatives) that the prompts might be less unclear than the relative ones ("such prompts are limited by fewer ways of expression and unclear descriptions which may not be discriminative").

1. The first sentence in Setection 3.2 has been updated to "The 3D object insertion approach creates new 3D scenes by combining original scenes that have been annotated with seen categories, along with new objects from large vocabulary categories that include seen and unseen ones." in our uptaded version.
2. The notation has been updated to " <target> <spatial relationship> <anchor>" in our updated version.
3. As for section states about "Absolute Location Prompt", this paragraph has been rewritten to be more clear: [Besides, we also propose the Absolute Location Prompt. The construction of this type of prompt relies solely on the object's position in the scene, and not on other objects in the scene. For example: "a table that is closer to the center/corner/wall of the room". Compared to the Relative Location Prompt, the prompt is simpler, but it offers a cruder positioning of the object.]