ImOV3D: Learning Open Vocabulary Point Clouds 3D Object Detection from Only 2D Images

Q1：
【Pre-training Stage】While our method, ImOV3D, has been primarily designed and evaluated for indoor scenes, we recognize the importance of assessing its performance on real-world 3D data beyond the benchmark datasets mentioned in our paper. To address this, we conducted experiments on the KITTI dataset, which consists of outdoor scenes, to evaluate how well our method generalizes to different datasets.

Table 1: Results from the Pre-training Stage Comparison of Open Vocabulary 3D Object Detection Performance on KITTI Moderate Difficulty Level

During pre-training stage, the results indicate that our method, while showing strong performance on indoor datasets, also exhibits a reasonable level of robustness and adaptability when confronted with outdoor scenes. Compared to other baselines, our mAP@0.25 is 4.1% higher. The pretraining images encompass both indoor and outdoor scenes, hence allowing us to naturally generalize to outdoor scenes.
【Adaptation Stage】We fine-tuned our model using real point cloud data, and the results show that our method outperforms OV3DET by 5.5% in the mAP@0.25, indicating a significant enhancement in performance after fine-tuning (compared to the data in Table 1). However, the relatively limited categories in outdoor scenes restrict the full potential of the open-vocabulary detector. Moreover, the differences between indoor and outdoor data also affect the outcomes. More refined adjustments and designs tailored for outdoor scenarios, our results could further improvement.

Table 2: Results from the Adaptation Stage Comparison of Open Vocabulary 3D Object Detection Performance on KITTI Moderate Difficulty Level

Q2：L59-61
In response to the question regarding the data used during the inference phase, we confirm that in our experiments on the ScanNet and SUNRGBD datasets, our method strictly used only point cloud data for inference and did not utilize image information. Within our framework, although it may appear that images and point clouds are used concurrently during the inference phase, we would like to clarify that the images presented are actually pseudo images generated by our rendering module based on the input point cloud data. While these pseudo images are visually similar to real images, they do not contain real image data inputs. We have adopted the previous baseline method (OV3DET and CoDA), it guarantees that our inference is exclusively based on point cloud data, aligning with the methodological rigor of using a single modality for input. You can see more detailed intermediate results of the pipeline visualized in Fig 2 and 3 of the global pdf response.\

【Dear Reviewer】 After reviewing our rebuttal response, do you have any further questions or areas that require additional clarification? We place great importance on your opinions and hope to continue the discussion and address any remaining concerns you may have in the upcoming communication session.

Dear Reviewer m5vv, thank you for your valuable comments. As the deadline is approaching, we kindly inquire whether our discussions have addressed your concerns. If you have any further questions, we would be happy to continue our conversation. If our response has resolved your concerns, we would greatly appreciate it if you could consider updating your score and providing feedback. Thank you again.

Q1：
We gladly accept the suggestion and are considering modifying it to “How far can 2D images drive 3D open-vocabulary object detection? ” We appreciate your advice, but our motivation is to explore how can we fully exploit information from 2D images and how far they can drive the performance of 3D understanding both in an adaptation-free setting and an adaptation setting. We are also open to any suggestions.
Q2：
For a better understanding, please refer to Figure 1 in the global pdf response.
【2D and 3D Fusion Details】
(1) Seed point generation and feature extraction
In the 3D branch, K seed points are initialized from the 3D point cloud, each defined by its coordinates in 3D space (Kx3). These seed points are further enriched by feature extraction, where each point, in addition to its coordinates, also acquires additional feature representations $F_{pc} \in \mathbb{R}^{K \times F}$.
(2) Lifting 2D information to 3D
Using a 2D OV detector to identify objects on RGB images generates 2D bounding boxes and semantic labels. These 2D information are transformed into 3D rays through the camera matrix, with these rays originating from the seed points and pointing towards the center of the objects in 3D space. This process elevates the geometric cues from the 2D image into 3D space, aiding in the precise localization of objects. These lifted image features can be represented as $F_{img} \in \mathbb{R}^{K \times (3+F')}$.
(3) 2D and 3D Feature Fusion
During the feature fusion phase, the point cloud features $F_{pc}$ are concatenated with the image features $F_{img}$ to form the joint feature representation $F_{joint} \in \mathbb{R}^{K \times (3+F+F')}$. As a result, each seed point contains not only the geometric information in 3D space but also the semantic information integrated from the 2D image.
(4) Multi-Tower Architecture
We employ a multi-tower architecture to process the point cloud features and 2D image features separately. Each tower focuses on handling one type of input feature and balances the contributions of different modalities through a gradient fusion strategy. The img tower has a weight of 0.3, the point tower has a weight of 0.3, and the joint tower has a weight of 0.4, consistent with the backbone ImVoteNet.
【Loss Details】
Finally, the loss function can be expressed as:

where $i$ represents different features, such as $\text{pc}$, $\text{img}$, $\text{joint}$. $W_i$ is the weight corresponding to feature $i$. $L_{\text{loc}}$ represents the original localization loss function used in ImVoteNet. $F_{\text{text}}$ denotes the feature extracted by the text encoder in CLIP.
Q3：
Our research paper demonstrates the immense potential of using a 2D-only setting. Through our approach, we primarily aim to address the question of how much 2D images themselves can enhance the performance of OV 3D object detection. In our work, this is not just a paper that solves existing problems but also a scientific exploration. We believe that the 2D-only version setting, as an important experimental conclusion, shows people that 2D images can also provide abundant information for 3D understanding. The current increase in the adaptation setting exceeds the improvements made by previous works, such as CoDA. Although our method increases the complexity of training, our ablation study has proven that every step in the training stage is crucial. However, in the inference stage, our scheme is no more complex than the existing OV3DET. In actual use, the complexity of our scheme does not affect the user experience. The latency remains comparable to OV3DET, and the additional memory cost is even smaller. To promote the reproducibility of research, we will also fully open-source our code for everyone to reproduce and further study.

【Dear Reviewer】After reviewing our rebuttal response, do you have any further questions or areas that require additional clarification? We place great importance on your opinions and hope to continue the discussion and address any remaining concerns you may have in the upcoming communication session.

Q1：
We focus on 2D images because real-world 3D point cloud data is not only limited in quantity but also has relatively few annotations. At the same time, we have observed that 2D image data is not only abundant in quantity but also rich in annotation information. Based on these observations, we wish to explore the potential of 2D images to enhance 3D understanding performance under the condition of having only 2D images.

In the adaptation stage, we do indeed use real 3D point cloud data, but the main purpose is to reduce the differences between the model's performance across different data domains, that is, to minimize the domain gap. The reason we choose not to use point clouds and images simultaneously during the training phase is that the existing RGB-D paired datasets are small in scale. We do not want the model to be optimized only for these limited datasets during the adaptation stage, thereby restricting the model's generalization ability.

Therefore, our approach aims to improve the performance of the 3D detection model by making full use of the rich 2D image data and utilizing the limited 3D point cloud data in the adaptation stage to optimize the model. This ensures that the model maintains efficient and accurate performance when facing real-world data, while also possessing good generalization capabilities.

Q2：
【3D Pseudo Annotation Generation】
Firstly, we determine the boundaries of the frustum based on the 2D bounding boxes, and then extract all points within the frustum from the 3D point cloud. The extracted point cloud may contain background points and outliers. To remove these points, we employ a clustering algorithm to analyze the point cloud, gathering points that belong to the same object together. Through the clustering results, we can identify and remove background points and outliers that do not belong to the target object, thus obtaining cleaner point cloud data. After removing non-target points, we calculate the 3D pseudo bounding box based on the remaining points. This includes determining the center position, size, and orientation of the pseudo box. Typically, the center position can be determined by the centroid of the point cloud, the size can be estimated by the boundary of the point cloud, and the orientation can be determined by the object's main axis or normal. Therefore, we can generate corresponding pseudo labels, including the position, size, and orientation information of the box.
【The Accuracy of the Pseudo Labels】
Since the depth images Dmetric obtained from monocular depth estimation may contain noise, this can affect the accuracy of the 3D bounding boxes. To address this issue, we use a 3D bounding box filtering module to filter out inaccurate 3D bounding boxes. We have constructed a median object size database using GPT-4, by asking for the average length, width, and height of specific categories, defining a filtering criterion for comparing the size L, W, H of each object in the scene with the median size. If the ratio of the object's dimensions to the median size is within a range of the threshold T, the object is retained. We filter out bounding boxes that do not meet the size criteria before training. During inference, we remove categories that are semantically similar but have different sizes to prevent errors such as misidentifying a book as a bookshelf. From Table 3 in the paper, the quantitative results of this part can be seen. We will elaborate this more clearly in the main paper.
Q3：
【Answer questions about the complex pipeline】
Understanding 3D scenes using only images is an incredibly challenging problem. That's why we've designed this component and the entire pipeline. Although it may appear complex at first glance, every part of it is essential. We haven't included any components to make it redundant or to add unnecessary complexity. Instead, this seemingly complex pipeline is carefully crafted to include everything necessary to effectively address the task.
【Regarding the possibility of removing image generation step】
Please review the global reply.
【Inference process】
The OV3DET operates on a point cloud only basis during the inference process, which is not what question mentioned as being able to input both image and point cloud simultaneously. On the contrary, our pipeline can incorporate images during the inference process. However, Tables 3 and 4 in the PDF indicate that using point clouds and images during inference leads to a decline in results. The reason is the domain gap between the rendered images from training and the real images from inference.
Q4：
In our paper, the training data for the depth estimation model comes from a mix of 12 datasets (e.g. NYU DepthV2,KITTI etc), excluding SUNRGBD and Scannet, which are used for detection. This ensures there is no overlap in the training data.
Q5：
Our apologies, we will cite "Controlnet" in the main paper subsequently.

【Dear Reviewer】After reviewing our rebuttal response, do you have any further questions or areas that require additional clarification? We place great importance on your opinions and hope to continue the discussion and address any remaining concerns you may have in the upcoming communication session.

Dear Reviewer TsJm, thank you for your valuable comments. As the deadline is approaching, we kindly inquire whether our discussions have addressed your concerns. If you have any further questions, we would be happy to continue our conversation. If our response has resolved your concerns, we would greatly appreciate it if you could consider updating your score and providing feedback. Thank you again.

Q1：
While our method, ImOV3D, has been primarily designed and evaluated for indoor scenes, we recognize the importance of assessing its performance on real-world 3D data beyond the benchmark datasets mentioned in our paper. To address this, we conducted experiments on the KITTI dataset, which consists of outdoor scenes, to evaluate how well our method generalizes to different datasets.

Table 1: Results from the Pretraining Stage Comparison of Open Vocabulary 3D Object Detection Performance on KITTI Moderate Difficulty Level

The results indicate that our method, while showing strong performance on indoor datasets, also exhibits a reasonable level of robustness and adaptability when confronted with outdoor scenes. Compared to other baselines, our mAP@0.25 is 4.1% higher. The pretraining images encompass both indoor and outdoor scenes, hence allowing us to naturally generalize to outdoor scenes.

Q2：
We fine-tuned our model using real point cloud data, and the results show that our method outperforms OV3DET by 5.5% in the mAP@0.25, indicating a significant enhancement in performance after fine-tuning (compared to the data in Table 1 of Q1). However, the relatively limited categories in outdoor scenes restrict the full potential of the open-vocabulary detector. Moreover, the differences between indoor and outdoor data also affect the outcomes. More refined adjustments and designs tailored for outdoor scenarios, our results could further improvement.

Table 2: Results from the Adaptation Stage Comparison of 3D Object Detection Performance on KITTI Moderate Difficulty Level

Q3：
【Computational Resources】 In our research project, the model's pre-training stage utilized 8 NVIDIA GeForce RTX 3090 GPUs, taking 72 hours, while the adaptation stage required only 12 hours, with both stages demanding approximately 16GB of GPU memory per GPU with a batch size of 12. 0.2s per scene on average on the scannet and sunrgbd dataset.
【Scalability and Latency】Our method is designed with scalability, suitable for large-scale applications. Its core strength lies in the flexibility of its backbone network, which can be seamlessly replaced with the latest efficient architectures, thereby further enhancing efficiency with future technological advancements.

【Dear Reviewer】After reviewing our rebuttal response, do you have any further questions or areas that require additional clarification? We place great importance on your opinions and hope to continue the discussion and address any remaining concerns you may have in the upcoming communication session.

Dear Reviewer teog, thank you for your valuable comments. As the deadline is approaching, we kindly inquire whether our discussions have addressed your concerns. If you have any further questions, we would be happy to continue our conversation. If our response has resolved your concerns, we would greatly appreciate it if you could consider updating your score and providing feedback. Thank you again.

Q1：
In depth estimation research, using fixed camera intrinsics is common practice [1][2]. While we've followed this approach, we recognize it may lead to inaccurate bounding box sizes. To address this, we've implemented a scale filter using GPT-4, adjusting bounding boxes based on empirically determined object sizes from LLMs. This correction has significantly improved performance. As shown in Table 3, the first row presents results without bounding box correction, and the second row shows improved performance with the filtering strategy: mAP0.25 increased by 1.65% on the SUNRGBD dataset and by 1.27% on the ScanNet dataset. Thank you for your valuable feedback; we will include a discussion on this limitation in the paper.
Q2：
The motivation for using this formula is explained in the main text from L143 to 150. We explore Equation (1) in depth within paper, utilizing Rodrigues' rotation formula [3]. It calculates the rotation matrix $R$ for a rotation by $\theta$ (less than $180^\circ$) around the axis defined by the unit vector $\hat{n}=(n_x,n_y,n_z)$. The formula is defined as: $R=I+(\sin\theta)N+(1-\cos\theta)N^2$ Where $I$ is the identity matrix, and the skew-symmetric matrix $N$ for the unit vector $\hat{n}$ is constructed as: $N=\begin{bmatrix}0&-n_z&n_y;&n_z&0&-n_x;&-n_y&n_x&0\end{bmatrix}$ To align unit vector $N_{pred}$ with unit vector $Z_{axis}$, we start from the definitions of the inner product $N_{pred} \cdot Z_{axis} = \cos\theta$ and cross product $v = N_{pred} \times Z_{axis} \Rightarrow v = \sin\theta\hat{n}, |v| = \sin\theta$. Accordingly, define: $K\stackrel{\mathrm{def}}{=}(\sin\theta)N= \begin{bmatrix}0&-v_z&v_y;&v_z&0&-v_x;&-v_y&v_x&0\end{bmatrix}$ Based on rotation formula, we have: $R=I+K+\frac{1-\cos\theta}{\sin^2\theta}K^2$ $=I+K+K^2\frac{1-N_{pred}\cdot Z_{axis}}{||v||^2}$ Q3：
【Generating a Visual Frustum】
We utilize the four corner points of a 2D bounding box, along with the minimum and maximum depth values from the depth image, to create a visual frustum. Specifically, we first convert the 2D image coordinates into 3D spatial coordinates using the intrinsic matrix. Then, based on the depth information, we calculate the actual positions of these corner points in 3D space, thereby generating the visual frustum. Within the visual frustum, we extract the corresponding point cloud data. To ensure the stability and efficiency of the calculations, we perform downsampling on the point cloud data. Subsequently, we apply the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) clustering algorithm to the extracted point cloud data to remove outliers and noise, retaining only the main point cloud clusters.
【3D Bounding Box Orientation】
By applying Principal Component Analysis (PCA) to the main point cloud clusters, we calculate the main orientation angle (yaw) of the point cloud. Specifically: PCA analysis is conducted on the X-Y plane of the point cloud data to obtain the principal component direction vector. The main orientation angle (yaw) is calculated based on the principal component direction vector. The point cloud is rotated using a rotation matrix to align the main direction with the X-axis. The range of the bounding box after rotation alignment is calculated. The bounding box is then rotated back to its original direction to obtain the final 3D bounding box position and orientation. We construct the final 3D bounding box description based on the calculated center point coordinates, dimensions (width, depth, and height), and orientation angle (yaw).
【Regarding the Orientation of Bounding Boxes in the ScanNet Test Dataset】
The official bounding boxes provided by the ScanNet dataset are axis-aligned, in our research, we have adopted the baseline method (OV3DET and CoDA), following the common practices established by previous studies, using Principal Component Analysis (PCA) to determine the orientation of the bounding boxes. In all our experiments, we uniformly used Oriented Bounding Boxes for testing, having standardized the representation of bounding boxes, thus eliminating any discrepancies in bounding box orientation. At lines L162-163, we have provided a description pertinent to the issue raised. We will elaborate more on this in the paper to ensure clarity.
Q4：
As mentioned in the global reply.
Q5：
Thank you for your suggestion, we have followed your request and displayed the visualization of each step in Fig2 and Fig3 of the global response PDF.
Q6：
【About lack novelty】
Our data generation pipeline is meticulously designed, and we do not agree that similar work implies a lack of novelty in our approach. The works you mentioned, such as Spatial-RGPT and Spatial VLM, are considered concurrent under the NeurIPS Policy, which treats works published within two months of submission as being from the same period. Our task design and complete pipeline differ significantly, particularly in our focus on 3D detection and detailed descriptions of 3D bounding boxes, including orientation and size. During the rebuttal phase, we will provide more detailed visualizations of the pipeline in the global PDF, Fig 2 and 3, to better illustrate the entire process. We are committed to open-sourcing our code to enable others to explore further applications based on our work, fostering development and innovation within the community.

【Dear Reviewer】We value your feedback and look forward to discussing any additional issues in the next communication phase.
References
[1]Pan Ji, Runze Li, Bir Bhanu, Yi Xu, "MonoIndoor: Towards Good Practice of Self-Supervised Monocular Depth Estimation for Indoor Environments"
[2]Vitor Guizilini, Igor Vasiljevic, Dian Chen, Rares Ambrus, Adrien Gaidon, "Towards Zero-Shot Scale-Aware Monocular Depth Estimation"
[3]Richard M. Murray, Zexiang Li, S. Shankar Sastry, pp. 26-28,"A Mathematical Introduction to Robotic Manipulation"