Training an Open-Vocabulary Monocular 3D Detection Model without 3D Data

We would first like to express our appreciation for your time and insightful comments. Please find our response to your concerns in the following:

---

1. Performance of Unidepth.

Thanks for your valuable suggestion. We experiment with different architectures of Unidepth, which exhibit different performance. We find that the more accurate the depth estimation model use, the better the performance of the trained open-vocabulary monocular 3D detection model. This also indicates that with the development of increasingly powerful depth estimation models, our method has a broad prospect.

---

2. Costs and benefits of GPT-4.

We would like to clarify that since we do not use any annotations in the dataset during training, we cannot obtain prior information about objects from the dataset. Therefore, we propose to utilize a large language model, which has seen many descriptions of object shapes on the web, to provide reasonable shape priors. As shown in Table 3(d), using LLM-generated priors yields comparable results to utilizing dataset statistics. This indicates that LLMs indeed provide a reasonable estimation of object shape priors.

Please note that our access to GPT-4 is not on a sample-wise basis; instead, we only need to access GPT-4 once for the priors of all the categories we interested. The cost can be negligible. Furthermore, we can utilize open-source large language models, such as LLaMA.

---

3. Missing of distant objects.

If distant objects are missing, it can lead to confusion during the model training.

However, we find that the self-training approach can continuously refine the pseudo labels, which involves employing the results from the previously well-trained model as pseudo labels in subsequent training processes, thereby enhancing the quality of the initially generated pseudo boxes. As shown in the table below, self-training can further enhance the model's performance across various distances.

Additionally, we can utilize techniques introduced in previous work [1], zooming in regions in the image that contains small or distance objects to improve the 3D detection performance of these instances.

---

4. Ablation studies.

We apologize for any confusion. In Table 3, our framework's default setting is indicated in grey, with each sub-table representing the ablation of one component. For example, in Table 3(b), compared to the fixed erosion method, our adaptive erosion component bring an increase of 2.1 AP (16.4 vs 18.5). We will improve the presentation of our ablation studies in the revision.

[1] ZoomNet: Part-Aware Adaptive Zooming Neural Network for 3D Object Detection. Xu et al. AAAI 2020.

We would first like to express our appreciation for your time and insightful comments. Please find our response to your concerns in the following:

---

1. nuScenes result.

We have conducted experiments on nuScenes, and our method has consistently achieved good results, demonstrating its generalizability. Please refer to the results in the global rebuttal.

---

2. Details of adaptive pseudo-LiDAR erosion module.

Thanks for your valuable suggestion. We will add these details below and the diagram in the attached PDF to the Sec 3.1 of the revision:

We propose to employ the image erosion operation to eliminate inaccurate object mask edges, thereby obtaining accurate pseudo-LiDAR after unprojection. Image erosion is a fundamental operation in digital image processing that involves shrinking or wearing away the boundaries of foreground objects in a binary image. $M_i$ is the binary mask of one object we interest, where each pixel $M_i(x,y)$ is either 0 (background) or 1 (foreground). Let $B(u,v)$ represent the elements of the structuring element $B$, which is a small binary matrix of size $m \times n$. Erosion of the mask $M_i$ by the structuring element $B$ is denoted as $M_i \ominus B$.

$$(M_i \ominus B)(x, y) = \begin{cases} 1 & \text{if } M_i(x+u, y+v) = 1 \text{ for all } (u,v) \text{ such that } B(u,v) = 1, \\\\ 0 & \text{otherwise.} \end{cases}$$

In our experiment, we employ a 3x3 structuring element $B$ where all elements are set to 1. The erosion operation can be applied repeatedly. The intensity of erosion can be controled by adjusting the number of erosion iterations. We note that aggressive erosion may completely remove small objects like distant pedestrians, while mild erosion may leave considerable noise on large objects. Therefore, for masks with a smaller area, we use a smaller number of iterations; conversely, for masks with a larger area, we use a larger number of iterations, ensuring more accurate pseudo-LiDAR.

---

3. More visualization of pseudo-LiDAR erosion module.

Please refer to the more visualizations in the attached PDF, and we will incorporate them into the camera-ready version.

---

4. Related works.

Thank you for pointing out. We will make sure add them in the revision.

---

5. Baseline trained without ground-truth for base classes.

Thanks for your valuable advice. We compare our framework to the open-vocabulary 3D detection method OV-3DET [1]. OV-3DET is trained and tested on point cloud data, but it does not require any annotations for training. To adapt it to the monocular detection scenario, we use pseudo-LiDAR as its input. Furthermore, to mitigate the data distribution shift between training and test for OV-3DET, we train the OV-3DET framework using pseudo-LiDAR.

However, their framework is designed to generate pseudo labels from real point clouds, failing to address the inherent noise in pseudo-LiDAR data, which results in low-quality labels.

In contrast, our framework is tailored to the noisy nature of pseudo-LiDAR, incorporating adaptive pseudo-LiDAR erosion and bounding box refinement techniques, and leveraging prior knowledge from large language models, which allows us to generate pseudo labels more effectively for pseudo-LiDAR. Please refer to the results in the global rebuttal.

[1] Open-Vocabulary Point Cloud Object Detection without 3D Annotation. Lu et al. CVPR 2023.

We would first like to express our appreciation for your time and valuable comments. Please find our response to your concerns in the following:

---

1. Evaluation on OV-3DET and larger-scale datasets.

Thanks for your suggestion. We show the results in the global rebuttal and will add them in the final revision.

---

2. Experimental setup.

Indeed, ideally, 3D open-world detection should be as generalized as 2D open-world detection, i.e., a foundation model can be applied to any class. However, this generalization has not been feasible yet in 3D for the two main reasons:

• The volume of data and the number of categories in 3D datasets are far less than those in 2D datasets.
• There is an enormous domain gap between different 3D datasets due to sensor configurations.

Especially for monocular 3D detection, the camera parameters can vary across different datasets, leading to poor transferability [1,2]. Therefore, we follow the current paradigm in many recent 3D open-world detection works which train models on a single dataset and test it on this dataset [3,4,5].

We would like to clarify that we do not need to split the datasets into base and novel splits to train our model. We introduce this base/novel setting only to compare our method with the constructed Cube R-CNN+Grounding DINO baseline due to the lack of a suitable baseline before. In fact, we do not need any ground-truth annotation, as shown in the last line in Tables 1 and 2 of our paper. Therefore, our framework is fully capable of applying and achieving open-world 3D detection when the data are sufficient.

---

3. Why pseudo-label 3D datasets?

Due to the scale ambiguity inherent in monocular depth estimation [6,7], it is always better to know the camera's focal length to perform accurate monocular depth estimation [8,9]. Therefore, we select datasets with camera focal lengths for our experiments. As for the COCO dataset, we do not know its camera focal lengths.

It is worth noting that our method can generalize beyond 3D datasets. It is to demonstrate our approach for us to conduct experiments on 3D data. In fact, our method can leverage any images along with their camera focal lengths for training. Data acquisition does not necessitate the use of costly LiDAR or 3D scanners.

Additionally, we can use various video data sources like YouTube and the newly released SAM 2 to obtain camera focal lengths through SLAM (Simultaneous Localization and Mapping) to generate pseudo labels and thus facilitate our framework to train 3D open-world detectors on video data.

---

4. Pseudo-labeler heuristics.

Thanks for your insightful comment. In our framework, the erosion and PCA to determine orientation are both robust. When searching for the bounding box, we assess whether the box falls within the reasonable range of prior knowledge with two thresholds $\tau_1$ and $\tau_2$. Table 3(f)(g) in the paper shows the selection of this threshold is robust and reasonable.

To further validate, we conduct experiments on additional datasets. We make no modifications to our framework, using exactly the same parameters on ARKitScenes and nuScenes as we do on SUN RGB-D and KITTI to generate pseudo labels. Despite ARKitScenes and nuScenes having completely different data distributions from SUN RGB-D and KITTI (due to different sensor models and parameters, different shooting angles, weather conditions, and geographical locations), our framework is still able to generate effective pseudo labels and achieve good results. Please refer to the results in the global rebuttal.

Additionally, we can refine the model using a self-training approach, which involves using the results from the previously well-trained model as pseudo labels in the subsequent training process. As shown in the table below, self-training can further enhance the quality of the initially generated pseudo boxes.

---

5. LLM-generated priors vs. real priors.

Thanks for your suggestion. In Tables 1 and 2 of the attached PDF, we provide a detailed comparison of shape priors on KITTI and SUN RGB-D. Since we only utilize the shape priors to filter out pseudo boxes that may be unreasonable due to noise or occlusion and to refine them, as long as the shape priors are within a reasonable range, they are sufficient for our purposes.

---

6. Out of Distribution Dataset.

In the following table, we present the evaluation on nuScenes using the model trained on KITTI. Unsurprisingly, the results are inferior to those obtained from direct training. Many previous studies indicate that there is a significant domain gap between different 3D datasets. Especially for monocular 3D detection, the camera parameters vary across different datasets, leading to poor transferability [6,7].

[1] MonoUNI: A Unified Vehicle and Infrastructure-side Monocular 3D Object Detection Network with Sufficient Depth Clues. Jia et al. NeurIPS 2023.

[2] FS-Depth: Focal-and-Scale Depth Estimation from a Single Image in Unseen Indoor Scene. Wei et al. TCSVT 2024.

[3] Open-Vocabulary Point Cloud Object Detection without 3D Annotation. Lu et. al. CVPR 2023.

[4] PLA: Language-Driven Open-Vocabulary 3D Scene Understanding. Ding et al. CVPR 2023.

[5] OpenScene: 3D Scene Understanding with Open Vocabularies. Peng et al. CVPR 2023.

[6] Is Pseudo-Lidar needed for Monocular 3D Object detection? Park et al. ICCV 2021.

[7] Pseudo-LiDAR++: Accurate Depth for 3D Object Detection in Autonomous Driving. You et al. ICLR 2020.

[8] Towards Zero-Shot Scale-Aware Monocular Depth Estimation. Guizilini et al. ICCV 2023.

[9] Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Image. Yin et al. ICCV 2023.

We would first like to express our appreciation for your time and insightful comments. Please find our response to your concerns in the following:

---

1. Dependence on depth estimation quality.

Indeed, our method relies on accurate depth estimation, similar to most monocular 3D detection methods. To address the issue of potential inaccuracies in estimating the depth of distant objects:

• We can continuously refine the pseudo labels using a self-training approach, which involves employing the results from the previously well-trained model as pseudo labels in subsequent training processes, thereby enhancing the quality of the initially automatically generated pseudo boxes. As shown in the table below, self-training can further enhance the model's performance across various distances.

• We can utilize techniques introduced in previous work [1], zooming in regions in the image that contains small or distance objects to improve the 3D detection performance of these instances.

---

2. Performance vary with the quality of the depth estimation.

Thanks for your insightful comment. We experiment with different architectures of Unidepth, which exhibit different performance. We find that the more accurate the depth estimation model use, the better the performance of the trained open-vocabulary monocular 3D detection model. This also indicates that with the development of increasingly powerful depth estimation models, our method has a broad prospect.

---

3. Evaluation with other OV-3Det methods.

Thanks for your valuable suggestion. We have provided the comprehensive comparison with OV-3DET [2] in the global rebuttal above and will add them in the revision. Considering OV-3DET [2] being trained and tested on point cloud data, to adapt it to the monocular detection scenario, we use pseudo-LiDAR as its input. Furthermore, to mitigate the data distribution shift between training and test for OV-3DET, we train the OV-3DET framework using pseudo-LiDAR.

However, their framework is designed to generate pseudo labels from real point clouds, without taking into account the noisy nature of pseudo-LiDAR, which results in low-quality labels.

In contrast, our framework is tailored to the noisy nature of pseudo-LiDAR, incorporating adaptive pseudo-LiDAR erosion and bounding box refinement techniques, and leveraging prior knowledge from large language models, which allows us to generate pseudo labels more effectively for pseudo-LiDAR.

CoDA [3] also faces the same difficulties as OV-3DET when adapting to pseudo-LiDAR. Additionally, CoDA requires training with annotations from base categories, but our method does not require any annotations, so it is unfair to directly compare with them.

---

4. Generalize to other data.

We have conducted experiments on additional datasets such as ARKitScenes and nuScenes, demonstrating the effectiveness of our method.

We make no modifications to our framework, using exactly the same parameters on ARKitScenes and nuScenes as we do on SUN RGB-D and KITTI to generate pseudo labels. Despite ARKitScenes and nuScenes having completely different data distributions from SUN RGB-D and KITTI (due to different sensor models and parameters, different shooting angles, weather conditions, and geographical locations), our framework is still able to generate effective pseudo labels and achieve good results. Please refer to the global rebuttal for the numbers.

---

5. Computational cost.

Thanks for your suggestion. We compare our computational cost on SUN RGB-D dataset to OV-3DET [2] in the table below. Since we do not infer 3D point clouds but only infer images, the inference latency is significantly reduced.

---

6. Challenges in real-world applications.

When applying in the real world, it is important to consider the differences in hardware, such as camera parameters and distortion issues that can lead to differences in imaging. Calibration of the hardware should be conducted before application.

Additionally, it would be best to fine-tune the depth model to achieve more accurate estimation results.

---

7. Complexity of Implementation.

We include details of our implementation in the paper and will release the code to make sure the reproducibility.

[1] ZoomNet: Part-Aware Adaptive Zooming Neural Network for 3D Object Detection. Xu et al. AAAI 2020.

[2] Open-Vocabulary Point Cloud Object Detection without 3D Annotation. Lu et al. CVPR 2023.

[3] CoDA: Collaborative Novel Box Discovery and Cross-Modal Alignment for Open-Vocabulary 3D Object Detection. Cao et al. NeurIPS 2023.