2D-Supervised Monocular 3D Object Detection by Global-to-Local Reconstruction

W3: Comparison with more recent methods

On the Waymo open dataset (WOD), we have compared BA2-Det with the fully supervised SOTA method BA-Det. On the KITTI dataset, we additionally compare with some recently published methods to show the results more comprehensively. Please refer to the revised paper.

W4: Analysis of false positives

• False positives at longer distances are mainly caused by the pseudo label distribution. As shown in Figure 3, our pseudo labels can be 0-200m, and so the depth prediction on training set (as labels for retraining) will be in 0-200m. However, in iteratively self-retraining stage, we constrain the depth estimation in 0-75m. That means many too-far objects are predicted near 75m, which are the FP predictions. (However, according to Table 6, 0-75m constraint in retraining is effective for general AP metrics.)

• In this work, we have alleviated this problem by removing objects that are 1 meter away from the farthest predicted distance, i.e., if predicting depth in [0,75m], we filter the object whose depth is more than 74m. The improvement is shown in the following table.

• In the future, we will try the label balancing algorithms to further solve this problem.

We sincerely thank you for the professional comments. We are inspired and hope our discussion brings more insights.

W1: Assumptions and limitations of monocular 3D object detection

We use 2D annotation to obtain pseudo 3D labels, and finally train a monocular 3D object detector with these pseudo labels. Similar to other monocular 3D detectors, we also encounter inherent issues such as inaccurate depth estimation and occlusion problems. The main contribution of this paper is proposing the use of weakly supervised methods to train a 3D detector. Compared to using 3D ground truth, our weakly supervised approach has the following limitations.

• The limitation of reconstructing videos: Our method relies on reconstruction to obtain pseudo labels, so it is limited by the usage conditions of SfM. It is difficult to achieve good reconstruction results when there is no ego-motion or when the camera undergoes severe movement in the dataset. The method doesn't work well for purely static video datasets. However, less than 5% of videos in the driving dataset do not meet the requirements, and we can use only pseudo labels with good reconstruction results to train the model. Therefore, this limitation can be greatly alleviated in driving scenarios. We also apply this method in more general scenes (Fig. 6), where most videos conform to the assumptions of this paper.
• The limitation of pseudo label quality: (1) Inaccurate orientation: Since many 3D clusters from the reconstructed scene are incomplete due to occlusion and self-occlusion, orientation estimation is more challenging than 3D location and 3D shape estimation. In section 4.2 and 4.3, we try our best to alleviate this problem. However, it still affects the performance and becomes an open problem. (2) Big cluster containing more than one object: When some objects are side-by-side, like in a parking lot, the clustering algorithm may output a big object cluster containing more than one object. We filter this kind of cluster by 3D shape threshold. In the future, we will improve the clustering algorithm to solve this problem. (3) Time-consuming iteratively self-retraining: Although iteratively self-retraining can improve performance a lot, it takes more than double the training time. We will try to decrease the time cost and keep the performance in the future work.

W2: Simplistic approach for 3D object label generation

The rationality of each module design can be verified in our ablation study (Table 3). As for why we use specific methods, algorithms, or models, the principle for using them is simple and effective.

• GBA and LBA are based on SfM, and we implement SfM using the COLMAP library. Implementing GBA and LBA using SfM/COLMAP is because it is a robust and widely-used temporal 3D depth/location estimation method. The main limitations and potential shortcomings using SfM/3D reconstruction are time-consuming and non-ego-motion videos. However, there are many works on 3D reconstruction acceleration [A,B,C]. Besides, non-ego-motion videos are less than 5% in autonomous driving datasets. The negative impact is relatively limited.
• We use ImmortalTracker as the tracking module, because it is 3D Kalman Filter-based tracker, which is a SOTA 3D tracker without a learnable module.
• The clustering method is based on the Connected Component (CC) algorithm, a simple graph algorithm implemented by the Scipy library. However, we design the two-stage object clustering method DoubleClustering, much better than just using the CC algorithm.

[A] Ozyesil et al. Robust camera location estimation by convex programming. In CVPR 2015.

[B] Cui et al. HSfM: Hybrid Structure-from-Motion. In CVPR 2017.

[C] Jin et al. Image matching across wide baselines: From paper to practice. In IJCV 2021.

We really appreciate your valuable comments. We are trying our best to address your concerns and are open to any further discussions.

W1: Discussion about relevant works

Although MonoDR and nerf-from-image also learn 3D cuboids from 2D supervision, there are significant differences in problem settings and methods between them and us.

• The core challenge is much different. These methods emphasize the complete object mesh/shape reconstruction, however, we do not reconstruct objects at the mesh level. The reconstruction stage is just for 3D box estimation. The main challenge for us is the object's depth estimation in complex scenes.
• Besides, they both rely on additional hand-craft class-wise CAD models. However, utilizing video-based 3D reconstruction, we do not need CAD models, as shown in Fig. 6 we can even detect open-set 3D objects.
• As for depth estimation, they either utilize pretrained depth or depth labels. MonoDR requires a pretrained depth estimator to supervise the depth map for depth estimation. nerf-from-image relies on 2D canonical map labels from synthetic datasets. However, our work does not require these additional data/models/labels.
• Moreover, since they are fine-grained shape/mesh-based reconstruction methods, an additional pretrained segmentation network is required for obtaining precise object masks. Our model is only supervised with 2D boxes instead of object masks.
• We also try nerf-from-image in our complex dataset WOD, and the reconstruction performance is not as good as the clean dataset nerf-from-image used. Please refer to the response to Q2 for more details.

W2: Technical contributions

Our pipeline is not just a combination of straightforward approaches based on existing techniques. There are many technical contributions in our work.

• Although we utilize the simple yet effective Connected Component algorithm in DoubleClustering, the two-stage local-to-global clustering method is totally new. No other work has designed a similar algorithm. The two-stage clustering is also shown effective in our ablation study (Table 12).
• We propose a novel network $G_\theta$ to learn object 3D size/orientation from the well-reconstructed objects. This network makes it possible to provide more pseudo-labels for occluded objects and these labels play an important role in the following monocular 3D object detector (LBA-Learner) learning stage. (Table 3 & 13)
• We design novel orientation loss to solve the problem of 180$^\circ$ flipping in pseudo label (Table 14).
• Our iteratively self-retraining procedure is much different from SDFLabel [3] used on both motivation and method details. What we do in iteratively self-retraining procedure is to evolve and generalize to the dynamic objects. SDFLabel uses retraining to sample good pseudo labels by rules to make self-improving. However, we show that in our method, heuristic selection cannot work well (Table 5) and we focus on label refinement.
• In all, we agree that we also use the pipeline of pseudo label generation, learning from pseudo labels, and retraining in high-level concepts. However, the proposed method is novel and solve the challenging problems in 2D supervised monocular 3D object detection task.

W3&Q1: Using 10% GT

• We supplement the experiment using the semi-supervised setting with the same 10% GT as BA-Det and MonoFlex.

With additional 10% GT, our method can outperform BA-Det and MonoFlex.

W4: Extra tracking module

• We use the 3D Kalman Filter-based tracker ImmortalTracker, without any learnable module. So we do not introduce any additional parameters.
• Besides, in both baseline SfM+BA-Det and fully-supervised SOTA method BA-Det, we also use the same tracking module.

W5: Tables

Sorry for the difficulty readers may have in reading the table. We have fixed it in our revised paper. Thank you again for pointing it out.

Q2: Comparing with nerf-from-image

• We try nerf-from-image in our complex dataset WOD. However, only for object reconstruction, the performance is not as good as it shows in clean datasets CUB Birds and Pascal3D+ Cars. We analyze and discuss why it doesn't work on WOD in the following. For more qualitative results, please refer to section D in the appendix of our revised paper. (1) The main problem is for the small objects, i.e., for the objects far from the camera. Since nerf-from-image is designed for object-centric datasets, in which objects are near the camera (<10m). However, for WOD, most objects are in [30m,60m]. Please refer to Fig. 3 in our paper. We find that if the box is less than 80 * 160 pixels, nerf-from-image cannot recognize the object, even if we resize the image to 4x and higher resolution. In WOD, objects less than 80 * 160 pixels are more than 90% of all objects. That means nerf-from-image is an unavailable method on WOD. (2) Another problem is occlusion. Nerf-from-image will fail to find the main object in the 2D box. (3) Nerf-from-image also fails at night. The object cannot be recognized.
• However, our method will not fail in these cases, because our 3D reconstruction is a temporal method. Objects are also easier to be separated in reconstructed point clouds.
• In summary, we show that even if combined with a 2D detector, nerf-from-image is not a comparable method with our proposed method.

Q3: Duration of training process

We require a total of 624 GPU hours for the entire training process. We utilize 8 NVIDIA A40 GPUs to train GBA-Learner for 10 hours, LBA-Learner for 26 hours in the initial iteration, and self-retrain LBA-Learner for 42 hours over two iterations. As for fully-supervised BA-Det, the training process is 280 GPU hours.

We are grateful for your valuable comments and constructive advice, which help us a lot to make the paper better.

W1: end-to-end manner

Thanks for your advice. Leaning in an end-to-end manner has many advantages, which will make the entire method more concise and practical. In our current solution (Figure 2), these two stages are learned separately. The key problem is the clustering algorithm, which is a heuristic algorithm, without the differentiable selection and sampling module. However, it is still possible for them to be jointly learned in an end-to-end manner. If the clustering module can be involved in a differentiable network, the end-to-end learning pipeline can be achieved. We will aim to make the learning process end-to-end in future work.

W2: 10% data

• We conduct the additional experiments using only 10% of videos.

It shows that with only 10% videos, the performance will not decrease so much, especially for AP$_\text{5}$ metrics.

• As for the performance gap when using the same 100% data, this phenomenon is common in most methods using weak annotations. However, we intend to emphasize the comparison between BA$^2$-Det with 100% data and BA-Det with 10% data. With the natural advantage that our method can use more unlabeled data, our method can be comparable to the fully supervised method.

W3: both 3D and 2D supervision

In Table 3, $N_\theta$ w/ 3D and $N_\theta$ w/ 2D mean 3D and 2D label assignment methods (Sec. 4.3) instead of with 3D and 2D supervision. We use both 2D and 3D supervision for the first row ($N_\theta$ w/ 3D) as well as the second row ($N_\theta$ w/ 2D). In detail, that is about prediction and (pseudo) ground truth matching strategy. $N_\theta$ w/ 3D assigns labels using generated 3D pseudo labels. Since we use a CenterNet-based detector, the implementation is supervising 2D/3D classification and regression predictions on the location corresponding to the projected 3D centers on heatmaps. That means some objects without 3D pseudo-labels are not supervised. For $N_\theta$ w/ 2D, we supervise 2D/3D classification and regression predictions on the location corresponding to the 2D centers on heatmaps. Those objects without 3D pseudo-labels only have 2D classification and regression loss. These two label assignment methods cannot coexist in a network. In other words, we cannot implement both $N_\theta$ with 3D and $N_\theta$ with 2D simultaneously.