ReSimAD: Zero-Shot 3D Domain Transfer for Autonomous Driving with Source Reconstruction and Target Simulation

Q1: Figures 1 and 3 can be made more illustrative of the proposed paradigm.

A1: We are grateful to the reviewers for pointing out these problems in the manuscript. We have polished Figures 1 and 3 of the manuscript to further clearly our method.

 

Q2: The formatting of the paper needs adjustment, for example, ensure that figures and corresponding sections are not too far apart.

A2: We have noticed the placement issue with Tables 2, 3, and 4. Thank you very much for pointing that out, and we will make the necessary corrections in the upcoming updated version.

 

Q3: To eliminate artifacts in Point-to-mesh Implicit Reconstruction, it should be further explained how the authors performed point cloud registration when consolidating all frames from the corresponding sequence.

A3: We utilize the well-annotated Waymo dataset, which provides precise vehicle and LiDAR poses for each frame in all sequences. It is worth mentioning that, to avoid motion blur, prior to consolidating all frames, we first remove dynamic objects from the point cloud and only stitch together multiple frames of the static background. We further filter out outlier point clouds in the scene using point neighborhood statistics. Subsequently, we align the point cloud data from each frame in the current sequence to the starting frame of the sequence by projecting them using the LiDAR poses, thus achieving point cloud registration.

 

Q4: The interpretation of "Closed Gap" in Table 2 and its analysis are not provided in the paper.

A4: Thanks. As stated in caption of Table 2 of the manuscript, the "Closed Gap" denotes the detection accuracy drop rate closed by various methods along the Source Only and Oracle results, which is a common evaluation metric used by previous Unsupervised Domain Adaptation (UDA) works [1, 2]. "Closed Gap" can be calculated by: $(AP_{methods} - AP_{Source-only}) / (AP_{Oracle} - AP_{Source-only})$.

References:

[1] Jihan Yang, Shaoshuai Shi, Zhe Wang, Hongsheng Li, and Xiaojuan Qi. St3d: Self-training for unsupervised domain adaptation on 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10368–10378, 2021.

[2] Jiakang Yuan, Bo Zhang, Xiangchao Yan, Tao Chen, Botian Shi, Yikang Li, and Yu Qiao. Bi3d: Bi-domain active learning for cross-domain 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15599–15608, 2023b.

 

Q5: The paper does not mention the significance or the specific impact of "the matching of vehicle traffic flow density" mentioned in Section 4 (Mesh-to-point Rendering).

A5: Due to the strong correlation between vehicle movement and road network structure, vehicles generally cannot travel to non-driving areas. In this paper, we rely on the Waymo Open Dataset as the source domain dataset to complete background reconstruction, including roads. If the matching of vehicle traffic flow cannot be ensured in rendering, it may result in abnormal vehicle movement, such as collisions with surrounding buildings. Therefore, it is crucial to ensure the same density of vehicle traffic flow as the source domain dataset.

 

Q6: In Table 5, what is the reason of the significant gap between using zero-shot and oracle methods? The details regarding the sample quantity and other experimental settings can be supplemented in the paper.

A6: Thanks for your valuable suggestions. In Table 5, the main reason for the significant accuracy gap between using zero-shot and Oracle is that, the Waymo-to-nuScenes cross-dataset setting is very challenging for 3D UDA study, as reported in other works [1, 2, 3], since Waymo and nuScenes datasets have totally different LiDAR beam and scene layout.

References:

[1] Jihan Yang, Shaoshuai Shi, Zhe Wang, Hongsheng Li, and Xiaojuan Qi. St3d: Self-training for unsupervised domain adaptation on 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10368–10378, 2021.

[2] Jiakang Yuan, Bo Zhang, Xiangchao Yan, Tao Chen, Botian Shi, Yikang Li, and Yu Qiao. Bi3d: Bi-domain active learning for cross-domain 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15599–15608, 2023b.

[3] Zhuoxiao Chen, Yadan Luo, Zheng Wang, Mahsa Baktashmotlagh, Zi Huang. Revisiting Domain-Adaptive 3D Object Detection by Reliable, Diverse and Class-balanced Pseudo-Labeling. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 3714-3726).

We sincerely appreciate the reviewer's valuable comments. We have also tried our best to improve the quality and presentation of this manuscript, according to this reviewer’s comments.

Q1: Poor presentation.

A1: Thanks for your suggestions. The usage of English language and grammar of the revised manuscript has been double-checked thoroughly, and all the mistakes and errors which we can find have been revised and corrected. And the layout and order of the various tables and diagrams in the manuscripts have been better organized.

Q2: Technical novelty: This paper focuses on the dataset simulation process.

A2: In order to provide a clearer view of why and how the proposed ReSimAD is novel in autonomous driving community, we give an in-depth discussion of the following three aspects:

• Motivation: First, based on the fact that the model pre-trained on source-domain still faces serious performance drop when deployed on the target domain (e.g., different weather or different LiDAR beam), this paper provides a new view of the source-domain reconstruction followed by the new-domain simulation. Basically, we assume that the data from source-domain are well-annotated and massive. However, direct training on the data from the source domain may result in the model overfitting to the data distribution of the source domain. Thus, we produce 3D mesh from the source domain to decouple the domain characteristics, and simulate the target-domain-like data according to the produced 3D mesh. We are the first to achieve zero-shot target-domain 3D object detection.

• Framework: Second, directly generating the 3D mesh using the off-the-shelf reconstruction method such as VDBFusion is challenging and cannot bring satisfactory performance gains for zero-shot 3D object detection, as demonstrated in Table 2 and Figure 6 of the revised manuscript. This is mainly because we assume that only point clouds are available and the 3D scene-level reconstruction based on 3D points is very challenging, due to: a) Sparse 3D points, and b) Insufficient reconstruction evaluation.

  a) To address the issue of sparse 3D points, we aggregate all frames from the same scene sequence in Waymo dataset. Besides, we merge point clouds from side LiDARs to fill the blind spots around the vehicle, utilizing denser point clouds as input for reconstruction.

  b) To address the issue of insufficient reconstruction evaluation, we render the Waymo points based on the generated 3D mesh, and calculate the Root Mean Square Error and Chamfer Distance between the simulated Waymo points and the original Waymo points. The results in Table 6 of the Appendix indicate that the sequences we used have low errors and exhibit high reconstruction quality.

• Task: Finally, aiming to reduce the annotation cost of autonomous driving manufacturers for an unseen domain, we present the zero-shot target-domain 3D object detection task, where it is assumed that all annotated data from the source domain are available and no target domain data can be accessed. The presented task setting is practical, and as illustrated in Tables 2 and 4, we have verified that the proposed ReSimAD can achieve a satisfactory detection performance on the target domain compared to baseline models and UDA models.

Q3: Questionable cross-domain setting selection.

A3: Thanks for your concerns. However, we would like to clarify that the proposed cross-domain setting (namely, adaptation from Waymo to nuScenes) is rational, due to the following reasons:

1. Fair comparison with the existing Unsupervised Domain Adaptation (UDA) methods: The existing UDA works [1, 2, 3, 4, 5] employ the Waymo-to-KITTI, Waymo-to-nuScenes setting to study the model cross-domain ability under the 3D scenario. In order to make fair comparisons with previous cross-dataset works, we align their cross-dataset setting, selecting the Waymo-to-KITTI, Waymo-to-nuScenes, and Waymo-to-ONCE as the baseline setting.

2. Reconstruction process: In order to achieve high-quality reconstruction results, we have to merge multiple frames sampled from the same 3D sequence to perform the reconstruction process, which is also inspired by UniSim and StreetSurf. Different from these works, we also utilize the side LiDARs to fill the blind spots around the vehicle to boost the reconstruction quality. Besides, we are the first to explore the so-called one-for-all setting, meaning that the source-domain reconstruction process is only performed once using the proposed ReSimAD, and simulated on different downstream datasets including KITTI, nuScenes, and ONCE.

3. ONCE is a challenging dataset, publicly available in 2022. We also select ONCE as the target domain, evaluating the cross-dataset performance of the proposed ReSimAD. On Waymo-to-ONCE setting, we also observe some interesting findings as reported in Table 2 of the manuscript: a) The existing UDA work, such as ST3D[1], only achieve 68.13% mAP compared to 68.82% mAP of the source only (baseline result). This is mainly due to that the data distribution of ONCE dataset is very different from the Waymo dataset, including category definition and scene layout. b) Our ReSimAD exceeds the cross-dataset performance achieved by UDA work under the challenging Waymo-to-ONCE setting. Based on this, we believe that ReSimAD would pave the way for future exploration in zero-shot cross-dataset 3D study.

According to the reviewer’s comments, we have supplemented detailed descriptions and explanations on Pages 7and 8 of the revised manuscript.

References:

[1] Jihan Yang, Shaoshuai Shi, Zhe Wang, Hongsheng Li, and Xiaojuan Qi. St3d: Self-training for unsupervised domain adaptation on 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10368–10378, 2021.

[2] Jiakang Yuan, Bo Zhang, Xiangchao Yan, Tao Chen, Botian Shi, Yikang Li, and Yu Qiao. Bi3d: Bi-domain active learning for cross-domain 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15599–15608, 2023b.

[3] Zhuoxiao Chen, Yadan Luo, Zheng Wang, Mahsa Baktashmotlagh, Zi Huang. Revisiting Domain-Adaptive 3D Object Detection by Reliable, Diverse and Class-balanced Pseudo-Labeling. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 3714-3726).

[4] Yi Wei, Zibu Wei, Yongming Rao, Jiaxin Li, Jie Zhou, Jiwen Lu. Lidar distillation: Bridging the beam-induced domain gap for 3d object detection. In European Conference on Computer Vision (pp. 179-195).

[5] Jiageng Mao, Shaoshuai Shi, Xiaogang Wang, Hongsheng Li. 3d object detection for autonomous driving: A review and new outlooks. arXiv preprint arXiv:2206.09474.

Thank you for your comments. We provide discussions and explanations about your concerns as follows.

Q1: Clarifying issues related to mesh reconstruction.

A1: The main contribution claimed in this paper is the unified reconstruction-simulation-perception paradigm that addresses the domain shift issue. Considering previous work, StreetSurf [1], current advanced reconstruction methods are already capable of achieving high-quality point cloud rendering in the target domain, and the specific reconstruction methods are not our main contribution. In the mesh reconstruction section, we opt for a LiDAR-only setting, which allows us to be unaffected by lighting conditions and to obtain a richer and larger dataset of the scene. Drawing inspiration from StreetSurf [1], the reconstruction input is derived from LiDAR rays, and the output is the predicted depth.

On each sampled LiDAR beam $\mathbf{r}\_{\text{lidar}}$, we apply a logarithm L1 loss on $\hat{D}^{(\mathrm{cr}, \mathrm{dv})}$, the rendered depth of the combined close-range and distant-view model:

$${\mathscr{L}}\_{geometry} = \ln \\left ( |\hat{D}^{(\mathrm{cr}, \mathrm{dv})} (\mathbf{r}\_{lidar}) - D(\mathbf{r}\_{lidar }) | + 1\\right )$$

To achieve improved reconstruction quality, we aggregate all frames from the same scene sequence in the Waymo dataset. Additionally, we merge point clouds from side LiDARs to fill the blind spots around the vehicle, utilizing denser point clouds as input for reconstruction.

Simultaneously, we have conducted a quantitative evaluation of the reconstruction. Within the target domain LiDAR FOV (which is smaller than the source domain LiDAR FOV), we sample and calculate the Root Mean Square Error and Chamfer Distance for reconstruction errors. The results in Table 6 of the Appendix indicate that, the sequences we used have low errors and exhibit high reconstruction quality.

Due to space constraints, the evaluation of reconstruction quality and the explanation of LiDAR rendering are placed in the Appendix section. We greatly appreciate your review of this material.

 

Q2: Clarifying issues related to LiDAR rendering. What the lidar rendering formulation is?

A2: In the LiDAR rendering section, we utilize the official specifications of the LiDAR used in the target domain to complete the modeling of scanning characteristics, while also considering the simulation of physical properties. The key technical details of this part are fully elaborated in previous work on LiDAR simulation [2]. A brief overview is also provided in Appendix A.2.

 

Q3: Comparation with point clouds rendered from VDBF.

A3: Regarding your mention of comparing mesh quality with the VDBF method, it is true that we have only conducted a preliminary comparison in terms of visualization. However, considering the numerous voids and artifacts introduced by VDBF, it is evident that this would impact the quality of LiDAR rendering in our downstream section, making it challenging to achieve the results of our current reconstruction method.

 

Q4: Why points generated from mesh are meaningful for detection?

A4: We decouple the sensor characteristics of the source domain through mesh reconstruction and employ sensors conforming to the target domain characteristics during the LiDAR rendering. This leads to generated points with smaller domain differences from the target domain, which is more conducive to learning the features of the target domain point cloud. Consequently, this can greatly enhance the performance of downstream tasks.

 

Q5: Confusion issue of Figure 2.

A5: With regards to Figure 2, our original intention was to intuitively illustrate that, the LiDAR data obtained through our method closely resembles the real data, without any noticeable variations between domains. We acknowledge your suggestion and will remove this figure in the upcoming updated version.

 

Q6: Confusion issue of Table 1.

A6: As for Table 1, which discusses the Waymo-like dataset, it refers to the target domain Waymo dataset obtained by rendering on the mesh reconstructed from the source domain Waymo dataset. We use this to assess reconstruction errors, as detailed in Appendix Table 6. If this causes any ambiguity, we will provide a more detailed explanation in the legend of the upcoming version.

References:

[1] Jianfei Guo, Nianchen Deng, Xinyang Li, Yeqi Bai, Botian Shi, Chiyu Wang, Chenjing Ding, Dongliang Wang, and Yikang Li. Streetsurf: Extending multi-view implicit surface reconstruction to street views. arXiv preprint arXiv:2306.04988, 2023.

[2] Xinyu Cai, Wentao Jiang, Runsheng Xu, Wenquan Zhao, Jiaqi Ma, Si Liu, and Yikang Li. Analyzing infrastructure lidar placement with realistic lidar simulation library. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 5581–5587, 2023. doi: 10.1109/ICRA48891.2023.10161027.

Q1. Thanks for clarification. Now I understand the part is based upon another paper [1].

Q2. Thanks for clarification. Now I understand the part is based upon another paper [2].

Q3. Thanks for clarification. Now I understand that the VDBF quality is so bad that it does not deverse a quantitative comparisons.

Q4. Now I understand that although the mesh quality of ReSimAD seems low, e.g., bottom row of Fig.5, it may be enough for LiDAR simulation.

Q5&6. Thanks, although I don't see the updated version. There are still 24 hours. Looking forward to these updates before the update panel closes.

Dear Reviewer:

Thanks for your precious time and valuable suggestions to improve the quality of our paper. We have also carefully revised the manuscript, with modifications marked with blue text.

We hope that our revision makes our presentation clearer and covers your concerns. We are happy to discuss any remaining questions about our work.

Best regards, Authors of Paper 1573

Q1: Effectiveness on larger detection models.

A1: Thanks for your valuable comment. Considering that we are the first work to explore the zero-shot cross-dataset ability of 3D detection model, we align with previous cross-dataset works [1, 2, 3, 4], where they often employ PV-RCNN and PV-RCNN++ as the baseline models (as reported in Table 1 of their papers). The purpose of employing the PV-RCNN model is to make a fair comparison with previous cross-dataset works [1, 2, 3, 4].

References:

[1] Jihan Yang, Shaoshuai Shi, Zhe Wang, Hongsheng Li, and Xiaojuan Qi. St3d: Self-training for unsupervised domain adaptation on 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10368–10378, 2021.

[2] Jiakang Yuan, Bo Zhang, Xiangchao Yan, Tao Chen, Botian Shi, Yikang Li, and Yu Qiao. Bi3d: Bi-domain active learning for cross-domain 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15599–15608, 2023b.

[3] Zhuoxiao Chen, Yadan Luo, Zheng Wang, Mahsa Baktashmotlagh, Zi Huang. Revisiting Domain-Adaptive 3D Object Detection by Reliable, Diverse and Class-balanced Pseudo-Labeling. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 3714-3726).

[4] Yi Wei, Zibu Wei, Yongming Rao, Jiaxin Li, Jie Zhou, Jiwen Lu. Lidar distillation: Bridging the beam-induced domain gap for 3d object detection. In European Conference on Computer Vision (pp. 179-195).

Q2: It would still be valuable to see whether the proposed method could benefit from increasingly more target-domain information.

A2: We are very grateful to the reviewer for the insightful suggestion. According to the reviewer’s comment, we have illustrated the experimental results using ReSimAD under the fully-supervised target-domain setting. As reported in Table 3 of the revised manuscript, ReSimAD can provide better network initialization parameters, compared to the baseline trained from scratch. Specifically, PV-RCNN model is firstly trained from scratch on the target domain (nuScenes) using fully annotated data, and its performance is 53.07% on the target domain. By comparison, PV-RCNN model is initialized using the checkpoint pre-trained by the proposed ReSimAD, and fine-tuned on the target domain. The target-domain performance ($AP_{BEV}$) can outperform the one trained from scratch, achieving 54.48%.

Q3: Expected to compare against more recent SOTA DA/UDA approaches to further support the evaluation.

A3: Thanks for the valuable recommendation. As you observe in Table 1, there is still a performance gap between our ReSimAD (81.01%) and the UDA method (84.10%) on the target domain (KITTI). This is mainly due to that, our ReSimAD aims to zero-shot cross-dataset setting, where it is assumed that no data from the target domain are available. But for the UDA method, they assume that the data from the target domain are available for model training. Considering this, we select the ST3D [1], the representative 3D UDA method, to show the cross-dataset detection accuracy of the network under different input conditions (zero-shot or UDA settings). Besides, we select two zero-shot methods, which are CARLA-default and Sensor-like, as the comparison baselines. The experimental results are shown in Table 2.

Reference:

[1] Jihan Yang, Shaoshuai Shi, Zhe Wang, Hongsheng Li, and Xiaojuan Qi. St3d: Self-training for unsupervised domain adaptation on 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10368–10378, 2021.