Unified Domain Generalization and Adaptation for Multi-View 3D Object Detection

(1) Discussion with RGB feature-based methods

Thank you for your constructive suggestion. As reported by DETR3D [4], CVT [5] and BEVFormer [6], RGB feature-based methods are considerably robust against calibration noise. However, these methodologies experience significant performance degradation in dynamic view shift environments, such as cross-domain scenarios where calibration changes dramatically, as shown in table below (Lyft $\rightarrow$ nuScenes).

Additionally, we explore how domain shift hinders stable 3D recognition in main text Sec. 3.2, Appendix C and Table 7. Although, fair comparisons are challenging due to architectural differences, we validate that our proposed modules quantitatively mitigate dramatic view changes in the cross-domain (up to 20.9% NDS gain).

(2) In-depth explanation of LEDA

We hope that additional explanation of LEDA in global response will help you understand better. To improve the development of 3D object detection, we design Label-Efficient Domain Adaptation (LEDA) framework that successfully adapts to novel targets by only a few data and extra parameters, introducing a practical solution for real-world Autonomous Driving. To this end, we conduct structural experiments (Rebuttal PDF Table 2 and 3) to explore the optimal architecture that is both effective and efficient. Also, we validate that our depth constraint method encourages transfer learning from pre-trained source potentials to novel targets as shown in Rebuttal PDF Table 1. Especially, by comparing our methodology with existing state-of-the-art models and evaluating NDS by efficiency across various setups (Rebuttal PDF Figure 1), we quantitatively showcase that our proposed methods successfully bridge the gap between the source and novel target using a limited amount of training budget (i.e., training parameters and data). It is noteworthy that our proposed methodology outperforms existing works by 1% on novel targets with only less than 20% extra parameters. Finally, we will obviously present and analyze in the revised version.

(3) Status of LiDAR-based cross-domain object detection

LiDAR object detection research includes addressing performance degradation caused by domain shifts. Wang et al.[1] proposed statistical normalization to mitigate the differences in object size distribution across various datasets. ST3D[2] leveraged domain knowledge through random object scale augmentation, and their self-training pipeline refined the pseudo-labels. STAL3D[3] extended ST3D by incorporating adversarial learning. These methods were all proposed under the assumption of target-aware conditions, which presents the limitation that they cannot be applied without prior access to the target domain. The table (Waymo $\rightarrow$ KITTI) below extracts a subset of experimental results from STAL3D. LiDAR-based methods were less affected by domain shifts compared to those of cameras, though their performance did not reach the oracle level.

Unlike LiDAR modalities, these approaches fail to address the domain shift problem due to the poor quality of pseudo-labels resulting from the inaccurate geometric information in image-based 3D detection. To address the domain generalization for camera-based systems, we propose a depth constraint module and introduce the LEDA module to achieve oracle-level performance with label efficiency in the target domain.

Plus, we will clearly address the weaknesses (1) More details of our methods and (3) More related works in the revised version.

Thank you.

[1] Wang, Chen, et al. Train in Germany, Test in The USA: Making 3D Object Detectors Generalize, Computer Vision and Pattern Recognition, 2020.
[2] Yang, Shi, et al. ST3D: Self-training for Unsupervised Domain Adaptation on 3D Object Detection, Computer Vision and Pattern Recognition, 2021.
[3] Zhang, Zhou, STAL3D: Unsupervised Domain Adaptation for 3D Object Detection via Collaborating Self-Training and Adversarial Learning, IEEE TRANSACTIONS ON INTELLIGENT VEHICLES, 2024. [4] Wang, Yue, et al. "Detr3d: 3d object detection from multi-view images via 3d-to-2d queries." Conference on Robot Learning. PMLR, 2022.
[5] Zhou, Brady, and Philipp Krähenbühl. "Cross-view transformers for real-time map-view semantic segmentation." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.
[6] Li, Zhiqi, et al. "Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers." European conference on computer vision. Cham: Springer Nature Switzerland, 2022.

I appreciate the authors' effort to answer my questions. I am convinced by the authors' responses about "RGB-features" and "In-depth explanation of LEDA".

However, the authors' response to "Status of LiDAR-based cross-domain object detection" made me sceptical about what the authors meant by "(1) These methods were all proposed under the assumption of target-aware conditions, which presents the limitation that they cannot be applied without prior access to the target domain." and (2) "LiDAR-based methods were less affected by domain shifts compared to those of cameras, though their performance did not reach the oracle level." in the rebuttal.

Regarding (1), did the authors mean that LiDAR-based self-training methods require target statistics, such as average object size, and self-training requires access to the target domain?

From the reviewer's understanding, the proposed method LEDA even requires labels directly from the target domain(although it's a comparably small amount, i.e. 1%- 5%), whereas the LiDAR-based self-training methods do not require such direct labels.

In the context that LEDA requires target labels and LiDAR-based methods require self-training, it seems to me that they both require prior access to the target domain.

Regarding (2), if the authors meant that LiDAR-based methods haven't met the Orcal performance in Waymo to KITTI adaptation task, I must say that this is false. For instance, DTS [1] and so on already outperformed Oracle with the justifiable reasons in Waymo to KITTI adaptation task.

I understand that the multi-view-based and LiDAR-based Domain adaptation/generalization methods have different characteristics and cannot be directly comparable. However, as a similar branch of work, the LiDAR-based domain adaptive 3D detection also needs to be addressed with proper description so that readers can understand the lines of similar works. Specifically, it does not sound very convincing to advertise that multi-view-based domain generalization is more challenging than LiDAR-based domain generalization because of the above-mentioned reasons for (1) and (2). I personally think that just mentioning LiDAR-based domain adaptive/generalization 3D detection works as relevant works without comparing which one is more challenging would make it clearer for readers to know what the relevant lines of research are.

[1] Density-Insensitive Unsupervised Domain Adaption on 3D Object Detection, Hu et al. CVPR 2023

We genuinely appreciate the reviewer's thoughtful and detailed feedback. Our proposed generalization technique, Multi-view Overlap Depth Constraint, does not require prior access to the target, unlike Statistical Normalization (SN) [1].
However, due to its limited performance, we designed LEDA to efficiently learn novel target knowledge, and we agree with your observation that LEDA indeed falls under the category of methods requiring direct labels from target domain. Furthermore, we acknowledge that our lack of insight into UDA for LiDAR 3D Object Detection led to an insufficient comparison in the answer (3) 'Status of LiDAR-based Cross-Domain Object Detection'. To address these issues, we will revise rebuttal phrases (1) and (2) as follows:
Regarding (1): “These methods were all proposed under the assumption of target-aware conditions, which presents the limitation that they cannot be applied without prior access to the target domain.” $\rightarrow$ “These methods adopt 'prior access to target' approaches (e.g., SN [1], self-training [2, 3]) to effectively align the domain gaps between the source and target.”
Regarding (2): “LiDAR-based methods were less affected by domain shifts compared to those of cameras, though their performance did not reach the oracle level.” $\rightarrow$ “LiDAR-based methods were less affected by domain shifts compared to those of cameras, though their performance did not reach the oracle level.”

More importantly, we observe that the challenges of LiDAR-based methods are also aligned with our problems in a certain aspect (i.e., label-efficient learning). LiDAR-based approaches utilize self-training strategies with pseudo-labeling; however, our UDGA framework introduces a method for fine-tuning LEDA using a small subset of novel target labels, and applying LEDA to LiDAR-based 3D object detection represents a promising direction for future work. We will clearly mention this discussion in the final manuscript and agree that this explanation will indeed enhance the readability of our paper.

Thanks for your thorough and professional comments on this comparison.

[1] Wang, Yan, et al. "Train in germany, test in the usa: Making 3d object detectors generalize." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
[2] Yang, Jihan, et al. "St3d: Self-training for unsupervised domain adaptation on 3d object detection." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.
[3] Hu, Qianjiang, Daizong Liu, and Wei Hu. "Density-insensitive unsupervised domain adaption on 3d object detection." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

(1) Explanation of Table 1

Thank you for your insightful review. We understand that there was some confusion regarding the misaligned results in Table 1 due to insufficient explanation. Since previous methods did not adopt consistent experimental methods, a fair comparison is only possible on the target domain (i.e., they did not evaluate the same model on both the source and target domains). To address this issue, we adopt their highest scores for comparison, as detailed in Appendix Table 5. We will make the necessary revisions to clarify this information and enhance the overall clarity of our manuscript.

(2) Discussion with other metrics

Note that, to ensure fair comparison, we adopt unified evaluation metric NDS as reported in DG-BEV and PD-BEV.
Waymo and Lyft calculate 3D AP by directly measuring the intersection over union (IoU) between predicted and ground truth bounding boxes. In contrast, NDS measures differences between predicted and ground truth objects leveraging various indicators such as translation error, size error, orientation error, and BEV AP. This approach enables capturing plausible failure cases (e.g., translation fails or orientation fails) and allows for practical analysis of domain shift issues, as shown in main paper Sec. 3.2 L#126-130, Sec. 4.2 L#207-209, Sec. 4.3 L#244-245 and L#250-251, and Appendix C L#505-513.

(3) Discussion with the 3D universal detection

Thank you for your meaningful discussion. [1] and [2] explore multi-domain learning and generalization to improve 3D object detection. Here, we present a brief overview as follows.

The Universal 3D Object Detection task is quite similar to our framework, and our proposed methods showcase considerable potential in this field (multi-source training and multi-target testing), as shown in main paper table 2. However, while universal training methods generalize external domain knowledge, we leverage internal knowledge, which do not rely on large-scale and diverse datasets. Furthermore, our framework can generalize without prior access to the target, making it more feasible to develop and deploy 3D detection in real-world scenarios.

(4) More details of Label-Efficient Domain Adaptation

We hope that this rebuttal will be helpful to you. We provide more details of LEDA in the global response, improving the motivation and potential of our proposed method. Although existing methods aim to generalize the domain shift between the source and the novel target, they often fail to provide a practical solution mainly due to unsatisfied generalization performance and show room for improvement. Additionally, costly resources exacerbate these issues and hinder the expansion of multi-view 3D object detection. To tackle these issues, we design a de-facto framework leveraging a efficient and effective learning strategy LEDA for multi-view 3D object detection.

First, we show that our depth constraint method smoothly deals with sensor misalignment between source and target domains and effectively boost the adaptation capability of LEDA (as shown in Rebuttal PDF Table 1). Also, our LEDA successfully transfers their own potentials to novel targets without forgetting previously learned knowledge as shown in main text Table 5. As a result, we note that our proposed framework impressively releases costly resource issues and unstable performance as shown in Rebuttal PDF Figure 1 and presents practical solutions for real-world autonomous driving.

We sincerely appreciate your constructive review and will address the points discussed above in the revised manuscript.

Thank you.

[1] Wu, Guile, et al. "Towards universal LiDAR-based 3D object detection by multi-domain knowledge transfer." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
[2] Zheng, Liangtao, et al. "Cross-Dataset Sensor Alignment: Making Visual 3D Object Detector Generalizable." Conference on Robot Learning. PMLR, 2023.

Dear Reviewer gahV,

We greatly appreciate your valuable efforts and professional feedback, which have indeed improved the quality of the final version of our manuscript. We’ve provided answers to your remaining concerns above, and it would be great to hear your feedback on our rebuttal so that we can further improve the final version.

Although the authors-reviewers discussion period is nearing its end, we are fully prepared to address any further questions you may have.

Best regards,

The Authors

(1) More details of LEDA.

Disconnected strategies
We hope that further explanations of LEDA in global response will enhance your understanding. We empirically observe that direct fine-tuning approaches (with a small fraction of data) often fail to align between source and target, mainly due to dynamic perspective shifts. To mitigate this issue, our proposed depth constraint method softly addresses perspective gaps and enhances adaptation capacity to novel targets. In Rebuttal PDF Table 1, LEDA (w/o $\mathcal{L_{ov}}$ and $\mathcal{L_{p}}$) struggles to bridge with novel target knowledge and show only 20.4% / 24.4% NDS / mAP gain. However, LEDA (w/ $\mathcal{L_{ov}}$ and $\mathcal{L_{p}}$) significantly transfer pre-trained knowledge to novel targets (34.2% / 42.5% NDS / mAP gain). As a result, we note that $\mathcal{L_{ov}}$ and $\mathcal{L_{p}}$ enable efficient learning by encouraging stable convergence even with tiny datasets.

Ablation of objectives
Additionally, we study the importance of our Optimization Objectives ($\mathcal{L_{ov}}$ and $\mathcal{L_{p}}$). Although, our depth constraint method effectively addresses domain gaps, it suffer from narrow overlap regions between adjacent views. To tackle this drawback, $\mathcal{L_{p}}$ effectively boosts stable recognition during both pre-training and fine-tuning phase. Specifically, $\mathcal{L_{p}}$ yields max 1.9% / 1.9% NDS / mAP gain during pre-training and max 2.1% / 1.6% NDS / mAP gain during fine-tuning. As a result, we demonstrate that our proposed two methods synergistically enhance UDGA performance.

In-depth explanation of LEDA.
In this paper, we advocate that depth consistency play a pivotal role to bridge domain gaps. Especially, we show that UDGA stably releases geometric differences from perspective view changes and encourages optimal BEV representation learning as shown in Rebuttal PDF Table 3. Precisely, UDGA achieves remarkable performance improvements of up to 10.0% in NDS and 9.5% in mAP in view transformation and BEV encoder layers, demonstrating its effectiveness.

(2) Comparison with Augmentation strategies.

We also measured generalizability of previous multi-view augmentation strategies as shown in Rebuttal PDF Tab.4. We adopt conventional augmentation strategies to multi-view 3D object detection as follows:

• GT sampling effectively address unbalanced labels by sampling ground truth objects in the 3D object detection.
• 2D aug. directly augment multi-view inputs (i.e., image resize, crop and paste, contrast and brightness distortion).
• 3D aug. globally rotate, re-scale and translate multi-view inputs and ground truths.
• Extrinsic aug. also adopts global yaw rotation towards the random direction
• CBGS re-balances to solve unbalanced ground truths.

These methods significantly enhance geometric understanding from input noises. However, in dynamic view changes (i.e., cross-domain), they still suffer from geometric inconsistency and show poor generalization capability. Moreover, various 2D approaches do not guarantee geometric alignments between 2D images and 3D ground truths and relevant studies have not been explored well, as reported in DG-BEV and [1]. To tackle these issues, we present a novel Multi-view Overlap Depth Constraint that effectively mitigate dynamic view changes in the cross domain. We also applied top 3 methods 2D, 3D, and extrinsic augmentation to our method and achieved state-of-the-art performance on both source and target. Unfortunately, we were unable to find any additional comparable papers for generalized multi-view 3D object detection. If you could provide us with relevant papers, we would be happy to analyze them together.

We sincerely appreciate your thoughtful review and will address above discussion in the revised version.

Thank you.

[1] Zhao, Yunhan, Shu Kong, and Charless Fowlkes. "Camera pose matters: Improving depth prediction by mitigating pose distribution bias." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
[2] Zheng, Liangtao, et al. "Cross-Dataset Sensor Alignment: Making Visual 3D Object Detector Generalizable." Conference on Robot Learning. PMLR, 2023.

Code release

We plan to release the source code upon acceptance.

Additionally, in this rebuttal, we provide further explanations, analyses, and results (please refer to global response and Rebuttal PDF). We hope that these materials will be helpful to you.

If you have any questions or need further discussion, please feel free to ask.

Thank you.