FCVL: Fourier Cross-View Learning for Generalizable 3D Object Detection in Bird’s Eye View

We have updated the reply above, which should also have addressed your comments. Please checked the updated version.

We would like to emphasize the significance of improving model generalization with single domain data once again.

1. Generalization ability refers to how well a model performs in the scenes it has not seen before. Since they are "unseen scenes“, this is hard to be solved by collecting more data alone. Data is infinite, and it is impossible to collect data on all possible scenarios.

2. The setting of single domain generalization is a realistic yet more challenging scenario. As deep neural networks are very good at memorizing all the training data, single domain generalization can truly reflect an algorithm's generalization ability in real-world environments. Humans beings do not have to be taught on all luminace and weather conditions to learn to drive.

3. For multi-view 3D detection task , the collection of data and annotating 3D bounding boxes is much more time-consuming and labor-intensive than common 2D classification tasks. There is no dispute about this point. Studying the single-domain generalization of multi-view 3D detection , (1) reduces the dependence on more annotated data, (2) allows the model to better cope with complex and variable driving environments, which is of great significance for the safety and reliability of autonomous driving.

We would like to thank the reviewer for taking the time to read our submission and for valuable feedback concerning our paper.

“Summary: This paper studies how to boost the generalization of monocular 3D object detectors, like BEVFormer, when only a single domain of data is available. ”

Firstly, we kindly remind the reviewer that it is not monocular 3D object detection. It is a multi-view 3D object detection approach, which is often referred to as the BEV object detections in autonomous driving. This is indeed a very different area: the same target object could appear in adjacent perspectives, resulting in cross-view situation (please kindly check Figure 3 in the paper) and one of our main contributions is leveraging the natural cross-view features from multiple inputs to enhance generalization, utilizing its multi-view nature.

W1

Although companies may collect data from multiple domains, it is still meaningful to study how to improve the generalization ability of models using only data from a single domain. For research purposes, we are interested in how to approaching human-level generalization abilities, which means you do not need the whole world's data for driving. Indeed, humans do not have to experience every domain (daytime, night, rainy, cloudy, and snowy) to learn to drive, which is in stark contrast with the common practice of collecting an enormous amount of data.

Furthermore, for democratic purposes, we may not wish the autonomous driving to be solely controlled in a small group. We wish with the development of the generalization research, more people including resource-limited non-profit academic institutions can produce generalizable and safe autonomous driving systems which could benefit every one. Besides, in certain situations, such as rapid deployment in emergencies or specific fields, there might indeed be only data from a single domain available. For example, if we would like to quickly develop an autocar for earthquake rescues, we cannot expect too many data in this kind of domains. This further justifies the necessities of this research direction.

How to improve the generalization ability of models with limited data in situations where resources are constrained or specific domain data is difficult to obtain is a worthwhile research question. Focusing on single-domain generalization (SDG) not only addresses practical constraints but also provides a more robust evaluation of model adaptability.

W2

Thank you very much for your comments. We will improve the fluency of this paper in later versions.

W3

Thank you for the constructive comment. Frequency-domain based methods and their variants are actually widely in commercial autonomous driving vehicles especially when the computation power is extremely limited. For example, in the image processing ISP pipelines, they can be used for denoising the analog signals. We hope our method could potentially be widely adopted in the future.

W4

Thank you for the feedback. Using mathematical formulations here is to help providing further theoretical understanding of the working mechanisms for this method. This can provide readers more information in addition to the empirical results present in the paper.

We thank again for the reviewer's helpful suggestions.

We would like to thank the reviewer for taking the time to read our submission and for valuable feedback concerning our paper.

W1 The proposed method has seveal hyperparameters to tune, in the paper, the authors did not specifically point it out how to make it work. If the authors could elaborate how to set up these hyperparameters and how do they affect the final performance, it will be better.

Thank you for the constructive suggestion. These hyperparameters mainly consist of two parts: the probability of augmentation and the intensity of augmentation (more details can be found in the methodology (Sec2.2,2.3) and hyperparameter analysis (Sec4.3)). As the probability $p$ and the intensity $\alpha$ increase, this method can create more samples with more diverse styles to improve the generalization. As is shown in the Fig.4(c) in hyperparameters analysis: initially, as the probability $p$ and the intensity $\alpha$ increase, the out-of-domain performance gradually improves. After reaching a certain level of probability and intensity, further changes in the parameters will no longer cause drastic changes in OOD performances, indicating that the model is stable against hyper-parameter misspecifications as long as the hyper-parameters are within reasonable ranges.

W2 The authors could discuss more on the failure cases or limitations

We have added subsection Limitations in the revised paper.

Currently, this method involves several hyperparameters that require fine-tuning. In our future work, we aim to explore supplementary techniques to minimize the time spent on hyperparameter optimization and to further augment the performance of FCVL. For snowy weather, we have already improved by 10 points, but the performance in snowy conditions is still much worse compared to the performance in other scenarios such as low light. Consequently, there is a substantial potential for enhancement in adverse weather conditions.

W3 additional evaluation results using different datasets

Thank you for the constructive suggestion. Following this suggestion, we have introduced a new state-of-the-art (SOTA) baseline called Far3D and conducted thorough comparative experiments on the updated dataset, Argoverse 2. Additional results across more datasets have been included in Table 2 of the revised paper. On Argoverse 2, our method still has a significant advantage over other domain generalization methods. More visualized results are shown in the anonymous link. https://drive.google.com/file/d/1p7ATTCM55GQNH7JltW6dJsBF3NWJaNXi/view?usp=drive_link

Secondly, we have assessed our method's performances in real-world scenarios, particularly in conditions with abrupt light changes. Our method was tested at night using a model trained solely on daylight samples. We have visualized several detection results from image sequences captured in real-world environments and provided an analysis on pages 19-20 in Appendix F.2 (Lines 1022-1050) of the revised paper. We have also uploaded the visualization results in the anonymous link. https://drive.google.com/file/d/15vYYmeviYDbLy9ugJOsfQ55z5XB3QBS2/view. Figure 10 illustrates a compelling example of our model's ability to robustly handle rapid environmental changes, such as fluctuations in lighting conditions. The model, trained exclusively on daylight samples with the proposed FCVL, demonstrates consistent performances in areas with dense vehicle traffic and intense lighting, effectively detecting targets under these challenging conditions. As the vehicles transition into areas with normalized lighting, the model's detection capabilities return to standard operation. Notably, despite being exposed only to typical daylight samples, the model, augmented with our proposed FCVL, performs significantly better even under the extreme lighting variations encountered at night.

Thank you again for your time and suggestions. Please do not hesitate to contact us if you have any further questions.

We would like to thank the reviewer for taking the time to read our submission and for valuable feedback concerning our paper.

W1 The Fourier Cross-View Semantic Consistency Loss constructs positive and negative samples by splitting adjacent perspectives into halves. However, this approach has a limitation: each segment contains not only foreground objects but also complex background interference, which requires further analysis.

This is indeed a valuable suggestion. In our experiments, we find that the target objects across different viewpoints occupy most of the foreground area (as we have split each perspective into halves). We conducted a separate experimental analysis of the Cross-View Loss. Under the current experimental setup, our cross-view learning loss has achieved good experimental results (Table 9). We have visualized more cross-view samples with GradCam, it can be seen that the same object from different perspectives has a stronger feature response compared with background. (https://drive.google.com/file/d/155vOi-sRGqk4P-DIYWJbXhgMLNiEKjTH/view?usp=drive_link)

Such cross-view targets are common in multi-camera inputs, providing natural opportunities to observe the same object from different perspectives. To exploit this, we propose the Fourier Cross-View Semantic Consistency Loss to help the model learn more domain-invariant features from adjacent views.

W2 supplement the analysis with commonly used feature distribution visualizations

Thank you for your constructive comment. We employ t-SNE for visualizing the Bird's Eye View (BEV) features across various domains, with the results presented in Section 4.5 of our paper. We have also uploaded the visualization results in the anonymous link. https://drive.google.com/file/d/199U0ufsXO0QeGGVWu9K4Vv2uxUaLR3fG/view?usp=drive_link. The visualization reveals that the features extracted from BEVDet for different domains are not only distant from one another but also loosely dispersed throughout the feature space. In contrast, after applying our FCVL optimization, the features from the four domains are more tightly clustered and interconnected, aligning with the principles of augmentation graph theory. FCVL enhances the connectivity of the augmentation graph between the source and unseen domains, thereby significantly bolstering the model's generalization capabilities.

Q2 How is the number of positive and negative samples set in the Fourier Cross-View Semantic Consistency Loss?

The ratio of positive to negative sample pairs is 1:10.

Q3 additional evaluation results using different datasets

Thank you for the constructive suggestion. Following this suggestion, we have introduced a new state-of-the-art (SOTA) baseline called Far3D and conducted thorough comparative experiments on the updated dataset, Argoverse 2. Additional results across more datasets have been included in Table 2 of the revised paper. On Argoverse 2, our method still has a significant advantage over other domain generalization methods. More visualized results are shown in the anonymous link. https://drive.google.com/file/d/1p7ATTCM55GQNH7JltW6dJsBF3NWJaNXi/view?usp=drive_link

Secondly, we have assessed our method's performances in real-world scenarios, particularly in conditions with abrupt light changes. Our method was tested at night using a model trained solely on daylight samples. We have visualized several detection results from image sequences captured in real-world environments and provided an analysis on pages 19-20 in Appendix F.2 (Lines 1022-1050) of the revised paper. We have also uploaded the visualization results in the anonymous link. https://drive.google.com/file/d/15vYYmeviYDbLy9ugJOsfQ55z5XB3QBS2/view. Figure 10 illustrates a compelling example of our model's ability to robustly handle rapid environmental changes, such as fluctuations in lighting conditions. The model, trained exclusively on daylight samples with the proposed FCVL, demonstrates consistent performances in areas with dense vehicle traffic and intense lighting, effectively detecting targets under these challenging conditions. As the vehicles transition into areas with normalized lighting, the model's detection capabilities return to standard operation. Notably, despite being exposed only to typical daylight samples, the model, augmented with our proposed FCVL, performs significantly better even under the extreme lighting variations encountered at night.

Thank you again for your time and suggestions. Please do not hesitate to contact us if you have any further questions.

As for 1& 2, the phenomenon of objects crossing viewpoints exist naturally and very common. We propose to utilize this point and improve the ood performance. In our experiments, we find that the target objects across different viewpoints occupy most of the foreground area (as we have split each perspective into halves). We conducted a separate experimental analysis of the Cross-View Loss. Under the current experimental setup, our cross-view learning loss has achieved good experimental results (Table 9). We have visualized more cross-view samples with GradCam, it can be seen that the same object from different perspectives has a stronger feature response compared with background. (https://drive.google.com/file/d/155vOi-sRGqk4P-DIYWJbXhgMLNiEKjTH/view?usp=drive_link) . Such cross-view targets are common in multi-camera inputs, providing natural opportunities to observe the same object from different perspectives. To exploit this, we propose the Fourier Cross-View Semantic Consistency Loss to help the model learn more domain-invariant features from adjacent views.

The main advantage of surround-view input is its ability to provide more comprehensive environmental information, which is very helpful for detecting and tracking objects across different views. There are also other works may not about generalization studies but related to cross-view features. This work[1] is about generating surround-view data and they design “a cross-view attention module, ensuring consistency across multiple camera views”. These can verify that the phenomenon of objects crossing viewpoints exist naturally and very common.

[1] Gao, Ruiyuan, et al. "Magicdrive: Street view generation with diverse 3d geometry control.” ICLR,2024

As for 3, we would like to emphasize the significance of improving model generalization with single domain data once again.

We would like to emphasize the significance of improving model generalization with single domain data once again.

1. Generalization ability refers to how well a model performs in the scenes it has not seen before. Since they are "unseen scenes“, this is hard to be solved by collecting more data alone. Data is infinite, and it is impossible to collect data on all possible scenarios.

2. The setting of single domain generalization is a realistic yet more challenging scenario. As deep neural networks are very good at memorizing all the training data, single domain generalization can truly reflect an algorithm's generalization ability in real-world environments. Humans beings do not have to be taught on all luminace and weather conditions to learn to drive.

3. For multi-view 3D detection task , the collection of data and annotating 3D bounding boxes is much more time-consuming and labor-intensive than common 2D classification tasks. There is no dispute about this point. Studying the single-domain generalization of multi-view 3D detection , (1) reduces the dependence on more annotated data, (2) allows the model to better cope with complex and variable driving environments, which is of great significance for the safety and reliability of autonomous driving.

We would like to thank the reviewer for taking the time to read our submission and for valuable feedback concerning our paper.

W1 Since the primary augmentation occurs on 2D images, it can be difficult to differentiate it from standard 2D augmentation techniques. A more detailed explanation of these methods might enhance clarity for readers less familiar with them.

Thank you for the valuable suggestion. In the introduction, we have outlined the challenges associated with directly applying standard 2D augmentation techniques—such as geometric transformations, style transfer, and data generation to the Bird's Eye View (BEV) task. Following your guidance for better clarity, we have included a more comprehensive overview of these 2D methods in Appendix A of the revised manuscript, owing to spatial constraints within the main text. Additionally, we have elaborated on the distinctions between our approach and other frequency-domain methodologies in Appendix A. This indeed improves the clarity of our paper.

W2 & Q2 additional evaluation results using different datasets

Thank you for the constructive suggestion. Following this suggestion, we have introduced a new state-of-the-art (SOTA) baseline called Far3D and conducted thorough comparative experiments on the updated dataset, Argoverse 2. Additional results across more datasets have been included in Table 2 of the revised paper. On Argoverse 2, our method still has a significant advantage over other domain generalization methods. More visualized results are shown in the anonymous link. https://drive.google.com/file/d/1p7ATTCM55GQNH7JltW6dJsBF3NWJaNXi/view?usp=drive_link

Secondly, we have assessed our method's performances in real-world scenarios, particularly in conditions with abrupt light changes. Our method was tested at night using a model trained solely on daylight samples. We have visualized several detection results from image sequences captured in real-world environments and provided an analysis on pages 19-20 in Appendix F.2 (Lines 1022-1050) of the revised paper. We have also uploaded the visualization results in the anonymous link. https://drive.google.com/file/d/15vYYmeviYDbLy9ugJOsfQ55z5XB3QBS2/view. Figure 10 illustrates a compelling example of our model's ability to robustly handle rapid environmental changes, such as fluctuations in lighting conditions. The model, trained exclusively on daylight samples with the proposed FCVL, demonstrates consistent performances in areas with dense vehicle traffic and intense lighting, effectively detecting targets under these challenging conditions. As the vehicles transition into areas with normalized lighting, the model's detection capabilities return to standard operation. Notably, despite being exposed only to typical daylight samples, the model, augmented with our proposed FCVL, performs significantly better even under the extreme lighting variations encountered at night.

Thank you again for your time and suggestions. Please do not hesitate to contact us if you have any further questions.

Thank you for your thoughtful response, which addressed most of my concerns. After carefully reviewing the comments from the other reviewers, I still have some concerns that need to be addressed. While the rebuttal argues that single-domain generalization accurately reflects an algorithm's generalization ability in real-world environments, I believe this should be validated through experiments rather than relying solely on theoretical reasoning.

We would like to thank the reviewer for taking the time to read our submission and for valuable feedback concerning our paper.

W1&W2 training and inference efficiency

Thank you for the valuable question. We add a new subsection "Efficiency Analysis" (Sec4.4, Page 9) in the revised paper. In this part, we make more analysis to delve into the proposed FCVL. We investigate how the method scales with increasing image resolution and computational complexity for practical implementation. The results are listed in Table 5 of the revised paper. As it can been seen, (1) at larger image scales, FCVL can still significantly improve the model's generalization performance. (2) There will be a slight increase in training time (+0.11s per training step) during the training phase. However, as the FCVL is only used during the training phase, it introduces no latency in inference phase. Without the need for more time-consuming and costly data collections, the FCVL can improve the generalization performance almost for free. (3) Besides, NVIDIA provides a library called cuFFT to accelerate the Fourier Transform. Many hardware vendors also provide Fourier transformation accelerations. Thus, the speed of our method can be further improved in the training phase.

W3 In table.1, it seems FCVL didn't show superior performance on the normal validation set.

Thank you for the constructive feedback. There are already many excellent works on improving the in-distribution performances, mostly from a neural architectural approach. Previous works mainly focus on improving the network’s capacity to improve the performances on the normal validation set, i.e. the in-distribution evaluation set.

In this paper, we provide a novel perspective on improving the performances on the data distributions unseen in the training phase. This arguably provides a more challenging but practical setting for BEV object detection methods evaluations. This is because we cannot collect all data in the world including all colors, shapes, combinations of objects and backgrounds images on the road. As shown in the paper, we achieved competitive performances on the normal validation set, which demonstrates that our method is in par with the baseline methods. We also achieved significant performances on the OOD set (increased by 5-6 %), this indicates that our method can significantly improve existing BEV object detectors’ generalization performances. Our approach is indeed orthogonal to previous approach which mostly focus on the neural network architectures.

W4 Missing references

Thank you for the reminder. The references have already been supplemented in the revised paper.

Q1 What is the rationale behind the specific choices of frequency domain transformations? Were other alternatives considered?

Thank you for the insightful question. We opt to operate in the frequency domain as it enables us to distinctly separate phase components, which include semantics and causal cues, from amplitude components, which encompass styles and non-causal cues. Causal cues are pivotal for bolstering the model's resilience to variations across disparate domains. While previous works propose methods to extract causal features or decouple them through sophisticated network architectures, these works focus on the image classification cases. For example, CIRL [1] creates augmented images through a causal intervention module that targets non-causal factors, and AGFA [2] employs adversarial training between a classifier and an amplitude generator to produce a challenging domain for model adaptation. We have tried to adapt these methods on the BEV object detections but they do not show good performances in the experiments. The reason is that for image classification tasks, the key information for classification is often concentrated in the central part of the image. Thus, it is closer to the ideal theoretical settings. However, the BEV object detection tasks are much more complex. Most of parts of the images are backgrounds. Our approach stands out for its stability and efficacy without the need for additional module design or specialized training regimens. In contrast to these techniques, our method offers a straightforward, plug-and-play solution that yields superior generalization outcomes with greater efficiency, particularly beneficial given the intricate nature of BEV-based 3D object detection models. As highlighted in our efficiency analysis, our method is utilized exclusively during the training phase and is not deployed during the inference phase, thus incurring no extra computational overhead for real-world applications.

[1].Lv, Fangrui, et al. “Causality inspired representation learning for domain generalization.” Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

[2] Kim, Minyoung, Da Li, and Timothy Hospedales. “Domain generalization via domain adaptation: An adversarial Fourier amplitude approach.” arXiv preprint arXiv:2302.12047 (2023).

Q2 Does the method maintain its effectiveness when dealing with rapid environmental changes (e.g., entering/exiting tunnels)?

Thank you for the valuable suggestion. We apologize that we were unable to acquire data for scenarios involving entering and exiting tunnels in the limited time frame. However, we have conducted tests on similar scenarios characterized by abrupt luminance condition changes. We have visualized certain detection results from image sequences captured in real-world settings and provided a detailed analysis on pages 19-20 of the revised paper, specifically in Appendix F.2 (Lines 1022-1050). We have also uploaded the visualization results in the anonymous link. https://drive.google.com/file/d/15vYYmeviYDbLy9ugJOsfQ55z5XB3QBS2/view. Our method was tested under nighttime conditions, despite the model being trained exclusively on daylight samples. Figure 10 illustrates a compelling example of our model's ability to robustly handle rapid environmental changes, such as fluctuations in lighting conditions. Trained solely on daylight samples with the proposed FCVL, the model demonstrates consistent performance in dense vehicle areas with intense lighting, successfully detecting targets even under these challenging conditions. As the vehicles move into areas with normalized lighting, the model's detection capabilities return to normal. Notably, despite being trained only on typical daylight samples, our model, augmented with FCVL, performs better even under the extreme lighting variations encountered at nights.

Thank you again for your time and considerations. Please do not hesitate to contact us if you have any further questions.