UniDrive: Towards Universal Driving Perception Across Camera Configurations

Dear Reviewer DD4T,

Thanks for devoting time to this review and providing valuable comments. Below, we provide detailed responses to your comments and concerns. The revised manuscript has been uploaded, with all changes clearly highlighted in red for your convenience.

W1 & Q1: Results with more camera-only 3D detection methods.

A: Thanks for the constructive suggestion! We agree that evaluating our framework on additional methods would further demonstrate its generalizability. We previously prioritized BEVFusion-C as our baseline method as it introduced an efficient BEV pooling operator and runs faster than the other methods, making it widely adopted in real-world applications. Due to the limited time in the discussion, we have conducted experiments with BEVFormer as shown below. UniDrive demonstrated promising adaptability, we will extend to the other models including Sparse4D.

W2: It is not specifically described that how to adopt the proposed method to existing network. It would be helpful as the proposed method serves as a plug-and-play component.

A: Thank you for the valuable comment! We add a pipeline (as shown in Figure 3) in the revised paper and show how to use our module. Our module operates at the data input stage of the model as a preprocessing step.

Q2: It is recommended to compare the optimized virtual camera configuration to randomly selected configurations and evaluate performance across different camera settings.

A: Thanks for your insightful question! As shown in Figure 5 of the paper, we conducted experiments comparing the virtual camera configurations with optimization (w/ optimization) to randomly picked configurations without optimization (w/o optimization). The results demonstrate that the average transfer performance is significantly improved after optimization. This highlights the effectiveness of our proposed metric and optimization process in Sections 3.3 and 3.4.

We sincerely thank you again for the valuable comments provided during this review.

Dear Reviewer DD4T,

Thank you for your positive recognition! We sincerely appreciate your constructive comments and will continue to refine our paper. Thank you once again for your time and effort during this review!

Q2: While BEVFusion is popular, the paper focuses on vision-centric autonomous driving. It would be beneficial to include experiments with popular detectors in the camera domain, such as BEVFormer and PETR. These differ from methods like BEVFusion-C and BEVDet, as they are based on query-based transformer solutions.

A: Thanks for the constructive suggestion. We agree that evaluating our framework on additional methods would further demonstrate its generalizability. We previously prioritized BEVFusion-C as our baseline method as it introduced an efficient BEV pooling operator and runs faster than the other methods, making it widely adopted in real-world applications. Due to the limited time in the rebuttal, we have conducted experiments with BEVFormer as shown below. UniDrive demonstrated promising adaptability, we will extend to the other models including PETR.

Q3: Were the number of virtual cameras optimized together?

A: Thanks for your question! The number of virtual cameras are not optimized together. In our experiments, we only consider the situation of 6 virtual cameras. Considering the increased workload for optimization and benchmark experiments with camera numbers, we will do our best to provide the results before the discussion ends.

Q4: The paper discusses that Inconsistent Intrinsics have a significant impact. Since real autonomous vehicles also have inconsistent heights, could you include an experimental analysis?

A: Thanks for your question! We conduct experiments on identical configurations as follows. Overall, the inconsistent heights can impact performance, but the impact on performance is not as significant as the changes in intrinsic parameters.

We also conduct transfer experiments.

Q4.1: Were the heights of virtual cameras optimized together?

A: Thanks for your question! Heights are optimized together. However, for cameras within the same multi-camera configuration, the heights are consistent across all cameras.

Q5: Given the various camera configurations, would training on datasets that mix various configurations to optimize a unified virtual camera space lead to improved results?

A: Thanks for your insightful question! Mixing various camera configurations for training might not lead to improved performance. Even when different camera parameters appear on the same vehicle, we observe a slight performance degradation (as we discussed in Q4). However, it remains unclear whether the performance will improve when the dataset size becomes sufficiently large. If time permits, we will include experimental results to address this question before the discussion session ends. We appreciate your patience and interest in this aspect!

Q6: All experiments are conducted on simulation data. Incorporating transfer experiments with real datasets would be more convincing, like the real dataset transfer experiments conducted in 3D Domain Generalization/Adaptation.

A: Thanks for your insightful comment! We acknowledge that Incorporating experiments with real datasets would be more convincing. However, our experiments require identical scenes across varying camera setups. This controlled setup is critical for validating our approach and ensuring that our results are truly representative of cross-configuration performance. Since no publicly available datasets provide identical scenes with varying camera setups and collecting such real-world data goes far beyond the scope of this paper, we currently use CARLA as our platform.

To address the reviewer's concern, we provide some experimental results on real-world datasets as follows. While it is challenging to fully decouple the impact of scene variations on the experiments, our method has shown promising effectiveness to some extent, indicating its potential applicability.

We sincerely thank you again for the valuable comments provided during this review.

Dear Reviewer 3zcR,

Thanks for devoting time to this review and providing valuable comments. Below, we provide detailed responses to your comments and concerns. The revised manuscript has been uploaded, with all changes clearly highlighted in red for your convenience.

Q1: Does the virtual configuration optimization strategy actually work effectively? (Core concern: I am highly willing to increase the score if this issue is explained thoroughly.)

A: Thanks for your question! The virtual configuration optimization strategy in our UniDrive framework is effective, as demonstrated by the experimental results. We answers detailed questions below.

Q1.1: In certain camera configurations, the results from a single dataset in the paper are not as good as those of the original BEVFusion-C (e.g., 6x60 69.3 vs. 64.6, 6x80b 69.1 vs. 63.1). Why can't the virtual configuration optimization improve to achieve a virtual camera space as good as or better than the original BEVFusion-C?

A: Thanks for your question! To clarify, the problem our paper aims to solve is the generalization capacity of models transferred between varied camera configurations, rather than the performance on the identical camera configurations. Our optimization strategy is specifically designed to improve this generalization.

Since the virtual projection would introduce information losses, our method theoretically cannot do better than BEVFusion-C (e.g., 6x80b, 6x70, 6x60) for individual inference. Our model achieves the best overall performance accross different configurations though. We also observe experimental outperforming on certain configurations. The reasons are as follows:

1. Inconsistent intrinsics would introduce performance degradation (Section 4.4). We found that even for the original BEVFusion-C, when tested on identical configurations, the performance of 5x70+110 was significantly lower compared to 6x80 and 6x70. Our virtual projection method eliminates the impact of inconsistent intrinsics, which is why it achieves better performance on 5x70+110.

2. Impact of camera numbers. Our virtual projection uses 6 virtual camera planes. This provides richer information compared to setups with only 4-5 actual cameras, thus enhancing performance on 5x75 and 4x95.

Q1.2: In some camera configurations, the results of individual optimization are not as good as those from transfer learning, such as 60x80b (train on 60x80b 63.1 vs. train on 60x70 67.9). Why can't the virtual configuration optimization achieve a virtual camera space as good as or better than the transfer setting?

A: Thanks for your question! Our optimization strategy is designed to find a virtual camera plane that balances and minimizes the projection errors from all configurations. In our experiments, we perform virtual projection for both identical-configuration inference (e.g. train and test on 60x80b) and cross-configuration inference (e.g. train on 60x80b, test on 60x70). Our experimental results show that the average performance for transfer inference is improved significantly, which addressed our problem setting well.

To address your concern, for most circumstances, the identical-configuration performance is better than the transfer inference. However, the identical-configuration inference will also be impacted by the projection errors, and it is possible that the projection errors in identical-configuration inference might be larger than those of some transfer inference settings (e.g., the virtual camera configurations are more closed to the transferred configuration), leading to a performance drop.

Q1.3: For different camera configurations, what do the optimized Virtual Cameras look like?

A: Thanks for your questions! We add visualizations of the optimized Virtual Cameras in Figure 4. The optimized camera parameters are as follows.

Q1.4: How well does the original view warp to the virtual view? Are there any areas that are not properly warped? Can you provide visual results?

A: Thanks for your question. Overall, the warping from the original view to the virtual view is highly accurate. The majority of the scene is seamlessly projected. We add visualizations in the Appendix Figure 7.

Dear Reviewer 3zcR,

Thank you so much for your positive recognition and for raising your scores! We sincerely appreciate the time and effort you invested in reviewing our work, as well as your thoughtful feedback that helped us improve our submission.

Dear Reviewer rjYz,

Thanks for devoting time to this review and providing valuable comments. Below, we provide detailed responses to your comments and concerns. The revised manuscript has been uploaded, with all changes clearly highlighted in red for your convenience.

W1: The innovation is somewhat limited, as the framework mainly leverages established virtual projection and optimization techniques without presenting substantially novel concepts.

A: Thanks for this comment! We appreciate the reviewer's feedback regarding the perceived limitations in innovation. However, we would like to clarify several key contributions of our work:

1. The research problem we address is novel and real-world demanding. Specifically, we focus on the generalization capability of models trained on one camera configuration and deployed/transferred on a different one. To the best of our knowledge, no prior work has systematically explored this cross-configuration transferability.
2. We introduced a ground-aware depth assumption technique that enables effective virtual projections without requiring depth priors.
3. We developed a comprehensive benchmark designed to systematically study the problem of cross-configuration generalization. This benchmark provides a standardized way to evaluate and compare methods in this new research area.
4. The optimization strategy we developed is specifically tailored to our problem setting. It addresses the challenges inherent in cross-configuration deployment.

These contributions, taken together, provide a substantial step forward in addressing the challenges of deploying vision-based models across varying sensor configurations.

W2: The approach relies heavily on simulated environments (e.g., CARLA), raising concerns about its applicability and robustness in real-world conditions.

A: Thanks for your insightful comment! We acknowledge the reviewer's concerns regarding simulation. However, the use of simulation is not merely to simplify the experiments, but rather it is essential for ensuring identical scenes across varying camera setups without interference from other factors. This controlled setup is critical for validating our approach and ensuring that our results are truly representative of cross-configuration performance.

Since no public datasets provide identical scenes with varying camera parameters and collecting such real-world data goes far beyond the scope of this paper, we use CARLA as our platform, which is broadly used in the Autonomous Driving community [1-3].

[1] B. Zhang, X. Cai, J. Yuan, D. Yang, J. Guo, X. Yan, R. Xia, B. Shi, M. Dou, T. Chen, S. Liu, J. Yan, and Y. Qiao, ReSimAD: Zero-Shot 3D Domain Transfer for Autonomous Driving with Source Reconstruction and Target Simulation, In ICLR 2024.

[2] H. Shao, Y. Hu, L. Wang, G. Song, S. L. Waslander, Y. Liu, and H. Li, LMDrive: Closed-Loop End-to-End Driving with Large Language Models, In CVPR 2024.

[3] Q. Li, X. Jia, S. Wang, and J. Yan, Think2Drive: Efficient Reinforcement Learning by Thinking with Latent World Model for Autonomous Driving (in CARLA-v2), In ECCV 2024.

To address the reviewer's concern, we provide some experimental results on real-world datasets as follows. While it is challenging to fully decouple the impact of scene variations on the experiments, our method has shown promising effectiveness to some extent, indicating its potential applicability.

Q1: To what extent does the reliance on simulated data affect the generalizability of the results to real-world settings, particularly in urban driving scenarios?

A: Thanks for your insightful question!

1. The primary limitation of CARLA lies in the lower complexity of traffic participant behavior compared to the real world, which can impact autonomous driving decision-making evaluation. However, CARLA's sensor physics closely aligns with those of the real world. This ensures that the generalizability impact on our cross-camera configuration problem setting is minimal.
2. We used 6 maps in CARLA, ensuring the simulated scenarios were diverse, covering a wide range of urban driving conditions, including intersections, pedestrian crossings, varying traffic densities, and occlusion cases.
3. CARLA provides a photo-realistic environment with accurate physics, which is recognized and broadly used within the Autonomous Driving community, as referenced in the above citations.

Q2: Does the framework introduce any latency or computational overhead that could impact its feasibility for real-time applications in autonomous driving?

A: Thanks for your insightful question! Basically, our framework does not introduce latency to the driving perception model.

1. Our framework introduces a plug-and-play module. This module operates at the data input stage of the model as a preprocessing step, which does not affect the neural network's training and inference.
2. We implemented our module using CUDA, which ensures high computational efficiency and fast runtime performance.
3. The optimization is designed to find the optimal virtual camera parameters and is entirely independent of the model’s deployment. It does not interfere with model loading, training, or inference.
4. To further validate the feasibility of our framework, we provided a comparison of inference speeds. We tested on one RTX 4090 GPU.

We sincerely thank you again for the valuable comments provided during this review.

Thank you for your further explanation. I have raised my scores.

Dear Reviewer rjYz,

Thank you so much for your positive recognition and for raising your scores! We sincerely appreciate the time and effort you invested in reviewing our work. Thank you once again for your insightful feedback that helped us improve our submission!

Dear Reviewer sBiy,

Thanks for devoting time to this review and providing valuable comments. Below, we provide detailed responses to your comments and concerns. The revised manuscript has been uploaded, with all changes clearly highlighted in red for your convenience.

W1: As the authors noted, one main limitation is that all analyses were conducted in a simulated environment. While this controlled setting is useful for evaluating the method’s potential, it raises concerns about real-world applicability, as training-based detectors are often sensitive to dataset-specific factors. Given the paper’s real-world motivation, validation on actual datasets would be valuable.

A: Thanks for the valuable comment. We only used simulated data to isolate the effect of camera configurations since we can control the scenes to be exactly the same across different camera configurations.

However, for real-world evaluation, it is not possible to obtain identical scenes across varying camera configurations. This controlled setting is critical for validating our approach and ensuring that our results are truly representative of cross-configuration performance.

Still, we have conducted some preliminary experiments on real-world datasets as follows. While it is challenging to fully decouple the impact of scene variations on the experiments, our method has shown promising effectiveness to some extent, indicating its potential applicability.

W2: It is a little counterintuitive that the transformation from the original view to a virtual view may cause information loss but can bring improvements.

A: Thanks for the question. Due to the information loss introduced by the virtual transformation, there is indeed a slight performance degradation sometimes when evaluated on identical training configurations (e.g., 6x80b, 6x70, 6x60).

However, our work mainly aims to enhance model generalization. Specifically, the main objective is to ensure that a model trained under one camera configuration can be deployed/transferred to other configurations while maintaining similar performance.

Our method shows an improvement in average performance on transfer tasks across different camera configurations. We also observe performance improvements on certain configurations (identical to the training setup). The reasons are as follows:

1. Inconsistent intrinsics would introduce performance degradation (Section 4.4). We found that even for the original BEVFusion-C, when tested on identical configurations, the performance of 5x70+110 was significantly lower compared to 6x80 and 6x70. Our virtual projection method eliminates the impact of inconsistent intrinsics, which is why it achieves better performance on 5x70+110.
2. Impact of camera numbers. Our virtual projection uses 6 virtual camera planes. This provides richer information compared to setups with only 4-5 actual cameras, thus enhancing performance on 5x75 and 4x95.

Apart from the issues I posed above, I also have some suggestions that will not prevent this paper to be accepted if unresolved. In real-world applications, there are several datasets of different camera configurations. As the proposed method can leverage the virtual view to unify these different configurations, I am curious whether the proposed method can leverage these datasets to train a better-to-all detector on each dataset.

A: Thanks for your constructive suggestion! We appreciate the idea of leveraging real-world datasets with different camera configurations to train a unified detector using our proposed method. However, due to time constraints, we are currently unable to explore this direction within the scope of this submission. Utilizing multiple datasets with different camera configurations has the potential to improve performance. However, increasing the number of datasets can also increase projection errors due to greater variability across camera configurations. This highlights the importance of our optimization process, which is designed to find the optimal virtual camera space. We believe this is a promising avenue for future research. Thank you again for your insightful feedback!

We sincerely thank you again for the valuable comments provided during this review.