CALICO: Self-Supervised Camera-LiDAR Contrastive Pre-training for BEV Perception

We are glad that the reviewer found our method effective and comprehensive. We appreciate the opportunity to address the points you have raised.

How does it identify the semantic-less and semantic-rich points? How does it calculate the 4th dimension of the points? What is the range of the 4th dimension of the points?

In our approach, the differentiation between semantic-rich and semantic-less points is achieved through a novel unsupervised technique termed semantic pooling. This process is designed to identify points potentially associated with objects of interest in an unsupervised manner, such as cars, cyclists, and pedestrians, categorizing them as semantic-rich. As our semantic pooling is based on the modified DBSCAN clustering algorithm with heuristics, these pooled points are extracted into different clusters based on distance metrics. The representation of these clusters can be seen in different colors in Figures 6, 7, and 8 in Appendix C. Points that are not selected by our semantic pooling, typically associated with less relevant features like buildings, ground, or grass, are marked as semantic-less.

Regarding the 4th dimension of the points, it serves as a cluster identifier for the semantic-rich points. Each point is assigned an integer value in this dimension that corresponds to the cluster it belongs to, as determined by our semantic pooling operation. For semantic-less points, this value is set to -1, distinguishing them from their semantic-rich counterparts. It's important to note that the 4th dimension is used as metadata during our pretraining process and does not directly feed into the model learning.

We have recognized the need for a more detailed explanation of these concepts and have included additional clarifications in the revised version of our paper.

Is $T_2$ identity transform in Fig. 1? If not, how does it generate the images from the original images?

The second transformation $T_2$ does not necessarily need to be the identity transformation. We would like to clarify that Figure 1 is only for illustrative purposes. There are three transformations in the actual implementation of our framework.

In the first stage of CALICO, namely point-region contrast (PRC), we employ two transformations, $T_1$ and $T_2$, to facilitate effective contrastive pretraining.

In the second stage of CALICO, which is the region-aware distillation (RAD), we introduce another transformation on the xy-plane. This transformation, denoted as $T_3$, is similar to $T_1$ and $T_2$ and serves as a data augmentation method for the LiDAR modality. Concurrently, the view transformation module in the camera modality, LSS [a,b], incorporates $T_3$ when transforming the features from the perspective view to the bird’s eye view (BEV). This ensures that the feature maps from both LiDAR and camera modalities remain well-aligned in the distillation process.

We hope our response addresses all the comments and that the reviewer will consider raising the rating accordingly. We are more than glad to answer any further questions.

[a] Philion, Jonah, and Sanja Fidler. "Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d." Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIV 16. Springer International Publishing, 2020.

[b] BEVFusion Codebase https://github.com/mit-han-lab/bevfusion/blob/main/mmdet3d/models/vtransforms/base.py (disclaimer: this link contains a public codebase that does not leak any information from the authors.)

The necessity of negative samples of $P_{PLRC}$ and $P_{RAPC}$ is unclear.

To address this, we first acknowledge the fundamental role of negative samples in contrastive learning. The absence of negative samples can lead to model collapse, as highlighted in several studies [a, b, c]. Additionally, methods focusing solely on positive pairs often implicitly benefit from statistics hidden negative samples during model training [d].

Therefore, we hypothesize that you are specifically inquiring about the necessity and role of negative sample augmentation through semantic-less points in our framework. Our approach addresses a unique challenge in LiDAR data, notably the presence of multiple objects of interest within a single frame. For instance, a LiDAR frame capturing a traffic intersection may contain numerous cars and pedestrians. If we restrict our framework to only use pooled points for generating negative pairs, we risk encountering the class collision problem. This occurs when points from different clusters (extracted by our semantic pooling), although distinct, belong to the same object class (e.g., car). There will be performance degradation when treating points from the same class of object as negative pairs. Our negative sample augmentation is thus proposed to alleviate this problem.

To empirically validate our approach and address the concerns raised, we conducted experiments specifically focusing on the impact and necessity of our negative sample augmentation strategy.

These experimental results clearly demonstrate the effectiveness and necessity of our negative sample augmentation method in improving the overall model performance.

We have included all the experimental results in the revised manuscript and hope our response addresses all the comments and that the reviewer will consider raising the rating accordingly. We are more than glad to answer any further questions.

[a] He, Kaiming, et al. "Momentum contrast for unsupervised visual representation learning." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

[b] Chen, Ting, et al. "A simple framework for contrastive learning of visual representations." International conference on machine learning. PMLR, 2020.

[c] Grill, Jean-Bastien, et al. "Bootstrap your own latent-a new approach to self-supervised learning." Advances in neural information processing systems 33 (2020): 21271-21284.

[d] Li, Alexander C., Alexei A. Efros, and Deepak Pathak. "Understanding collapse in non-contrastive siamese representation learning." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

We are glad that the reviewer found our method effective and comprehensive. We appreciate the opportunity to address the points you have raised.

The proposed method adopts the BEVFusion architecture. Lidar backbone is PointPillars, and Image backbone is Swin-T. Evaluation of different Lidar/Image backbone models could have a high impact.

We adopted the BEVFusion codebase for its state-of-the-art performance and its versatility as a general sensor fusion framework at the time of our implementation. In Section 3.5 of our paper, we have conducted an ablation study using the VoxelNet backbone and Transfusion head to specifically demonstrate the effectiveness of our point-region contrast (PRC) approach in the LiDAR modality. The results, as illustrated in Figure 4, clearly show that our PRC method consistently outperforms other baseline approaches in terms of performance. Additionally, in response to your valuable suggestion, we have integrated ResNet as an alternative backbone for the camera modality (as configured in the BEVFusion codebase). Specifically, we use our PRC pretrained LiDAR backbone as the teacher model and leverage the 10% finetuning data setting in this experiment. This further step is taken to demonstrate the adaptability and generalizability of our framework across different backbone architectures.

The results in the above table illustrate a consistent trend in performance between Swint-T and ResNet as the image backbone. More importantly, our CALICO using the ResNet backbone consistently achieves the best performance compared to other baseline methods.

[a] He, Kaiming, et al. "Momentum contrast for unsupervised visual representation learning." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

[b] He, Kaiming, et al. "Masked autoencoders are scalable vision learners." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

[c] Sun, Jiachen, et al. "Towards robust {LiDAR-based} perception in autonomous driving: General black-box adversarial sensor attack and countermeasures." 29th USENIX Security Symposium (USENIX Security 20). 2020.

It appears that semantic pooling (feature rich vs feature less regions) plays a part from the figure 5, but I would like more examples of failure cases. The clustering methods look rather empirical. More experiments on how they work in different cases would be useful.

In our exploration of the unsupervised nature of our semantic pooling operation in the CALICO framework, we encountered certain limitations. For instance, the semantic pooling sometimes erroneously clusters tree points as semantic-rich. Nonetheless, these instances did not significantly impact overall performance. The primary objective during the pretraining phase is to learn robust and prominent representations within the model’s backbone, a process that remains largely independent from downstream tasks. Misclassifying a small number of random objects as semantic-rich does not detrimentally affect the contrastive pretraining for the model backbone because the objective is to learn the correspondence between regions. Furthermore, the core concept of semantic pooling is pivotal, transcending the specifics of its implementation. Our current implementation utilized a basic approach to achieve region partitioning in LiDAR BEV space relying on independent LiDAR frames. Enhancements to semantic pooling could include integrating temporal data to refine point aggregation or to better aggregate points and identify moving points. However, such an approach necessitates temporally continuous data, introducing an additional layer of assumption and complexity. CALICO is designed to be versatile with respect to the choice of clustering algorithms, as long as the region partitioning is reasonable, as the core of CALICO is its following contrastive learning design based on the partitioned regions.

We have added an ablation study in Appendix B to quantitatively demonstrate the effectiveness of our semantic pooling compared to the random partitioning leveraged in original PLRC. We have included visualizations and a discussion of this in our revised manuscript, as shown in Figure 7 and 8 in Appendix C. In particular, Figure 7 visualizes a LiDAR frame with some failure clusters and Figure 8 showcases that our semantic pooling could successfully cluster some small but interesting objects.

I think it would help to learn if the system can demonstrate robustness against calibration error, a common occurrence in autonomous driving setups.

We are grateful for the reviewer's insightful suggestion to assess our system's resilience to calibration errors, a challenge in autonomous driving systems. Initially, it's worth clarifying that the primary objective of our framework is to enhance the efficiency and overall performance of self-driving perception tasks, with the robustness analysis serving as a supplementary aspect.

Our current evaluation framework operates under the premise of accurate alignment between LiDAR and camera data. This alignment is pivotal for the efficacy of our pretraining methodology, particularly due to the necessity of data synchronization for successful knowledge distillation in RAD. In scenarios where we intentionally alter the calibration matrix between the two modalities, our method demonstrates similar performance to other baseline approaches.

On one hand, we acknowledge that in real-world scenarios, misalignments due to calibration errors can occur, potentially impacting the performance of systems like ours. Although our current study did not specifically focus on this aspect, we recognize its importance in the broader context of autonomous driving. On the other hand, evaluating calibration errors is challenging, as misalignments result in differing localization ground truths for both modalities (i.e., the ground truth bounding boxes also do not align with each other). Therefore, determining a method and metric to quantify the evaluation is controversial.

In response to the reviewer's valuable feedback, we have incorporated the above discussion in our revised manuscript about the potential implications of calibration errors on systems pre-trained using CALICO. This addition not only acknowledges the issue but also underlines it as a vital area for future research, aiming to further refine the robustness and reliability of autonomous driving technologies.

We thank the reviewer for pointing out the superfluous lines in our manuscript and we have removed them in our revised manuscript.

We hope our response addresses all the comments and that the reviewer will consider raising the rating accordingly. We are more than glad to answer any further questions.

We are glad that the reviewer found our study valuable to this field and that our method is effective and generalizable. We appreciate the opportunity to address the points you have raised.

The paper is entirely empirical, and is as such a purely application based work. One may or may not take this as a weakness, of course.

We agree with the reviewer's observation regarding the empirical nature of our study. While it is true that our paper focuses on application-driven research, we believe this approach is not only valid but also crucial in the context of pretraining methods. As the reviewer notes, the empirical focus should not inherently be seen as a weakness.

In the field of pretraining, whether contrastive (e.g., MoCo [a]) or generative (e.g., MAE [b]), the practical application of methods often precedes and informs theoretical understanding. This is particularly true in rapidly advancing areas where empirical results can guide and refine theoretical models. Our work contributes to this tradition by providing valuable insights through real-world scenarios in autonomous driving perception.

To comply with the suggestions from the reviewer, we have included a new discussion in Appendix D.1 of our revised manuscript on potential avenues for theoretical exploration in pretraining methods, acknowledging the current limitations while highlighting the importance of empirical contributions like ours in laying the groundwork for future theoretical advancements.

The paper is heavy on tables, but I feel that qualitative analysis of where the improvement comes from is light.

We appreciate the reviewer's insightful observation regarding the balance between quantitative and qualitative analysis in our paper. We agree that the abundance of tables could potentially overshadow the qualitative insights. The choice to emphasize tabular data was driven by our intent to provide a comprehensive and detailed comparison across different metrics.

In response to the reviewer's suggestion, we have included a more focused qualitative analysis in Appendix B.1. This analysis presents key insights derived from our results. Notably, we observe that the benefits of pretraining diminish with the increase in fine-tuning data. This highlights the nuanced trade-off between pretraining and fine-tuning in model performance, which is as expected. Furthermore, when comparing our proposed PRC method (augmented with the RAPC component) to the PLRC approach, it becomes evident that our method demonstrates more significant improvements, especially when finetuning on 50% of the data. Moreover, our ablation study in Table 5 also showcases that the RAPC component in our design serves as a good regularization term to balance the performance of our framework among different levels of availabilities in finetuning data.

We hope that this enhanced qualitative discussion complements the extensive quantitative data presented and provides a more holistic understanding of our research contributions.

Could the authors explain how adversarial robustness is relevant in this context?

The relevance of adversarial robustness in our study is directly tied to the phenomenon of LiDAR spoofing attacks, as presented in [c]. These attacks involve the injection of malicious data points into LiDAR sensors, posing significant driving hazards. Our work included an evaluation of the resilience of models pretrained using our CALICO framework, as well as other baseline methods, against these black-box sensor attacks. To simulate the real-world implications of such attacks, we crafted scenarios with varying degrees of complexity, introducing 60, 100, and 200 spoofing points to create the illusion of false objects 10 meters in front of the ego vehicle, adhering to the protocols established in [c].

Our results are promising. We observed that CALICO notably diminished the success rates of these attacks in comparison to models trained from scratch and LiDAR-only methods. This increased robustness can be attributed to CALICO's ability to achieve a more balanced pretraining process, integrating both LiDAR and camera modalities effectively. In contrast, models trained from scratch or using LiDAR-only pretraining methods exhibited a higher susceptibility to attacks, because we found their end-to-end models mainly or entirely rely on the LiDAR modality.

In conclusion, while CALICO's primary aim is to enhance the efficiency and accuracy of 3D object detection models, it also plays a crucial role in fortifying these models against sophisticated adversarial attacks.

Thank you for your detailed and thoughtful response. I still feel that the work is very empirical and does not shed theoretical insight (despite the additional detail provided in the appendix). Nonetheless, I really enjoyed reading the paper and can see that distillation + sensible ideas such as clustering could be very useful in a practical problem such as sensor fusion.

The ablations in appendix B show that the distillation idea works for different backbones. I would have hoped for more ablations on methods but am fairly convinced with the response.

Perhaps I am not reading this correctly, but I notice that the paper does not beat (nor even match) BEVFusion's results - the paper reports an mAP of 60.1 (table 1) for Lidar+Camera, whereas in BEVFusion, they get a number of 75 or so (table 1 of BEVFusion [1]). As the paper uses the BEVFusion architecture, one might hope that it can match BEVFusion's results. Could the authors explain this? Can the authors also report distance based results - how does detection perform in near and far distances, and per-class metrics (car, pedestrian, etc.)? Can the authors shed light on these aspects?

[1] BEVFusion: https://arxiv.org/pdf/2205.13542.pdf

Dear reviewers, Could you please read the authors' responses and give your feed back? The period of Author-Reviewer Discussions is Fri, Nov 10 – Wed, Nov 22. Many thanks, AC