Fusion Is Not Enough: Single Modal Attacks on Fusion Models for 3D Object Detection

We express our deepest gratitude for the reviewer’s time and astute comments. It is our pleasure to address the primary concerns in the following sections.

Q1: The effectiveness of the proposed two-stage optimization approach needs further justifications.

A1: Thank you sincerely for providing valuable insights. In our original manuscript, we took care to include a comprehensive comparative analysis, particularly with respect to other single-stage attacks, notably the state-of-the-art (SOTA) one-stage camera-only patch region optimization methods discussed in Appendix J. The results of this comparative study underscore the limitations of the SOTA method in generating physically deployable patches when considering the entire scene. Additionally, in the motivation section, we presented an evaluation of traditional single-stage patch attacks, emphasizing their failure in achieving success if the predefined patch region does not align with a vulnerable region for the subject fusion model. Our two-stage attack framework, through its adept identification of vulnerable regions and utilization of distinct attack strategies based on sensitivity types, aims to generate both deployable and effective adversarial patches. This, we believe, substantiates the efficacy and versatility of our proposed method.

Q2: How to ensure the feasibility of the adversarial patches?

A2: Your thoughtful question is much appreciated. In contrast to the SOTA one-stage gradient-based patch optimization approach (as shown in appendix J), our two-stage approach has solved the problem of undeployable adversarial patches. Notably, the patch's location in our approach is not optimized with gradients concerning the entire scene; rather, it is determined by highly practical projection functions. The optimization with gradients in the second stage is exclusively applied to the patch texture. As detailed in Section 4, following the recognition of sensitivity types of the fusion model in the first stage, different attack strategies are deployed in the second stage based on sensitivity types. The patch's location on the image is contingent on the projection function (i.e., proj_x()) in specific attack strategies (See Figure 4). For scene-oriented attacks, we project the patch onto a predefined physical location on the ground, and for object-oriented attacks, we dynamically project the patch onto a target object using reverse-engineered projection parameters based on different 3D locations and orientations of the object from the driving footage. Consequently, the patch generated by our method is highly deployable and feasible in the real world.

We posit a reasonable assumption that our patch does not affect LiDAR since it is designed to be affixed to the ground (for scene-oriented attacks) or a target vehicle (for object-oriented attacks) during deployment. Our primary alteration occurs in the surface texture of the 3D scene, preserving the 3D structure captured by the LiDAR sensor. The material of the patch (e.g., paper) is common and has little influence on the LiDAR intensity. Hence the impact of our approach on the LiDAR data is minimized and it is also difficult to recognize the adversarial information using only the LiDAR sensor.

Q3: What is the sensitivity of single-modal methods vs multi-modal (LiDAR/camera) methods?

A3: Your question is both thoughtful and appreciated. Our attack framework extends its applicability to single-modal methods, as evidenced by our evaluation with a camera-only model (BEVFormer) in the original manuscript. The last row of Figure 5 shows the sensitivity heatmap of BEVFormer, which demonstrates higher sensitivity in object areas and the model is classified as object sensitivity. Similar to other object-sensitive models, BEVFormer is robust to scene-oriented attacks and vulnerable to object oriented attacks as evidenced in Table 1 and Table 2. Compared with multi-modal models, the single-modal model exhibits worse benign performance and experiences more pronounced performance degradation under object-oriented attacks.

Thanks for the detailed comments and the great efforts in rebuttal. All my questions are addressed. Considering the novel physically plausible attack methods and technical contributions for multi-modal sensor fusion adversarial attack, I would like to increase the ratings. The analysis and additional explanations in the rebuttal should be added in the final manuscript for better readability. Thank you!

Thanks for the detailed feedback.

Q1: Appendix J only shows the qualitative visualizations while quantitative results are needed to justify the performance boost.

Q2: How to ensure the projection always projects to the ground?

Thanks a lot for your continuous help in improving our paper. We answer your new questions below.

Q1: quantitative results are needed to justify the performance boost.

A1: Your thoughtful comments are appreciated. Allow us to clarify that the contribution of our approach is not the boost of attack performance in the digital space, but the practicality and effectiveness of the patch attack in the physical world. The qualitative visualization in appendix J demonstrates that the patch generated by the SOTA method, which encompasses a one-stage optimization of the patch’s content and location, spans over multiple physical objects and is not deployable. In contrast, our two-stage approach decouples the location and content optimization, and it is guaranteed to generate both deployable and effective patches for any scene (see Figure 12 and Figure 13). Hence our approach is more versatile and universal.

Q2: How to ensure the projection always projects to the ground?

A2: For the scene–oriented attacks, as shown in Figure 4a, we pre-define the 3D location of the patch with parameters like height $g$, distance $d$ and viewing angle $\alpha$, in the coordinate system of the ego-vehicle's front camera. Then we project the 3D coordinates of the patch back onto the 2D scene image (step 2). Given that we know the camera's height $g$ above ground from the dataset, we ensure by definition that the patch is positioned on the ground when using this camera height.

We express our deepest gratitude for the reviewer's invaluable time and astute comments. We have made every effort to address the principal concerns as follows, and we sincerely appreciate the opportunity to enhance our work through your insightful feedback.

Q1: Some methods do not conduct feature-level fusion between image and LiDAR. The two modalities are decoupled in these methods.

A1: We appreciate your insightful remarks. The methods you have brought up are in the category of data-level fusion. As expounded in Appendix B of our manuscript, for the data-level fusion strategy, some studies use the extracted image information to augment the LiDAR points. In the evaluation section, we demonstrated the effectiveness of our proposed attack on a data-level fusion model (i.e., PointAugmenting) that utilizes image features extracted by CNN to hence LiDAR data. In the first study you mentioned, the authors employ CNN-based 2D object detection to identify object regions in the image input, then use the extruded 3D viewing frustum to extract corresponding points from the LiDAR point cloud for bounding box regression. In the second study, the authors use semantic segmentation on the image input to generate the frustum. Both methods depend on information from the camera input (i.e., 2D object detection or semantic segmentation results) to augment point cloud-based object detection. Therefore, adversarial attacks on images can deceive object detection or semantic segmentation as in [1-3], leading to the failure of the extruded 3D frustum in capturing the corresponding LiDAR points of each object. This would subsequently affect the 3D bounding box regression. Your insights are greatly appreciated, and a discussion on these studies has been included in Appendix B, with appropriate citations.

Q2: Is the noise patch shared by the whole dataset?

A2: Thank you for your comments and queries. The adversarial patch is shared across various frames in the dataset, with each patch optimized for a scene (in scene-oriented attacks) or a target object (in object-oriented attacks). In the context of object-oriented attack strategies, we project the patch image onto the rear of the target vehicle in each scene image using varying projection parameters based on the target object's location in 3D space (see Figure 4b). The varying patch appearances in object-oriented attacks result from the changing projection parameters and the interpolation among pixels in each data frame. We have included a detailed threat model in Appendix Q of our revised manuscript.

Q3: Is the mask in Sensitivity Distribution Recognition shared by the whole dataset?

A3: No, the mask is not shared. Each mask (i.e., sensitivity heatmap) is defined and optimized for a specific frame of data extracted from the dataset. For the sensitivity type classification, as per Equation 4, the average sensitivity of object areas and non-object areas is calculated across various data frame $x$’s sensitivity heatmaps. We have refined the description and notations in the revised version for clarity.

Q4: I do not understand the form of the proj_x in Eq. 6 and why it is necessary.

A4: proj_x() signifies the synthesis of the original patch image (see the “2D patch image” in Figure 4a) onto a specific area of the scene image (see the “captured image” in the center of Figure 4a) to simulate how the patch would look once it's physically deployed. This projection is necessary for enhancing the physical-world robustness of the generated patch and minimizing the disparity between digital space and the physical world. As outlined, equations 7-9 provide an expanded representation, where ($u^p$, $v^p$) denotes the coordinates of a pixel on the original patch image, and ($u^s$, $v^s$) represents the pixel's corresponding coordinates on the scene image. proj_x() establishes this mathematical connection based on projection parameters (e.g., lateral and longitudinal distance, and viewing angle of the patch in the physical world). Specifically, for object-oriented attacks, as the target object’s location and orientation vary across data frames, the patch would move with the target object once deployed physically. Hence, the patch’s projection parameters should differ across frames for a faithful simulation during patch training before deployment. We innovatively extracted those parameters dynamically from the detected 3D position of the target object as we stated in our methods. We have revised the presentation of our method in the updated manuscript to make it more clear.

Q5: Are there any special designs to solve problems in image-only attacking or autonomous driving scenes? What is the difficulty of image-only attacks?

A5: We acknowledge your concerns and would like to highlight the unique challenges inherent in image-only attacks that have guided our method's unique design. Most prominently, depending on model architectures and fusion strategies, the vulnerable regions change across different fusion models. As a result, directly attacking a specific area, akin to the SOTA patch attack [4], may fail when the targeted area is not a vulnerable region (e.g., attacking the road for object-sensitive models, as depicted in Figure 2). Hence, a unique challenge is to determine the vulnerable region, and then the corresponding attack strategy can be employed. Our sensitivity distribution recognition algorithm addresses this challenge. A straightforward extension to the SOTA attack would be to directly optimize some patch on entire scene images. However, the generated patch may span over multiple physical objects and is not deployable (see Appendix J). Our two-stage design ensures physical deployability, and its compatibility with SOTA camera attacks reinforces it as a more universal solution. Rather than viewing the generality of our attack framework as a limitation, we consider it a strength. Another addressed challenge is reverse engineering precise projection functions for different target object locations and orientations from driving footage, facilitating faithful scene synthesis during offline patch generation (as elucidated in Q4).

Reference

[1] Xie, Cihang, Jianyu Wang, Zhishuai Zhang, Yuyin Zhou, Lingxi Xie, and Alan Yuille. "Adversarial examples for semantic segmentation and object detection." In ICCV, 2017.

[2] Huang, Lifeng, Chengying Gao, Yuyin Zhou, Cihang Xie, Alan L. Yuille, Changqing Zou, and Ning Liu. "Universal physical camouflage attacks on object detectors." In CVPR, 2020.

[3] Nesti, Federico, Giulio Rossolini, Saasha Nair, Alessandro Biondi, and Giorgio Buttazzo. "Evaluating the robustness of semantic segmentation for autonomous driving against real-world adversarial patch attacks." In WACV, 2022.

[4] Cheng, Zhiyuan, James Liang, Hongjun Choi, Guanhong Tao, Zhiwen Cao, Dongfang Liu, and Xiangyu Zhang. "Physical attack on monocular depth estimation with optimal adversarial patches." In ECCV, 2022.

Thanks a lot for your continuous help in improving our paper. We answer your new questions below.

Q1: If the author solved a unique problem of image-only attacking. If so, what is the problem?

A1: Thank you for your comments and queries. We agree that the proposed method can be adapted for LiDAR modality, which we leave as a future direction. In this context, the unique problem of image-only attack that we solve in this work is to reverse-engineer precise projection functions for varying target object locations and orientations from driving footage, which facilitates faithful scene image synthesis during patch generation (as elucidated in our initial response to Q4).

Q2: Will the authors open-source the project?

A2: Yes, we will open-source the project after the anonymous period. At present, our code with detailed instructions to run are available in the supplementary materials.

We are genuinely grateful for your thoughtful considerations, and we address the new questions below:

Q1: Why the project step ① in Figure 4 is necessary?

A1: We agree that $d$, $g$ and $\alpha$ (the so-called projection parameters) are numbers that will not be optimized, but these parameters are not the same for different scene images in the driving footage, especially for object-oriented attacks. In other words, $proj_x()$ is the projection function for a specific scene image $x$ and the projection parameters (e.g., $d_x$, $g_x$ and $\alpha_x$ ) are also unique for $x$. However, the 2D patch image ($p$) optimized by us should be universally effective across frames, hence we need to use $proj_x()$ to project the 2D patch image $p$ onto each scene image $x$ sampled randomly from the dataset during optimization, simulating the patch's appearance on each scene image $x$ (i.e., $p_x = proj_x(p)$ in Equation 6). Step 1 is necessary to connect the 2D patch image $p$ optimized by us and the randomly sampled scene image $x$ that will be fed to the fusion model, as $p$ should be universally effective across various scene images.

Q2: why is it a problem to be solved?

A2: Despite the form of $proj_x()$ is a basic spatial transformation and projection, the projection parameters (e.g., $d_x$, $g_x$ and $\alpha_x$) in $proj_x()$ for a specific scene image $x$ is dynamically extracted. We solved the problem of dynamically projecting the patch image $p$ onto the target object in randomly sampled scene images during the patch optimization process. This is accomplished by extracting these unique projection parameters for each scene image from the target object's 3D bounding box that is predicted by the fusion model in benign case.

Please kindly let us know if we have any misunderstanding about your questions. Thanks.

We express our gratitude for your comments and reevaluation of our contribution. Although extracting those projection parameters may not be technically difficult, we think it is a necessary and novel step for the process of dynamic projection in our proposed optimization approach. Thank you for your valuable feedback and continuous help in making our work better.

We express our deepest gratitude for the reviewer’s invaluable time and astute comments. It is our utmost pleasure to address the principal concerns as follows.

Q1: Limited Fusion Model Efficacy.

A1: Your thoughtful comments are appreciated. As we elaborated in Related Works, Discussion of Other Fusion Strategies (Appendix B) and Limitations (Appendix P), our research primarily focuses on data-level and feature-level fusion strategies. This focus is driven by the fact that advanced fusion models, which demonstrate superior detection performance, predominantly employ these fusion strategies [1]. These strategies are favored due to their enhanced feature extraction capabilities. The adoption of such early-fusion schemes is also a general trend with the increasing popularity of end-to-end autonomous driving [2-3]. Decision-level fusion, the other strategy, is less prevalent in recent fusion models. In such a setup, our camera-only attacks would only influence the output of the camera-based model. Therefore, the impact of such an attack is significantly dependent on the relative weights assigned to the two modalities during the fusion process. This dependency is a common limitation for single-modal attacks. While prior LiDAR-only attacks on Baidu Apollo have shown success [4], this can be attributed to Apollo's current higher weighting of LiDAR results. Enhancing the success rate of camera-only attacks against decision-level fusion models poses an open problem, and we acknowledge it as a promising area for future exploration and research. Thanks for your insightful comments, we have highlighted the fusion strategies we focused on in the introduction of our revised version and discussed more about decision-level fusion in Appendix B.

Q2: Data Dependency and Generalizability Concerns.

A2: We express gratitude for your valuable feedback. The Nuscenes dataset employed in our experiments is a real-world dataset encompassing diverse scenarios across driving conditions. Models trained on this dataset exhibit robust generalization to real-world and simulation scenarios, as scrutinized in the practicality section of our manuscript. The fusion model, trained with the Nuscenes dataset, successfully detects all objects near the victim vehicle in benign cases. This performance across the Nuscenes dataset, real-world conditions, and simulation environments underscores the generalizability and minimum data dependency of our fusion models. Furthermore, our attacks demonstrate success across three types of data, affirming the generalizability and robustness of our proposed attack approach.

Q3: Inadequate Security Analysis and Potential Resource Constraints.

A3: We appreciate your insightful comments and suggestions. In response, we have enriched the revised version of our manuscript with a more comprehensive analysis of autonomous vehicle security implications in Appendix R. To address potential resource constraints, we've introduced a detailed threat model for our attack in Appendix Q, outlining the assumptions made about attackers. Additionally, specific devices and the detailed computational overhead required for generating adversarial patches are disclosed in Appendix H, providing a thorough dissection of our approach's resource requirements and inherent limitations.

Q4: Comparison with Pre-existing Methodologies.

A4: Your comments are valued. In the Related Work section of the original manuscript, we have thoroughly discussed the relationship between our attack and prior attacks against fusion models. Succinctly, some previous attacks are multi-modal attacks that fool camera and LiDAR modalities either separately (with colors and shape) or concurrently (with shape). Others are single-modal attacks against only the LiDAR modality. In contrast, our approach pioneers a single-modal attack exclusively targeting the camera modality. As stated in the Introduction, prior LiDAR-related approaches pose implementation challenges, necessitating additional equipment like photodiodes, laser diodes, or industrial-grade 3D printers to manipulate LiDAR data, thereby increasing deployment costs for attackers. Our camera-only attacks, in comparison, are more affordable and easier to deploy, enabling attackers to print generated adversarial patches with home printers. This reduction in effort heightens the threat to autonomous vehicle security.

We express our deepest gratitude for the reviewer’s time and insightful comments. It is with great respect that we have endeavored to address the primary concerns as follows.

Q1: What is the threat model of the proposed attack?

A1: We sincerely appreciate your insightful question. Our attack assumes that the attacker has complete knowledge of the camera-LiDAR fusion model used by the target autonomous driving vehicle. Therefore, our attack model is considered to be in a white-box setting. This assumption is congruent with existing endeavors in the literature dedicated to adversarial attacks on autonomous driving systems [1-5]. To achieve this level of knowledge, the attacker may employ methods such as reverse engineering the perception system of victim vehicles [6], employing open-sourced systems, or exploiting information leaks from insiders. In addition, the attacker is clear about the target object and driving scenes he wants to attack. For example, he can record both video and LiDAR data using his own vehicle and devices while following a target object (for object-oriented attacks) or stopping at a target scene (for scene-oriented attacks). Leveraging this pre-recorded data, the attacker undertakes a one-time effort to generate an adversarial patch using our proposed approach. The attacker can then print and deploy the generated patch on the target object or in the target scene for real-time attacks against victim vehicles. The detailed elaboration of our threat model can be found in the appendix Q of our manuscript.

Q2: The method’s ineffectiveness on decision-level fusion models.

A2: We express our gratitude for your thoughtful comments. As we elaborated in Related Works, Discussion of Other Fusion Strategies (Appendix B) and Limitations (Appendix P) sections, our research primarily focuses on data-level and feature-level fusion strategies. The decision-level fusion is less prevalent in recent fusion models. In such a setup, our camera-only attacks would only influence the output of the camera-based model. Therefore, the impact of such an attack is significantly dependent on the relative weights assigned to the two modalities during the fusion process. This dependency is a common limitation for single-modal attacks. Although prior LiDAR-only attacks have succeeded on Baidu Apollo [10], it can be attributed to the current higher weighting of LiDAR results in Apollo. Our focus on data-level and feature-level fusion is driven by the fact that advanced fusion models, which demonstrate superior detection performance, predominantly employ these fusion strategies [7], due to their enhanced feature extraction capabilities. The adoption of such early-fusion schemes is also a general trend with the increasing popularity of end-to-end autonomous driving [8-9]. The challenge of enhancing the success rate of camera-only attacks against decision-level fusion models remains an open question, and we duly acknowledge this as a promising area for future exploration and research. Based on your comments, we have highlighted the fusion strategies we focused on in the introduction of our revised version and discussed more about decision-level fusion in Appendix B.

Q3: The positioning of this paper. What is the major advantage of this work compared to existing attacks against camera-LiDAR fusion models?

A3: We are genuinely grateful for your thoughtful considerations. In the Related Work section of the original manuscript, we have discussed the interplay between our work and the works you referenced. In essence, the referenced works encompass multi-modal attacks that deceive camera and LiDAR modalities either individually (with colors and shape) or concurrently (with shape). In contrast, our focus is on single-modal attacks directed exclusively at the camera modality. As stated in the introduction, prior approaches, as you indicated, present implementation challenges due to additional equipment requirements, such as photodiodes, laser diodes [10], or industrial-grade 3D printers [1], for manipulating LiDAR data. This increases deployment costs for attackers. In contrast, our camera-only attacks offer enhanced affordability and ease of deployment, enabling attackers to print the generated adversarial patch with home printers. This reduction in effort amplifies the threat to autonomous vehicle security.

Q4: The practicability of the adopted adversarial patch is questionable. How to place the patch on the back of a vehicle? Does the patch hide the license plate of the vehicle? How to place the adversarial patch on a pedestrian in the real world (as shown in Figure 10)?

A4: We sincerely appreciate your inquiries. Allow us to clarify that the 1m * 1m is not the minimum patch size required for attacks. The minimum patch size necessary for a successful attack is contingent upon the distance between the target object and the victim vehicle. The critical factor is the pixel count influenced by the patch in the input images, and a smaller patch can exhibit efficacy when in close proximity to the victim vehicle. Table 5 in our study demonstrates that the attack performance of adversarial patches diminishes with increasing distances. Our additional exploration of object-oriented attacks, considering patch size, shape and varying distances between target and victim vehicles, is detailed in Table R1. As shown, the 0.5m * 0.5m patch still performs well at a closer distance (e.g. less than 8m). It is crucial to underscore that failure to detect the target object at a close range is still hazardous, as the driver may lack sufficient reaction time, leading to a braking distance exceeding the actual distance.

Additionally, the patch does not need to be in square shape. Attackers have the flexibility to define patches with rectangular or arbitrary shapes to circumvent covering the target vehicle's license plate.

Figure 10 in Appendix G (Property Validation of Sensitivity Heatmap) aims to demonstrate the property of the sensitivity heatmap instead of real-world attacks against pedestrians. Attackers may use a smaller patch on T-shirts (like [11]) to attack the pedestrians at a closer distance (e.g. 5 m).

The additional experiments and explanations have been added to the revised manuscript in Appendix M.

Table R1: Detection score of the target object under different distances and patch sizes.

Q5: The real-world evaluation is weak.

A5: We express our gratitude for your observation. In Section 4, our approach accounts for diverse lighting conditions, distances, and viewing angles during the patch optimization process. This is achieved by randomizing brightness, contrast, and projection parameters of the patch and incorporating Estimation of Transformations (EoT) in training. The evaluation of attack performance under varied distances and viewing angles is meticulously presented in Appendix M. To assess the practicality of our attacks, we conducted experiments in both real-world and high-fidelity simulator environments following the practical attack procedures. Responding to your insightful suggestion, we augmented the practicality evaluation by conducting additional experiments at different times of the day to simulate lighting changes. The experimental settings remain consistent with those detailed in Appendix K, and the results are presented in Table R2. The second and third columns indicate the benign and adversarial performance of BEVFusion-PKU respectively. The fourth column presents the percentage of model performance degradation, indicating our attack effectiveness. As shown, our patch remains a robust attack performance at different lighting conditions.

Table R2: Attack performance at different times of the day with various lighting conditions.