APD: Boosting Adversarial Transferability via Perturbation Dropout

2. Targeted attacks

We chose not to conduct targeted attack because we consider it a more specialized task that requires dedicated methods to solve. For instance, as indicated in the appendix of the RAP [1] method, on page 22, there's a table for Inc-v3 → ResNet-50 target attack. The best attack achieved only a 16.7% attack success rate, and some fundamental methods had a mere 0.1% attack success rate. In our experiment, we used models that were more complex and difficult to attack than theirs, including Inc-v3, Inc-v4, IncRes-v2 and Res-101, the targeted attack success rates among these models do not exceed 0.1% using MI-FGSM.

However, we do test our method's effectiveness using an method specifically designed for target attacks, namely CFM[2]. Combining our method with this specialized attack for target attacks can improve attack success rate significantly. Here are our results in Table[2]. CFM-RDI-MI is their SOTA attack in their paper and APD-CFM-RDI-MI is our attack, which achieves new SOTA results.

[1] Boosting the transferability of adversarial attacks with reverse adversarial perturbation. NeurIPS 2022. [2] Introducing Competition to Boost the Transferability of Targeted Adversarial Examples through Clean Feature Mixup. CVPR 2023.

Table[2]: Attack Success Rate(%) of target attack

1. Relevance to existing attention-based attack methods

Indeed, you're right. Attention-based attack methods have been a classic tool within the realm of studying adversarial transferability. However, it's essential to note that the attention-based method is a commonly used technique in computer vision, especially in semantic segmentation, serving as a fundamental method. First and foremost, it needs to be clarified that existing methods(including ours) do not use attention as innovative point itself but rather utilize it as a tool. So, we did not specifically discuss attention in our paper, but rather focused on noise effectiveness of I-FGSM, TI-FGSM, and PI-FGSM.

Below, I will introduce the relationship between our method and existing methods[1][2][3] using attention.

[1] Evading defenses to transferable adversarial examples by translation-invariant attacks. CVPR 2019. [2] Boosting the transferability of adversarial samples via attention. CVPR 2020. [3] Dual attention suppression attack: Generate adversarial camouflage in physical world. CVPR 2021.

1.1 Evading defenses to transferable adversarial examples by translation-invariant attacks.

In our Section 1, i.e., Introduction, we have already discussed it.

1.2 Boosting the transferability of adversarial samples via attention.

"Boosting the transferability of adversarial samples via attention" is highly relevant to our approach, we were not aware of it previously. We appreciate your sharing this paper and will include relevant discussions in the final paper.

1) Their method, illustrated in Figure 1 (refer to their paper), highlights attention heatmaps of various models on a cat image, primarily focusing on the cat's face. They observe that most models extract similar important features, leading to their method's design to increase the perturbation norm in these important features' regions while reducing the perturbation norm in unimportant regions.

2) Our method: We observe that not all models focus on the same position, indicating their observation is somewhat insufficient. Our Figure 2(refer to our paper) displays a turtle image, which could have multiple important features, and different models might identify different ones. Our method decouples different regions, enhancing their independence to improve transferability.

3) Comparision: Their method succeeds because they reduce the norm of noise in unimportant regions while increasing the norm of noise in important regions. In contrast, our method succeeds because it decouples the effects of noise on different blocks in important regions. In principle, the two methods are inconsistent.

4) More Discussion: Inspired by this paper, we also discover a weakness of our method, but we demonstrate that this weakness has an almost negligible impact on our method. If an image aligns with their cat image, which just has one important region and all models share attention on it, looks like our method's gonna lose its powers. Specifically, our method can only decouple the background from the important feature region, which will not improve transferability significantly in this situation. To explain this, we utilize different models to generate CAM on the dataset and compute the Number of Hotspot Regions for each image. Next, we count the Number of Images that own specific Number of Hotspot Regions, as shown in Table [1].

Crafted on IncRes-v2

Crafted on Res-101

3. Comparisons with more relevant methods

Thank you for your valuable suggestion. We have incorporated additional comparisons with advanced methods, namely PI-FGSM[1], RAP[2], SSA[3], PGN[4], TAIG[5], and CFM[6]. Partial results are presented in Table[3], with the remaining outcomes slated for inclusion in the final version of our paper. Note that we have not included the method "Toward understanding and boosting adversarial transferability from a distribution perspective," as provided by you. This is due to the significant time consumption associated with their approach, as indicated in their paper: "The computation cost of fine-tuning the surrogate model for one epoch requires 8 hours on Tesla V100 using ResNet-50 on ImageNet." Therefore, we require additional time and plan to incorporate comparative results with this method in the final version of the paper.

We appreciate your feedback and believe these enhancements strengthen the comprehensiveness of our comparative analysis.

[1] PI-FGSM: Patch-wise Attack for Fooling Deep Neural Network. ECCV 2020.

[2] Boosting the transferability of adversarial attacks with reverse adversarial perturbation. NeurIPS 2022.

[3] Frequency Domain Model Augmentation for Adversarial Attack. ECCV 2022.

[4] Boosting Adversarial Transferability by Achieving Flat Local Maxima. NeurIPS 2023.

[5] Transferable Adversarial Attack based on Integrated Gradients. ICLR 2022.

[6] Introducing Competition to Boost the Transferability of Targeted Adversarial Examples through Clean Feature Mixup. CVPR 2023.

Table[3]: Attack Success Rates (%) of adversarial attacks. We integrate our method into all the baselines, such as we let 'PI/+APD' denote PI and APD-PI. The best results are bold. * indicates the white-box setting.

Note that for all methods, we integrate momentum because we take momentum as a fundamental method.

Crafted on Inc-v3

Crafted on Inc-v4

Dear Reviewer 3YWF,

Thank you again for your time. As the deadline for discussion is approaching, we do wish to hear from you to see if our response resolves your concerns. We are happy to provide any additional clarifications if needed.

4. Comparisons with more relevant methods

Thank you for your valuable suggestion. We have incorporated additional comparisons with advanced methods, namely PI-FGSM[1], RAP[2], SSA[3], PGN[4], TAIG[5], and CFM[6]. Partial results are presented in Table[2], with the remaining outcomes slated for inclusion in the final version of our paper. It's worth noting that we opted not to include "Improving Adversarial Transferability via Neuron Attribution-based Attacks," as this work pertains to feature-level attacks. Furthermore, we have already integrated a more advanced feature-level attack[6].

We appreciate your feedback and believe these enhancements strengthen the comprehensiveness of our comparative analysis.

[1] PI-FGSM: Patch-wise Attack for Fooling Deep Neural Network. ECCV 2020.

[2] Boosting the transferability of adversarial attacks with reverse adversarial perturbation. NeurIPS 2022.

[3] Frequency Domain Model Augmentation for Adversarial Attack. ECCV 2022.

[4] Boosting Adversarial Transferability by Achieving Flat Local Maxima. NeurIPS 2023.

[5] Transferable Adversarial Attack based on Integrated Gradients. ICLR 2022.

[6] Introducing Competition to Boost the Transferability of Targeted Adversarial Examples through Clean Feature Mixup. CVPR 2023.

Table[2]: Attack Success Rates (%) of adversarial attacks. We integrate our method into all the baselines, such as we let 'PI/+APD' denote PI and APD-PI. The best results are bold. * indicates the white-box setting.

Note that for all methods, we integrate momentum because we take momentum as a fundamental method.

Crafted on Inc-v3

Crafted on Inc-v4

Crafted on IncRes-v2

1. Why do we use hotspots instead of using thresholds to determine the dropped regions.

Thank you for bringing this to our attention. We appreciate your feedback, and we acknowledge that we inadvertently overlooked an important aspect. We will include the following clarification in our paper:

In our motivation experiment depicted in Fig1(b), we employed a threshold to identify regions where the source model focuses while the target model does not. However, in our formal method, we opted to utilize midpoints with square regions.

Here's the rationale behind this choice:

1. Search for Effective Dropout Method: Our goal is to discover a more effective dropout method by exploring regions where the synergy is more pronounced across models.

2. Observation from Experiment (Fig1(b)): Through our experiments, especially in Fig1(b), we found that the "regions where the source model focuses while the target model does not" have a significant negative impact on transferability. This observation led us to conclude that differences in attention regions between the two models contribute most to the negative impact on transferability.

3. Identification of Attention Region Differences: Combining the findings from points 1 and 2, it became crucial to identify the differences in attention regions across different models. From our observations in Fig2, we noted that a model's attention region comprises some meaningful blocks. By decoupling perturbations in these different blocks, we can effectively improve transferability.

4. Challenges with Threshold-Based Approach: When using a threshold to choose dropped regions, two challenges arise: i) if the threshold is set too low, different blocks merge together, and ii) if set too high, each dropped region does not cover one entire block. Additionally, different attention regions have different values, making it challenging to determine an optimal threshold. In contrast, the hotspot method accurately identifies each block. Therefore, we opted for the hotspot midpoint approach. The use of a square region is more of a customary choice. Shapes like circles or triangles are also workable. The emphasis is on choosing the hotspot's center rather than fixating on the shape.

2. Time consumption comparison.

Here is our analysis of Computational Cost:

In our proposed method, the time cost primarily arises from forward propagation, backward propagation, and input transformation. Notably, the computation process of the CAM heatmap can be seamlessly integrated into the forward propagation, rendering the heatmap computation time negligible. As the primary time cost is associated with forward and backward calculations, we omit other factors like input transformation.

Algorithm 1(see in our paper) involves a global iteration number T, implying 2T forward and backward calculations, local maximum points number n, and size scale number m. Utilizing the parameters from MI, DIM, TIM, SIM, and AAM, we summarize the time cost relative to MI, where MI's cost is considered as 1x:

The process of generating adversarial examples is conducted offline. As the primary evaluation metric is the attack success rate, with time cost being a secondary consideration, we have provided additional discussion and experiments in Appendix A.3 of our original paper. Given that our method incurs additional computational cost compared to the original MI-FGSM, we aim to illustrate in the A.3 section that the improved transferability results from our dropout mechanism rather than the increased computational cost.

We appreciate your consideration and are open to any further questions or clarifications.

3. Evaluation on diversity of model architectures

Thank you for highlighting this concern. We appreciate your suggestion and would like to inform you that our paper already includes an evaluation of the diversity of model architectures. In Table 3 of our original paper, we considered various network architectures, including sequencer deep LSTM, ViT-B/16, and MnasNet.

Upon reviewing the paper you provided, we recognized the opportunity to further enhance the diversity of our model evaluations. As a result, we have expanded our architecture diversity by incorporating additional models, including Inc-v3, Inc-v4, IncRes-v2, Res-101, ViT-B/16, TNT, Swin, PiT-B, CaiT-S/24, Visformer-S, and SNNs (Spiking Neural Networks). It's worth noting that we selected four CNN-based models (Inc-v3, Inc-v4, IncRes-v2, Res-101), six Transformer-based models (ViT-B/16, TNT, Swin, PiT-B, CaiT-S/24, Visformer-S), and introduced SNNs to broaden the spectrum of architecture diversity. Deep learning, as a broader category, encompasses artificial neural networks (ANNs). SNNs, on the other hand, represent a distinct type of neural network within the realm of artificial neural networks.

To provide you with a glimpse of the results, we have included part of the findings in Table[1]. Rest assured, we will incorporate the complete set of results into our paper in the final paper:

Table[1] Attack Success Rate(%) of models of diverse architectures

Dear Reviewer e8ZY,

Thank you again for your time. As the deadline for discussion is approaching, we do wish to hear from you to see if our response resolves your concerns. We are happy to provide any additional clarifications if needed.

4. About the Regions that Both Source and Target Models Neglect

We employed different drop strategies to eliminate noise, and the results are presented in Table[4]. 'Attention Removal:S-T' refers to removing noise in regions where the source model focuses but the target model does not. 'Random Removal' involves randomly dropping noise of equal size to 'Attention Removal:S-T'. 'Attention Removal:-S-T' indicates removing noise in regions neglected by both the source and target models, using the same size as 'Attention Removal:S-T'. It is observed that noise in regions neglected by both the source and target models has the least influence on transferability. This aligns with the ideas presented in our paper.

Table[4]: Attack Success Rate(%) of when inflicting different drop strategies

5. About PGD Attack

The selection of adversarial example generation methods in our experiments focused on several fundamental techniques, namely FGSM, I-FGSM, PGD, C&W, and DeepFool. Our proposed method is based on I-FGSM, and consequently, we considered baselines within this branch, such as MI-FGSM, while excluding other fundamental methods like FGSM, ILCM, PGD, C&W, and DeepFool. We made this choice believing that including one fundamental baseline is sufficient for our study.

Moreover, although PGD attacks were not considered in a series of related works - MI, DIM, TIM, SIM, Admix - we still conducted experiments by comparing PGD with the PGD version of our method. The results, demonstrating the effectiveness of our approach, are presented in Table[5].

Table[5]: Attack Success Rate of PGD Version of Our Method

6. About the expression of dataset.

Thank you for your reminder. The expression of dataset in our paper might not has been entirely clear. The random selection mentioned in the paper was not conducted by us but by the organizers of the NIPS 2017 adversarial attack challenge. They randomly selected 1000 images from ImageNet, which later became the most widely used dataset for adversarial examples transferability. The models we selected exhibit high classification accuracy on these images.

7. The flow chart.

See "flow_chart.pdf" in Supplementary Material.

9. Why do we use hotspots instead of using thresholds to determine the dropped regions.

Thank you for bringing this to our attention. We appreciate your feedback, and we acknowledge that we inadvertently overlooked an important aspect. We will include the following clarification in our paper:

In our motivation experiment depicted in Fig1(b)(see in our paper), we employed a threshold to identify regions where the source model focuses while the target model does not. However, in our formal method, we opted to utilize midpoints with square regions.

Here's the rationale behind this choice:

1. Search for Effective Dropout Method: Our goal is to discover a more effective dropout method by exploring regions where the synergy is more pronounced across models.

2. Observation from Experiment (Fig1(b))(see in our paper): Through our experiments, especially in Fig1(b), we found that the "regions where the source model focuses while the target model does not" have a significant negative impact on transferability. This observation led us to conclude that differences in attention regions between the two models contribute most to the negative impact on transferability.

3. Identification of Attention Region Differences: Combining the findings from points 1 and 2, it became crucial to identify the differences in attention regions across different models. From our observations in Fig2, we noted that a model's attention region comprises some meaningful blocks. By decoupling perturbations in these different blocks, we can effectively improve transferability.

4. Challenges with Threshold-Based Approach: When using a threshold to choose dropped regions, two challenges arise: i) if the threshold is set too low, different blocks merge together, and ii) if set too high, each dropped region does not cover one entire block. Additionally, different attention regions have different values, making it challenging to determine an optimal threshold. In contrast, the hotspot method accurately identifies each block. Therefore, we opted for the hotspot midpoint approach. The use of a square region is more of a customary choice. Shapes like circles or triangles are also workable. The emphasis is on choosing the hotspot's center rather than fixating on the shape.

10. The attack performance on adversarial training models?

Indeed we have the evaluation on adversarial training models in our paper. The $Inc-v3_{ens3}$, $Inc-v3_{ens4}$ and $IncRes-v2_{ens}$ are adversarial training models. We test the signal model attack setting on these adversarial training models in Table 1(see in our paper) and ensemble model attack setting in Table 2(see in our paper). Putting these three classic adversarial training models and normal models together is a usual practice, and we apologize the confusion it may have caused you.

8. Evaluation on diversity of model architectures

Thank you for highlighting this concern. We appreciate your suggestion and would like to inform you that our paper already includes an evaluation of the diversity of model architectures. In Table 3 of our original paper, we considered various network architectures, including sequencer deep LSTM, ViT-B/16, and MnasNet.

Upon reviewing the paper you provided, we recognized the opportunity to further enhance the diversity of our model evaluations. As a result, we have expanded our architecture diversity by incorporating additional models, including Inc-v3, Inc-v4, IncRes-v2, Res-101, ViT-B/16, TNT, Swin, PiT-B, CaiT-S/24, Visformer-S, and SNNs (Spiking Neural Networks). It's worth noting that we selected four CNN-based models (Inc-v3, Inc-v4, IncRes-v2, Res-101), six Transformer-based models (ViT-B/16, TNT, Swin, PiT-B, CaiT-S/24, Visformer-S), and introduced SNNs to broaden the spectrum of architecture diversity. Deep learning, as a broader category, encompasses artificial neural networks (ANNs). SNNs, on the other hand, represent a distinct type of neural network within the realm of artificial neural networks.

To provide you with a glimpse of the results, we have included part of the findings in Table[8]. Rest assured, we will incorporate the complete set of results into our paper in the final paper:

Table[8] Attack Success Rate(%) of models of diverse architectures

3. Comparisons with more relevant methods

Thank you for your valuable suggestion. We have incorporated additional comparisons with advanced methods, namely PI-FGSM[1], RAP[2], SSA[3], PGN[4], TAIG[5], and CFM[6]. Partial results are presented in Table[3].

We appreciate your feedback and believe these enhancements strengthen the comprehensiveness of our comparative analysis.

[1] PI-FGSM: Patch-wise Attack for Fooling Deep Neural Network. ECCV 2020.

[2] Boosting the transferability of adversarial attacks with reverse adversarial perturbation. NeurIPS 2022.

[3] Frequency Domain Model Augmentation for Adversarial Attack. ECCV 2022.

[4] Boosting Adversarial Transferability by Achieving Flat Local Maxima. NeurIPS 2023.

[5] Transferable Adversarial Attack based on Integrated Gradients. ICLR 2022.

[6] Introducing Competition to Boost the Transferability of Targeted Adversarial Examples through Clean Feature Mixup. CVPR 2023.

Table[3]: Attack Success Rates (%) of adversarial attacks. We integrate our method into all the baselines, such as we let 'PI/+APD' denote PI and APD-PI. The best results are bold. * indicates the white-box setting.

Note that for all methods, we integrate momentum because we take momentum as a fundamental method.

Crafted on Inc-v3

Crafted on Inc-v4

Crafted on IncRes-v2

1. Thoughts is similar to "AdvDiffuser: Natural Adversarial Example Synthesis with Diffusion Models. International Conference on Computer Vision (ICCV). 2023".

Introduction of AdvDiffuser's Method

AdvDiffuser proposes a method that utilizes Diffusion Models to generate natural adversarial examples. In Section 3.2 of their paper, they introduce the concept of Adversarial Inpainting, which utilizes Class Activation Maps (CAM) to derive a mask matrix. The primary goal is to ensure that the generated adversarial example closely resembles the reference image by minimally perturbing areas containing important objects, while mainly perturbing less critical areas.

Similarities with Our Work

The similarity between AdvDiffuser's approach and ours lies in both methods employing CAM for masking. However, it's crucial to note that the CAM-based mask method is a widely used technique in deep learning, especially in semantic segmentation, serving as a fundamental method. The technical novelty of both our method and AdvDiffuser's do not lie in the CAM-based mask itself.

Novelty of Our Method

The innovation of our method is twofold:

1. We discovered that a complete attention region comprises multiple blocks.
2. We indicated substantial synergy between these distinct blocks, as illustrated in Section 3.3 of our paper.

As a result, we utilized CAM to identify positions of multiple important regions, effectively decoupling noise from these blocks.

Similarly, the CAM-based mask method is not the sole innovation of AdvDiffuser's method. Their novelty lies in reducing noise in important areas to achieve the appearance of a reference image. CAM serves as their tool for identifying important objects’ positions. Moreover, the difference lies in their method normalizing the mask values to [0, 1], reducing the noise weight in important objects’ positions and increasing the noise weight in non-important positions. Moreover, their method does not employ a dropout mechanism, unlike ours, and thus, they don't achieve the decoupling effect that our method accomplishes.

2. About Attention Regions Shift During Attack Iterations

Yes, the attention regions will shift during the attack iterations. Actually, our paper contains relevant content. The shift in attention regions is not a drawback. In fact, we utilize this shift to enhance our method. In section 3.4 of our paper, we have mention that: "It’s worth noting that we use CAMs at each attack iteration instead of just the initial image’s CAM because the attention region expands over the attack steps. The initial image’s CAM provides limited attention blocks. Using CAMs at each step allows us to identify as many attention blocks as possible over the complete attack sequence, providing a more comprehensive set of regions to drop for better transferability."

About "it is unreliable to rely on the CAM from the previous iteration as guidance for the current one": Actually, we update the adversarial examples based on the CAM of the current iteration. And the shifting attention doesn’t make our method unreliable. We only aim to discover additional blocks to enable the generated adversarial samples to transfer across more models.

We will incorporate visualizations illustrating the attention region shifts in the Appendix section of our paper. We appreciate your comment to enhance the clarity of our work. For now, the visualizations can be seen in "attention_region_shift.pdf" in Supplementary Material.

Dear Reviewer PxjZ,

Thank you again for your time. As the deadline for discussion is approaching, we do wish to hear from you to see if our response resolves your concerns. We are happy to provide any additional clarifications if needed.

1. Motivation of using Class Activation Mapping (CAM)

Here's the motivation:

1. To find more effective dropout methods, we need to explore where the synergy is more pronounced across different models.

2. Through our experiments in Fig1(b)(see in our paper), we discovered that the regions where the source model focuses while the target model does not have the most substantial negative impact on transferability. This leads us to the conclusion that the differences between the attention regions of two models have the most negative impact on transferability.

3. Combining 1 and 2 above, we need to identify the difference of attention region across different models. Observing from Fig2(see in our paper), it is noted that a model's attention region consists of some meaningful blocks. Hence, if we decouple the perturbations in different blocks, we can dropout more effectively.

2. About Feature-level attack

Firstly, we discover that the synergy among noises at the image-level affects the transferability of the model, so we propose to enhance transferability by suppressing this synergy at the image level. As a result, directly implementing feature-level dropout seemed obtrusive, so we did not adopt feature-level dropout in our paper.

However, here, we are considering the preliminary design of a feature-level dropout method to improve the transferability of adversarial examples. We carefully redesign a feature-level dropout method (called FAPD). Specifically, during the iteration process, when an image generates a feature, we apply dropout with a probability of 0.1 to this feature. When we have decided to perform dropout on a feature, we replace each neuron by the clean image's feature with a probability $p$. Inspired by [1], we choose the layers to be all convolutional layers and linear layers when the output's spatial size is less than or equal to 1/16 of the original input size.

[1] Introducing Competition to Boost the Transferability of Targeted Adversarial Examples through Clean Feature Mixup. ICCV 2023.

Results in Table[1] show a comparison between the baseline (RDI-MI) and ours (FAPD-RDI-MI). Results in Table[2] show a comparison among different dropout probabilities $p$.

Table[1]: Attack Success Rate(%) of feature level attack ($p=0.5$)

Table[2]: Attack Success Rate(%) of feature level attack with different dropout probability $p$

Dear Reviewer at4m,

Thank you again for your time. As the deadline for discussion is approaching, we do wish to hear from you to see if our response resolves your concerns. We are happy to provide any additional clarifications if needed.