Rethinking Model Ensemble in Transfer-based Adversarial Attacks

Thank you for appreciating our new contributions as well as providing the valuable feedback. We have uploaded a revision of our paper. Below we address the detailed comments, and hope that you may find our response satisfactory.

Question 1: The description of the experimental part could be improved.

Thank you for the suggestion. We have improved the description of the experimental results and discussed the results of VMI and SSA in the revision. The detailed discussion is as follows.

By integrating our proposed CWA with recent state-of-the-art attacks VMI and SSA, the resultant attacks VMI-CWA and SSA-CWA achieve a significant level of attacking performance. Typically, SSA-CWA achieves more than 99% attack success rates for most normal models. VMI-CWA and SSA-CWA can also outperform their vanilla versions VMI and SSA by a large margin. The results not only demonstrate the effectiveness of our proposed method when integrating with other attacks, but also prove the vulnerability of existing image classifiers under strong transfer-based attacks.

Question 2: How to explain the significant performance degradation of VMI-CWA compared to VMI in Tab 1 when the target models are Swin-S and Max ViT-T?

We are sorry that we showed the wrong results of VMI-CWA in Table 1. In our submitted code, we implemented two versions of VMI-CWA. The first one is based on Algorithm 4 in Appendix B.3, in which we adopt the gradients of VMI and perform iterative updates based on our algorithm. The second version is slightly different, which adopts the gradients of our method to perform updates based on VMI. In practice, we observed that the first version performs better, but we showed the results of the second version in Table 1. In the revision, we have corrected the results of VMI-CWA in Table 1. Now the performance of VMI-CWA is better than VMI for the transformers Swin-S and MaxViT-T.

Thanks for your reply. My concern has been addressed and I will keep the rating, i.e., tend to accept this paper.

Thank you for appreciating our new contributions as well as providing the valuable feedback. We have uploaded a revision of our paper. Below we address the detailed comments, and hope that you may find our response satisfactory.

Question 1: The intuition behind SAM and CSE is not clear enough.

Thank you for pointing this out. SAM is an effective method to acquire a flatter landscape, which is formulated as a min-max optimization problem, in a similar way to adversarial training. The inner maximization aims to find a direction along which the loss changes more rapidly; while the outer problem minimizes the loss at this direction to improve the flatness of loss landscape. We have added the introduction of SAM in the beginning of Section 3.3.

For CSE, to maximize the cosine similarity between the gradients of different models, we are inspired by Nichol et al. [1] to develop a first-order algorithm. Nichol et al. propose a meta-learning algorithm that iteratively performs gradient updates for each task, and they theoretically show that this procedure can increase the inner product between gradients of different tasks. We extend this method to also increase the cosine similarity between the gradients of different models, with theoretical analyses in Appendix B.2.

Question 2: Overclaim on fooling all target models.

Thank you for pointing this out. We have revised our paper to avoid this overclaim. Now the text becomes

Given a natural image $\boldsymbol{x}_{nat}$ and the corresponding label $y$, transfer-based attacks aim to craft an adversarial example $\boldsymbol{x}$ that could be misclassified by the models in $\mathcal{F}$.

Question 3: The perturbation is Figure 3 is not imperceptible to humans.

We are sorry that the first image in Figure 3 was placed wrongly -- we used the adversarial image with $\epsilon=32/255$, such that this image is more perceptible. In the revision, it is changed to the correct image with $\epsilon=16/255$. The perturbations are visually similar to those generated by other attack methods with the same perturbation budget.

Reference:

[1] Nichol et al., On first-order meta-learning algorithms, 2018.

Thank you for the supportive review. We are encouraged by the appreciation of the significance, novelty, and effectiveness of our method. We have uploaded a revision of our paper. Below we address the detailed comments, and hope that you may find our response satisfactory.

Question 1: The paper was bit hard to follow.

Thank you for pointing out the issues. We further improve the writing to address them in the revision. Below are the details.

• Improve the readability of SAM. We have provided a brief introduction of SAM in the beginning of Section 3.3, to provide the background knowledge of SAM and explain why it can be used for our problem.

• Figures 1 and 2 are not very informative. We have updated Figures 1 and 2 to be more informative, including adding the axes and labels, changing the colors, and specifying the symbols.

• Include a mathematical description of the prior ensemble attacks. We have added a new subsection 3.1 in the revision to introduce the backgrounds of adversarial attacks.

We hope that the new revision can be easier to follow. We will further improve the paper in the final.

Question 2: Lack of analysis on complexity.

Thank you for the suggestion. We discussed the computational efficiency in Section 4.4 and Appendix D.4. We further report the actual runtime of different attack methods in Table D.2. Our method incurs only a slight computational overhead compared to the baseline attacks. This demonstrates that our method can serve as a plug-and-play algorithm to effectively and efficiently enhance transfer attack performance. As also discussed in Section 4.4, given a fixed computation overhead, we can use a smaller number of attack iterations in our method, which also leads to better results, as shown in Figure 4(c).

Question 3: Various typos and minor problems.

Thank you for pointing them out. We address these issues and carefully proofread the paper. We have uploaded a new revision. Below are the details.

• What is being plotted in Figure 1. Figure 1 shows an illustration of common weakness, where we plot the synthetic loss curves of different models w.r.t. the input $\boldsymbol{x}$. We show that the generalization performance is correlated with the flatness of loss landscape and the distance to the local optimum of each model.

• Typos in Sec. 3.1, Theorem 3.1, Sec. 3.2. Thank you for pointing them out. We have corrected them in the revision.

• The symbols in Figure 2 are not defined. We have revised the caption to link the symbols to the equations in the main text.

• The order of classifiers in Alg. 1. The order of classifiers is randomly set and fixed during the optimization process.

• Some highlighting numbers are not the best in Table 1. Thank you for pointing this out. We carefully checked the results and marked the best results in bold in the revision.

• Can the method be applied in the white-box setting. Yes, our method can be applied in the white-box setting. But when there is only one white-box model, our method would degenerate to PGD or MI since we focus on ensemble-based attacks. Therefore, we further show the attack performance of our method against the six white-box models in the ensemble, as detailed in Section 4.1, and compare with PGD-10 and MI. The results are shown below.

It can be seen that our method also leads to higher attack success rates over PGD and MI with a small number of attack iterations. It indicates that our method converges much faster than the baselines since it can align the gradients of different models.

Thank you for the supportive review. We are encouraged by the appreciation of the novelty, clarity, and thorough experiments of this paper. We have uploaded a revision of our paper. Below we address the detailed comments, and hope that you may find our response satisfactory.

Question 1: The performance of CWA only.

Thank you for the suggestion. We further conduct experiments of SAM, CSE, and CWA without integrating with other methods. The detailed results are shown in Table C.5 in Appendix C.7. The average performance of different methods over all black-box models are shown below.

It can be seen that SAM, CSE, and CWA outperform the baselines FGSM, BIM, and MI by a large margin. However, their performance is inferior to their counterparts with MI, indicating that our methods and MI are orthogonal and can be integrated together to further improve the performance.

Question 2: Different norm decomposition.

Thank you for the suggestion. It is true that we can decompose the second-order term $(\boldsymbol{x}-\boldsymbol{p}_i)^\top \boldsymbol{H}_i (\boldsymbol{x}-\boldsymbol{p}_i)$ with different norms (see Appendix A.1 for a more general result). However, with a different norm of $\boldsymbol{H}_i$, it is hard to interpret the meaning of minimizing this norm, and we can hardly associate it with generalization/transferability of adversarial examples. Besides, we need to calculate the third-order derivative of the loss function w.r.t. input $\boldsymbol{x}$, which would be intractable for deep neural networks. Therefore, it is hard to deal with other norms of $\boldsymbol{H}_i$, and we only focus on the Frobenius norm of $\boldsymbol{H}_i$ with a clear connection to loss flatness and an alternative method based on sharpness aware minimization for optimization.

Question 3: Clarify the novelty compared to the previous methods.

The main novelty of this paper lies in a new adversarial attack method from the perspective of model ensembling. Specifically, we use a quadratic approximation of the loss function and discover that the second-order term can be upper bounded by the product of the Frobenius norm of the Hessian matrix and the distance to the local optimum of each model. Therefore, we propose a common weakness attack to minimize the upper bound with great effectiveness. The existing attacks (e.g., MI, VMI, SSA) mainly focus on developing optimization algorithms or data augmentation techniques to improve transferability, while do not consider model ensembling in adversarial attacks. The sharpness aware minimization (SAM) is commonly used to train deep neural networks, but we extend it to adversarial attacks to improve the transferability based on our analysis.

Question 4: Why is the method more effective for adversarial trained model?

The main reason is that our method can better exploit the vulnerabilities of different surrogate models in the ensemble by finding their common weakness. As we included two adversarially trained models (which are publicly available by [1,2]) in the ensemble, the adversarial examples generated by our method can better transfer to attack other black-box adversarially trained models. But for baseline attacks that directly average different surrogate models, they tend to attack easier models (i.e., normally trained models) [3], such that the useful gradient information would be overwhelmed by these normal models, leading to inferior results for adversarially trained models.

Reference:

[1] Salman et al., Do adversarially robust imagenet models transfer better? NeurIPS 2020.

[2] Debenedett et al., A light recipe to train robust vision transformers. IEEE Conference on Secure and Trustworthy Machine Learning, 2023.

[3] Dong et al., Discovering adversarial examples with momentum, https://arxiv.org/abs/1710.06081v1.