Flatness-aware Adversarial Attack

Thank you for your efforts on our paper and your valuable suggestions. We answer your questions point-by-point as follows:

Question 1: The launch experiment is not solid enough.

Response: We would like to clarify some misunderstandings concerning our launch experiment. Our launch experiment aims to show that adversarial examples residing in flat regions enjoy better transferability (i.e., the necessary condition), instead of demonstrating the equivalence between the transferability of adversarial samples and flat regions. As shown by you, the results support our objective (the necessity).

In fact, this paper delves into the transferability problem, with the hope of augmenting the transferability for the effectiveness of black-box attacks. In Section 1, drawing inspiration from input regularization methods, we hypothesize that adversarial examples located in flat regions enjoy better transferability. This conjecture is preliminarily supported by visualizations of the loss landscape (Figure 1). To further substantiate this hypothesis, we conduct the launch experiment. Leveraging the empirical evidence gathered, we are led to the conviction that adversarial examples in flat regions indeed possess superior transferability. Consequently, we propose our method, FAA, to improve the transferability of adversarial examples. By the way, Appendix C also offers a theoretical demonstration to support this hypothesis.

Question 2: The novelty of this paper.

Response: We carefully check [1] and think it is enlightening. However, this paper distinguishes itself from [1] by several key parts, constituting our contributions. The significance of transferability research to our community is two-fold: first, it enhances the understanding of transferability, which has implications for various related fields; second, it improves the effectiveness of black-box attacks, with potential implications for the development of defenses and increasing awareness of vulnerabilities in neural networks. Herein, we expound on how our work provides new insights into these two aspects.

(I) Our motivation varies. Existing literature, including [1,2], analogizes the transferability of adversarial examples to models' generalization ability, inspiring exploration of techniques that enhance generalization to boost transferability, such as SAM-like optimization formulation [2]. Unlike these, to our best knowledge, we are the first to observe the potential associations between input regularization methods and flat regions (paragraph 1 in Intro). By clarifying how input regularization methods enable adversarial example convergence towards flat regions, we illuminate their limitations (paragraph 3 in Intro). This insight motivates the design of FAA (paragraph 3 in Intro). Thus, this paper deepens our understanding of existing input regularization methods by rethinking them from a unique perspective.

(II) On the technical pathway, despite the similarity between Eq. 3 in [1] and Eq. 4 in our paper, their underlying technical trajectories differ fundamentally. They penalize maximum norms while we penalize gradients of samples around $x+\delta$. This subtle variation can indeed lead to significant differences. For instance, L1 and L2 regularizations appear highly similar in form, but their underlying mechanisms and effects are substantially different (L1 regularization induces sparsity). Specifically, from the perspective of conjugate functions [3], loosely speaking, Eq.3 in [1] would be a relaxation of Eq. 4 in our paper. In other words, our penalty term provides a tighter bound for flat regions, resulting in the superior performance of our method.

(III) Our theoretical contribution is pioneering (Appendix C). To the best of our knowledge, we are the first to provide theoretical proof validating the relationship between transferability and flat regions. While [2] speculates on this relationship based on intuition, [1] follows suit. Their results are empirical, whereas ours possess theoretical underpinnings, positioning our paper in parallel rather than merely building upon (or overlapping) their work.

(IV) The effectiveness of our method is significant. FAA is the first to achieve a success rate of over 90% in attacking nearly all mainstream models, spanning CNNs and transformers, with or without defenses. Our attacks on real-world systems further demonstrate its effectiveness in the physical world. As evidenced in responses to questions 4 and 5, our attack success rate surpasses baselines by a clear margin. For practitioners and real-world applications, the primary concern lies in effectiveness rather than other factors. From this standpoint, our attacks hold immense value for industrial and DNN applications.

In summary, these four significant parts constitute the contributions that set this paper apart from others, paving the way for advancements in this field.

Question 5: I would be interested in $l_0$/patch-based optimization and transferability, because it is the most applicable techniques for the real-world adversarial attacks, and the authors' thought on extensibility of their framework to this case (taking into account the differentiability problem).

Response: We believe the extension of FAA to patch-based attacks is easy. We can directly add our gradient penalty term to the optimization targets in [4].

In contrast, the constraint of L0 norm is quite hard to handle, primarily stemming from the difficulty in identifying the optimal set of pixels to optimize. In response, we advocate for an approximation solution as a remedy. Specifically, we relax L0 norm constraint by introducing an L1 regularization term (or other sparsity-inducing terms) in the optimization target (Eq. 4) to supplant the L0 norm constraint. This ensures the differentiability of the optimization target (Eq. 4). Finally，we retain perturbation elements that exert the most substantial influence on the optimization target (Eq. 4)，thereby satisfying L0 constraint.

---

Question 6: Why we need the Appendix B (minmax problem complexity)? It seems like a very obvious thing and doesn't provide any insight at all.

Response: To our knowledge, some researchers seem to not be very familiar with the difficulty of employing gradient descent algorithms to address the bi-level optimization problem. Thus, we provide an example to supplement this.

---

Reference:

[1] On the Second Mean-Value Theorem of the Integral Calculus, 1909

[2] On the Robustness of Split Learning against Adversarial Attacks, ECAI 2023.

[3] Explaining and harnessing adversarial examples, ICLR 2014.

[4] Adversarial T-shirt! Evading Person Detectors in A Physical World, ECCV 2020.

Thank you for the efforts on our paper and your valuable suggestions. We have included the following response in the revised paper. We will upload the revised version before the end of the discussion period. We answer your questions point-by-point as follows.

---

Question 1: I am very interested whether the theoretic part (see my notes above) can be corrected and improved as now it seems to have a lot of mistakes.

Response: For the first one, I'm not sure that the Eq. (9) is correct... In Equation 9, we treat $L(F(\cdot),\cdot)$ as a holistic function and differentiate it, so there is no mathematical error in this expression. Regarding the effectiveness of linear expansion, we want to provide some discussion to support Equation 9. Firstly, for the domain of adversarial examples, the assumption of neural networks as approximately linear is common, as seen in [3]. The supporting argument is that modern neural networks still exhibit a high degree of linearity, as the activation functions of neural networks often demonstrate favorable linear properties, such as ReLU. Furthermore, albeit empirically, assuming linearity in neural networks when studying adversarial examples often yields good results. Our attack results also support this. Lastly, within a small neighborhood, this assumption is often feasible, as per the Taylor series expansion. Our perturbations are typically constrained to a small magnitude to maintain the human-imperceptibility of crafted adversarial examples.

For the second one, 'we use $p(x) \leq p(x + \delta)$ which is incorrect, and should be $p(x) \geq p(x+\delta)$'. We have revised this typo. Thanks for your careful check again.

For the third one, "Notice that the flatness-aware item punishes the norm of gradients of samples around $x + \delta$. Therefore, this induces..." We want to clarify some things. In fact, according to Equation 4, our gradient penalty term is $||\nabla L(F(x+\delta+\Delta),y)||, \Delta \sim U(-b,b)$ instead of $||\nabla L(F(x+\delta),y)||$. The second derivative can be considered as a metric for the change rate of the first-order derivative. Intuitively speaking, when the penalty strength is sufficiently large, there is $||\nabla L(F(x+\delta+\Delta),y)|| \rightarrow 0$. Consequently, the second derivative at $x + \delta$ naturally becomes zero. Formally, the norm of the second derivative can be expressed by $||\nabla^2 L(F(x+\delta),y)|| = ||\lim_{\Delta \rightarrow 0} \{ (\nabla L(F(x+\delta+\Delta),y) - \nabla L(F(x+\delta),y)) \Delta^{-1} \}||$. Punishing $||\nabla L(F(x+\delta+\Delta),y)||$ indeed minimizes $||\nabla^2 L(F(x+\delta),y)||$.

For the fourth one, at the end of Appendix C, when doing all the approximations, nothing... When the proxy and target models exhibit similarity, the gradients of the proxy and target model also present high similarity. In fact, as demonstrated in [1] (the conclusion of Section 5.1), The distance between $log p(x)$ and $log F(x)$ can control the magnitude of $||\nabla log p(x) - \nabla log F(x)||$. In other words, a smaller distance between $log p(x)$ and $log F(x)$ results in smaller gradient distances. Empirically, a well-trained model $F(x)$ typically yields a smaller $||\nabla log p(x) - \nabla log F(x)||$. Consequently, we are also able to impose a certain degree of penalty on $\nabla log p(x+\delta)$.

---

Question 2: Page 4, the ref. to Mean Value Theorem: it'd be better to refer to some classic mathematical results (I guess they are discovered hundreds of years ago, not just 15).

Response: To the best of our understanding, the origins of the Mean Value Theorem can be traced back to 1823, when its formal proof was established by mathematician Cauchy. We have updated our references accordingly.

---

Question 3: Page 16, Table 5: It's not clear what "Approximation Error" exactly means: is it the overall Hessian-based additive term, or Term1, or Term2 (Equation 8)?

Response: The approximation error is the overall Hessian-based additive term. We have revised the manuscript to enhance its clarity.

---

Question 4: Page 19, Appendix E.8: "As shown in Figure 9, FAA produces flatter adversarial examples than FAA" -> "As shown in Figure 9, FAA produces flatter adversarial examples than RAP"?

Response: Thanks for your careful check. This is a typo error and it is "As shown in Figure 9, FAA produces flatter adversarial examples than RAP". We have revised this typo.