Transferability Bound Theory: Exploring Relationship between Adversarial Transferability and Flatness

Q1: The presentation of the the proof of Thm. 3.1 could be improved & The usage of D in the statement of Thm. 3.1 makes a mapping of terms in the proof to the result unnecessarily cumbersome.

Response: In response to your suggestion, we have revised the proof and replaced $D$ to enhance clarity and readability.

Q2: why is $p(x+\delta) \leq p(x)$?

Response: Natural samples are drawn from the real-world distribution that the model was designed to handle. In contrast, adversarial examples are not a direct product of this distribution; they are artificially crafted through specific techniques that exploit the model’s vulnerabilities. Therefore, their occurrence in real-world scenarios, where data is not typically manipulated in such a adversarial manner, is far less common, i.e., $p(x+\delta) \leq p(x)$.

Q3: How does the proposed method compare to PGN [1] which improves transferability through flatness?

Response: We employ ResNet50 as the proxy model to craft adversarial examples for 1000 natural samples, using our attack and PGN. The results, summarized in the table below, demonstrate the superior performance of our attack over PGN. For instance, our attack achieves an impressive ASR of 99.7% on EfficientNet, significantly surpassing the 81.6% achieved by PGN.

Q4: The computational problem of measuring flatness, i.e., the second order gradient components, can be alleviated by considering relative flatness [2], which has also been applied to adversarial examples [3]. Could this be used as an alternative or means to improve the proposed method?

Response: We have carefully read [2,3] and found them to be quite enlightening. Indeed, relative flatness shares a significant similarity to the second-order gradient component in our bound. Nonetheless, calculating relative flatness presents a considerable challenge, as shown in [3], which only addresses the relative flatness concerning the penultimate layer. Similarly, directly penalizing the relative flatness of adversarial examples poses substantial computational difficulty. We value the implications of these inspiring works [2,3] and will include them in the revised manuscript. We also consider exploring relative flatness as a promising avenue for future research.

Question 1&2: The theoretical claims lack a strong and important assumption. & Theorem 3.1 cannot well reflect the bound of the adversarial transferability of inputs.

Response: For first issue, we do not assume $L(F'(x),y)-L(F(x),y) \approx 0$ (See the derivation below).

For second issue, it is a typo and it should be $D(x+\delta,y)= L(F'(x+\delta),y) - L(F(x+\delta),y)$. As you mentioned, the left side of our bound (Theorem 1) should be in squared terms.

Finally, based on revised Theorem 3.1, we can derive the bound of adversarial transferability. Let us briefly restate the revised proof to address your questions.

Based on $\mathcal{L}(F'(x),y)-\mathcal{L}(F(x),y) = L(F'(x+\delta),y) - \nabla L(F'(x+\delta),y)^\top \delta- L(F(x+\delta),y)+\nabla L(F(x+\delta),y)^\top \delta$, we have $D(x+\delta,y) = D(x ,y) + \nabla L(F'(x+\delta),y)^\top \delta - \nabla L(F(x+\delta),y)^\top \delta$.

Taking the $L_2$-norm on both sides and then taking expectiation, we obtain $\int p(x) ||D(x+\delta,y)||_2^2dx\leq \int p(x) ||D(x,y)||_2^2dx+\int p(x)|| \nabla L(F'(x+\delta),y)^\top \delta-\nabla L(F(x+\delta),y)^\top \delta||_2^2dx$.

The main result in Appendix A is Equation 21, which does not use $L(F'(x),y)-L(F(x),y) \approx 0$. Therefore, we have $\int p(x) || \nabla L(F'(x+\delta),y)^\top \delta-\nabla L(F(x+\delta),y)^\top \delta||_2^2 dx \leq (1+C) \int p(x) ||\delta||_2^2 || \nabla \log F(x+\delta) ||_2^2 dx + 2 \sum \int p(x) ||\delta||_2^2 |\nabla^2 \log F(x+\delta)[i,i]| dx + C \int p(x) ||\delta||_2^2 ||\nabla (\log F'(x) - \log F(x))||_2^2 dx.$

Combining the above equations, we get our revised bound (revised Theorem 1): $\mathbb{E} ||D(x+\delta,y)||_2^2 \leq \mathbb{E}\{ ||D(x,y)||_2^2 + C ||\delta||_2^2 ||\nabla D(x,y)||_2^2 \} + (1+C) \mathbb{E} \{ ||\delta||_2^2 || \nabla \log F(x+\delta) ||_2^2 \}+2\mathbb{E} \{ \sum \mathbb{E} ||\delta||_2^2 |\nabla^2 \log F(x+\delta)[i,i]| \}.$

Now, let us consider the bound for adversarial transferability. Specifically, based on $L(F'(x+\delta),y) = D(x+\delta,y)+L(F(x+\delta),y)$ and taking the $L_2$-norm on both sides and applying basic norm properties, we get: $||L(F'(x+\delta),y)||_2^2 \geq | || L(F(x+\delta),y) ||_2^2 - || D(x+\delta,y) ||_2^2 |.$

Since $L \geq 0$ and higher loss of $x+\delta$ on the proxy model than the target model, there is $\mathbb{E} ||L(F'(x+\delta),y)||_2^2 \geq \mathbb{E} || L(F(x+\delta),y) ||_2^2 - \mathbb{E} || D(x+\delta,y) ||_2^2.$ The bound of $\mathbb{E} || D(x+\delta,y) ||_2^2$ is already provided. Thus we can obtain the bound of $\mathbb{E} ||L(F'(x+\delta),y)||_2^2$ (See Response for Reviewer #x5ig's Q1). The loss of adversarial examples on the proxy model is our "local effectiveness term". This inspires design of our attack, as shown in Equation 4 in the original manuscript, where the first term maximizes the local effectiveness term and the second term minimizes the bound about transfer-related loss term.

Question 3: The attack success rates in experiments are not reported with the error bar or the standard deviation.

Response: We have conducted experiments to calculate the error bars and standard deviations and included them in revised manuscript. Below are some results, where we use ResNet50 as the proxy model and run attacks in five trials to report (ASR $\pm$ standard deviation). Our method not only achieves ASRs but also enjoys smaller deviations.

Question 4: It is unknown in Section 6 how the used 100 adversarial examples are crafted, e.g., which proxy model is used, which dataset is used, and how 100 examples are selected.

Response: We randomly select 100 samples from the benchmark evaluation dataset ImageNet [1]. Using these samples, we generate 100 adversarial examples with our attack method, employing the default hyperparameters and ResNet50 as the proxy model. We have added these details in the revised manuscript.

Question 5: The evaluation in Section 6 is conducted by only one volunteer, which is unreliable.

Response: We have recruited two additional volunteers to conduct evaluation. The table below reports the average scores and variance from all three volunteers (the original evaluator + two additional volunteers). The results show a high degree of consistency provided by the three volunteers, reinforcing the superior performance of our method.

[1] On success and simplicity: A second look at transferable targeted attacks

Thank you for your feedback. We have explicitly made the assumptions ($p(x+\delta) \leq p(x)$ and $F'(x+\delta) \geq F(x+\delta)$) in the revised manuscript to avoid any potential confusion.

To clarify, $F'(x+\delta)$ represents the probability that the target model correctly classifies $x+\delta$, i.e., $0 \leq F'(x+\delta) \leq 1$. Regarding $F'(x+\delta) \geq F(x+\delta)$ and the confusion surrounding Equation 12, we have provided a detailed explanation in our response to Reviewer x5ig. Please refer to that for further details.

Concerning $p(x+\delta) \leq p(x)$, there is substantial supporting evidence from literature [1,2,3] and practical considerations. Adversarial examples are artificially crafted through specific algorithms and optimization processes, tailored to particular models and tasks. In practice, the distributions we encounter are typically those generated by the natural processes that generate the data, and these do not favor adversarial examples over natural ones. As such, they are not naturally occurring, and their probability of appearance in the real world is significantly lower compared to natural samples.

Formally, let us consider the following derivation:

Since the ground-truth label $y_g$ for a specific natural sample $x$ is fixed, we can simplify the above equation to:

Intuitively, $\delta$ should satisfy $p(x|y_g)^T \delta \leq 0$, which means that $\delta$ does not increase the probability of the ground-truth label $y_g$ after being applied to $x$. This is because $\delta$ is generated by a proxy model, which is trained on the data distribution $p(x,y)$. Thus, $p(x|y_g)^T \delta > 0$ is counterintuitive and unlikely, unless the proxy model has learned a distribution that is negatively associated with $p(x,y)$. In practice, DNNs, especially those used in real-world applications, perform well on $x$. For poorly performing DNNs, they may already have low accuracy on $x$, and thus generating adversarial examples for them is trivial. Therefore, the assumption that $p(x+\delta) \leq p(x)$ is reasonable.

We hope these clarifications help to strengthen the rationale behind our assumptions and provide a clearer understanding of the context. We also highlight that this manuscript serves as the first theoretical study on adversarial example transferability, and we believe it can offer the community deeper insights into understanding adversarial example transferability.

We look forward to your reply and hope that this addresses your concerns.

[1] On the (Statistical) Detection of Adversarial Examples

[2] Out-of-Distribution Data: An Acquaintance of Adversarial Examples -- A Survey

[3] Interpreting Adversarial Examples in Deep Learning: A Review

Question 1: The tightness of our bound.

Response: We would like to provide some clarifications regarding our bound. First, as pointed out by Reviewer QVii, the first and second terms in Eq.3 should be squared. The revised bound in Theorem 1 is: $\mathbb{E} \{ ||D(x+\delta,y)||_2^2 \leq \mathbb{E} ||D(x,y)||_2^2 + C \mathbb{E} ||\delta||_2^2 ||\nabla D(x,y)||_2^2 \} + (1+C) \mathbb{E} \{ ||\delta||_2^2 || \nabla \log F(x+\delta) ||_2^2 \} + 2 \{ \sum \mathbb{E} ||\delta||_2^2 |\nabla^2 \log F(x+\delta)[i,i]| \}.$ We denote the terms on the right-hand side of the inequality as $Q$. Moreover, in our response to Reviewer QVii, we detail the lower bound for the squared loss of the generated adversarial examples on the target model: $\mathbb{E} ||L(F'(x+\delta),y)||_2^2 \geq \mathbb{E} || L(F(x+\delta),y) ||_2^2 - \mathbb{E} || D(x+\delta,y) ||_2^2 \geq \mathbb{E} || L(F(x+\delta),y) ||_2^2-C.$

We here conduct an empirical evaluation to examine the effectiveness of our bound. We craft 1000 adversarial examples using ResNet50 against DenseNet121. Our estimates show that the sum of squared losses for the examples on the proxy and target models is approximately 280.90 and 76.37, with $\mathbb{E} ||D(x+\delta,y)||_2^2$ of 69.27. We calculate the value of $Q$ to be 170.75, setting $C$ to 1 due to $C \leq 1$. The difference between 170.75 and 69.27 is somewhat non-trivial. However, when we translate this difference into probabilities, it becomes quite minor. Specifically, the loss for the target model on the generated adversarial examples should approximate $10.50$ ($\mathbb{E} || L(F(x+\delta),y) ||_2^2-C=280.90-170.75=10.50^2$). This implies that the probability of the target model correctly classifying the examples is at most $e^{-10.50} \approx 2.75 \times 10^{-5}$, which is a rather small number. In summary, our bound is indeed tight and practical, due to the exponential mechanism and the typically high loss of the generated adversarial examples in the target model. We also evaluate several other models, as detailed in the table below.

Question 2: The target model is black-box and unknown.

Response: What we commonly refer to as a black-box scenario actually permits some queries to the target model but restricts access to its architecture and parameters. This is common across various AI applications, e.g., Google's AI services, which allow users to input data and receive predictions. Moreover, a truly inaccessible target model would negate the possibility of feeding adversarial examples into it, making black-box attacks trivial. Therefore, limited access is practical and reflects actual attack scenarios.

Question 3: Why does the gradient of the target model $F'(x+\delta)$ disappear from the fourth to the fifth line in Equation (12).

Response: The probability of the target model correctly predicting $x+\delta$ should be higher than that of the proxy model, that is, $F'(x+\delta) \geq F(x+\delta)$. Consequently, we have $-F'(x+\delta) \leq -F(x+\delta).$ Additionally, considering the second derivative of $log(x)$, which is $-\frac{1}{x^2} \leq 0$ we have: $- \sum \int p(x)||\delta||_2^2 F'(x+\delta) \nabla^2 \log F(x+\delta)[i,i] dx \geq - \sum \int p(x)||\delta||_2^2 F(x+\delta) \nabla^2 \log F(x+\delta)[i,i] dx.$ We have added a clear explanation of this step to improve the manuscript's readability.

Question 4: Comparison of the TPA method with [11, 41].

Response: We employ ResNet50 as the proxy model to craft adversarial examples for 1000 natural samples, using our attack and the attacks proposed in [11,41]. The results, summarized in the table below, demonstrate that our attack significantly outperforms those described in [11,41]. Notice that the attacks presented in [11] and [41] are almost identical in terms of their objective functions and optimization methods, resulting in nearly identical attack success rates.

Formally, the key difference between our attack and those in [11, 41] lies in the use of uniformly distributed noise. Despite its simplicity, this additional random noise plays a unique and fundamentally important role in enhancing performance, as our theoretical analysis illustrates. This is also the primary reason why our method achieves notably higher success rates compared to [11, 41].

[11] Boosting adversarial transferability by achieving flat local maxima

[41] Gnp attack: Transferable adversarial examples via gradient norm penalty

We appreciate your feedback. We would like to provide further clarification as we sense some misunderstandings here. We also look forward to your reply and hope that this addresses your concerns.

Question: Our bound.

Response: Notice that the expected square loss of the generated adversarial examples on the target model (DenseNet121) is $10.50^2$. In statistical terms, while individual sample's squared loss may vary around this expected value, we expect that the majority of samples will exhibit comparable losses. To be more specific, we empirically evaluate the variance of the squared losses to be 54.62 (out of 1000 adversarial examples generated by our attack). According to Chebyshev's inequality, at least 96% of the samples should lie within 5 standard deviations of the expected value. Within this range, the squared losses of our generated adversarial examples should be at least $10.50^2 - 5 \times \sqrt{54.62} = 73.30 \approx 8.56^2$.

This implies that our generated adversarial samples incur a loss greater than 8.56 on DenseNet121 with at least 95% probability. A loss of 8.56 indicates that the probability of correct classification by DenseNet121 for these samples is approximately 0.0002. In other words, out of every hundred samples, at least 95 can effectively mislead the target model. Doesn't this high probability of successfully attacking the target model sufficiently demonstrate the effectiveness and practicality of our bound?

Question: The effectiveness of TPA.

Response: As stated in our introduction, inspiring works such as [1, 11, 41] prompt us to investigate the theoretical relationship between flatness and the transferability of adversarial examples. Our theory suggests that penalizing the first and second-order gradients of generated adversarial examples can effectively enhance their transferability. Notably, prior works [1, 11, 41] did not include penalties on second-order gradient. Our proposed method is simple yet quite effective, which can generate more transferable adversarial examples by penalizing both one-second and second-order gradients via additional noise. We acknowledge the contributions made by these existing works in our introduction and clarify the distinctions between our method and theirs.

Thank the authors for your rebuttals carefully. My concerns about unclear details and experiments are well addressed. Thus, I would like to raise my final score. Besides, the assumptions must be explicitly presented and explained in the final version.

Question: The assumption about $F'(x+\delta) \geq F(x+\delta)$.

Response: Regarding the second concern, there are some typos in the appendix of the original paper. Let us consider the following derivation: $\sum \int p(x) \Vert \delta \Vert _2^2 \nabla F'(x+\delta)[i] \cdot \nabla \log F(x+\delta)[i] dx$

$= \sum p(x) \Vert \delta \Vert_2^2 F'(x+\delta) \cdot \nabla \log F(x+\delta)[i] |_a^b - \sum \int p(x) \Vert \delta \Vert_2^2 F'(x+\delta) \nabla^2 \log F(x+\delta)[i,i] dx$

$= - \sum \int p(x)\Vert \delta \Vert_2^2 F'(x+\delta) \nabla^2 \log F(x+\delta)[i,i] dx \geq - \sum \int p(x) \Vert \delta \Vert_2^2 F(x+\delta) \nabla^2 \log F(x+\delta)[i,i] dx,$ where $a=+\infty, b=-\infty$. The second equality utilizes integration by parts, the third equality uses $p(\infty)=0$ and the fourth equality leverages $F'(x+\delta) \geq F(x+\delta)$ and the fact that the second derivative of $\log$ is negative. This expression indeed derives $\int p(x) \Vert \delta \Vert_2^2 \nabla \log F'(x+\delta)^\top \nabla \log F(x+\delta) dx \geq - \sum_{i} \int p(x) \Vert \delta \Vert_2^2 F(x+\delta) \nabla^2 \log F(x+\delta)[i,i] dx.$

We rely on $F'(x+\delta) \geq F(x+\delta)$, a generally valid assumption. $\delta$ typically maximizes the loss function on the proxy model, implying $F'(x+\delta)$ optimally suits the proxy, i.e., $F'(x+\delta) \geq F(x+\delta)$. Moreover, we conduct experiments using BIM and our method to generate adversarial examples for 1000 natural samples on ResNet50. For DenseNet121, EfficientNet, VGG19, ConvNet, and Vision Transformer (ViT), all correctly predict these generated adversarial examples with higher confidence compared to the proxy model, meaning they assign a higher prediction probability to the ground-truth class. Specifically, in the presence of adversarial examples generated by our attack, DenseNet121, EfficientNet, VGG19, ConvNet, and ViT have correct prediction probabilities that are 61 times, 197 times, 406 times, 1058 times, and 3674 times higher, respectively, than those of the proxy model ResNet50 (note that since the proxy model's correct prediction probability for adversarial examples is typically very tiny, such as around $10^{-7}$, the target models are still easily misled by these adversarial examples). Thus, assuming $F'(x+\delta) \geq F(x+\delta)$ in transfer attack scenarios is justified.

Furthermore, even if we consider the existence of peculiar samples satisfying $F'(x+\delta) > F(x+\delta)$, there is no need to investigate these further because $F'(x+\delta)>F(x+\delta)$ already suggests that $x+\delta$ can trick the target model (since adversarial examples generated on the proxy model often mislead the proxy itself). In other words, transfer-based attacks are meaningful only if $F'(x+\delta) \geq F(x+\delta)$. For theoretical rigor, we can adjust our analysis as follows: $- \sum \int p(x)\Vert \delta \Vert_2^2 F'(x+\delta) \nabla^2 \log F(x+\delta)[i,i] dx = - \sum \int_{U_1} p(x)\Vert \delta \Vert_2^2 F'(x+\delta) \nabla^2 \log F(x+\delta)[i,i] dx - \sum \int_{U_2} p(x)\Vert \delta \Vert_2^2 F'(x+\delta) \nabla^2 \log F(x+\delta)[i,i] dx$

Here, we split the entire integration domain into two non-overlapping parts: $U_1$ where $F'(x+\delta) \geq F(x+\delta)$ and $U_2$ where $F'(x+\delta) < F(x+\delta)$. Due to $- \sum \int_{U_2} p(x)\Vert \delta \Vert_2^2 F'(x+\delta) \nabla^2 \log F(x+\delta)[i,i] dx \geq 0$, we can disregard this term. This adjustment does not compromise the integrity of our theory; it merely restricts the integration interval to $U_1$. We hope this can address your concerns.