We are grateful to the reviewer for their valuable feedback, and we will address the following issues in our response. (If the latex formula is not rendered, please refresh the page.)

Q1: Explanation of Feature Importance Assessment Methods

In Sec. A.1 of the appendix, we provided a brief introduction to the feature importance assessment methods used in different feature-level attacks. However, we did not offer sufficient details regarding the implementation of these methods or the distinctions between them. Next, I will provide a detailed explanation.

• FIA: In FIA, feature importance $W$ can be expressed as

  $$W = -\bar{\Delta} _ k^x = -\frac{\sum _ {n=1}^N \Delta _ k^{\mathcal{T} _ n(x)}}{|| \sum _ {n=1}^N \Delta _ k^{\mathcal{T} _ n(x)} || _ 2}, ~~ \mathcal{T} _ n(x) = x \odot M _ {p _ d}^n, ~~ M _ {p _ d}^n \sim Bernoulli(1 - p _ d),$$

  where $\bar{\Delta} _ k^x$ denotes the aggregate gradient, $\Delta_k^x = \frac{\partial l(x, y)}{\partial f_k(x)}$, $l(\cdot, y)$ denotes the logits output concerning the ground-truth label $y$, and $M_{p_d}^n$ is a binary matrix that satisfies the $Bernoulli(1 - p_d)$ distribution, where $p_d$ represents the random pixel dropping rate. Therefore, the aggregated gradient $\bar{\Delta}_k^x$ can be summarized as performing a batch of random pixel-dropping transformations on the original image and then unitizing the gradient sum of the feature maps of the transformed images.

• RPA: The difference between RPA and FIA lies only in the different transformation $\mathcal{T}$, which results in different feature importance $W$. Specifically, FIA applies random pixel-dropping transformations to the image, while RPA first generates a mask $M$ of the same size as $x$, which is a matrix with all elements of 1. Then, $M$ is divided into regular, non-overlapping patches $P$, with each patch size being $n^2$. Next, we randomly select a subset of patches with the probability of $P_m$: $P_{p_m}=Rand(P,p_m)$ and modify them to follow a uniform distribution: $P_m \sim U[0,1)$. Therefore, the transformation $\mathcal{T}$ of RPA can be expressed as $\mathcal{T}_n(x) = x \odot M$.

• NAA: In NAA, the feature importance $W$ can be expressed as follows:

  $$W = -\frac{\sum_{m=1}^n \frac{\partial F(x_m, y)}{\partial f_k(x_m)}}{|| \sum_{m=1}^n \frac{\partial F(x_m, y)}{\partial f_k(x_m)} ||_2}$$

  where $F(\cdot, y)$ denotes the softmax output of the true label $y$, $x_m = (1 - \frac{m}{n})x' + \frac{m}{n} x$ and $x'$ denotes a baseline image. Therefore, the feature importance $W$ of NAA can be understood as taking $n$ points along the linear path from the baseline image $x'$ to the input image, calculating the unitized result of the gradient sum of the feature map of these $n$ points, and finally adding a negative sign.

• DANAA: The difference between DANAA and NAA lies only in that the paths used to compute $x_m$, which results in different feature importance $W$. DANAA uses a non-linear path instead of the linear path. Specifically, $x^m = x^0 + \sum_{k=0}^{m-1} \Delta x^k$, where $\Delta x^k = lr \cdot \text{sign}\left( \frac{\partial F(x^k)}{\partial x^k_i} \right) + N(0, \sigma)$, $\frac{\partial F}{\partial x^k_i}(\cdot)$ is the partial derivative of $F$ to the $i$-th pixel, $lr$ denotes learning rate and $N(0,\sigma)$ denotes Gaussian noise.

• SFVA: In SFVA, the feature importance $W$ can be expressed as follows:

  $$W = -\hat{W}^*=-\frac{1}{c} \sum_{i=1}^{N}\frac{\partial F(x'_i)}{\partial f_k(x'_i)}$$

  where $\hat{W}^*$ denotes the optimal feature weights, $x'_i=Scale(Mask(x)+\gamma_i)$. Specifically, we sequentially perform random mask, random addition of noise, and scale transformation on the original image $N$ times, and then compute the unitized result of the gradient sum of the feature map of the transformed images.

• BFA: In BFA, feature importance $W$ can be expressed as follows:

  $$W =I = \frac{1}{N}{\sum_{m=1}^N \frac{\partial F(x_m^{IF}, y)}{\partial f_k(x_m^{IF})}}$$

  where $x_m^{IF}$ denotes the fitted image at the $mth$ iteration. Specifically, we compute the average of the fitted gradients of the fitted images with different degrees of fit to represent the feature importance $W$.

Q2: Lack of Transformer-based models

The target models we selected, while classic, may not be sufficiently advanced and are exclusively CNN-based. We have additionally included four Transformer-based models, i.e., PiT-S, CaiT-S, DeiT-B, and Swin-B, as target models. The experimental results are presented in the response to Reviewer ASTV’s Q3 and Reviewer Li9f’s Q1. These results demonstrate that our proposed P2FA continues to exhibit higher transferability, further substantiating the effectiveness of our approach.

Finally, we express our gratitude for your valuable feedback, which will greatly contribute to improving the quality of our manuscript. We look forward to your response.

Q1: Lack of Transformer-based models

You rightly pointed out that Transformer-based models dominate contemporary research, and there is a legitimate interest in understanding how our proposed attack performs against these more advanced architectures.

In response to your comment, we have followed your suggestion and expanded our experimental evaluation to include four Transformer-based models: Swin Transformer (Swin-B)[1], Data-efficient Image Transformer (DeiT-B)[2], Pooling-based Vision Transformer (PiT-S)[3], and Class-Attention in Image Transformers (CaiT-S)[4]. These Transformer-based models were used as target models by the recent work RPA[5]. The preliminary results are shown in Table 1 (more experimental results can be found in Reviewer ASTV's Q3), demonstrating that our attack method remains effective against these models and outperforms the current SOTA feature importance-based attack, BFA. Detailed results and analysis will be incorporated into the revised manuscript.

Table 1. Attack Success Rates of P2FA(Ours), BFA, and their Combinations with Input Transformations on Transformer-based Models, Using Inception-v3 as the Surrogate Model

We believe this extension not only addresses the limitation you identified but also enhances the practical relevance of our findings to the current research landscape. We will include a discussion of this limitation, the newly added experimental results, and our planned follow-up experiments in the revised manuscript to reflect your suggestion adequately.

[1] Liu Z, Lin Y, Cao Y, et al. Swin transformer: Hierarchical vision transformer using shifted windows[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2021: 10012-10022.

[2] Touvron H, Cord M, Douze M, et al. Training data-efficient image transformers & distillation through attention[C]//International conference on machine learning. PMLR, 2021: 10347-10357.

[3] Heo B, Yun S, Han D, et al. Rethinking spatial dimensions of vision transformers[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2021: 11936-11945.

[4] Touvron H, Cord M, Sablayrolles A, et al. Going deeper with image transformers[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2021: 32-42.

[5] Zhang Y, Tan Y, Chen T, et al. Enhancing the Transferability of Adversarial Examples with Random Patch[C]//IJCAI. 2022, 8: 13.

Q2: Clarification of innovations

Regarding innovation concerns, this paper offers two key contributions.

• Innovation 1: Discover the inefficiency of existing feature importance-based attacks through mathematical proofs. We would like to highlight that our work stems from a comprehensive analysis of existing feature importance-based attacks. Through mathematical proofs (the latest proof see Reviewer ASTV's Q1) and experimental validation, we reveal that existing feature importance-based attacks, relying on multiple pixel-space perturbations, equate to just one step in the feature space along feature importance. Therefore, the perturbation efficiency of existing feature importance-based attacks is inefficient, which also motivates the proposed P2FA to apply feature-space perturbations directly to improve the perturbation efficiency.

• Innovation 2: A new paradigm for transforming perturbed space from pixel to feature space. Based on the above proof and experiments, P2FA shifts perturbation from pixel to feature space, applying multiple efficient perturbations to critical features, boosting transferability across models. This redefines feature importance-based attack paradigms, and we will clarify these innovations in the revised introduction and methodology sections.

We are grateful for your feedback, which has prompted us to refine our exposition. In the revised manuscript, we will expand the discussion in the introduction and methodology sections to clearly articulate this innovation and underscore our contributions.

Q3: Poorly expressed

We thank you for noting ambiguity in our phrasing: “is effectively equivalent to perturbing the features only once along the direction of feature importance in the feature space.” We meant one perturbation step in the feature space matches the effect of multiple pixel-space perturbations. To avoid confusion, we will revise it to: “is effectively equivalent to perturbing the features in one step along the direction of feature importance in the feature space.” This clarification will improve readability in the updated manuscript.

We thank the reviewer for their valuable feedback, which we address below. (If the latex formula is not rendered, please refresh the page.)

Q1: Theoretical Proof

We appreciate the reviewer spotting an error in Eq. (11). We omitted the constraint on $x^{adv}$. Next, we will focus on rigorously proving that multiple pixel-space perturbations in Eq. (8) equal a single feature-space perturbation along feature importance $W$. First, we equivalently rewrite Eq. (8) as:

$$\underset{x^{adv}}{\arg \max} \langle W, f_k(x^{adv}) - f_k(x) \rangle,~ s.t.~||x^{adv} - x ||_p \leq \epsilon.$$

Then, using the Cauchy–Schwarz inequality ($\langle u, v \rangle \leq ||u||_2||v||_2$, equality when $u = s \cdot v, s \geq 0$), the following inequality still holds:

$$\left \langle W, f_k(x^{adv}) - f_k(x) \right \rangle \leq ||W||_2 || f_k(x^{adv}) - f_k(x) ||_2$$

with equality holding when $f_k(x^{adv}) = f_k(x) + s \cdot W (s \geq 0)$ and $||x^{adv} - x||_p \leq \epsilon$, also achieving the optimum. The role of $s$ is to ensure that the adversarial example obtained through feature inversion satisfies $||x^{adv} - x||_p \leq \epsilon$. In practice, we additionally add a clip function to ensure this. In other words, we only need a single feature-space perturbation $s \cdot W$ and satisfy the $\epsilon$-ball to achieve the optimum of Eq. (8). Experiments in Sec. 3.2 also validates the correctness of the claimed conclusion. We’ll update the manuscript with a more detailed version of this proof and revise Fig. 2.

Q2: Efficiency

We clarify that efficiency in the submitted manuscript refers to fewer perturbation iterations ($T=3$ for P2FA vs. $T=10$ for baseline). As you rightly noted, efficiency also includes time efficiency. To fairly compare time efficiency, we tuned BFA’s hyperparameters ($T$, $N$) to match P2FA’s success rate on ImageNet-NIPS with Inception-v3 on an RTX 4090. P2FA ($T=3, N=30$) achieves 84.1% average success in 0.616s/example, while BFA ($T=50, N=200$) reaches 83.9% but in 1.577s/example. BFA is 2.5x slower than P2FA. We’ll add detailed data to the revised manuscript.

Q3: SIA and BSR

SIA and BSR are two highly commendable works. In response to your feedback, we have conducted additional experiments integrating P2FA with SIA and BSR. Partial experimental results are presented in the table below.

Results show P2FA+SIA and P2FA+BSR achieve higher transferability than SOTA BFA with SIA and BSR. Full results will be added to the revision.

Q4: Step Size

Q4.1: Large Step Size

P2FA requires a large step size for effective perturbations due to small $W_t$ values (the average order of magnitude is between $10^{-6}$ and $10^{-5}$) relative to the intermediate layer’s feature $f_{k,t}$ (the average order of magnitude is $10^{-1}$).

Q4.2: Low Sensitivity Beyond $10^3$

Clipping within the $\epsilon$-ball limits the feature-space perturbation range when $s > 10^3$, reducing sensitivity. The average values of pre- and post-clip features confirm this.

Q4.3: Meaning of $s$

$s$ is the scaling factor from the Cauchy–Schwarz inequality, acting as the step size along $W$. We’ll clarify this in the revision.

Q5: Defense Models

Regrettably, we were unable to find PyTorch versions of Ens3-Inc-v3, Ens4-Inc-v3, and Adv-IncRes-v2, but tested P2FA on PiT-S, CaiT-S, DeiT-B, and Swin-B from RPA, showing higher transferability (see Reviewer Li9f’s Q1). Lastly, if you could kindly provide a solution for utilizing above defense models in PyTorch, we would be delighted to conduct further validation of our proposed method.

Q6: Eq. (12)

In gradient-based attacks, cross-entropy $J(x, y) = -\mathbb{1}_y \cdot \log \text{softmax}(f(x))$ is used to update the input image $x$. We shift it to $J(f_k, y) = -\mathbb{1}_y \cdot \log \text{softmax}(f_k^{post}(f_k))$ for feature updates, where $f_k^{post}$ is the post-$k$-th-layer model part. We’ll add this explanation to the revision.

---

We sincerely thank you for enhancing our manuscript’s quality.