Boosting Adversarial Transferability with Spatial Adversarial Alignment

Thank you for your recognition and support of our work. We hope you can be our champion in future discussions.

---

Weakness 1: Difference between alignment and direct ensemble model attack.

We sincerely appreciate the reviewer's insightful question. To clarify the difference between alignment-based methods and direct ensemble model attacks: While both aim to enhance adversarial transferability, their approaches are distinct. Alignment-based methods, such as ours, focus on improving surrogate models themselves by leveraging witness models to align structural features, thus generating more transferable adversarial examples. complementary strengths. These two strategies are complementary rather than conflicting.

Notably, our alignment-based method (SAA) can further enhance ensemble attacks, as demonstrated in Table 5 of the main body. For example, when combined with an ensemble of Res50+ViT-B, SAA improves the attack success rate (ASR) by 20.8% compared to the direct ensemble baseline, outperforming even the state-of-the-art MA method (+9.1%). This confirms that Alignment-based refinement of surrogate models provides orthogonal benefits to ensemble strategies, strengthening their transferability in black-box settings. We hope this clarifies their distinct contributions.

---

Weakness 2: The improvement of the self-aligned strategy.

While the unique spatial features in the self-aligned strategy are relatively limited, the surrogate models can still learn features related to adversarial patterns and category relationships. In terms of adversarial features, the adversarial-aware alignment enables dynamic learning of adversarial characteristics. Since the witness model itself is frozen, the surrogate models can align with adversarial features without deviating significantly from their own inherent properties. Additionally, the surrogate models can still acquire high confidence in the correct categories under soft labels, which also includes the "ambiguity" towards similar categories. Such soft labels guide the surrogate models to learn more robust feature representations, thereby enhancing the transferability of the generated adversarial examples. We believe these factors collectively contribute to the improvements brought by the self-aligned strategy. Thank you again for your valuable feedback.

---

Weakness 3: The differences between SAA and other attacks on ViTs such as SGM, PNA, and TGR.

We appreciate your attention to the differences between SAA and other attacks on ViTs, such as SGM, PNA, and TGR. As shown in Table 4 of our paper, SAA demonstrates stronger cross-architecture transferability compared to these methods, which preliminarily reflects the superiority of SAA in common settings.​

Furthermore, we have provided a detailed discussion on SGM, PNA, and TGR in Appendix A.2. Specifically, the Pay No Attention (PNA) method extends the Skip Gradient Method (SGM) to ViTs by omitting the gradient computation of attention blocks during back-propagation, thereby enhancing adversarial transferability. Token Gradient Regularization (TGR) operates from the perspective of variance reduction, stabilizing the gradient direction to prevent adversarial examples from getting stuck in poor local optima.​

Distinct from these methods, SAA is the first method to specifically analyze architectural differences across models. It enables the creation of more generalized surrogate models that integrate seamlessly with optimization and data augmentation methods without modifying the forward and backward propagation of the model. This unique design is what ultimately allows SAA to achieve state-of-the-art transferability.

Weakness 1: More modern methods is missing.

In fact, we compared many transfer attacks from the past three years. Among aligned-based methods, Model Alignment (MA) is quite new, having been accepted by ECCV 2024, not 2022. The fact that SAA outperforms MA clearly demonstrates its state-of-the-art performance. Furthermore, experiments combining it with other transfer attacks also incorporate a lot of work from the past three years, such as GI (accepted by Expert Systems with Applications 2024).

---

Weakness 2 & Weakness 3: Limited insight and Weak qualitative evidence in section 2.3.

Thank you for your insightful comments, which help us clarify the focus and limitations of our analysis in Section 2.3. We appreciate the opportunity to address your concerns.

First, we acknowledge that qualitatively and quantitatively dissecting differences between model features is inherently challenging. In Section 2.3, our core focus is to verify whether SAA improves the spatial and adversarial features of surrogate models after alignment, rather than analyzing local vs. global features in spatially-aware alignment. This distinction is important: the link between such feature improvements and practical transferability has already been validated through quantitative transferability analysis in Table 7, which demonstrates that enhanced feature alignment directly translates to stronger attack performance across architectures.

Second, regarding the use of cosine similarity: since spatial features here refer to representations of entire images (not localized regions), cosine similarity serves as an intuitive and widely adopted metric to quantify global feature similarity between models. As shown in Table 1, two key observations emerge:

1. there is a significant inherent feature gap between diverse architectures (e.g., Res50 and Swin-B);
2. SAA consistently reduces this gap, with meaningful improvements (e.g., the 49.3% gain for Res50-Swin-B on adversarial examples). These results quantitatively confirm that SAA effectively identifies shared features between distinct models, which in turn enhances adversarial transferability.

Finally, we recognize the limitation of the Grad-CAM examples provided in the main body, which are constrained by space. However, these visualizations do illustrate that aligned surrogates learn features more consistent with the witness model—for instance, in Figure 2, aligned surrogates on clean inputs exhibit feature distributions closer to the witness model’s patterns. To address the need for more evidence, we have included additional visualizations and analyses in Appendix E: these show that adversarial examples from unaligned surrogates retain heatmap distributions similar to clean inputs (indicating poor transfer), while those from aligned surrogates effectively disrupt the target model’s attention patterns, leading to more thorough attacks. This strengthens the case that SAA-driven alignment modifies features in a functionally meaningful way.

We hope these clarifications address your concerns. Thank you again for pushing us to refine our analysis.

---

Question 1: Learning from the fact that some architectures of ViT transfers better to some CNN architectures? e.g., Swin-B to VGG?

Thank you for this insightful question, which prompts us to delve deeper into the transferability patterns between ViT and CNN architectures.

As highlighted in prior work [1], ViTs and CNNs learn distinct feature representations, with notable differences particularly in their early-layer computations. Specifically, [1] demonstrated that aligning ResNet (a CNN) layers to match the hidden representations of a ViT requires modifying more layers than the reverse process (aligning ViT layers to match ResNet). This observation reveals an inherent asymmetry in cross-architecture alignment: ViTs are more adaptable to aligning with CNNs than CNNs are to aligning with ViTs.

This asymmetry sheds light on why certain ViT architectures (e.g., Swin-B) transfer better to specific CNNs (e.g., VGG). ViTs, by nature, exhibit a stronger capacity to capture unique structural features—features that are generalizable enough to align with the representational spaces of CNNs. In contrast, CNNs, with their inductive biases toward local spatial hierarchies, are less flexible in adapting to the global, token-based features learned by ViTs. Consequently, when generating adversarial examples, ViTs can encode features that resonate more effectively with the internal representations of CNNs, leading to stronger transferability from ViT to CNN architectures.

We appreciate this opportunity to clarify this mechanism, and we will emphasize this insight in the revised manuscript. Thank you again for your valuable feedback.

[1] Do vision transformers see like convolutional neural networks? NeurIPS 2021

Thanks for your thoughtful suggestions. During the rebuttal period, we have made careful efforts to address your concerns. Additionally, we have provided detailed responses to your other questions. We earnestly hope that you will reconsider our paper.

---

Weakness 1.1: Supports on SA.

Thanks for your valuable feedback on the motivation in lines 48–61. The analysis in lines 48–61 focuses on potential reasons for the limited cross-architecture transferability of MA, and this is directly validated by the qualitative and quantitative experiments in Section 2.3—our core effort to verify whether SAA improves spatial and adversarial feature alignment.

• For quantitative evidence: As detailed in Section 2.3, we use cosine similarity to measure global feature similarity (since spatial features here refer to entire image representations). Table 1 clearly shows two key results:

  1. there is a significant inherent feature gap between diverse architectures (e.g., Res50 and Swin-B), confirming that spatial features of CNNs and ViTs are distinct;
  2. SAA consistently reduces this gap (e.g., a 49.3% improvement for Res50-Swin-B on adversarial examples), demonstrating that unaligned spatial features hinder transferability—directly supporting the claim that "features are not aligned in space." These results empirically validate the existence of architectural feature differences and misalignment.

• For qualitative evidence: Grad-CAM visualizations in Figure 2 illustrate that unaligned surrogates exhibit feature distributions distinct from witness models, while aligned surrogates (via SAA) learn more consistent patterns—confirming improved spatial alignment. Additionally, Appendix E provides further visualizations: adversarial examples from unaligned surrogates retain heatmap distributions similar to clean inputs (indicating poor transfer due to misaligned features), whereas aligned surrogates produce adversarial examples that disrupt witness model attention, validating the impact of spatial feature alignment on transferability.

To address the need for citations, we will explicitly reference relevant literature in the revised version: "features are not aligned in space" (supported by PNA [51]), and "the spatial features of ViTs and CNNs are different" (supported by [1], [2]).

---

Weakness 1.2: Final prediction.

Thank you for pointing out this concern—we appreciate the opportunity to clarify our statement about Model Alignment (MA, ECCV 2024).

Our claim that "MA only uses the final prediction of the model" is grounded in both MA’s official paper and its implementation, and we aim to clarify this alignment:

1. First, while MA’s Section 3.1 presents formalizations for alignment across different layers, their Section 3.2 explicitly states: "Specifically, when $q = l$, the alignment loss measures the divergence between the probability distributions generated by the two models. In this scenario, the Kullback-Leibler (KL) divergence is particularly suitable, due to its effectiveness in measuring distribution differences and its relative ease of implementation in practice", which aligns with our description of "using the final prediction-related representations" of the model.

2. Second, MA’s publicly available code (model-alignment/src/align.py, lines 35–37) explicitly uses the model’s final output as the basis for computing the alignment loss. This implementation directly confirms that MA focuses on the final prediction-related representations rather than intermediate layers in practice.

Thus, our statement is consistent with MA’s own paper and official code. We welcome reviewers to verify this via their publicly accessible repository.

---

Weakness 1.3: References on SA.

We will add citations to MA [32] in lines 104–105 and 108–112, and to [2] in lines 115–122. In addition, we will further check the rigor of the description, add the corresponding citations, and update them in the final version.

---

Weakness 1.4: Global features.

Thank you for pointing this out. We would like to clarify that the term "global feature" is primarily used to distinguish it from local features in spatial-aware alignment. This term is used because, in the vision backbone, the final output features reflect global image information.

We will further refine the description in the main body. First, we will highlight the definition and description of global and local features in lines 130-144. Then, we will further deepen the intuitive understanding of global and local features by combining them with Figure 1. We have provided code in the supplementary materials, which is consistent with the description in the paper and hopes to further bridge any gaps in understanding.

---

Weakness 1.5: Dimension explanation.

Thank you for pointing out this issue. The $H$ and $W$ here refer to the dimensions of the local features, not the image dimensions. In image classification models, the downsampling factor is typically 32. For a $224 \times 224$ image, downsampling results in local features of $(224/32)\times(224/32)=7\times 7$. In this case, $H=W=7$ so that the dimension of $z$ is $7 \times 7=49$. We will update this description in the final version.

---

Weakness 1.6: Feature misalignment.

Thank you for this insightful question, which prompts us to clarify the explicit connection between our motivation (addressing cross-architecture misalignment) and the proposed local/global feature perspective.

As summarized in lines 48–61, the limited cross-architecture transferability of MA stems from its exclusive reliance on aligning surrogate and witness models via their final global outputs. While MA assumes feature similarity correlates strongly with transferability, its poor performance across architectures reveals a critical gap: relying solely on final global features fails to capture meaningful similarity in cross-architecture settings. This gap arises because CNNs and ViTs inherently differ in structural designs—including receptive fields (local vs. global), layer stacking mechanisms, and normalization techniques—leading to misaligned spatial and structural features that hinder transferability.

Our proposed spatial-aware alignment directly addresses this by explicitly bridging local and global features, creating a clear link between motivation and method:

• Beyond aligning final global features (as MA does), we additionally focus on learning local regional features. This dual focus targets the structural differences between CNNs and ViTs: local features capture fine-grained spatial patterns critical for CNNs, while global features retain the holistic token-based representations central to ViTs.
• By integrating both perspectives, surrogate models learn not just high-level global similarity but also low-level structural nuances unique to each architecture. This enables them to acquire "unique structural features" that transcend architectural differences—features that are aligned both spatially (locally) and semantically (globally), thereby mitigating cross-architecture misalignment.

In essence, the local/global feature perspective directly responds to the root cause of misalignment (structural differences between CNNs and ViTs) by ensuring alignment at both granularities, strengthening transferability where MA falls short.

---

Weakness 2: Feature similarity.

Yes, we have explained in line 188 that feature similarity is calculated using cosine similarity of global features. In Section 2.3, we focus on verifying whether SAA improves the spatial and adversarial features of surrogate models after alignment. Therefore, we need to verify the feature similarity before and after alignment and the corresponding visualization. Table 2 of the main body already fully demonstrates that the adversarial transferability brought by SAA significantly exceeds that of MA. Therefore, we focus more on the nature of SAA itself in this section. For more qualitative and quantitative analysis, please refer to Weakness 1.

While it might be obvious that a model with better transferability can improve a surrogate model with poor transferability when used as a witness model, in SAA, a model with average transferability can further improve a surrogate model with better transferability. For example, when Res50 is aligned with ViT-B, Res50 learns unique structural features, allowing the generated adversarial examples to still improve transferability. SAA reveals this phenomenon and demonstrates that there are still many areas of urgent research in feature similarity and adversarial transferability.

---

Weakness 3 & Question 1: Computational overhead.

In "GPU Memory and Computing Cost" of Appendix C, we report the training time (GPU hours). For computing cost, training SAA for one epoch on ImageNet (with a batch size of 64) requires about 10 hours using an Nvidia RTX 3090.

Here, we analyze two alignments: spatial-aware alignment supports parallelization, so only one forward and backward propagation is required for loss calculation; for adversarial-aware alignment, since adversarial examples need to be generated, $t$ forward and backward propagations are required. Therefore, the time complexity of training one epoch is $O(1+t)$. In the above practice, spatial-aware alignment and adversarial-aware alignment take about 2 and 8 hours respectively.

Note that the main computational bottleneck is actually obtaining adversarial examples, not model training. Some newer adversarial example techniques [3] may be able to accelerate this process, which is a very promising research direction.

[1] Do vision transformers see like convolutional neural networks?, NeurIPS 2022

[2] Understanding the robustness in vision transformers, ICML 2022

[3] Revisiting Adversarial Training at Scale, CVPR 2024

Weakness 1 & Question 2: The compute budget is not reported (total training time, epochs, GPU hours, or number of adversarial examples).

We would like to clarify that the relevant information is already included in the paper, and we will consolidate it into the "Implementation Details" section in the final version for better accessibility. Specifically:

1. In "Training Epochs" of Appendix C, we note that all experiments in this paper use 1 training epoch. As described in Algorithm 1, the number of adversarial examples generated by SAA is in a 1:1 ratio with the training dataset. Additionally, we explore performance differences across multiple training epochs and find that compared to MA, SAA achieves higher transferability even with fewer epochs, and its transferability reaches a higher upper bound after multiple training rounds.

2. In "GPU Memory and Computing Cost" of Appendix C, we report the training time (GPU hours). For computing cost, training SAA for one epoch on ImageNet (with a batch size of 64) requires approximately 10 hours using an Nvidia RTX 3090.

---

Weakness 2: The security implications are unexplored.

We would like to clarify that the relevant information is already included in Lines 333-335 of the main body: Adversarial examples by SAA exhibit enhanced adversarial transferability, especially in cross-architecture capabilities. This poses a huge threat to the deployment of real-world applications. Simultaneously, it is also conducive to better evaluating their adversarial robustness.

---

Weakness 3: Overreliance on cosine similarity.

In Section 2.3, our core focus is to verify whether SAA improves the spatial and adversarial features of surrogate models after alignment. Regarding the use of cosine similarity: since spatial features here refer to representations of entire images (not localized regions), cosine similarity serves as an intuitive and widely adopted metric to quantify global feature similarity between models. As shown in Table 1 of the main body, two key observations emerge:

1. there is a significant inherent feature gap between diverse architectures (e.g., Res50 and Swin-B);
2. SAA consistently reduces this gap, with meaningful improvements (e.g., the 49.3% similarity gain for Res50-Swin-B on adversarial examples). These results quantitatively confirm that SAA effectively identifies shared features between distinct models, which in turn enhances adversarial transferability.

Cosine similarity quantitatively demonstrates that SAA can improve the feature similarity between surrogate and witness models, proving the validity of this paper's hypothesis. Regarding adversarial transferability, a more direct comparison can be found in Tables 2 and 3, where ASR (Adversarial Success Rate) better demonstrates attack performance. Given the diversity of CNN and ViT models, the relationship between feature similarity and attack performance has been underexplored. SAA opens this door, making it a promising future research topic.

---

Weakness 4: Label correctness.

By definition, adversarial examples are crafted with human-imperceptible perturbations (under $l_ₚ$ norm constraints) that mislead model predictions while preserving the original semantic content. In the field of adversarial machine learning, this property is foundational: the perturbations are strictly bounded to ensure they do not alter the image’s meaning as perceived by humans. This is a well-established convention, as $l_ₚ$-constrained perturbations (e.g., $ε = 16/255$ for $l_∞$ setting) are designed to be visually negligible and thus do not interfere with human recognition of the semantic label.

Given this standard, verifying individual adversarial examples via manual or automatic checks for semantic label preservation is generally unnecessary in the literature, as the perturbation bounds inherently guarantee that the original label remains semantically valid. Our work adheres to this convention, with perturbations constrained to lₚ norms that ensure such preservation.

---

Weakness 5 & Question 4 & Question 5: Pseudo-labels' noises.

We should note that local pseudo-labels are not directly related to true labels because we learn local features of the surrogate and witness models, not global features.

Furthermore, we add experiments using soft labels (logits) and the KL divergence loss function for local alignment, as shown below. The surrogate model is Res50 and the witness model is ViT-B. In this scenario, using hard labels (local pseudo-labels) achieves higher adversarial transferability (improving ASR by 6.1%).

Finally, we include experiments with different distance metrics (which can be understood as embedding metrics) in the "Distance Metric" section of Appendix C. Here, we select Kullback–Leibler Divergence (KL), Total Variation (TV), Jensen-Shannon Divergence (JS), and NCE loss for evaluation. We select ResNet-50 as the surrogate model and ViT-B as the witness model for ablation studies. As shown in Table 8, KL exceeds the next best JS by 2.5%, so we choose KL as the metric.

---

Weakness6: Overfitting the witness.

The ultimate metric for evaluating the effectiveness of aligned features is the average attack success rate on the black-box model, not the accuracy of the surrogate model itself on ImageNet. In this paper, our alignment was trained for only one epoch, making it less susceptible to overfitting. Secondly, in "Training Epochs" of Appendix C, we further explore performance differences across multiple training epochs. While the transferability of MA converges rapidly with increasing epochs, SAA continues to improve until the ninth epoch. This indicates that MA easily overfits to the witness model, limiting its adversarial transferability. SAA, on the other hand, is less susceptible to overfitting, and its transferability reaches a higher upper bound after multiple training rounds, demonstrating the importance of using spatial and adversarial features for model alignment.

---

Question 1 & Question 3: Witness model choice.

In Section 3.2 of the main body, we have conducted a large number of experiments to demonstrate that SAA is robust to model selection. Although we conduct 16 combinations and evaluate on 6 CNNs and 4 ViTs, showing that SAA’s improvement is not limited to specific choices of witness models. Based on the above experiments, we further summarize the empirical guidance for the selection of surrogate and witness models:

1. Self-alignment (surrogate and witness models are consistent) usually only provides minor improvements (it is difficult to learn unique structural features), and alignment between different models improves more significantly.
2. When the surrogate model is a ViT-like model and the witness model is a CNN-like model, better cross-architecture transferability can be obtained. This is mainly because ViT has stronger performance, while CNN further provides unique structural features.

If the shallow CNN is a well-trained model (for example, with normal performance on ImageNet), then there can be an improvement. Even when the surrogate model and witness model have no architectural overlap, SAA still improves adversarial transferability. As shown in Table 2 of the main body, when the surrogate model is DN121 and the witness model is Swin-B, SAA improves the average ASR by 10.9% and 12.6% compared to MI and MA, respectively.

---

Question 6: Known target models.

We should clarify that in a transfer-based black-box setting, no details of the target model are known. We chaim this in Lines 284-286: It should be noted that the target models are unknown in the transfer attack, so no information about the target models can be used for alignment. Therefore, your assumption is invalid. If the target model is known (but frozen), then the target model can be directly selected as the surrogate model, which is equivalent to a white-box attack.