Understanding Model Ensemble in Transferable Adversarial Attack

Thank you very much for your constructive comments! We address all your questions and concerns in the following responses.

Q1: In transfer attacks, the surrogate model and the target model have different model architectures and parameter numbers. However, the authors implicitly assume that the surrogate and target models share the same parameter space (right column, Line 111-112), which significantly limits the generalizability of their theoretical results.

A1: Thank you for your comments.

• Firstly, in Section 4.2 (Remark 2), we explicitly discuss how Theorem 4.3 can be generalized to cases where the surrogate and target models have distinct parameter distributions. This addresses part of the concern regarding parameter space assumptions.
• Secondly, as the first theoretical work to formally link adversarial transferability with generalization theory, our primary contribution is a foundational framework rather than an exhaustive treatment of all scenarios. To our best knowledge, transferable model ensemble adversarial attacks remain less theoretically explored. The field still lacks papers that understand it theoretically. We hope our work inspires follow-up studies to fully address the problems you mention in the future.

Q2: I suggest revising some of the notation. It is strange that (x, y) represents an adversarial example (right column, Line 121-122).

A2: Thank you for your constructive comments. We agree with your suggestion and will use $(x^{\text{adv}}, y)$ instead to represent an adversarial example in our final version.

Q3: The theoretical results do not seem particularly novel, as they represent a widely accepted conclusion.

A3: We sincerely appreciate the reviewer's comments. Firstly, we're encouraged that multiple other reviewers (uk2i, kypp, and Ejwu) have explicitly recognized the novelty of our theoretical framework in their comments. It suggests our approach does offer fresh insights to the field. Secondly, we theoretically analyzes and validates three important practical guidelines to improve adversarial transferability, including (1) incorporating more surrogate models, and (2) increasing their diversity, and (3) reducing their complexity in cases of overfitting. Our work provides the first theoretical framework to explain these empirical observations in adversarial transferability, while offering practical insights for algorithm design. We hope this foundation inspires deeper understanding and future advances in the field.

Q4: The experiments are limited. The authors' experimental results do not differ significantly from those in previous empirical studies. Additionally, the authors mentioned conducting experiments in ImageNet, yet I could not find any results in ImageNet. The model architectures and attack methods used in the experiments are also limited. The authors could include visual transformers and more transfer attack methods.

A4: We appreciate the reviewer's constructive feedback, which has helped us strengthen our work. Our experimental design was carefully crafted to validate our novel theoretical contributions previously unexplored in the literature. In response to the reviewer's valuable suggestions, we have expanded our experiments to include additional attack methods, diverse model architectures (including transformers) and larger dataset (ImageNet). These new experiments, detailed in our response to Reviewer Ejwu's Question 3, provide important insights for future research in transfer adversarial attacks. Furthermore, in our response to Reviewer uk2i's Question 1, we have also expanded the experimental validation by employing another effective attack methodology. This additional analysis not only confirms our theoretical findings but also broadens the scope and strengthens the practical implications of our research.

Q5: Some theoretical papers have been overlooked, e.g., [1].

[1] Transferability Bound Theory: Exploring Relationship between Adversarial Transferability and Flatness, NeurIPS 2024.

A5: We appreciate you bringing this work to our attention. We will ensure proper citation of [1] in our final version. While [1] examines flatness and transferability, we establish the theoretical link between statistical learning theory and adversarial transferability. Our framework provides new insights into transferability through the lens of generalization theory, offering complementary (rather than overlapping) perspectives to [1]. We’re happy to further clarify these differences if needed.

Thank you very much for your insightful review of our work!

Q1: The authors should include direct comparisons with state-of-the-art adversarial attack methods in the experiments.

A1: We sincerely appreciate the reviewer’s constructive feedback. In direct response to Reviewer uk2i’s Question 1, we have significantly expanded our evaluation to incorporate additional attack algorithms as suggested.

Q2: The mathematical derivations in the paper are highly complex and lack intuitive explanations, which may pose difficulties for readers, especially those without a deep background in information theory.

A2: Thank you for your constructive comments. We will provide more step-by-step explanations about information-theoretic analysis in Appendix B.5 in the final version.

Q3: The paper could benefit from a more detailed discussion of the practical implications of the theoretical results, particularly in terms of designing more effective adversarial attack algorithms.

A3: We thank the reviewer for the constructive suggestion. We show an example here to investigate how properly controlling the model complexity of surrogate models can contribute to more effective adversarial attack algorithms, which is in line with our theory. We use ImageNet as the dataset here. To conduct model ensemble attack, we fine-tune surrogate models—VGG16, InceptionV3, and Visformer—using a sparse Softmax cross-entropy loss [1]. This modification encourages sparsity in the model’s output distribution, and we observe in our experiments that the model complexity (L2 norm of the weight matrix) are reduced after using such a loss:

Building upon three advanced transfer attack methods—MI-FGSM [2], SVRE [3], and SIA [4]—we propose their sparsity-enhanced variants (MI-FGSM-S, SVRE-S, and SIA-S) through the integration of sparse Softmax loss during surrogate model training. We consider eight kinds of model architectures and measure attack performance via attack success rate, where higher value corresponds to stronger attack effectiveness.

As can be seen in the table, these variants outperform their standard counterparts in most cases, demonstrating the benefit of controlling model complexity in both CNNs and visual transformer settings to improve adversarial transferability. Beyond this example shown above, we believe that our work can also inspire the development of more stronger attack algorithms in the future.

[1] From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification. ICML 2016

[2] Boosting Adversarial Attacks with Momentum. CVPR 2018.

[3] Stochastic Variance Reduced Ensemble Adversarial Attack for Boosting the Adversarial Transferability. CVPR 2022.

[4] Structure Invariant Transformation for better Adversarial Transferability. ICCV 2023.

Q4: To improve the paper's usability, the authors should consider releasing the code and providing practical examples to help other researchers implement and test the framework.

A4: We appreciate the reviewer's valuable suggestion. We are happy to release the code and provide implementation examples to facilitate reproducibility and adoption by the research community.

Thank you very much for your insightful review of our work!

Q1: ...However, in the standard model ensemble attack scenario, the models may have been trained with fully or partially identical training data, and therefore the models could be quite correlated. Therefore, it seems to me that the term $H\_\alpha^{\frac{1}{\alpha}}\left(\mathcal{P}\_{\Theta^N} \\| \mathcal{P}\_{\bigotimes\_{i=1}^N \Theta} \right)$ could be quite large and make the bound vaccuous in practice.

A1: To make it clear and easy to understand, we provide an intuitive approximation below. We choose $\alpha=10$, $\delta=0.01$, $\beta=1$ in our Theorem 4.3. Let $P=\mathcal{P}\_{\Theta^N}$ and $Q=\mathcal{P}\_{\bigotimes\_{i=1}^N \Theta}$. We consider the model parameters for a given precision so that $P$ and $Q$ are discrete distributions.

• Equation (8) from [1] tells us that $H\_\alpha(P \\| Q)=e^{(\alpha-1)D\_\alpha(P,Q)}$, where $D\_\alpha(P,Q)$ is the Rényi divergence.
• Let $\delta \in [0,1]$ be the TV distance between $Q$ and $P$, and $\beta_1=\min\_{a \in \mathcal{A}} \frac{Q(a)}{P(a)}$ be defined in Equation (8) from [2], i.e., the minimum of the ratio of the probability density function of distributions $Q$ and $P$.
• Now we approximate $\beta\_1$. Consider there are $t$ parameter configurations for each model. For simplicity, we assume that part of the models ($f(N)$ models) play a key role in adversarial transferability, and the other $N-f(N)$ models are random sampled from these $f(N)$ models.
  • For the product of marginal distribution $Q$, the parameters from each model are random. Consider the case of uniform distribution, where every parameter in the $N$ models share the same probability, i.e., $Q(a)=\frac{1}{t^N}$.
  • For the joint distribution $P$, we also consider the case of uniform distribution, where $f(N)$ models are fixed and $N-f(N)$ models are randomly sampled, i.e., $P(a)=\frac{1}{t^{N-f(N)}}$.
  • Therefore, $\beta\_1 \approx\frac{Q(a)}{P(a)}=t^{-f(N)}$, which is less than 1.
• Substitute the above into Theorem 3 from [2], we have $H\_\alpha(P \\| Q) \le 1+\frac{\delta\left(\beta\_1^{-1}-1\right)}{1-\beta\_1} \le \beta\_1^{-1} \approx t^{f(N)}$
• Substitute the above into Theorem 4.3 in our paper, we have

Thank you very much for your insightful review of our work!

Q1: ...Comparing another attack method (either extremely strong or weak) would give us more insights about the vulnerability-variance tradeoffs.

A1: We sincerely appreciate the reviewer's constructive suggestion. We have conducted additional experiments using the VMI-FGSM attack [1] on MNIST.

$\lambda=10^{-4}$

$\lambda=10^{-3}$

The observed vulnerability-variance tradeoffs demonstrate consistent alignment with our paper. We will include the complete experimental details and analysis on other datasets in our final manuscript.

[1] Enhancing the transferability of adversarial attacks through variance tuning. CVPR 2021.

Q2: Please clarify the effect of increasing $\lambda$ in the paper and explain how it relates to the model complexity...

A2: Thank you for your insightful question. Firstly, as suggested in Lemma 4.2 and Appendix A, the empirical model ensemble Rademacher complexity can be upper bounded by model complexity and the number of ensemble components (for instance, the analysis in Section 4.2 states that reducing the weight norm of the model and increasing the number of the models will reduce the empirical model ensemble Rademacher complexity). These two kinds of effect have been reported in our experiments:

• In Section 5.1, we adjust the weight decay factor $\lambda$ to change the model complexity and investigate the trends in how complexity interacts with other factors.
• In Section 5.2, we improve the number of ensemble components and observe an increasing trend of attack success rate.

Following the reviewer’s suggestion, we conduct a deeper investigation into the impact of model complexity by applying a max norm constraint to the model parameters. This technique limits the L2 norm of each weight vector to a predefined threshold to control the model complexity.

• Larger max norms enable richer feature representations at the cost of potential overfitting.
• Smaller max norms promote simpler models and may induce underfitting by limiting model capacity.

As illustrated in the table below, this trade-off manifests across architectures (MLPs and CNNs with 1–3 layers) and varying max norm values. We measure attack performance via accuracy, where lower accuracy corresponds to stronger attack effectiveness.

The results demonstrate a clear trend: as the max norm constraint is relaxed from very small values (e.g., 0.1) to moderate levels (e.g., 5.0), the attack effectiveness first increases and then decreases. This pattern indicates that excessively restrictive constraints can impair model expressiveness, whereas an optimally tuned max norm effectively balances model complexity and representational capacity. More importantly, these findings support our paper's claim regarding the influence of the weight decay factor $\lambda$ on model complexity.

Q3: A comparison between CIFAR-10 and CIFAR-100 could also be interesting...

A3: We sincerely appreciate your insightful question. As shown in our paper, the attack success rate and loss exhibit an initial increase followed by stabilization with growing steps, while the variance first rises and then declines. We fully agree with the reviewer's insightful comment regarding dataset-dependent characteristics; indeed, the inflection points vary across datasets due to their distinct characteristics. Due to the space limitations of this rebuttal, we will extend the step range and include a comparative analysis of CIFAR-10 and CIFAR-100 in Appendix E in our final version.