Evaluating Model Robustness Against Unforeseen Adversarial Attacks

Thank you for your careful analysis of our work. We hope the following response addresses your concerns.

Distinction between our attacks and pixel-space $L_p$ attacks.

I disagree with the authors stating that the proposed attacks are beyond the $L_p$ norm. In particular, it can be still computed the amount of perturbation and bounded to the maximum budget. The whole optimization problem can also be written as an $L_p$ problem, and constrained to be such a problem. Maybe the projection is a bit different, because the authors want to use specific patterns. Hence, all the discussion regarding going beyond that threat model does not apply on this paper.

We certainly agree that our attacks are $L_p$ perturbation sets in a latent space! In fact, we point this out on line 204. However, in the context of adversarial robustness the phrase “$L_p$ attack” almost always refers to “pixel-space $L_p$ norm bounded additive perturbations”. This is commonly understood to be the meaning of “$L_p$ attack”. Even though attacks that are not pixel-space $L_p$ attacks may still be formulated as $L_p$ perturbations in some latent space, these are typically not referred to as $L_p$ attacks. For examples of well-known papers demonstrating this, see [1, 2, 3, 4]. Thus, our discussion of going beyond the space of “$L_p$ attacks” remains valid and in keeping with standard terminology. We have clarified this in the updated paper.

[1]: “Towards Deep Learning Models Resistant to Adversarial Attacks”. Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, Adrian Vladu. ICLR 2018 [2]: “Certified Adversarial Robustness via Randomized Smoothing”. Jeremy M. Cohen, Elan Rosenfeld, J. Z. Kolter. ICML 2019 [3]: “Adversarial Training for Free!”. Ali Shafahi, Mahyar Najibi, Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S. Davis, Gavin Taylor, Tom Goldstein. NeurIPS 2019 [4]: “Spatially Transformed Adversarial Examples”. Chaowei Xiao, Jun-Yan Zhu, Bo Li, Warren He, Mingyan Liu, Dawn Song. ICLR 2018

Descriptions of differentiable attacks are already in the main paper.

No details on the attack. The authors spend a lot of space to present the manipulations, and they vaguely state they are differentiable. However, I could not understand how. This content must be present in the main paper, since it is a claimed contribution. Placing them in the appendix is not the right thing to do.

This information is already provided in the main paper, taking up most of page 5. Please see Section 3.3, in which we briefly describe how each of our 8 core attacks are designed in a differentiable manner. These descriptions include the most relevant details for each attack, while still being fairly brief to conserve space. For example, we explain that our Gabor attack is obtained by “optimizing the underlying sparse tensor which the Gabor kernels are applied to”. In Appendix B, we provide corresponding descriptions for the 11 additional attacks and a highly detailed description of the Wood attack (corresponding to Figure 3).

Clarifying technical novelty of attacks.

I debate that the nine attacks are novel, since they are all well-known noise patterns already present in the literature. … I would frame that the real contribution is how the authors attach those noise patterns in a differentiable way (see previous comment).

We fully agree that many of the noise patterns themselves are well-known (e.g., Gabor noise), and we are glad that you agree that making differentiable versions of these noise patterns / corruptions constitutes a real contribution. In many cases, making these noise patterns differentiable was nontrivial and required considerable manual effort. For example, our Fog attack uses a custom differentiable version of the diamond-square algorithm. We will clarify this in the updated paper. Thank you for your suggestion.

Additional requested information on PGD implementation.

Also, the authors do not provide their implementation of PGD, which might also contain bugs.

Please see this code for our $L_\infty$ PGD implementation (anonymized): https://pastebin.com/b99EJSmi

We are quite confident the implementation is correct, as PGD is a simple attack to implement. In case it helps, we asked o1-preview whether our implementation is correct, and it said yes. Removing torch.sign on the gradient results in o1-preview changing its answer to "no", as expected. Please feel free to check the implementation yourself. If there are any other questions or concerns we can address, please let us know and we would be happy to help.

Thank you for your careful analysis of our work. We hope the following response addresses your concerns.

Evaluations on 11 additional attacks.

One thing that could be incorporated for strengthening the paper could be adding some evaluations on the attacks left out of the 8 core attacks and demonstrating that the performance measured over the left out attack set is correlated with performance measured over the 8 core attack set. This would empirically demonstrate that the performance on the 8 core attacks are highly representative of the 19 attacks.

We actually do include full evaluations on the 11 additional attacks (with a medium distortion level). These results are Figure 26, but are not properly referenced elsewhere in the paper. We have updated the paper to remedy this.

We have also added an evaluation of the correlation between the average accuracy on the 11 additional attacks and the average accuracy on the 8 core attacks. The Pearson correlation is 0.92, indicating that the 8 core attacks are a good representation of the full set of 19 attacks. Thank you for your suggestion.

Thank you for your careful analysis of our work. We hope the following response addresses your concerns.

Please see Table 2 for comparisons to $L_p$ attacks.

In section 4.1 the authors compare UA2 to other distribution-shift benchmarks, and they claim, based on the results (Table 5), that UA2 is a measure of worst-case robustness, like $L_p$, but they did not assess the distributional shift of $L_p$ perturbed samples.

In Table 5 and Appendix I, we compare to prior distribution shift benchmarks, which naturally does not include $L_p$ attacks. Our comparison to $L_p$ attacks appears in Table 2. In particular, we compare to the popular PGD attack. We also include evaluations with the more recent AutoAttack method in Appendix F.4.

Comparisons to distribution shift literature.

there have been recent workshops and efforts to address the real-world issues applied to ImageNet, and the authors should provide some references and possibly differentiate their work from these. One example is the ShiftHappens workshop at ICML 2022

The distribution shift literature is indeed highly relevant to our work. In Table 5 and Appendix I, we already include comparisons to ImageNet-C and ImageNet-Sketch—two foundational datasets from this literature. The ICML 2022 ShiftHappens workshop included papers presenting other similar datasets, such as ImageNet-Cartoon and ImageNet-D. Thus, our existing comparisons already have fairly good coverage of these kinds of distribution shift datasets.

Our comparisons in Table 5 and Appendix I demonstrate that our differentiable worst-case corruptions have distinct properties from the average-case corruptions that are studied in the distribution shift literature. For example, Table 5 shows that the rankings between methods is similar for ImageNet-C and our unoptimized corruptions. But when we optimize our corruptions, accuracy drops substantially and the ranking between methods changes. This suggests that optimized, worst-case corruptions have different properties than average-case corruptions commonly studied in the distribution shift literature.

Importantly, we do not claim that our benchmark is “better” than ImageNet-C or other distribution shift benchmarks. Rather, our point is to demonstrate that they have very different properties and that worst-case corruptions may require novel methods to address. This is a desirable quality for any new benchmark. We will clarify these points in the updated paper. Thank you for your suggestion.

Clarifying $L_p$ attack selection and hyperparameters.

the authors based this claim on the evaluation of just one $L_p$ attack method (PGD) and just on $L_\infty$ bound constraints, with a fixed number of optimization steps (only $50$, very limited for ImageNet, and without providing evidence of convergence).

Please note that we do evaluate AutoAttack in Appendix F.4, although for the specific reason of checking the unusually high robustness of DINOv2.

We agree that checking convergence of attacks is important. We did in fact check convergence when selecting hyperparameters for all our attacks. The PGD attacks in our main evaluations do converge for nearly all models. We will add loss plots to the updated paper to make this easy for readers to check for themselves. Thank you for your suggestion.

Thank you for your response. We believe we can address many of the concerns in your response, so we would be grateful if you could consider the following points.

Overfitting to the attacks is not allowed, as stated in our threat model.

The attacks used for testing have different "rules" each, it might be possible to "overfit" to the benchmark by submitting a model that is robust only to the specific perturbations (and perturbation sizes) used here;

In our threat model (Section 3.1), we describe how the defender does not have access to the attacks at training time. This is what we mean by the attacks being "unforeseen". All of our attacks are held-out for evaluation, which fully addresses the concerns of overfitting.

Also note that the description of attacks having rules also applies to all previously proposed attacks, including PGD. A key strength of our benchmark is that the attacks cannot be used at training time, allowing us to measure robustness to out-of-distribution adversaries.

Clarifying why averaging over perturbation sets is sensible.

The discussion on the UA2 metric still needs improvement, as it's still not clear what is the output of this evaluation. Worst-case evaluations are commonly used to evaluate the models, not the attacks. So from my perspective it's still relevant to take the worst rather than the average effect. It's not clear to me why the average would be helpful.

It may be easier to see why averaging over perturbation sets is sensible by looking at the related research problem of multi-attack robustness. In multi-attack robustness, the goal is to improve adversarial robustness against multiple different perturbation sets, oftentimes by training against multiple attacks at once. It is standard practice in this area to average over perturbation sets. For examples of papers that do this, see [1, 2]. The fundamental reason for taking the average is that different perturbation sets are not easily compared with each other. E.g., see our earlier response about how it is nontrivial to say that a patch attack is stronger than a warping attack; there is simply not a clear basis of comparison in most cases.

However, we do agree that taking the minimum accuracy across attacks (i.e., the union attack) can be informative if all the attacks are normalized in some manner. In our benchmark, we carefully tuned our attacks to ensure that all severity levels still preserve semantics of the image, which provides a basis of comparison so that different perturbation sets can still be compared. For this reason, we will add a minimum accuracy metric (union attack) to the updated paper. Thank you for your suggestion.

[1]: "Learning to Generate Noise for Multi-Attack Robustness". Divyam Madaan, Jinwoo Shin, Sung Ju Hwang. ICML 2021

[2]: "MultiRobustBench: Benchmarking Robustness Against Multiple Attacks". Sihui Dai, Saeed Mahloujifar, Chong Xiang, Vikash Sehwag, Pin-Yu Chen, Prateek Mittal. ICML 2023

The 8 core attacks are a good proxy for the full suite of 19 attacks.

The selection of the 8 core attacks as well is not convincing, as different models might be affected more by one of the attacks that are not selected.

Please note that we include extensive evaluations on the 11 additional attacks in Figure 26 of the submission. These results show that the extra attacks do not significantly change the ranking of models. The top 5 models shared between the Figure 26 and Figure 8 (the corresponding core attack evaluation) are exactly the same, with the top 2 models having the same ranking. This figure is not currently referenced in the text, so it was easy to miss. We have updated the paper to link to this figure in the main text.

We have also added an evaluation of the correlation between the average accuracy on the 11 additional attacks and the average accuracy on the 8 core attacks. The Pearson correlation is 0.92, indicating that the 8 core attacks are a good representation of the full set of 19 attacks.

Clarifying why we create a descriptive name for our metric.

the evaluation metric is the same as ImageNet-C, but why do we need a new name?

There may be a misunderstanding here. In our initial response, we mentioned that the mCE metric from ImageNet-C is an analogue to our UA2 metric. We did not mean that the metrics are the same. Rather, we meant that they are analogous in certain ways. Namely, our benchmark can be thought of as an adversarial version of ImageNet-C in which the corruptions are differentiable and can be optimized. Just as how ImageNet-C uses the name "mean corruption error" (mCE) as a descriptive name for their metric, we use the name "unforeseen adversarial accuracy" (UA2) for our metric. Coming up with descriptive names for metrics is common in benchmarking and can help readers follow along with the results.

Thank you for your careful analysis of our work. We hope the following response addresses your concerns.

Added reference to Figure 5.

Key elements of the logic should be presented within the main text. For instance, Figure 5 should either be compressed and included in the main paper or replaced with an equivalent narrative explanation.

We agree that Figure 5 would be good to reference early in the main text, since it shows our full suite of attacks. We have added a reference to Figure 5 in the introduction of the updated paper. Thank you for your suggestion.

Clarified Figure 3 caption.

In Figure 3, the term $m \times m \times 2$ lacks a corresponding definition, which should be clarified.

The $m$ value is the spatial dimension of the latent variable that we optimize in the Wood attack. This allows us to reduce the spatial frequency of the distortion by setting $m < n$, where $n$ is the input image dimension. We have clarified this in the caption of Figure 3 in the updated paper. Thank you for your suggestion.

Fixed typos.

Equation 2 should indeed be the argmax, not argmin. We have fixed this typo in the updated paper. Thank you for noticing it.

Clarifying novelty of contributions.

The study mainly employs various image augmentation techniques to examine model performance without contributing significant advancements in model verification methods (i.e., assessing stability across augmented samples), which falls short of the standards for ICLR.

We agree that simply providing new image augmentations would not be as interesting, given that many works have already proposed image augmentations and distortions to evaluate robustness (e.g., ImageNet-C).

However, this would not be an accurate description of our work. Our work is very different from prior work on model verification, since we propose differentiable image corruptions that can be optimized to find worst-case corruptions. There have been multiple prior works accepted to ICLR proposing one new attack of this kind [1, 2], but we propose eighteen new attacks (nineteen in total, counting the elastic attack), all of which are manually curated to provide a high-quality evaluation.

If we have addressed the thrust of your concerns, we kindly ask that you consider raising your score.

[1]: “Spatially Transformed Adversarial Examples”. Chaowei Xiao, Jun-Yan Zhu, Bo Li, Warren He, Mingyan Liu, Dawn Song. ICLR 2018

[2]: “Unrestricted Adversarial Examples via Semantic Manipulation”. Anand Bhattad, Min Jin Chong, Kaizhao Liang, Bo Li, D. A. Forsyth. ICLR 2020