On Adversarial Training without Perturbing all Examples

1. no analysis: We agree with the reviewer, that an additional analysis regarding the 'why' would be of interest. At this point, we cannot offer one but refer the reviewer to their own statement that the "results are informative". We also believe that our results are informative and are likely to spur follow up work. We believe this is a strong reason to publish this work. We restate the contribution of our paper: in contrast to related work on robustness and its transfer capabilities (as discussed in related work and our method section), we conduct a systematic analysis where we perturb only a fixed subset of training examples. This reveals a surprising phenomenon: adv. training a single class, transfers robustness to all other classes above non-trivial levels. Moreover, the hardest class provides the best transfer. As a consequence, we observe that just perturbing $50$% of training examples is sufficient to reach baseline AT performance and only $30$% of examples is sufficient in the task transfer setting.

2. a) no theory: While additional theory is useful, we think that a lack of it does not reduce the importance of our work. Our work describes a phenomenon of adversarial training --few attacked examples/classes transfer robustness surprisingly well-- comprehensively. This is confirmed by the reviewer itself and also R3h75, RgZFF and RQzqJ.

   b) effect is just side-effect of generalizability: We thank the reviewer for a highly relevant question that is addressed by providing additional figures in the new paper revision reporting the defense rate -- robustness independent of clean accuracy. While the trend between clean and robust accuracy seems highly correlative, in section A.6 and figure 17 in the appendix, we show that the increase in robustness is not due to clean accuracy alone. By providing robustness on accurately classified examples only, we remove the influence of clean accuracy entirely. We can see that robustness transfer is best when SAT is performed using the hardest examples.

3. no TRADES and AWP evaluation: We agree with the reviewer that integration of TRADES is especially interesting in our setting, since it provides a means to trade-off clean with robust accuracy. However, we highlight that selecting the best trade-off between clean and robust accuracy with TRADES does not alter the main findings of our paper: adv. training on few examples/classes transfers robustness surprisingly well. Moreover, adversarial weight perturbation would be interesting to investigate, yet we believe that it goes beyond the scope of this work. Regarding TRADES, we have added results for CIFAR-10 for CSAT $L_2 \epsilon=0.5$ $|A|=1$ and the general case. For $|A|=1$ we conducted a coarse hyperparameter search for TRADES $\beta$ parameter and found $\beta=12.0$ to provide good robustness transfer (see table below). This configuration achieves $42.9$% $(+10.7)$ robust accuracy on $B$ while also increasing rob. acc. on $A$ to $78.2$% $(+28.6)$, with the expected decrease in overall clean accuracy ($81.0$% $(-10.2)$).

In a second experiment, we use linearly decreasing $\beta$, from $12$ down to $6$, for our ESAT $L_2 \epsilon=0.5$ experiment on CIFAR-10, corresponding to figure 6. That is, when $|A|$ is small, we trade off some of the gained clean accuracy for increased robustness with large $\beta=12$, but decrease to $\beta=6$ with increasing size of $|A|$. We choose $\beta=6.0$ for $|A|=9$ based on the same parameter choice for the full CIFAR-10 set in (Zhang et al. 2019). We compare ESAT-hardest first with and without TRADES in the table below. We observe quicker robust accuracy convergence to the baseline ($66$% robust accuracy is achieved at $20k$ samples, instead of at $25k$) with an expected drop in clean accuracy below full AT baseline clean accuracy ($85.6$% clean accuracy at $20k$ vs $89.7$% at $25k$ vs $89.2$% for full AT).

Thanks for the response of the authors. I have read other reviewers' comments and responses. There are some additional questions.

1. The authors mention that the proposed method can reduce the time cost. Could the authors discuss related works, such as Free-AT[1]?

2. I still think it is important to give a deep analysis of the experiment results. For example, what is the reason that the model can obtain more robustness from the 'cat' class? Is it just because images in 'cat' classes are harder to learn? And why model obtain robustness for cars from cat images? They are very important questions to figure out the reason why we do not need to perturb all classes.

3. There is no theoretical analysis is a main flaw. For CIFAR-10, it is okay to study images in each class. And find the most efficient strategy. However, if there is no theoretical guarantee, it is difficult to design strategies for larger datasets, like ImageNet-1k. I hope the authors can prove they can efficiently find proper methods for ImageNet even without theoretical analysis.

4. I think that the authors misunderstand the question of generalizability. I mean that as the perturbed data are selected with the metrics entropy during the training process, it is possible that the perturbed data for the next training epoch will not be chosen. Because the model has the generalizability to give a clean version of the AE. Therefore, during the whole training process, all data could be selected to be perturbed, but they are selected in different epochs. If true, all data are perturbed, instead of only part of the training data.

[1] Adversarial training for free!

4. what is difference to contrastive AT?: We base our discussion around (Kim et al.), in which the authors provide a self-supervised adversarial training method, which does not require explicit labels. We highlight, that our proposed analysis is complementary to this work. (Kim et al.) perturb all training examples, similar to the vanilla AT approach. Ours, instead, investigates the impact of robustness transfer when we perturb only a fixed subset. Our SAT analysis is therefore applicable to the proposed contrastive learning approach.

   Kim, Minseon, Jihoon Tack, and Sung Ju Hwang. "Adversarial self-supervised contrastive learning." Advances in Neural Information Processing Systems 33 (2020).

5. some figures' labels are blocked: We thank the reviewer for raising this issue. We have fixed the blocked labels in all figures.

6. data augmentation different to related work: We adopted the adversarial training methodology from the GitHub repository github.com/MadryLab/robustness. Consequently, we use the same data augmentation as they did. That is, on CIFAR-10, the data augmentation involves cropping, flipping, color jittering and rotating (for details see github.com/MadryLab/robustness/blob/master/robustness/data_augmentation.py). Irrespectively, we repeated our CSAT $|A|=1$ experiments with the reviewers suggested data augmentation pipeline: only performing cropping and flipping. Results, reported in the table below, show similar results to our reported in our submission. We conclude that the data augmentation is not the reason for our observed robustness transfer.

1. no (theoretical) explanation for the observed transferability: While we agree with the reviewer, that additional theoretical or empirical based explanations would be beneficial, we believe that our paper has a valuable contribution to the scientific community. We highlight a surprising phenomenon and counter-intuitive results -- e.g. (i) CIFAR-10 SAT on cat provides better robust transfer to car than truck on car or (ii) $30$% of training examples achieve baseline robust accuracies in the task transfer setting -- that could lead to new research directions.

   1. theoretical justification for why harder examples transfer robustness better: At this point, we cannot provide an underlying theory. We speculate however, that classifying hard examples requires the detection of more diverse features, which are trained adv. robust. Given the compositional nature of deep networks, it is plausible to assume that these robust features are reused by other classes.

   2. reasons behind diminishing returns when dataset complexity increases: We conjecture this to be similar to our point made for the previous question. Datasets with a larger number of classes and larger input images likely require the detection of more diverse features.

   3. why does robustness transfer increase for smaller $\epsilon$?: We conjecture, that with increasing $\epsilon$ each feature becomes more class-specific and is thus less reusable. In contrast, with decreasing $\epsilon$ the generalizability is improved.

1. what are the benefits of considering a subset:?

   The benefits of considering a subset are twofold: (a) practical, and (b) analytical.

   Practical: We observe significant savings in terms of training cost with our protocol. For example, training our ResNet-18 models on CIFAR-10 takes approx. 3h without any adversarial perturbations. Full adversarial training takes about 7h, and only perturbing 50% of the data results in a training time of 5h. Thus reducing the cost of training considerably. These savings are expected to be even larger at scale, that is when training large models on large datasets. E.g. consider the power consumption of training foundational models, which consume hundreds of Mega-Watt-hours without adversarial training (e.g. nearly 500MWh for LLaMa-65B [LLama]). Adding adversarial training roughly increases the training time -- and thus the power consumption -- by a factor of PGD-steps. Our insights suggest that training time/consumption could be at least halved.

   Analytical: While it is true that "the proposed method does not lead to better robust accuracy than full adversarial training", we show that you can trade-off a minor loss in robust accuracy against significant savings in training time -- by not perturbing all classes/examples. However, we also believe that this loss in robust accuracy should motivate further research into more effective defenses against adversarial attacks: Why do we need to perturb all classes/examples to achieve the full potential of adversarial training? Can we train models to be robust in a class-agnostic manner? These are important questions raised by our work, which we believe to be of interest to the scientific community.

   [LLama] Touvron, Hugo, et al. "Llama: Open and efficient foundation language models." arXiv preprint arXiv:2302.13971 (2023).

2. adversarial training protocol: For vanilla adversarial training, we perturb all training examples. There is no mix between clean and perturbed examples in a batch as described in earlier work on adv. training. For SAT on the other hand, we randomly sample a training batch and perturb only those examples that are contained in $A$. We have updated the appendix, section A.1 in our manuscript to make this clear.

3. during task transfer, why not retrain with full adv. training? Our argument is similar to question 1. We see an increase in foundational models, which provide strong off-the-shelf performance which can be improved via fine-tuning. Given the computational demand that is required to train such large models, it is often sufficient to only train a shallow classifier on top of intermediate outputs. In this case, adversarial robustness can only be provided when the foundational model is robust. We have shown that it is sufficient to perturb only $30\%$ of examples during the initial (foundational) training to achieve high adversarial robustness after fine-tuning.

4. mismatched reporting on page 6: Thank you for highlighting this issue. We have fixed it in the new revision. Note that, now, the numerical values are actually better than what was reported in our initial text.

5. unreadable figures: Thank you for highlighting a readability issue. We will improve the readability for the final revision.

Dear reviewer, thank you for the response.

We would like to emphasize that our paper should not exclusively be read as providing “guidelines“ for more efficient adversarial training. As described earlier in this thread, there is also an analytical component to our paper centered on the question: Do we need to adversarially perturb images from every class to attain good robustness, and what are the implications?

There is a “glass half full” answer that we back up with our experiments, namely that one can get away with a minor loss of robustness by not perturbing examples from every class. This has positive practical implications that we have discussed at length.

There is however also an interesting “glass half empty” answer, namely that to achieve the full potential of adversarial training in terms of robustness, it would seem that one indeed has to perturb representative images from every class, but also that the choice of classes is inferior when random. This has all kinds of interesting implications and suggests further questions and directions for research, e.g. to name a few: While we provide some measures that correlate with robustness transfer, what really are the underlying mechanisms? What theoretical tools are needed to distinguish between individual object and scene classes in terms of their visual characteristics? What does this say about adversarial training as a general framework if robustness ultimately has to be learned for every class? Can we be satisfied with visual recognition methods that must gain robustness in this manner? Is this scalable?

Note: Regarding the fast adversarial methods (esp Wong et al 2020) our initial experiments show that these may well be complementary but we acknowledge that further investigation is warranted.

We thank the reviewer for the encouraging scores on our manuscript.

1. guidelines for selecting subset size in practice?: We would like to emphasize that our proposed SAT methodology provides means to analyze the robustness transfer across classes and examples, yet for practical purposes we would give the following guidelines: for downstream task transfer, we recommend using the hardest $30$% of training data and otherwise $50$%. If calculating the entropies during training is intractable for whatever reason, it should be sufficient to use randomly sampled data.

Dear authors,

Your comments will be fully considered in the final recommendation. Many thanks for your efforts made during the rebuttal.

Best regards,

Your AC