Advancing the Adversarial Robustness of Neural Networks from the Data Perspective

We thank reviewer "pfh7" for the criticism and the recommended literature.

---First, we address the mentioned weaknesses regarding the lack of comparison with prior works to delineate our contribution.---

There is indeed a vast corpus of literature regarding adversarial machine learning in general and adversarial training in particular. In order to (re)position our work with respect to the listed papers, we have revised our exposition to include the previous advancements and clarify which “gap” we intend to close with our contribution. More precisely, we focus on “why“ adversarial training shows a weaker effect in regions of non-robust data which cannot be explained entirely by small margins between differently labelled points.

To stress this point, we added to the updated version that the minimal $l_\infty$ distance between two non-robust CIFAR-10 images ($\approx0.2118$) is more than six times as large as the maximum perturbation radius ($\approx0.0314$) which, geometrically, leaves more than enough space for a robust decision boundary. However, it is not found consistently in these regions, as seen in Fig. 1.

Our primary goal and contribution in this work is to provide a new perspective on this phenomenon which builds on a deeper geometric concept, namely curvature (as perceived by a model during training). Whereas one part of this contribution is the theoretical framework connecting the robustness of data to the perceived curvature, our empirical results show that this perspective is meaningful in practice. Regarding the adversarial training experiments, this view of perceived curvature can explain why focusing on a negligible number of non-robust points during training leads to an increased robust generalisation accuracy beyond the expected local improvements. Indeed, as we clarify in the updated version, the focus on the least robust points entails a better understanding of the pixel distances (the local improvement) and a better understanding of semantic distances (judging by the improved performance on unseen data). Together, this leads to a better understanding of curvature.

Whereas we do improve upon the current state-of-the-art approach from Wang et al. (2023) in a cost-neutral way, the main contribution of our work is a deeper understanding of representation learning. Our focus is not to provide a new method of adversarial training (which is beyond the scope of this work) but to motivate why developing methods based on this deeper geometric insight is promising in the first place.

---Regarding the question.---

First, if shifting points away from the true data manifold alone would impede a model's performance, we would expect a much worse accuracy on data augmented with, for example, slight uniformly distributed noise. (It is mathematically impossible to generate noise tangent to the manifold without knowing its shape.) However, as shown recently (Xiao et al. (2022); Chen et al. (2023)), it is possible to generate (unrestricted) semantic adversarial examples which are not only close but on the data manifold.

What the view of a model perceiving curvature allows is to combine both types of vulnerabilities described above. Adversarial noise normal to the data manifold is detrimental even when the model's internal representation of the data concurs with the true representation. On the other hand, adversarial variations of the data may align with the data manifold if the model's learned representation of the latter is distorted in the first place. Our proposed view of a model perceiving curvature can explain how and where these distorted representations appear and why, for example, adversarial training shows a weaker effect in the least robust regions.

Intuitively, in the regions of perceived high curvature, the training algorithm forces a model to connect a binary "semantic" change (labels are either different or not) to a relatively small change in terms of pixels (as is the case in the least robust regions). This may lead to the model adapting by learning spurious correlations to account for this perceived imbalance between semantic and pixel variation, ultimately making the model more susceptible to (adversarial) noise. This can explain why models can even be susceptible to noise perturbations tangent to the data manifold (because they are not aligned with the model's internal representation of the same).

We thank reviewer "vUSY" for the criticism and the suggestions to improve our work.

---First, we address the mentioned weaknesses individually.---

The clarity of the presentation and the quality of interpretation regarding the empirical observations could benefit from further refinement to enhance understanding.

To clarify the presentation of empirical results, we rephrased and refined Section 4 to make it less convoluted and more accessible. We focused on more intuitive descriptions, especially for readers who are not necessarily familiar with the topics.

The improvement in adversarial robustness is very marginal.

In absolute terms, an improvement of 0.2 percentage points seems small but, as stated in the paper, amounts to the same increase of extending the training by 800 epochs, that is, one-third of the entire training (based on results from Wang et al. (2023)). Importantly, this benefit is achieved using identical computational resources. However, the primary goal of these experiments is to show that focusing on a negligible number of points from non-robust regions leads to an increased robust generalisation accuracy beyond the expected local improvements. In other words, a better understanding of the pixel distances between points is paired with a better understanding of semantic distances (hence the improved performance on unseen data).

Regarding the questions.

Figure 1: I suppose this is on CIFAR10. How are the most/east robust elements defined?

The robustness of a data point is the minimal distance to a differently labelled data point. We added an explicit definition in Section 3 to avoid confusion. However, the absolute robustness (or, reciprocally, sensitivity) value is of secondary importance. It is the order of points when sorted according to their robustness (or sensitivity) values which lets us connect these concepts to the perceived curvature of a model. For this reason, we always refer to the "least and most robust points" instead of "points with a robustness of <x> or smaller/larger than <x>". To make the concept more accessible to readers, we changed the introduction by taking a more intuitive approach to explaining data robustness.

The claim at the bottom of page 1 is essentially that by training for another 2000 epochs, the most robust training data (at epoch 400) become 1% more robust (at epoch 2400); and the least robust training data (at epoch 400) becomes 7.5% more robust (at epoch 2400). Is that correct?

Correct.

However, are we tracking the same training data? Is "eval_adversarial_acc" the validation accuracy?

The "eval_adversarial_acc" traces the adversarial robustness against a PGD-40 attack with maximum perturbation radius 8/255 and step size 2/255 on the 1024 least and most robust elements, respectively. We added more information to the introduction to clarify this. However, these subsets are also fixed parts of the training data in every epoch, allowing us to enforce a focus on these points during adversarial training. This example demonstrates that the effect of adversarial training is much weaker in non-robust regions compared to robust regions. We modified the exposition of this example by (i) removing two plots and (ii) changing its explanation, where we chose a more natural way to introduce data robustness beforehand.

Also, it seems that the author uses the term "validation set" as a parts of the training set, which is odd.

To avoid confusion, we changed the nomenclature at several points.

I did not understand "For CIFAR-100, we disregarded the 40 least robust elements because of distorted robustness values due to differently labelled duplicate images". What are "differently labelled duplicate images"? What happens to the result if we include them?

As we found out, the CIFAR-100 set contains identical images with different labels. For such data points, the sensitivity is not defined due to the vanishing quotient (comp. Equation 1). Based on the implementation, the sensitivity computation for these points would either return “Nan”s or extremely large values (as for us), both of which are inadmissible and distort the sensitivity distributions.

Figure 2: My understanding is that the absolute sensitivity is computed using Eq1. How are the relative sensitivity computed?

The relative sensitivity distributions are empirical probability distributions over the absolute sensitivity values (similar to histograms). However, since these plots confuse readers (and add barely any meaningful information), we removed them.

We thank reviewer "8FHc" for the criticism and their literature recommendations.

---First, we address the mentioned weaknesses individually.---

Contribution / significance

To explain and delineate our contribution from existing work and to show the significance of the proposed link between data robustness and the perceived curvature of a model, we reframed our exposition in the updated version.

As mentioned by the reviewer, it is known that nearby differently labelled points (= non-robust points) enforce small margins, which naturally impair adversarial training in such regions (as showcased in the first example). However, while "less space" may explain the weaker effect of adversarial training in such non-robust regions, it fails as an explanation if "less space" is "still more than enough space". To strengthen our argument, we included the following point: the minimal $l_\infty$ distance between two non-robust CIFAR-10 images ($\approx0.2118$) is more than six times as large as the maximum perturbation radius ($\approx0.0314$) which, geometrically, leaves more than enough space for a robust decision boundary. Nevertheless, a robust decision boundary is not found during training for a significant portion of the non-robust points (comp. Fig. 1). As our primary contribution in this work, we provide a new perspective for this phenomenon to explain why these non-robust points are "difficult" for any model and why adversarial training may not lead to a robust decision boundary, even if we know it exists.

Whereas this motivates new adversarial training methods, the development of such a method is beyond the scope of this work. Instead, we provide theoretical and empirical evidence to show that this curvature-based perspective is meaningful in practice and can motivate new methods of adversarial training in the future. We made significant changes in the introduction and Section 4 to clarify this point.

Missing relevant work

We included the listed works to explain what phenomenon we focus on in this work (which cannot be explained by traditional margin-based optimisation strategies; see explanation above). We highlight that our primary contribution is a new and deeper geometry-based perspective to explain why adversarial training shows a weaker effect in certain regions. We clarified this point in the updated version.

Writing and composition

We reframed several sentences in the updated version to make the content less convoluted and clarified some essential aspects to make them more accessible. To this end, we redesigned the example in the introduction as well.

---Regarding the questions.---

Could the authors clarify their contribution in the context of existing methods / analysis (e.g. by providing some explanation for the efficacy of existing methods to enhance robustness)?

Our primary contribution is an explanation for the weaker effect of adversarial training in non-robust regions, which small margins between differently labelled points cannot entirely account for. This explanation, building on connections between a model’s perception of curvature and the robustness of data, can motivate new methods of adversarial training in the future. In this work, we aim to provide theoretical and empirical evidence to demonstrate that this connection exists and is meaningful in addition to the development of new methods (which is beyond the scope of this paper).

One claim is that diffusion model connects regions of non-robust data to more prominent semantic changes, which we take as the model accounting for a more significant perceived curvature. Can this be made more precise?

We rephrased the explanation in the paper to clarify what we mean. In essence, we use the diffusion model to generate new images from the most and least robust regions. As we demonstrate, the semantic differences between the original and generated images are much smaller for the most robust elements than the least robust ones. Hence, the semantic distance (or manifold distance $\tilde{d}$) is larger for the latter. By comparing the corresponding $l_2$ and $l_\infty$ distances between original images and clones, we notice that the converse holds for the pixel distance. More precisely, the pixel distance ($d$) between the original and generated images is more significant for the most robust elements compared to the least robust elements.

By comparing the semantic and pixel distances in the form of the Finsler-Haantjes curvature, we see that the diffusion model connects regions of lower/higher curvature with regions of robust/non-robust data. Here, we refer to regions instead of single points, as curvature and robustness (or, reciprocally, sensitivity) are local concepts (comp. Theorem 1).

We thank reviewer "yQLb" for the criticism.

---First, we address the mentioned weaknesses regarding the exposition and clarity of our paper.---

In order to make the content of our paper more accessible, we have rewritten the introduction. The primary goal was (i) to make abstract concepts such as data robustness and the data manifold's curvature more accessible and (ii) to clarify what our contribution in this context is. We removed two plots in Figure 1 and added a schematic to clarify what we mean by the different distances in feature space and along the data manifold (as a preliminary for curvature).

To show why perceived curvature is meaningful, we outlined how it can explain the weaker performance of adversarial training in regions of non-robust data where robust decision boundaries exist but are not found during training.

To clarify how our experimental evidence supports our theoretic view (of a model perceiving curvature differently in robust and non-robust regions), we have also rewritten Section 4. One goal was to convey an intuition for why we observe a diffusion model generating semantically very different images in the least robust regions but not in the most robust regions. Overall, we explained the experimental setups in a less convoluted way to avoid confusion.

Finally, the gains in adversarial robustness appear small but are indeed significant in this area, where we refer to the work of Wang et al. (2023). However, the surprising fact we stressed in the updated version is that focusing on a negligible number of points from non-robust regions leads to an increased robust generalisation accuracy beyond the expected local improvements. In other words, a better understanding of the pixel distances between points is paired with a better understanding of semantic distances (hence the improved performance on unseen data).

---Regarding the question.---

Yes. This “sensitivity” information was used in the experiments. However, as shown theoretically, we are not interested in the absolute sensitivity value but the value compared to the entire sensitivity spectrum. In this way, sensitivity has only meaning in relative terms. This is why we always refer to the “most and least sensitive or robust points” instead of the “points with a sensitivity above/below x”. Ultimately, this allows us to connect the primitive measure of sensitivity to the elaborate concept of curvature.

---References---

Zekai Wang, Tianyu Pang, Chao Du, Min Lin, Weiwei Liu, and Shuicheng Yan. Better diffusion models further improve adversarial training. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 36246–36263. PMLR, 23–29 Jul 2023.