The Uncanny Valley: Exploring Adversarial Robustness from a Flatness Perspective

Dear Reviewer 3F1W,

There seem to be multiple important misunderstandings that we wish to clarify. In addition, we give background knowledge on adversarial examples and flatness.

1. Significance of the Contribution

We disagree that an investigation into the phenomenon of the uncanny valley and the relationship between flatness and adversarial robustness is not well-motivated. As our broad empirical evaluation shows, the uncanny valley phenomenon can be observed in a large variety of datasets and model architectures, including large language models. Our theoretical contribution shows that we can connect generalization and adversarial robustness through flatness of the loss, addressing a significant gap in the literature.

1.a Related Work

The phenomenon of the uncanny valley is novel. To the best of our knowledge there exists no study of the flatness of the loss around adversarial examples in the literature. Existing work on adversarial attacks and defenses can be related to flatness and we discussed those in Sec. 2. It is often claimed that models that generalize better – as measured in terms of flatness – are better protected against adversarial attacks, which is at odds with the common observation that adversarial robustness often leads to worse performance.Overall, this reflects a contradictory existing evidence of links between flatness and adversarial robustness.

1.b The Uncanny Valley Phenomenon is Counterintuitive

Flatness has traditionally been associated with desirable properties like good generalization and robustness, which Schmidhuber and Hochreiter proposed in 1995. Given that adversarial examples are by definition, a consequence of poor generalization, one would expect that the sharpness of the model evaluated at the adversarial distribution is high. In layman's terms, the model lies in a sharp area. Hence, it is unintuitive that adversarial samples do not lead to a high sharpness.

To reiterate: Since adversarial examples cross the decision boundary, it is to be expected that sharpness increases until the label flips. Although it has not been studied before, a decrease in sharpness afterwards is not unexpected. What is unexpected, however, is that adversarial examples then move into a very flat extended region, or valley. Since, intuitively, adversarial examples, which exploit vulnerabilities, should lie in sharp regions where the model is unstable. Thus, the fact that adversarial examples instead reside in very flat regions, as revealed by our empirical results, contradicts this intuition and raises critical questions about the sufficiency of flatness as a robustness metric. Our theory then complements these findings by showing that robustness is indeed related to flatness, but only around a given dataset.

1.c Conclusions from Experiments with Adversarial Training

These experiments show that our theory is correct, because adversarial training increases flatness around clean examples. It also supports our crucial conclusion that while we can guarantee adversarial robustness around the training examples, flatness cannot inform on adversarial robustness away from it.

To cite our manuscript: “If we attack these more robust models with stronger attacks, we again observe that the adversarial examples lie in flat regions (Fig. 4c and 5c). Hence, the uncanny valleys still exist and can be reached via stronger attacks.”

Dear Reviewer nqir,

We appreciate your detailed review and the effort you put into evaluating our submission. However, we strongly disagree with your assessment of the paper’s soundness, presentation, and contribution. Below, we provide a point-by-point rebuttal to address your concerns and demonstrate that our work makes a significant and well-supported contribution to the field.

Theoretical Contribution and the Uncanny Valley Phenomenon

Indeed, in Prop. 5 and Cor. 6, relative sharpness is computed on the unperturbed training set. We then use the relation between flatness and robustness to random perturbation established in Petzka, et al [1] to prove adversarial robustness around training examples. This offers a novel perspective by linking flatness of the loss curve, generalization, and adversarial robustness, addressing a crucial gap in existing literature. Our experiments confirm that for well-generalizing models, the area around training examples is flat. The fact that the flat area around training examples is increased through adversarial training (Fig. 4 and Fig. 5) confirms our theory. Moreover, [7] empirically observes what our theory predicts, that is, flatter regions are more robust to adversarial examples and generalize better.

Definition of Adversarial Examples

The definition of adversarial examples has always implicitly depended on the choice of loss function. In Szegedy [4], an adversarial example changes the prediction which is captured by Def. 2 using the assumption of locally constant labels and any common loss function with $\epsilon = 0$. In order to relate adversarial examples to sharpness of the loss function, it is essential to define adversarial examples wrt a loss function.

The proposed Def. 2, which is a generalization of existing definitions, offers an improved framework over the literature for three reasons.

1. The existing definitions in the literature relate to the optimization problem of finding adversarial examples, but do not provide a general formal and practical definition. E.g., in Def. 1, only the closest point to the clean example that flips the label is considered an adversarial example, whereas a point that is just an $\epsilon$ further away would not be considered one. In practice, the attack algorithm does not converge to the optimum, so that the points found in the experiments are formally not adversarial examples. Def. 2 instead allows us to define the set of adversarial examples independent of an attack method.

2. As stated in Sec. 3, definitions of adversarial examples in the literature make several implicit assumptions (e.g., locally constant labels). These assumptions are reasonable in practice for image classification tasks, but should be made explicit. For example, in a regression task it is often the case that small changes to the input do lead to large changes in the output. Thus, existing definitions would not apply here.

3. It is a general definition of adversarial examples, independent of the use case or data type. As we show in Appendix D, Def. 2 can be instantiated to cover existing definitions in the literature, including targeted attacks, such as Szegedy, et al [4], Papernot, et al [5], Carlini and Wagner [6], as well as untargeted attacks [6].

Dear Reviewer xr3Q,

Thank you for your positive assessment of our submission, in particular its soundness and concise presentation. Below, we address your concern and question.

Formal Definition of "Valley"

We do not provide a formal definition of valley, but use the term uncanny valley metaphorically to describe a set of observations: By valley we mean a constantly very flat region along the path of the adversarial example. This valley is uncanny in the sense that it is still very close to the original example, but confuses the predictor; it is also very flat and the predictor is very certain, yet the predictor is wrong.

Practical Implications for Defense Techniques

Our findings directly inform the development of adversarial defenses. Specifically, they reveal that adversarial examples tend to settle in unusually flat regions of the loss landscape post-label flip, which contradicts the assumption that flatness universally correlates with robust generalization. This insight underscores the need for hybrid defense strategies that combine flatness with smoothness constraints, as detailed in Section 6. Moreover, our preliminary experiments suggest the potential for leveraging relative sharpness as an effective tool for adversarial detection (Appendix E).

Conclusion

We hope these clarifications address your questions and highlight the significance of our contributions.

Dear Reviewer tZta,

Thank you for your detailed review and feedback. However, we strongly disagree with your assessment, particularly your conclusion that our findings are unsurprising and that the contributions of this work are insufficient. We firmly believe that our paper makes a significant and novel contribution to the field by establishing a robust empirical and theoretical connection between adversarial robustness and the flatness of the loss landscape, leading towards the discovery of the uncanny valley phenomenon. This phenomenon, which we rigorously demonstrate across multiple models and datasets, is far from trivial and provides novel insights into the geometry of adversarial examples. Moreover, our theoretical framework offers a novel perspective by linking generalization, relative sharpness, and adversarial robustness, addressing a crucial gap in existing literature. We contend that this paper is both sound and impactful, and thus merits acceptance.

The Uncanny Valley Phenomenon

You correctly point out that the sharpness for an example depends on the confidence of the predictor. That is, the form of the trace of the Hessian in the penultimate layer we derived reveals that very high-confidence predictions are usually flat. This does not explain the observations, though. Indeed, confidence is directly related to the distance to the decision boundary. Consider a binary classification task using logistic regression and CE loss. The prediction goes to $1$ for $w^⊤x+b\rightarrow\infty$ and $0$ for $w\top ⊤ x+b\rightarrow −\infty$. Keeping $w,b$ fixed, this is achieved only when the distance of $x$ to the decision boundary $d=\frac{|w^⊤x+b|}{|w|}$, grows to infinity. High confidence predictions are in regions far away from the decision boundary, where predictions are stable, i.e., flat regions. The flatness measure exactly captures that. A well-trained network is flat around training points, but not perfectly flat. The curious phenomenon is that, when moving away from the training point during the attack, sharpness first increases around the decision boundary (as it should be) and then drops to an extremely low value. This low value persists as the attack progresses. Interestingly, the adversarial example never comes close to any other decision boundary again. Thus adversarial examples are not tiny regions close to clean examples in the otherwise reasonable decision boundary, but rather lie in a broad uncanny valley. We demonstrate this across diverse datasets and architectures (including LLMs), namely that adversarial examples often settle in these unusually flat regions of the loss landscape post-label flip (see Section 5, Figure 2). This behavior persists even under adversarial training, which traditionally aims to eliminate such vulnerabilities.

Practical Implications

The goal of our paper is to reconcile flatness, which is related to generalization, with adversarial robustness, since it has been put into question whether generalization and adversarial robustness are at odds [1]. Methods to improve adversarial robustness, like randomized smoothing and SAM indicate that flatness improves adversarial robustness. Together with the theoretical link of flatness to generalization, one could come to the conclusion that flatness implies that a predictor is correct. Our empirical results, however, show that adversarial examples live in wide, very flat regions. This contradicts the previous intuition that flat regions imply good performance. Our theoretical contribution makes this point more precise: flatness around training examples indeed implies both good generalization and adversarial robustness on those examples, but does not guarantee adversarial robustness on unseen data points. While the focus of our paper is to investigate this relationship, we did perform a preliminary experiment on whether flatness can be used to detect adversarial examples. We show that indeed a detection accuracy of $>90$ is possible using a simple threshold.