Relating Implicit Bias and Adversarial Attacks through Intrinsic Dimension

We thank the Reviewer for their comments on our work, and respond to the weaknesses and questions that were outlined in the review.

On the small decreases in intrinsic dimension: It is true that in some cases the decreases in the intrinsic dimension (ID) compared to the shuffled data sets are relatively small. This is the reason why we decided to adopt a statistical test to assess the significance of such variations.

Correlation is expected: We agree that a strong connection between the implicit bias and the adversarial vulnerability of a neural network was to be expected. However, we point out that, to the best of our knowledge, this work is the first one to explicitly pinpoint such a connection. Besides, our result is not limited to prove the correlation: we give a very precise image-specific map of the spectral implicit bias. This information is potentially useful to design new strategies for adversarial defense and interpretable networks, topics which we are addressing as future goals. Moreover, motivated by this comment and by points raised by other Reviewers, we decided to test our approach on adversarial attacks with higher perturbation budget (namely, PGD with $\epsilon=0.03$). Our findings are not harmed by this choice, as shown in the revised pdf (Appendix, section A.8). Thus, the correlation is not solely justified by the fact that adversarial noise is small.

Issues of the correlation method: While it is clear that some limitations emerge in the case of very high-dimensional data and/or very high-ID data (the case of Fourier masks on Imagenette is particularly challenging as it falls into both categories), we believe that the range of feasible applications of our ID-based correlation method is still very broad.

Paragraph spacing: We thank the Reviewer for pointing out this issue, we fixed the paragraph spacing in the new version of our manuscript.

Question: All attacks are used in their untargeted version, meaning that the only goal of the attack is to produce misclassification, regardless of the adversarial class. We thank the Reviewer for giving us the opportunity to clarify this point in the revised pdf file.

We thank the Reviewer for their feedback on our work. Please see our response below.

On the cosine similarity: The fact that the cosine similarity between masks is not sharp is one of the crucial points of our work. We find that the correlation between sets of masks is highly non-linear and thus not detectable by cosine similarity. This is the reason why we designed and applied a novel non-linear correlation method based on intrinsic dimension estimation.

On additional results: We thank the Reviewer for suggesting to inspect how correlations are related to adversarial success rates. We ran an experiment using PGD with variable perturbation budgets and found that correlation appears in any case. Results are available in the Appendix of the revised pdf (section A.8).

The proposed method is limited and incompleted: We are aware that there are limitations to our ID-based correlation approach, mainly related to the computational issues of ID estimation in the case of very high-dimensional or high-ID data sets. We clearly state these limitations in the paper (sections 3.2 and 4.6). However, we do not fully understand what the Reviewer refers to with the term ‘incompleted’.

Why we need to relate the implicit bias and adversarial attack: The problem of adversarial vulnerability of neural networks remains largely unresolved and, to a certain extent, not fully understood. We believe that our work represents a step towards understanding adversarial attacks and, potentially, defending against them.

On the adversarial class: The Reviewer is right in saying that the attacks we employ are untargeted. We state this point explicitly in the revised version of our pdf file. The adversarial class is simply known by feeding the adversarial input to the classifier.

New methods or strategies on top of our findings: We believe that this work can be a starting point towards new strategies for adversarial defence, as indicated in the future directions (section 5). Moreover, we sketch what we believe to be a promising idea in this direction in the last lines of the section that deals with class-specific masks (section 4.7).

We thank the Reviewer for their comments and for giving us the opportunity to improve the robustness of our work. We respond here to the weaknesses and questions raised in the review.

Minor weaknesses:

1. The Reviewer is right when assessing that the true distribution of the ID cannot be Gaussian, due to the impossibility of obtaining negative values. However, in all our tests, the distribution of the IDs computed after shuffling was nearly Gaussian, somehow justifying the choice of the Z-test. In any case, we coincide with the Reviewer's statement that, although the results of the paper are not affected by this issue, it should be at least commented on, so we added a sentence to the paper on the limitations of the Z-test (section 3.2).

2. As we detail in the paper (section 4.3), we chose $\epsilon=0.01$ for PGD to maintain consistency with the perturbation sizes found by the other attacks. This choice does not harm the efficacy of the attack, which succeeds in almost 99% of the cases (please see Table 4, in the Appendix). However, motivated by this suggestion and by other points raised in the Reviews, we decided to test our method on other perturbation magnitudes, including $\epsilon=0.03$. The results confirm our findings and they are reported in the Appendix (Section A.8) in the revised version of our pdf.

Major weaknesses:

1. Please refer to the response to question 2.

2. We could only provide a compact literature review due to space constraints. However, in section 2.1 we provide pointers to a number of papers that have analysed implicit bias in Fourier space (e.g. Rahaman et al., 2019 and Fridovich-Keil et al., 2022). The connection between implicit bias and adversarial attacks has not been as thoroughly investigated and it is, in fact, the focus of our work.

Questions:

1. We believe that understanding the phenomenon of adversarial vulnerability is a fundamental milestone towards building more robust and reliable models. As anticipated in the previous point, the literature on the relationship between implicit bias and adversarial attacks is, to the best of our knowledge, quite limited at the moment. We cite one paper by Faghri et al. (2021) as the main inspiration four our work. However, it is a purely theoretical paper and it deals with linear networks. Our work extends their results to more complex and useful models using an empirical approach.

2. The strong connection between correlations among data features and the intrinsic dimension of a dataset is a widely acknowledged concept in the field of unsupervised learning. For instance, in the context of hyper-planar manifolds, these correlations manifest in the PCA spectrum, creating a discernible gap that establishes the correct projection dimension. In the case of non-linear manifolds, where correlations exhibit non-linear characteristics, this connection persists, although its mathematical definition becomes somewhat elusive and hinges on the specific method employed for determining the intrinsic dimension. Since our method is designed to be agnostic to the intrinsic dimension determination technique, we purposefully refrain from providing a detailed mathematical description, opting instead to showcase the intuitive aspects of the method and its application to both toy and realistic models. We hold the firm belief that the tests we present are sufficient to support the conclusions outlined in the paper. This confidence is rooted in our verification that artificial uncorrelated by-design datasets with the same macro-features as those investigated in the paper yield clearly different results than those that we found correlated.

We thank the Reviewer for carefully reading our manuscript and for providing interesting points of discussion. We try here to address the weaknesses pointed out in the review.

W1: The Reviewer correctly references Karantzas et al. (2022) as the main inspiration for our Fourier mask framework. However, we would like to note that, while the concept of essential frequency mask had already been explored in the aforementioned paper, the adversarial frequency masks are a novel contribution of our work.

W2: Indeed, it is crucial to consider the variance associated with ID estimation. Employing a maximum-likelihood approach, one can obtain an expression for the variance and incorporate this insight as a correction term in the Z-score test. This would result in a slight modification of the p-value while maintaining unchanged conclusions. Additionally, addressing the bias in ID estimation could likely enhance the results and enable the extension of the test to datasets with a higher intrinsic dimension. However, tackling this remains an open problem in the unsupervised learning community, and its solution falls beyond the scope of this work. It is noteworthy, though, that the method proposed here remains independent of the ID estimator used (that is, if one had a perfect ID estimator, it would equally work on the method). Moreover, by comparing measures of the ID estimated on the same embedding dimension, one could expect that many of the potential biases vanish – or, at least, are mitigated – by error compensation.