Detecting Brittle Decisions for Free: Leveraging Margin Consistency in Deep Robust Classifiers

Thank you for your review! Please find below our responses to your concerns.

a/ Lipschtz Smoothness vs. Margin Consistency

Thank you for raising this point. Lipschitz smoothness is an essential property of robust models since a small Lipschitz constant $L$ guarantees the network's output cannot vary more than a factor $L$ of the input variation. Empirical adversarial training strategies aim in a certain way to achieve Lipschitz's smoothness indirectly with various levels of Lipschitz constants, and it is also possible to directly constraint the Lipschitz constant to achieve $1$-Lipschitz networks ([1] and references therein). However, Lipschitz smoothness does not imply margin consistency. Indeed:

Let $f$ be an $L$-Lipschitz neural network i.e. $||f(x_1)-f(x_2)|| \leq L||x_1-x_2||$, $\forall x_1, x_2$. Let us consider two points $x_1$ and $x_2$ with $0<d_{in}(x_1)<d_{in}(x_2)$. Note that the $L$-Lipschitz condition implies that $d_{out}(x_i)\leq L d_{in}(x_i)$ for $i=1,2$. However, as long as $d_{out}(x_1)>0$, it is clearly possible a priori to have $d_{out}(x_2)<d_{out}(x_1)$, thus violating the margin consistency condition, while still satisfying the previous relations.

This theoretical possibility is supported by the empirical evidence that we found two robust (hence probably Lipschitz) models which are not margin-consistent. The question of how both properties influence each other is out of the scope of this work and may be grounds for future work. We propose to add this clarification to the paper.

The theorem itself establishes that margin consistency (preservation in the feature space, of the order of the samples' input space distances to the decision boundary) is a necessary and sufficient condition for using the logit margin as a proxy score for the input margin.

b/ Possibility of adaptive attacks

Implicitly, margin consistency, like standard accuracy, measures a property of the model over test samples iid with the training data. We leverage it to detect vulnerable iid test samples susceptible to being adversarially attacked. We concede that it may indeed be possible to craft examples that are non-robust ($d_{in} \leq \epsilon)$) but for which $d_{out}>\lambda$, and therefore would bypass the proposed detector. We believe studying the margin consistency on adversarial examples or designing adaptive attacks to be out of the scope of this work.

c/ (Q1) Equidistance Assumption

In Appendix C, by "values," we refer to the distances between the classifiers $||w_i-w_j||_q$ and by a small range, we mean that the interquartile range is small compared to the mean value of the distance. The empirical evidence shows that using the logit margin instead of the exact feature margin (minimum over margins to other classes) marginally affects the results and avoids the computational overhead of searching the minimum (See the general response section and Table 1 of the attached PDF file).

d/ (Q2) The property regarding the feature space distance

In Figure 5, by the feature space distance, we mean the distance $||h(x)-h(x’)||$ in the feature space between the representations of a sample $x$ and the representation of its adversarial example $x’$. Although the correlation is a bit stronger than with the logit margin, it is as hard to compute as the input margin ($||x-x’||$) since they both require finding the closest adversarial example in the input space.

[1] Araujo, Alexandre, et al. "A unified algebraic perspective on Lipschitz neural networks." arXiv preprint arXiv:2303.03169 (2023).

Thank you for the response. While the response cleared up my confusion on Q2, I still have some remaining questions.

• (Minor) I agree that Lipschitzness and margin consistency are different concepts and it is possible to find counter-examples that satisfy one but not the other. That being said, they are still intuitively and empirically connected. The authors found two models that are "likely Lipschitz" but not margin-consistent, but also found 17 models that are robust margin-consistent (Figure 2). Hence, the theoretical results, while valid, are unsurprising in my opinion.

• (Major) As mentioned in multiple places in the paper, this work focuses on detecting non-robust decisions with logit margins, which often manifest as adversarial examples. If it is possible to construct adversarial examples that can break the margin consistency and deceive the logit margin-based detection, then the purpose of the proposed method is defeated.

Hence, I kindly disagree that studying the margin consistency on adversarial examples or designing adaptive attacks is out of the scope of this work, and believe that they must be carefully analyzed. This is especially important because existing work has shown that adaptive attacks can significantly compromise adversary detectors [1].

• (Minor) I am still confused about Figures 8 and 9. The authors mentioned that the distances between the classifiers vary by a small range. However, the data points in Figures 8 and 9 seem all over the place.

Therefore, for now, I maintain a rating of 4.

[1] Carlini, N. and Wagner, D. Adversarial examples are not easily detected: Bypassing ten detection methods.

Thank you for the clarification, which cleared up my confusion on the two minor points. However, I would like to follow up on the possibility of adaptive attacks. I will raise my score if these questions can be sufficiently addressed.

We aim at detecting (clean) samples on which the network’s decision is non-robust (susceptible to being attacked).

In this case, are the applications of the proposed method restricted to the case where 1) adversarial robustness is of interest; 2) clean examples are always provided?

Could you please provide a motivating scenario that satisfies this assumption?

We observe that the logit margins on adversarial examples are almost always (much) smaller than that threshold.

This is where an adaptive attack is important. With a white-box adaptive attack, is it possible to find adversarial examples (which trick the model into mispredictions) with larger logit margins? Is it possible to find examples that do not necessarily lead to mispredictions, but induce large logit margins without actually being far from the decision boundaries (i.e., lead to robustness overestimation)?

Also, what types of attacks are used to generate the tables?

We are not trying to differentiate between adversarial samples and clean samples that are close to the decision boundary. Instead, we both flag them as non-robust.

I agree that the proposed method can detect clean examples close to the decision boundary. However, for the above reasons, I am not fully convinced that adversarial examples can also be flagged.

Thank you for the response.

Regarding the two applications:

Given a large dataset, the logit margin can provide a reasonable estimate of the empirical robust accuracy by only performing attacks on a small subset of the dataset.

This makes sense. Hence, I have increased my rating to 5. Please add this result to the paper.

In a real-time deployment scenario, you can determine in real-time, just from the forward pass, which samples are vulnerable to adversarial attacks without actually performing an attack.

In a deployment scenario, attacks would be performed by some attacker outside the control of the deployer. Hence, the deployer may only receive adversarial examples and not have access to clean examples. Since the proposed method is only proven to work on clean examples, I am not convinced that it would be effective in this scenario.

Regarding FAB attack:

Is it the case that FAB attack finds minimally perturbed adversarial examples, and stops as soon as it finds one? If this is the case, then it makes sense why the output margins of adversarial examples are tiny. Does a similar phenomenon still hold for other attack algorithms that do not terminate early, such as untargeted and targeted PGD?

Regarding adaptive attack:

While we have given it some thought, it is not clear to us at this point, how to generate such adversarial examples.

As mentioned above, PGD is an option. You can modify the attack objective, so that increasing output margin becomes part of the goal.

You may also try AutoAttack. However, since the original AutoAttack also stops as soon as it finds an adversarial example, some modifications are needed to increase output margin (see Minimum-Margin AutoAttack in Appendix B.1 of [1] for an example).

Regarding generating examples that lead to margin overestimation (but do not necessarily change the predicted class), one possibility is to run a PGD attack with an objective that maximizes output margin while restricting the distance from the nominal point (thereby restricting the input margin).

[1] Bai et al. MixedNUTS: Training-Free Accuracy-Robustness Balance via Nonlinearly Mixed Classifiers.

Thank you for your review! Please see our comments in the general response for answers to your questions (Q1 and Q2 are addressed in the third point of the general response, and Q3 is addressed in the last point). Below are our responses to your other concerns.

a/ Bias across subpopulations:

Local robustness bias exists indeed in models; thank you for bringing up this interesting comment. Interestingly, when margin consistency (approximately) holds, we observe that robustness discrepancies across classes in input margin can also be observed using the logit margin (see top row, Figure 1 of the PDF attached to the general response section). We also observe that margin consistency (approximately) holds for each class individually, so no bias seems present as far as margin consistency goes. For the weakly margin-consistent models (example shown in the bottom row of Figure 1, attached PDF), there are significant disparities between the correlations across classes, which is also why using the logit margin for such models would be problematic.

b/ False sense of security and detection of adversarial examples:

Margin consistency, like standard accuracy, measures a property of the model over test inputs sampled iid from the training data distribution. From the objective they optimize, adversarial examples are close to the decision boundary and, therefore, have very small input margins, and we also observe that they have very small logit margins when compared to clean examples. However, we agree that it could, in principle, be possible for specially crafted adversarial examples to bypass margin consistency. This could be interesting for future work. The detection of adversarial examples itself is a different task, considered a defence strategy, and [1] shows that it is an equivalent task to robust classification. We believe studying margin consistency on adversarial examples or designing adaptive attacks is out of the scope of this work.

[1] Tramer, Florian. "Detecting adversarial examples is (nearly) as hard as classifying them." International Conference on Machine Learning. PMLR, 2022.

Thank you for your review! Please see our comments in the general response about standardly trained models with zero adversarial accuracy and results on ImageNet. Below are our responses to your other concerns.

a/ (W2) No additional analysis for why margin consistency fails for the 2 CIFAR10 models.

We agree that further analysis of these two models may be helpful. However, we believe it may be closely related to a broader question on the relationship between robustness and margin consistency, which can be an avenue for future exploration. We have added an analysis in Figure 1 (PDF attached to the general response section) that gives further insights. It shows a disparity of correlations across different classes, while it is almost uniform in margin-consistent models (top row).

b/ (Q1) Computational cost.

Estimating input margins requires running the FAB attack with a sufficient budget [1]. The average time to estimate the input margins over 1000 samples is the following (on 16GB GPU-Titan XP):

• CIFAR10: ResNet-18 (6min), ResNet-50 (19min), WideResNet-28-10 (35min)
• CIFAR100: ResNet-18 (58min), ResNet-50 (7h), WideResNet-28-10 (6h)

For ImageNet models, a ResNet-50 takes about 1 day 20hrs to 2 days on V100-32GB.

c/ (Q2) How successful would an adversarial attack maximizing the difference of the two max logits be in bypassing the proposed detection?

We believe it is possible to attack a sample to bypass that detection, but we think this is beyond the scope of this work. Nevertheless, in our case, we checked (line 178) that our non-robust samples are almost similar to the ones found by the standard AutoAttack evaluation which includes APGD-DLR that maximizes the DLR loss (Difference in Logits Ratio), which is a rescaled difference between the logit of the true class and the biggest logit among the others. This would seem to indicate that the logit margin can also detect vulnerable samples to this sort of attack.

[1] Xu, Yuancheng, et al. "Exploring and exploiting decision boundary dynamics for adversarial robustness." arXiv preprint arXiv:2302.03015 (2023).

Thank you for your review! We will take the corrections into account. Please see our comments in the general response about defining the margin consistency in terms of the logit margin and results on the $\ell_2$ norm.

a/ The proof of Theorem 1.

Thank you for reading the proof carefully; we agree that there was some clumsiness in the sufficiency part. In this sense, we suggest modifying line 625 by:

"Let $x_0$ be an element of the finite set $A^S_\epsilon$ such that $d_{in}(x_0)=\max \{d_{in}(x): x\in A^S_\epsilon\}$ and let $\lambda = d_{out}(x_0)$."

We agree that formulating perfect separation for finite samples only is not fundamentally necessary. However, it appears to us that this opens the way for a simpler proof which avoids dealing with the intricacy of the continuum. We propose to add a comment in this sense in the paper.

As for the necessity part, we need to indicate the proof is by contradiction, thank you for noticing this. We agree that our notion of perfect separability is oriented'' (i.e. non-robust points are required to be the ones with small $d_{out}$) and that we use this property in the proof. We understand that "it is always possible to separate the points with the inequality direction reversed", but this happens if and only if the model is "reversed margin consistent" ($d_{in}(x)<d_{in}(y)$ iff $d_{out}(x)>d_{out}(y)$), which cannot happen for the logit margin which is always non-negative and takes the value zero on that lie on input space decision boundaries (i.e. $d_{in}(x)=0$ implies $d_{out}=0$). While the three points counter example is attractive, it seems to us that the existence of such an example does not follow from the negation of margin consistency. We, therefore, suggest keeping this proof as it is while adding some comments that address the points you rightfully raised.

b/ $DB_j$ In Equation 2 should be the boundary $DB_{ij}$ defined similarly in the paragraph above

$\text{DB}_{ij} = \\{z' \in \mathbb{R}^m: (w_i-w_j)^\top z'+(b_i-b_j)=0\\}$

It is a good catch; we will correct it.

c/ Yes, in Line 133, we indeed refer to the logit margin as a score computed from the neural network's output (logits). We will clarify that.

d/ Lines 191, 192: the statement ... margin consistency is a property orthogonal to the robust accuracy ... seems like a strong statement to make.

We recognize that it is not the correct way to convey our message here, which is that being robust does not imply margin consistency (cf. discussion on Lipschitz smoothness vs. margin consistency, the first point of the answer to Reviewer WVo4). There may be indeed more to explore about the connection between the two properties. We propose to remove that statement and add the discussion about Lipschitz's smoothness and margin consistency.

e/ Pseudo-margin learning and the Sigmoid activation

The purpose of our study on pseudo-margin learning is to provide evidence that it is possible to learn to map the features $h_\psi(x)$ to a proxy $d_{out}$ that simulates the margin consistency. Our approach is inspired by the setup used for confidence learning [1] (ConfidNet): a target confidence $y \in [0,1]$, Sigmoid as output activation with MSE loss. In our case, we use $y=\frac{d_{in}(x)}{\max \\{ d_{in}(z): z\in S\\}} \in [0,1]$ as a target, where S is the training set. In L2R, the earliest and simplest methods are point-wise methods, which are no different from standard regression with the MSE loss [2]. This worked worked well for our purpose.

[1] Corbière, Charles, et al. "Addressing failure prediction by learning model confidence." Advances in Neural Information Processing Systems 32 (2019).

[2] He, Chuan, et al. "A survey on learning to rank." International Conference on Machine Learning and Cybernetics. Vol. 3. Ieee, (2008).