Towards Certification of Uncertainty Calibration under Adversarial Attacks

We thank the reviewer for taking the time to read our work, providing such detailed feedback and recognising the novelty of our work. We are pleased they find the work interesting and our presentation well detailed.

We thank the reviewer for raising minor issues on language. We will not comment individually here, but appreciate the feedback and incoporate them into our revision.

W1: [...] this technique relying on an approximation to construct ACCE. I'm intrinsically cautious about any result that presents itself as a certification while relying on breaking the guarantees of the certification in order to calculate results. Expectations around certifications guide their use, and violating the precepts of a certification while still labelling it as a certification is problematic to me. To me, an approximation to a guarantee is only useful in rare circumstances, where other insights can be gained by the approximation process - something that I would argue is not present within this work.

The reviewer is correct in noting that the ACCE is indeed approximate, which is a limitation of the work presented. We would like to note that we are very transparent about it, naming the metric approximate.

Further, in this work we propose multiple metrics. The certified Brier score (CBS) is available in closed-form and an exact bound (see Eq 3). Second, we show that the certified calibration error can be seen as a mixed-integer program (see Eq 6). The solution to this MIP is an exact bound. The approximate nature of the ACCE comes from solving this using ADMM.

We believe this inexact solution to the MIP is still very useful.

1. The ACCE provides a pessimistic view on the calibration error under attack. Certificates are often used in an apriori risk assessment of the model under deployment. While the ACCE cannot provide an exact guarantee, we argue that it still provides useful information on possible model failures. The practitioner should, of course be weary to make a clear distinction between an exact and an approximate solution.
2. Many certication methods such as Gaussian smoothing provide non-deterministic certificates (Cohen et al 2019). Hence, with small probability ($< \alpha$) the certificate is false. As noted by the reviewer, our ADMM solution to the MIP does not enjoy such exact bounds on falseness. However, we believe this shows that the community is generally interested in fallable certificates, if they are useful. Further, we should note that many certification techniques provide an exact guarantee for the accuracy on a specific testset failing to establish exact certificates on metrics w.r.t. the samples presented during deployment.
3. The inexact nature allows us to perform adversarial training as we present with our ACT-ACCE method. Performing certified training has been vital in the field to improve certicates.

W2: [...] motivation for their work [...] is a bit sparse - it is sufficiently detailed for comparisons, but the ultimate applicability of this attack could require more justification. This perception is reinforced by statements being made regarding risk and processes that might support this work without citations - leaving this reader curious as to if the justifications actually exist.

Before diving into motivation of attacking uncertainty, we would like to reiterate that any adversarial attack will yield changes in calibration. If a system uses uncertainty estimates, adversarial attacks will impact the reliability of the system. That includes the classical attack maximizing the cross-entropy. This should provide considerable motivation to many practitioners to pay more attention to uncertainty and not just accuracy when attacks are expected.

Now towards uncertainty attacks: We are not the first to argue that attacks on uncertainty estimates are possible. Galil & El-Yanif (2021) argue that attacks on individual confidence scores (i.e. sample-level uncertainty estimates) are plausible and motivate this extensively. Kumar et al. (2020), who present the certification of confidence, conclude their paper stating that "However, this method does not produce any guarantees on the calibration of the underlying model itself." recognising the need for another level of certification for calibration of the model, which we provide.

As we are the first to show direct attacks on the model reliability, there is no literature that we can cite on their reception. We believe that these attacks are conceivable and will eventually be carried out.

Regarding our statements on the risk: Any decision system has associated risks that clearly change when the decision model is misspecified (e.g. due to adversarial attacks). In the credit lending example, credit risk management is a core discipline of finance. We gladly elaborate and cite literature on this. Could the reviewer clarify where they do not find our arguments convincing.

Thank you to the authors for your detailed response - I was starting to wonder if my review had been ignored due to the time delta in responses, but clearly you've put work into this, and that's appreciated. This message is primarily just to note that I've read your response and I'm digesting it, and I will get back to you if I have any additional questions.

I will stress, however, that while the updates to the paper are appreciated, I hold significant reticence relating to changing your score due to the presentation of approximation certifications. I know we disagree on this point, and that this is an unappreciated comment, however in my eyes, labelling anything as a certification, even with a qualifying statement like calling it approximate, conveys a level of expectation in a user. Because fundamentally, it is still being read as a guarantee.

Apologies for taking a few days to get back to you on this (and for this discussion happening so late into the discussion period), but I wanted to make sure I wasn't missing any potential applications due to the nature of my background in this space.

In all honesty: I do not see a context where a certification is required but where an approximation like ACCE could be useful. Certificates are designed for safety critical applications, and approximate-safety means nothing if we are in a position to require it. And in your case, this is an approximation without even a guide as to how tight it is.

To flip this around - under what conditions do you as the authors see ACCE as being usable? What benefit would it have to the community? And if you were deploying a certification in a context where it was necessary, under what conditions would you prefer to use an approximation rather than expending extra compute for the whole thing?

We thank the reviewer for their continued engagement in the review process and appreciate the discussion. While we disagree on the utility of the ACCE at this point, it seems that we have addressed the reviewers remaining concerns.

In all honesty: I do not see a context where a certification is required but where an approximation like ACCE could be useful. Certificates are designed for safety critical applications, and approximate-safety means nothing if we are in a position to require it. And in your case, this is an approximation without even a guide as to how tight it is.

We agree that an approximation will fall short "if we are in a position to require" an exact certificate on the ECE, but we believe, the ACCE is still significantly more informative than not having it. If certified calibration in general is sufficient, the CBS covers that.

To flip this around - under what conditions do you as the authors see ACCE as being usable? What benefit would it have to the community? And if you were deploying a certification in a context where it was necessary, under what conditions would you prefer to use an approximation rather than expending extra compute for the whole thing?

We will outline some benefits the ACCE brings to the research community and practitioners.

Research Community

• To reiterate, the MIP in (6) with the ACCE solution obtained through ADMM is useful to augment training as we propose with the MIP in (8).
• We are the first to study protecting the ECE and believe our work marks a major step forward.
• Optimisation of the expected calibration error has been studied in recent years. For instance, the $dECE$, $MMCE$ and $ECE^{KDE}$ have been proposed (Bohdal 2023, Popordanoska 2022, Kumar 2018; in our preliminary experiments $dECE$ uniformly outperformed the others.). The ACCE at convergence is an exact expected calibration error estimator (see Figures 10 and 11) and hence our work marks a major step forward in the literature on optimisation of calibration properties, in our case the actual ECE rather than some approximation.
• We believe the MIP in (6) provides the basis for others to employ better optimization techniques to extend it / solve it.
• Our ACCE is a sets a new SOTA for calibration attacks.

Practitioners

We assume, a practitioner want deploys a model and is concerned about accuracy and uncertainty calibration under attack. The practitioner would like to estimate the apriori risk under attack. Hence, the practitioner chooses to use a model with smoothness guarantees and certifies accuracy.

Without this work. They would be lost to judge how poor calibration can be. They might use standard $\max \mathcal{L}_{CE}$ attacks on the smooth model to observe a) Brier score and b) expected calibration error.

With this work. They observe the certified Brier score (CBS) in closed form as exact bound providing clear apriori understanding of miscalibration under attack. However, this does not tell them how poor the expected calibration error (ECE) metric can become. In the absence of an exact certificate on the calibration error, the strongest empirical calibration error, i.e. ACCE, is very informative providing apriori information on how poor the ECE can become. If the ACCE is large and the deployer decides that the associated risks are too high, the deployer might consider not deploying the model, having gained valuable information from the ACCE. Contrary, should the ACCE be sufficiently small, the practitioner reaches a decision based on CBS and ACCE, knowing that a) miscalibration is truly bounded through the CBS (recall Brier score decomposition in refinement and calibration), b) there is a potential excess risk due to ECE exceeding the ACCE. At this moment, no attack method is known that induces significant excess risk, but it is possible.

We further discuss motivations to care about the ACCE in Section 2.2.

To the best of our knowledge there does not exist a single method providing meaningful, exact guarantees during inference (including certified accuracy) and hence all are an estimate.

In the vast majority of applications, we do not forsee a dichotomy of exact certificate or no certificate. While such might exist, in most cases protecting the CBS exactly and estimating the CCE through ACCE (given how strong it is in comparison) will be significantly more informative than not doing this. We hope the reviewer agrees with us on this.

W3c: [...] most SOTA defense methods protecting model accuracy against attackers actually sacrifice accuracy and thus even these have a hard time finding real-world use. The premise of this paper leans on these methods being used but not being protective enough of calibration (although the papers experiments do show that their method can be used without sacrificing much certified accuracy).

About general robustness literature: We agree with the reviewer that a significant portion of robustness and certification literature has trade-offs on the accuracy that are unfavorable, thus reducing their deployment in the real world. We still believe, however, that this research has merits and that there are a wide-range of security-critical applications where such trade-offs can be justified.

As to our paper: If a system uses uncertainty estimates, estimating its calibration is vital. This incurs no cost towards accuracy. Hence, estimating the ACCE and CBS should not be considered part of a trade-off. For the adversarial calibration training (ACT), it is true that such trade-off can occur and that is a weakness in which our work follows existing work. However, as the reviewer notes, we find that this trade-off is very weak and accuracy can be retained when using our methodology. Hence, we think we make a compelling offer with ACT.

Questions

We incorporate all of our discussions here into the revised paper.

Q1: (line 160) When $\omega = \hat{y}$ it seems that Equation 2 would read as maximizing $\mathcal{L}_{CE}(f(x,\gamma), f(x,\gamma))$, I think the notation here is a bit unclear, I think you mean $\hat{y} = f(x)$?

There was indeed an error in notation. We define $\hat{y} \triangleq F(x)$. E.g. when $\eta=1$, we maximize the loss w.r.t. the true label $y$ or the predicted label $\hat{y}$. We updated this in the revised paper.

Q2: (table 1) What metric is actually given here? The drop in AdaECE compared to the untracked version? I think at this point it is still slightly confusing how one should interpret these numbers, is higher better? worse? does it depend on $\eta$ and/or $\omega$?

For the AdaECE and ECE, lower is better (here, we use a $ECE \in [0, 100]$ notation rather than [0, 1]). The table shows absolute AdaECE values. For instance, for ImageNet the calibation error (AdaECE) is 3.70 without attack on a vanilla model ("ST"=Standard Training). Using all combinations of $\eta$ and $\omega$ in Eq 2, yields calibration errors that range from AdaECE=1.06 (actually the attacked improved calibration) to AdaECE=47.23 (a 12-fold worsening of the calibration error).

Q3: (line 168) Can you be a bit clearer about how this adversarial training is happening? The existing model is adversarially finetuned? A new model is trained with adversarial training? Standard PGD adversarial training? Does the accuracy of the adversarially trained model change? Or reference somewhere where this information can be found.

We believe this question is about the methodology of the motivating example. We fit the model using standard PGD training that solves the classical min-max-problem over the cross-entropy. We added a more detailed overview of the experimental setup to Appendix B.2.

Q5: (line 190) I think it is important to note that Gaussian smoothing gives probabilistic certificates [...]

See our response to W1c.

Q6: I think throughout the Section 3 it would make the argument easier to follow if there was a toy example of some sort.

We thank the reviewer for this great suggestion and agree that this would be helpful. The revised version now includes a working example in Appendix E.2.

Literature

• Wang, C. (2024). Calibration in deep learning: A survey of the state-of-the-art. arXiv.
• Gawlikowski, J., Tassi, C.R.N., Ali, M. et al. A survey of uncertainty in deep neural networks. Artif Intell Rev 56 (Suppl 1), 1513–1589 (2023).
• Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U. Rajendra Acharya, Vladimir Makarenkov, Saeid Nahavandi, A review of uncertainty quantification in deep learning: Techniques, applications and challenges, Information Fusion, Volume 76, 2021

We thank the reviewer for taking the time to read our paper and the incredibly detailed feedback. We appreciate the reviewer agrees that our proposed metrics and training methods are "novel" and that our experiments are extensive and show the effectiveness of ACT. We are pleased they find our work "very well written and argued". We thank for the number of minor comments. We will not respond individually, but appreciate the feedback and incorporate the remarks into the paper.

W1a: [...] this paper [...] does not at all address the extensive work in non-probabilistic verification/certification of Neural Networks [...]

As our method is agnostic towards how certificates C1 and C2 are obtained, a comparative study of verification techniques remains out-of-scope for this work. We utilize Gaussian Smoothing because it is popular and (as the reviewer notes) performs well. Most importantly, it scales well to modern sized neural networks (at the expense of inference cost). To the best of our knowledge none of the mentioned verification methods scale comparably.

However, we agree with the reviewer that these methods should not remain unmentioned and now mention them in Appendix A.1 in which we provide a detailed background on certification and Gaussian Smoothing.

W1b: I am also curious as to what the authors think about using non-probabilistic verification/certified training methods in the calibration certification case. [...] Do the authors think that adapting abstract interpretation based approaches could work in this case for verification or training?

We thank the reviewer for this interesting question. Yes, certified calibration is compatible with non-probabilistic / abstract interpretation methods as long as certificate C1 and C2 can be established. We also think that our ACT training can be combined with other certified training objectives (e.g. see Appendix J.4.2. L1869 we propose an alternative objective for ACT using the ACCE as penalty when updating weights). We leave a closer investigation of this for future work.

As mentioned above, we use Gaussian smoothing for scalability. However, when using smaller networks, deterministic certification methods (e.g. IBP) can be superior and would likely be preferred in combination with certified calibration.

W1c: The paper defines formal certificates (C1, C2) but does not note that the certificates it proves/approximates are based on randomized smoothing and thus probabilistic in nature (i.e. certifying a score above a certain probability threshold).

We agree with the reviewer that randomized smoothing practically is probabilistic. With small $\alpha$, however, the behavior approximates a deterministic certificate.

Due to space constraints we omit this in the first part of Section 3. However, we do note this in Appendix A.1 (L796-L802) where we provide a more detailed account of Gaussian smoothing.

W2: [...] a total of 5000 GPU hours for the experiments in the paper (although admittedly there are many experiments). [...]

The reported GPU hours are not indicative of the computational cost that practitioners would face applying our method. As the reviewer noted, our experiments are extensive across multiple datasets, hyperparameters and repetitions; they cover pre-training, fine-tuning, Gaussian Smoothing and our certification. The GPU hours refer to the cost of replicating our entire experiments. Practitioners can certify calibration error in <1 GPU hour and perform fine-tuning in low 10s of GPU hours. Certifying the Brier score in closed form has (almost) zero cost.

W3a: Currently accuracy still dominates as a metric for most practical applications

Accuracy, while important, does not inform calibration (see Appendix B.1 for an explanation) and hence there is a need for both, accuracy and calibration. There are many applications where calibrated uncertainty quantification is very important (see surveys e.g. Wang 2023, Abdar 2021, Gawlikowski 2023). We believe, our work sheds some light on the over-reliance on accuracy in the robustness literature.

We thank the reviewer for their efforts to review our work. We appreciate the reviewer agrees that certified calibration is "important", understudied, and our methods are "novel". We are pleased the reviewer finds the paper "easy to follow".

W1: It is unclear how large the effect is in Table 2. The Brier Score although defined in simple terms is appears somewhat uninterpretable.

Could the reviewer please clarify what they find confusing. We are more than happy to clarify and update the table, but are uncertain what the reviewer is referring to.

W2: The authors did not report mean and std of their results over different seeds.

This is indeed correct. Generally, in this work we estimate worst-cases and hence $mean$ aggregation is often less appropriate than $max$ aggregation. Where possible, we do provide insights into variation through replications. Two examples:

• Figure 3: We replicate Figure 3 (showing the ACCE computed through ADMM vs dECE) in Figures 13 and 14. Specifically, in Figure 13 we show the maximum achieved by any numerical method (i.e. the closest approximation to the worst case achievable). This is done over 1000s of initializations. In Figure 14, we show that one set of hyperparameters for ADMM outperforms the $max$ over 1000s of hyperparameter combinations for the dECE.

• Table 2: This table shows the minimum calibration error across many fine-tuned models with accuracies within a certain range. In addition, we present results from a slightly different angle, actually plotting raw data in Figure 4.

We hope the reviewer agrees that we provide insights into the reliability of our results. If we missed the reviewers point, could the reviewer please be more specific and we will gladly consider making adaptations.

W3: The reviewer has the feeling that explanations could be made more concise from time to time while improving readability further.

This work requires previous knowledge on uncertainty calibration and certification, and hence we anticipate many readers might not be familiar with one or the other. We agree that this paper could be more concise, but in the interest of helping those readers we opted for a slightly slower approach.

Q1: For Table 1: could the authors (possibly via bootstrapping) estimate variances here?

We agree with the reviewer that variances of the attack results could be estimated. However, as we do not make comparative claims on the magnitude of the effect (e.g. we do not claim SOTA in Table 1), we opted to invest our compute in other parts of the paper. We believe the results in Table 1 in their form sufficiently motivate the paper by showing that the calibration is vulnerable.

Q2: What is the final loss that is optimized to solve equation 2? As in how did you encode it?

We use $\eta \mathcal{L}_{CE}(f(x+\gamma),\omega)$ (see Eq 2) directly as loss. We perform projected gradient descent to satisfy $\|\gamma\|\leq \epsilon$ and stop optimisation if the next step would violate $F(x+\gamma)=F(x)$. As described in our answer to Q1, the purpose of this attack is to show how trivially it is possible to alter calibration, not establishing any SOTA. Hence, we focused our attention on our own methods rather than iterating to improve the attack in Eq 2.

Q3: Is the certificate actually sound or just approximate (see Line 238)?

The certified calibration error (CCE, Eq 4) by definition is the true upper bound. The exact solution to the MIP in Eq 6 is also a true upper bound. The approximate calibration error (ACCE) is a numerical approximation. Obtaining an exact CCE is challenging due to the discrete, non-convex nature of the problem. Finding non-vacuous upper bounds is also challenging (e.g. tools such as Minkowski inequality yield vacuous bounds). The certified Brier score (CBS) is a true upper bound and tight.

Q4: Is $a_{i,j}$ only for 1 $j$ allowed to be 1 while the others are 0?

Yes, that is correct and enforced through the "Unique Assignment Constraint" (see L279 to L283).

Q5: What where the considerations for choosing ADMM?

The optimization problem is mixed-integer and non-convex, for which no "canonical" optimizer exists to the best of our knowledge. ADMM has been shown to posses good convergence on other non-convex problems (see L314, Wang, 2019), and is simple to implement using software packages such as PyTorch or JAX. After initial experiments, we found it easy to get consistent results with fairly standardized hyperparameters. In addition, it is very fast and scalable.

Q6: Does the reviewer suppose correctly that the runtimes in G1 were reported for CIFAR-10?

Yes, that is correct. We updated G.1 to clarify.

We thank the reviewer for their efforts and recognizing the importance of our work in identifying a "critical vulnerability in NN classifiers" and the "well-formulated mathematical foundation" of certified calibration.

W1: The entire certification framework relies heavily on the binning strategy used in ECE computation Q3: How sensitive is the certification to the choice of binning strategy in the ECE computation?

We do not see a strong impact of the binning scheme on the ACCE. The ACCE does indeed depend on the binning as we use the canonical estimator (see Eq 1). However, we find strong correlations between certified Brier score (CBS) and the ACCE (r=0.98) (see L494-495). The CBS does not depend on any binning and is available in closed-form, suggesting that binning does not confound the ACCE significantly.

W2: Complex hyperparameter tuning required for ADMM solver

While the reviewer is correct in noting the large number of hyperparameters required for ADMM, we have performed extensive experimentation and identified a set of hyperparameters for the ADMM solver that works very well across all our experiments. We list these parameters in Appendix I.1. Using any of these or the ensemble (as we do in Section 5.3) will likely yield good convergence characteristics.

Additionally, in Figure 14 we show that a single set of hyperparameters for the ADMM solver (almost uniformly) outperforms the maximum over 1000s of hyperparameter sets for the dECE across datasets and certified radii. We strongly believe the evidence provided suggests that the hyperparameters we present can be expected to work well in a very wide range of settings.

The certified Brier score is available in closed form and does not require any hyperparameters.

Q1: How does the certification process scale with the number of classes? Would the approach be practical for problems with hundreds or thousands of classes?

The answer depends on the metric:

CBS: The Brier score itself depends on the number of classes $K$ linearly, $\mathcal{O}(K)$. The CBS inherits that.

ACCE: The certification only depends on the correctness of each prediction. Once that is established, obtaining the ACCE has constant complexity in the number of classes $\mathcal{O}(1)$.

Hence, we propose that our method scales favourably with large number of classes, far beyond the 1K classes in our ImageNet experiments.

Q2: Could the proposed methods be extended to other uncertainty estimation approaches beyond softmax confidence scores?

Yes, our method is agnostic to uncertainty estimation techniques. All that matters is that the certificates C1 and C2 can be established. For instance, Gaussian smoothing can be performed with temperature scaling or over the composition of neural network + Platt scaling establishing C1 and C2 without any adaptations (i.e. Lemma 2 in Salman et al. (2019) still holds when the smoothed function is the composition of neural network + confidence postprocessing).

Literature

• Hadi Salman, Jerry Li, Ilya Razenshteyn, Pengchuan Zhang, Huan Zhang, Sebastien Bubeck, and Greg Yang. Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers. Neurips 2019.