Calibration Attack: A Framework For Adversarial Attacks Targeting Calibration

Q1: The x-axis represents the confidence range, and the y-axis is the accuracy in that range of confidence scores.

Q2: Average confidence in the tables is referring to the averaged predicted confidence score for all of the tested data. In the context of the ECE calculation it is the same but relative to a specific bin.

Q3: Although it would be interesting to test these additional baselines, we have already tried many baseline defense methods and we cannot test every possible method of this type. We believe the current methods we tested are a good overview of the general performance.

Q4: The code for the results in table 1 and the rest of the results will be provided later as we will put the full code on GitHub. For now, are there any specific questions about the implementation as we cover our algorithm and training details in detail in the paper?

Weaknesses 1 and 2: We greatly appreciate making us aware of these other previous works that are in a similar vein to our paper, and we will acknowledge and reference them in our revision. Nevertheless, we still believe that our contributions over these works are substantial, as none of the prior works comprehensively study the range of the 4 attacks we introduce and cover in the context of calibration rather than uncertainty estimation, nor do they systematically investigate how well victim models would remain well calibrated under such attacks, including under a range of different calibration methods and adversarial defense methods, including a few novel ones like CAAT and CS. Unlike previous works, we examine in detail the effects of attacks targeting overconfidence as well, and evaluate the detectability relative to standard attacks both in terms of detectability methods and visualization approaches like Grad CAM.

Weakness 3: Indeed, they are not traditional defenses, but in the context of our work we are trying to examine whether by trying to better calibrate a model through traditional calibration methods leads to any sort of robustness against calibration attacks.

Weakness 4: Although this baseline would be an interesting point of comparison, we do already test methods meant for robust confidence scores like AAA. In addition, the L2 bounded norm used as the bases in the Kumar et al. (2020) paper means it would be difficult to use it as a comparison given the primary focus of our analysis and results in terms of defenses is in terms of ℓ∞

Weakness 5: That is true that it requires the attack to know the ground truth, but for most conventional adversarial attack papers it is already assumed the attack knows the correctly classified samples and attacks them to result in a misclassification, hence we do not see how this is all that different from that. In addition, the maximum attack is meant to be the strongest case. One can still utilize all of the other attacks, and in many cases the random attack performs quite similarly to the maximum miscalibration attack, and it requires no knowledge about the labels, so the harm from calibration attacks is still great.

Q1: We believe our experiments and analysis are novel, particularly for the overconfidence attack threat model which has not been studied before in previous work.

Q2: That is an interesting notion. We focus on the predicted class as it is more query efficient than trying to target every single possible class, especially in the black box setting, especially when the number of classes is very high (like 100) and many have very low probabilities which would be difficult to raise to yield a uniform distribution given the relative inefficiency of the overconfidence attack. (What’s the application of the suggested experiments)

Q3: We chose the maximum because it is the strongest form of attack and hence offers the best stress test to the defense methods. While its true that CS targets being effective against the maximum attack, it is primarily designed to be against it to avoid worse case miscalibration through the attacks. There is no perfect method to resist all forms of calibration attacks yet, which is why we open up the issue to the community to find methods that work across all forms of attacks.

Q4: Even the overconfidence attacks can affect the accuracy as we have noted in practice, particularly on samples close to the decision boundary, so we must reject a certain number of updates because of this, so we believe attack transfer might be difficult, at least with maintaining the goal of leaving the accuracy unaltered.

Motivation: The effect on average classification confidence can be seen, including a t-sne diagram in the appendix showing the effects of the attacks on the decision boundary. Calibration error is also the primary means of determining how suitable a model’s predicted confidence scores are. We can refer to papers such as Confidence Estimation and Deletion Prediction Using Bidirectional Recurrent Neural Networks (Ragni et al.) where in an ASR system downstream confidence scores are used for flagging up words that need further review for not wasting resources annotating data that is too easy, in which case edge cases are identified through confidence scores. Both of these require low calibration error for example. In the both cases, overconfidence attacked data will make it impossible to do these tasks. We can add these discussions in our revision.

Novelty: Please refer to rebuttal to Reviewer PTEC

Q1: That is an interesting question. We thought about this previously, but determined that it does not make that big of a difference targeting a model’s existing weaknesses by estimating its distribution because the max and random attacks tend to be more effective in the long run. One can implement targeting the specific calibration distribution this naturally by choosing the corresponding one of the attack types if one wishes (e.g., if average predicted confidence is far higher than accuracy then use overconfidence attack). The 4 different attacks we create offer a lot of flexibility in terms of how you would want to attack a model, and on which. Many derived attacks can be created, if same one determines that at a confidence level of 70% the model is very inaccurate, hence we should use the underconfidence or overconfidence attack to place more incorrectly classified samples into that range. This could be more efficient if one is using very small query sizes, but in the long run the maximum attack will be more effective, and the random attack will completely result in scrambled confidence scores.

Q2: Based on the current implementation, it is unlikely, though you could pick a black box attack that only makes use of predicted class labels and run the maximum attack on the samples based only the class labels and we believe the effect should be the same, although with less efficiency.

Q3: This is interesting but out of scope for this paper as we assume that classifiers are already trained and this scenario would be more of a data poisoning attack. In addition, existing models tend to lean towards overconfidence particularly when training on cross-entropy loss and that this is an inherent learning bias. We know that by feeding calibration attacked samples the inherent calibration of the model (on clean data) is made worse as it does with all of the adversarial training methods, but a sort of data poising approach we do not think it is as viable a scenario as we primarily want to introduce the issue in its most likely form.

It is true that the extension is not complicated, but many papers present techniques or methods that are simple modifications or variations of previous one, like the temperature scaling technique in the famous Guo et al. 2017 calibration paper, which was a simple modification of the previous Platt Scaling method, but provide strong results or present an interesting empirical analysis. We believe our paper fits into this category as no one has looked at adversarial attacks from the standpoint of attacking calibration without affecting the accuracy, and particularly examined aspects like efficiency, detection rate, defense method effectiveness etc. This is simply not an issue that the community is aware of, and without this paper vulnerable systems may not have the insights to see how these kinds of attacks can occur and comprise their performance. In particular studying attacking through overconfidence is not something that was really featured in previous works, and we believe that the results are interesting, including the insight of how it is more difficult to attack overconfidence than underconfidence. The autonomous driving example is only one such example that we put in because of the GPTSRB dataset. We can add and discuss an example in the medical domain where a doctor is using a model to determine whether x-ray scans contain cancer. In this case by attacking the scans and making the model miscalibrated without compromising the accuracy, edge cases that may or may not contain cancer can be made to be overconfident, meaning it is easier for the domain expert to ignore them. Vice versa, easy cases that contain no cancer can be made significantly underconfident, flooding the domain expert (doctor) with a lot of unnecessary work double checking scans. We will update this in our revision to make it clearer.