The Pitfalls and Promise of Conformal Inference Under Adversarial Attacks

We thank the reviewer's positive evaluation and constructive comments. Here are our responses.

Q1: There is only one attack method used in this paper.

R1: We use the strong AutoAttack (AA) to evaluate our method with AT. See the table below (each cell has the format of coverage, PSS) and the revised paper (Table 5). Under AA, the performance of our approach is also substantially better than the AT baseline. Interestingly, the robust accuracy under AA drops while the prediction set size reduced (CP efficiency is improved).

Q2: would it be better to normalize the improvements by the number of classes?

R2: Yes, we agree on this point. Thus, in the updated paper, we normalize the PSS by the number of classees and show the result in Table 3.

Q3: Maybe you can merge those sections into one section (that summarizes the discovery regarding CP against adversarial examples) with two subsections.

R3: We agree on this point. We have revised the paper accordingly, see Section 4 with the name NECESSITATE AT FOR ROBUST AND EFFICIENT COVERAGE.

Thanks again for your review. We are happy to answer any further questions.

Q14: Is the ratio reported not the fraction of points in that region? The caption and the text supporting this figure are confusing.

R14: It is the ratio of p(r) instead of the ratio of samples, which is explained in the paragraph following Equation (7). We will make it clear in the next version.

Q15: What is the reason for only looking at $l_{\infty}$ deviation adversarial examples? RSCP handles $l_2$ deviations; how do the experimental results differ?

R15: We explicitly explained this at the first section in the first page. We only handle the l_inf as it is a more challenging case than l_2.

Q16: What is the pre-trained model used for? Are the adversarial examples not constructed based on the model at hand?

R16: As we have explained in Section 7.1, we use pre-trained model as the initialization instead of random weights following Liu et al., 2023.

Q17: APS is claimed to be more stable than RAPS in Section 7. What is this stability concerning?

R17: As the prediction set is a quantification of prediction uncertainty, we use the stable APS instead of RAPS to have a low-variance uncertainty quantification to have a better contrast between different AT methods.

Q18: MART is used for the initial experiments but not the main ones. Is there a reason for that? How does the proposed method perform with MART?

R18: The reason why we did not include MART is that MART is not quite competitive in terms of CP efficiency, see Table 1.

Q19: The generalizations for which version of the proposed method to use in Section 7.2 are done for datasets. However, it seems to be dependent on the adversarial training method used.

R19: We provide our observations in Section 7.2. It makes sense that the performance depends on the AT method, as different AT has different prediction set sizes as Table 1 shows.

Q20: Prediction set size and conformal prediction inefficiency are used synonymously. However, the latter is not defined.

R20: It is defined in the caption of Fig. 1 and the reference is cited in the introduction.

Q21: What is the assumption of empirical risk minimization? What are adversarial examples? What is the PGD attack? What is adversarial training?

R21: As in every machine learning paper uses empirical risk minimization, we assume the data are i.i.d.. We cite the corresponding papers (adversarial examples, PGD attack and adversarial training) in our paper as most adversarial robustness papers do, as we all assume a basic understanding into the research topic. We have explicitly defined the adversarial training in Equation (2).

Q22: The conformal prediction framework is not explained well.

R22: We pointed the readers to Romano et al. 2020 for more details about CP in our paper.

Q23: The non-adversarially trained models are "completely broken" is incorrect.

R23: When the prediction set size is almost as large as the class numbers, the conformal prediction provides no uncertainty information, thus we think it is broken. We will revise the paper to make it clear.

Q24: It is worth mentioning in the main paper that the top-1 accuracy decreases when using the proposed method. It is also worth including details of the experimental setup in Sections 4-6 or pointing to where they are in the paper.

R24: We have revised the paper to make it clear.

We thank the reviewer for the detailed comments. Here are our responses.

Q1: The fact that a model trained without adversarial training does not produce small conformal prediction sets on adversarial examples is not surprising.

R1: Note that we never claim this effect is surprising in our paper. We show this effect to make the point that adversarial training is necessary for conformal prediction to work in an adversarial environment.

Q2: The correlation between the prediction set size with the prediction entropy and the predicted rank of the ground-truth class is not surprising. Using the 0-1 loss makes the argument in this paper not true.

R2: Our paper is an empirical study into the prediction set size of adversarially trained models and how to make the conformal prediction more efficient in such robustly trained models. We are glad to hear that our proposed method makes sense intuitively, as the reviewer admits, while our paper empirically proves that this intuitive design works quite well across several datasets and AT methods. However, it is unclear to us what is the difference between the mentioned 0-1 loss function and top-1 accuracy and how to compute the non-conformity score based on the mentioned 0-1 loss function. It is appreciated if the reviewer could elaborate on this point to make it a valid weakness.

Q3: While this paper highlights the comparisons with Gendler et al. (2021) in Section 1, the more recent work of Ghosh et al. (2023) is not.

R3: The reason why we did not compare with Ghosh et al. (2023) is that Ghosh et al. (2023) only considers the l_2 norm instead of l_inf norm and it does not consider the training stage, same as Gendler et al. (2021). Thus, similar to what the reviewer claims in Q1, we believe that the coverage of Ghosh et al. (2023) would be unsurprisingly bad at adversarially robust coverage.

Q4: Did the authors compare the proposed beta importance weighting method against that of Einbinder et al. (2022)?

R4: We are running this experiment now and will update the paper once it is ready. It is worth mentioning that Einbinder et al. is not explicitly for improving CP efficiency in adversarially trained models but regularizing the non-conformity scores so that they are uniformly distributed. We compare with the focal loss in our ablation study, which is shown to be comparable with focal loss in Einbinder et al., and find that the focal loss is worse than our method with a huge performance gap.

Q5: Was it meant to be Gendler et al. (2021) instead of Salman et al. (2020)?

R5: This is a typo, we have fixed it in the updated paper.

Q6: Theory issues.

R6: We have explained how to get the $A \leq cK^c$ in the updated version, page 17. Other theoretical issues like using general Gamma function for real-value $\alpha$ and $\beta$ will be fixed in our future work.

Q7: Theorem 1 is not empirically supported (cf. Table 3).

R7: Note that Theorem 1 does not guarantee that the top-1 test accuracy will be improved. Theorem 1 indicates that the generalization error of the beta-weighting method is comparable to that of ERM, so we can train a classifier with the beta-weighting in practice.

Q8: What is the evidence to show that the majority of data lies in the promising region (stated in Section 6.2)?

R8: Fig. 5 shows that most training samples are in the promising region. We compute the ratio of test samples in the promising region (Fig. 6) to all test samples on CIFAR100. The ratio is 44.36%.

Q9: What is done to reduce the importance weight when $r_i$=1 (since this is not part of the promising region)?

R9: We use small importance weights for those samples. See Fig. 4 and Equation 6 for the importance weight for samples outside the promising region.

Q10: The paper should explicitly mention how the predicted ranks are normalized.

R10: We will make it explicit in the next version.

Q11: Fig. 5 - The histograms look indistinguishable. Can the entire x-axis be included to highlight the difference, if there is one?

R11: Including the whole x-axis reduces the visual difference. The quantitative difference is shown in the bottom of Fig. 4.

Q12: How is the promising region determined in this illustration?

R12: According to the Beta-distribution shown in Fig. 4, we choose top 25% as the promising region for visualization purpose.

Q13: Is the ratio reported not the fraction of points in that region? The caption and the text supporting this figure are confusing.

R13: It is the ratio of p(r) instead of the ratio of samples, which is explained in the paragraph following Equation (7). We will make it clear in the next version.

We thank the review for the insightful comments. Here are our responses.

Q1: Do you still observe the improvement of using the entropy minimization loss using other conformal prediction constructions?

R1: When we use the RAPS as the conformal prediction method, the entropy minimization loss is also better than the AT baseline on Caltech256, see the following table. We think that the mentioned method Sadinle et all is not comparable in our context since their target coverage is not for adversarial examples.

Q2: How does the choice of beta distribution shape influence the results? Are the results sensitive to the beta-weight parameters?

R2: We use different beta-weight hyperparameters on Caltech256. The performance is stable within a range of $\beta$ (3.0, 4.0, 5.0), see the table below. The numbers in the parenthesis mean ($\alpha$,$\beta$).

Q3: How do different attack budgets influence the results?

R3: In addition to the $\epsilon$=8.0, we test different attack budgets $\epsilon$=4.0, 12.0, 16.0 and add the result on Caltech256 below. It shows that across the attack budgets, our method is consistently better than the AT baseline.

Q4: How do different training budgets and step-sizes influence the results?

R4: We use the default setting in this paper following existing papers like Liu et. al. We will do more experiments using different training budgets in the updated version.

Q5: The assumption on how $p_k$ distributed in Theorem 1 seems very stringent.

R5: Assumption on $p_k$ does not necessarily require that the polynomial distribution exactly hold. Instead, in practice, it can be generalized by setting a worse scenario as an upper bound.

Q6: The first term in Lemma 1 seems to be dropped in Theorem 1's proof?

R6: In generalization analysis, the dominating term is the second error term $O(\sqrt{ d_2(\mathcal P || \mathcal P / \omega) / m })$, rather than the first one $O(1/m)$, since $1/m$ is in higher order compred with $1/\sqrt{m}$. As a result, it is common to make effort to improve the second term. See the variance-based robust learning [R1] and [R2] as examples.

[R1] Namkoong, Hongseok, and John C. Duchi. "Variance-based regularization with convex objectives." Advances in neural information processing systems 30 (2017).

[R2] Maurer, Andreas, and Massimiliano Pontil. "Empirical bernstein bounds and sample variance penalization." arXiv preprint arXiv:0907.3740 (2009).

Q7: What is d(x,y) in the proof of Theorem 1?

R7: $d(x,y)$ means the differential of the variable in the calculus. Since we couple the input $x$ and output $y$ together as a joint random variable, we use $d(x,y)$ as the differential of the joint variable.

Q8: The summation in d2(P||P/w) should be r=k instead of r =1.

R8: Thanks for pointing out this typo. We will revise it.

Q9: Some brief explanations about the evaluation metrics will be helpful in Table 1

R9: We will revise the paper and add some brief introduction to the evaluation metrics.

We hope that our responses could address your concern and it is appreciated if the reviewer could raise the score accordingly.

Q1+Q2: There are quite a bit variability, both from varying beta and from comparing different methods in the original paper and the table in Q1. This makes me feel a very transparent and replicable experiement, including the parameter tuning, is really important. alpha = 1.1 is also a very interesting value, looks like a value from fine tuning. Currently, the parameter candidates are described as from pilot study, and details about the pilot study and how the parameters are determined should be provided for replicability and making sure there is no information leak.

Q3: no further question.

Q5: The faster the tail decays, the better the classifier needs to be, right? Does this mean that this Theorem only works for a very accurate classifier? (The current Theorem should also be stated using the tail bound version, it is better than the exact equality, although I still feel it is very stringent.)

Q6: Should state it precisely in the Theorem and make it rigourous in the proof (show distance < 1 instead of <= 1). (Also, I need to mention that I did not check the proof carefully. )