Conservative Prediction via Data-Driven Confidence Minimization

Thank you for your feedback. We address your comments below. Please let us know if our response addresses all of your concerns.

The theoretical guidance is not that clear: Are the two theorems practically useful and how do the authors capitalize on these theorems in the experiments?

Yes, the two theorems are practically useful. We explain their practical implications in the last part of Section 4. They show that it is important to use the right uncertainty dataset, one that includes examples close to the unknown examples that the model may encounter at test time.

In our experiments (Section 6), we capitalize on these findings by using the following uncertainty datasets: (1) unlabeled ID and OOD examples for OOD detection, and (2) misclassified validation examples for selective classification. These choices are validated by our results. We hope this clarification helps.

In particular, as per Line 2, Page 5, what kind of "appropriate threshold" is used? Did the authors use the value of the left-hand side of the inequality in (4)?

Our empirical evaluation considers different thresholds of confidence, as is standard in prior selective classification works [7,8,9,10,11]: Acc@90 measures with a threshold such that the model predicts on 90% of the data, AUC is an aggregate metric that considers all thresholds, etc.

Yes, we use a version of the LHS of (4), since all of our evaluation is based on the interplay between confidence (=MSP) and accuracy.

Practicality of the assumption that known and unknown datasets are disjoint. The unknown/misclassified points (I follow your definition of "unknown") in selective classification could be those hard points from some classes that intrinsically overlap with each other, then "known" and "unknown" are not disjoint. In OOD detection, now that we have the assumption of "disjoint", why do we still bother with the unlabeled data? Why do not generate auxiliary data around but separable from ID?

For selective classification: even though the known and unknown examples come from the same classes, they can be disjoint partitions of the overall distribution. For OOD detection: it’s not that any disjoint distribution works equally well; confidence minimization performs best when the uncertainty set is, in addition to being disjoint from the known data, close to the hard examples that the model may see at test time. Being separate from ID by itself does not make a dataset a useful uncertainty dataset.

Moreover, there are prior work that use synthetic outlier data: VOS [4], NPOS [5] and Dream-OOD [6]. We have provided comparisons to these methods on CIFAR-10 and CIFAR-100 as OOD datasets. Our method outperforms them in most cases, showing the efficacy of using the unlabeled auxiliary set.

Additionally, DCM significantly outperforms VOS [4] and NPOS [5] with larger datasets and model architectures. We ran additional OOD detection experiments using a pretrained CLIP ViT-B/16 with ImageNet-1K as ID.

I am confused about the definition of "unknown" in selective classification.

“Unknown” is a term we use to talk about the two problem settings in a unified way: (1) for OOD detection, it is the OOD data, and (2) for selective classification, it is misclassified validation set examples. For selective classification, misclassified examples are "unknown" to the model because they indicate a gap in the model's knowledge due to insufficient coverage of the training distribution.

Proposition 4.1 and 4.2 then sounds targeting OOD detection with the conclusion "DCM provably detects unknown examples". Here I presume that "detects unknown examples" solely means "detects OOD examples", excluding the "unknown" examples in the selective classification task.

By “detection”, we mean that the confidence values of a model trained with the DCM loss is high for known examples and low for unknown examples. This property is relevant to both the OOD detection and selective classification problem settings.

The authors need to well-articulate the notations before these are used. For examples, What is $P_{ID}$? It is not friendly for readers not familiar with this topic.

$P_{ID}$ represents the distribution of in-distribution, or “known”, examples. It is the distribution of the training data. We’ve revised Section 2.2 to introduce this notation.

Please add the appendix reference for the definition of $\delta$-neighborhoods in the comment after Proposition 4.1.

Thank you for the suggestion; we have added a pointer to the appendix when $\delta$-neighborhoods are first introduced in the main text.

What does $i$ in (4) stand for?

$i$ was not needed in (4) and the proof; this was something we wrote in an earlier version of the proposition but forgot to delete. Thank you for catching this!

The uniform label distribution used for "unknown" observations is ad-hoc. It sounds like you are assuming all unknown examples overlap together and hence you cannot clearly distinguish them. However, what if there are only several overlapped classes? For example, some unknown observations from class 1 overlap with class 2, but they are disjoint with other classes. In this case, it is not appropriate to give non-zero probability to generate other “pseudo” labels for these unknown examples from class 1. Moreover, if this unknown example is an OOD point that is unlike any of the "known" classes, does it still make sense to give a non-zero probability to be labeled as "known" classes?

We respectfully disagree with the claim that the uniform label distribution is ad-hoc. While we agree that it is possible for classes to have fine-grained relationships as the reviewer mentions, we are not assuming such additional knowledge. The uniform label distribution is the maximum entropy distribution among categorical distributions, and thus, regularizing towards it is a natural choice for increasing uncertainty. We note that regularizing towards the uniform distribution is a standard choice that has been successful in several prior works [1,2,3].

What is (5) used for?

(5) in the appendix states the DCM objective before we move on to the propositions and proofs. It is implicitly used in all later equations that involve the loss (6, 9, 10…)

What is $M$ in (15)?

$M$ refers to the number of classes. We realize that this was a duplicate notation; $M$ was used in an earlier version, and we did not revise it. We have replaced $M$ with $C$ in the text.

Proposition A.1: I think the dimension of $p$ is $C + 1$ when it comes to OOD detection (as the authors mentioned $p$ is the true label distribution). However, the dimension of $s$ is $C$ as the authors explicitly showed. Then is that legitimate for the expression $s - p$ and $s - \frac{1}{C}$ in (6)

The dimensionality of $p$ is always $C$, as Appendix A states throughout. Our analysis operates in a setting where $p$ assigns a $C$-way categorical distribution to all possible inputs, including OOD ones. Note that this is consistent with the evaluation of ours and many prior OOD detection papers, which uses the confidence of a $C$-way prediction to separate ID inputs from OOD ones.

Please consistently add a comma after "i.e."

Thanks for the suggestion: we’ve added commas after all instances of “i.e.” in the revised version (in red text).

[1] "Deep anomaly detection with outlier exposure." ICLR 2019

[2] "When does label smoothing help?." NeurIPS 2019

[3] "Conservative q-learning for offline reinforcement learning." NeurIPS 2020

[4] VOS: Learning What You Don't Know by Virtual Outlier Synthesis, ICLR 2022

[5] Non-parametric Outlier Synthesis, ICLR 2023

[6] Dream the Impossible: Outlier Imagination with Diffusion Models, NeurIPS 2023

[7] "Selective Classification for Deep Neural Networks." NeurIPS 2017

[8] "A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks." ICLR 2018

[9] "Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks." ICLR 2019

[10] "Deep Gamblers: Learning to Abstain with Portfolio Theory." NeurIPS 2019

[11] "Self-Adaptive Training: beyond Empirical Risk Minimization." NeurIPS 2020

[12] On the Opportunities and Risks of Foundation Models,

In Algorithm 2, since there are no "easily misclassified" examples with known labels like the validation data in Algorithm 1, why do not just train the model based on $L_{xent} + \lambda L_{conf}$? In other words, is there any necessity for the prior step to optimize $L_{xent}$?

The primary reason we use only $L_{xent}$ is computational efficiency: the fine-tuning step is much shorter than the pre-training steps, and the ratio of compute required becomes smaller when we use larger and larger datasets. When we train one epoch using $L_{xent} + \lambda L_{conf}$, the model essentially sees 2x the number of images compared to training using $L_{xent}$ only and hence takes 2x compute, since it sees equal number of images from the training and auxiliary datasets. Also this lets us take one ID pre-trained model and adapt it relatively quickly to different test distributions, avoiding the costly pre-training process for each different test distribution. We note that our method is resonant with the philosophy of using large pre-trained foundation models and fine-tuning them for individual downstream tasks [12]. Indeed, as the reviewer notes, it is possible to train the model based on $L_{xent} + \lambda L_{conf}$. We have added experiment results in Appendix K for this: in general, we see that training directly from scratch using $L_{xent} + \lambda L_{conf}$ leads to slightly lower ID accuracy but comparable performance on OOD detection tasks. This is possibly because the cross-entropy loss on ID training set and confidence minimization loss on the ID examples within the unlabeled uncertainty set work in opposite directions, making learning the ID classification task harder. Whereas if we fine-tune an ID pre-trained model, the model already has learned the ID task, and the fine-tuning step only further modifies the decision boundary, maintaining higher ID performance.

Equation (9) and the comment "resulting in a mixture between the true label distribution $p$ and the uniform distribution $U$, with mixture weight $\lambda$" after Proposition 4.2: Is this rearrangement (9) correct? Since $P_u = \alpha_{test} P_{ID} + (1 - \alpha_{test})P_{OOD}$, why is the mixture weight just $\lambda$, wouldn't there be an extra factor $\alpha_{test}$?

In our theoretical setup, we have assumed $\alpha_{test} = \frac{1}{2}$ for simplicity, which is mentioned at the start of Appendix A.1:

Let $D_u$ be an unlabeled test set where half the examples are sampled from $P_{ID}$, the other half are sampled from $P_{OOD}$.

Lemma A.3 also deals with the simplified transductive setup (which is generalized in proposition A.4), where the ID examples in $D_u$ also appear in $D_{train}$. However, the more general version can be proved similarly, assuming $\alpha_{test}$ is absorbed into the constant $\lambda$.

$\epsilon$ in Proposition 4.2 and the involved proof: The exact value of $\epsilon$ depends on the model performance or $L(\theta)$, how can we allow $\epsilon \leq \frac{1}{2N} \left(\frac{M - 1}{(1 + \lambda)M}\right)^2$ to conclude the inequality (15)?

The reviewer’s statement, “the exact value of $\epsilon$ depends on the model performance…”, is incorrect. $\epsilon$ is a free variable, and the lemma claims that there exists some $\epsilon$ such that (8) is true.

This is a standard trick in proving inequalities. We know that (14) holds for all $\epsilon > 0$. We can set $\epsilon$ to any positive value; by setting it to a value lower than $\frac{1}{2N} \left(\frac{M - 1}{(1 + \lambda)M}\right)^2$, we get equation 15.

We wanted to follow up to see if the response and revisions address your concerns. We are open to discussion if you have any additional questions or concerns, and if not, we kindly ask you to reevaluate your score. Thank you again for your reviews which helped to improve our paper!

Since the discussion period is ending soon, we wanted to check if you had any feedback based on our response. Specifically, let us know if we have addressed your concerns sufficiently. If you have any questions/suggestions based on our response, we are happy to discuss them.

Thanks for putting thoughts into our paper and your valuable insights.

“The dimension of $p$”: Your presentation “Let $P_{ID}$ be a distribution over $\mathcal{X} \times \{1, \ldots, C\} \subset \mathcal{X} \times \mathcal{Y}$ i.e., there are $C$ classes” indicates $C$ is the number of known/inlier classes. Now when it comes to OOD detection, there is an extra label for the OOD class. Then how could the label distribution $p$ have the same dimension as the number of inlier classes?

Most of the OOD detection literature does not consider a separate C+1-th class. Instead, the goal is to have the predictions be close to uniform on OOD inputs. This is a standard choice in the OOD detection literature [1,2,3] and has the advantage of not needing architectural modifications. Our work follows this convention.

[1] "Enhancing the reliability of out-of-distribution image detection in neural networks." arXiv preprint arXiv:1706.02690 (2017).

[2] "Deep anomaly detection with outlier exposure." ICLR 2019

[3] "A simple unified framework for detecting out-of-distribution samples and adversarial attacks." Advances in neural information processing systems 31 (2018).

Thank you for your thoughtful feedback. We address your comments below. Please let us know if you have any remaining questions or concerns.

It seems that the Figure 2 is not in correct order.

Thanks for catching this; we have fixed the caption and associated text in the paper related to this.

By observing Table1, we can find a performance drop on iid setting for relative simple datasets that can achieve classification accuracy more than 99%. And we can also find an enhancement in the ood setting. However on relatively hard setting like FMoW, the iid performance is enhanced by the performance, but the ood and iid+ood performance does not show a significant gap compared with other methods.

First, there is an inherent tension between ID and OOD performance. Many interventions for making models more robust (in different senses of the word) improve OOD performance at the cost of a slight drop in ID performance. We note that Camelyon17 is a bigger dataset than FMoW (455k vs 141k images) and involves a more severe drop in ID->OOD performance: the Acc@90 for MSP drops by 21.6% in Camelyon vs 7.4% in FMoW. In this challenging setting, we see substantial benefits over existing works. While other methods are also competitive on FMoW, DCM is consistently among the highest performing in all settings involving OOD data.

‘Could the authors show the result of adding a strong label smoothing or similar regularization during pretraining for more complete comparison?’

Thank you for the suggestion. We ran additional experiments comparing pre-training with label smoothing for both selective classification and OOD detection. Adding label smoothing to pre-training does not enhance performance.

Below, we compare selective classification performance of DCM with an MSP classifier on FMoW pre-trained with varying degrees of label smoothing.

(Selective classification AUC on FMoW)

We also conducted experiments comparing pre-training with label smoothing and weight decay as forms of regularization for OOD detection, detailed in Appendix K of the revised paper. We present results for label smoothing with CIFAR-100 as the ID dataset below; pre-training with label smoothing does bridge the performance gap to DCM.

(OOD detection AUROC on CIFAR-100 as ID)

We suspect that label smoothing during pre-training reduces the model’s ability to differentiate between classes, degrading selective classification and OOD detection performance.

In addition, could the author show the results of larger datasets? I am wondering whether the regularization still works as the classification task becomes harder.

Thank you for the suggestion. We ran our method with a ViT-B/16 model pre-trained on ImageNet-1K, and tested on 4 large OOD datasets, including iNaturalist and Places365. Our method outperforms all baselines on all 4 OOD datasets by a large margin, including several strong recent approaches such as VOS [4] and NPOS [5].

We also note that our evaluations for selective classification included two sizable datasets: Camelyon-17 (around 450,000 examples), and FMoW (around 150,000 examples, 63 classes).

We wanted to follow up to see if the response and revisions address your concerns. We are open to discussion if you have any additional questions or concerns, and if not, we kindly ask you to reevaluate your score. Thank you again for your reviews which helped to improve our paper!

Thank you for your thoughtful feedback. We address your comments below. Please let us know if you have any remaining questions or concerns.

For the thoretical part, what can we say when the following does not hold "(1) if the auxiliary set contains unknown examples similar to those seen at test time, confidence minimization leads to provable detection of unknown test examples". Specifically, what if the auxiliary set DOES NOT contain unknown examples similar to those seen at test time. It is important to know the thoretical property in this case.

In the absence of any assumptions about how the auxiliary dataset relates to the test distribution, we cannot make any theoretical claims about the effect of confidence minimization on test inputs. In this scenario, Proposition A.1 still informs us about the resulting model’s behavior on the auxiliary data distribution: DCM will make the model more conservative in the sense that the predictive probability will be upper-bounded by the true probability distribution. How this affects the test distribution is a matter of generalization from auxiliary->test, which we would need further assumptions to study. We note that empirically, unrelated auxiliary distributions still help in making the model more overall conservative [1,2], though not as much as more relevant auxiliary data as in ours.

‘The experiment results are shown in CIFAR…’

We thank the reviewer for their suggestions. We have run DCM on a ViT-B/16 model pre-trained on ImageNet-1K, and tested on 4 large OOD datasets, including iNaturalist and Places365. Our method outperforms all baselines on all 4 OOD datasets, including recent baselines such as VOS [3] and NPOS [4], by a large margin.

[1] Deep Anomaly Detection with Outlier Exposure, ICLR 2019

[2] Energy-based Out-of-distribution Detection, NeurIPS 2020

[3] VOS: Learning What You Don't Know by Virtual Outlier Synthesis, ICLR 2022

[4] Non-parametric Outlier Synthesis, ICLR 2023

We wanted to follow up to see if the response and revisions address your concerns. We are open to discussion if you have any additional questions or concerns, and if not, we kindly ask you to reevaluate your score. Thank you again for your reviews which helped to improve our paper!

Thank you for your thoughtful feedback. We address your comments below. Please let us know if you have any remaining questions or concerns.

‘There isn't much technical novelty on top of OE…’

While our method is similar to OE in that both methods minimize confidence, we note the following key differences between our work and OE: We analyze the role of the uncertainty set both theoretically and empirically. We extend this confidence minimization framework to selective classification, which prior works such as OE do not explore. Our method outperforms prior works that share our data assumptions (WOODS, ICML 2022 [1]; ERD, UAI 2022 [2]) while being simpler and computationally cheaper to run. Our method requires fewer algorithm-specific hyperparameters than WOODS [1] (1 for DCM, 8 for WOODS) and 5x reduction in compute compared to the ensemble-based ERD [2].

The proof seems fairly obvious; it seems to be saying that datasets are separable even when the training data are noisy. I may have missed some details, but surely this is already well-known and a standard result in learning theory. I'm worried that this proof doesn't contribute new knowledge to the field and may give rise to a false impression.

We believe that our analysis establishes a novel and useful result in the context of using auxiliary datasets to minimize confidence. We agree that the proof technique itself is straightforward; we are not claiming that the proof itself constitutes a theoretical advance. However, its application in the realm of using auxiliary datasets for confidence minimization is a novel contribution. We welcome any suggestions for relevant prior work to compare and cite. To our knowledge, our analysis in the context of conservative prediction hasn't been explored in existing literature.

‘There are numerous more recent baselines…’

Thank you for the suggestions.

We added comparisons to VOS [3], NPOS [4] and Dream-OOD [5] on CIFAR-10 and CIFAR-100 to Appendix H of our revised paper. DCM outperforms these methods by 2.5% on CIFAR-10 and 2.9% on CIFAR-100, in terms of OOD detection AUROC, averaged over 5 OOD datasets each.

Additionally, DCM significantly outperforms VOS [3] and NPOS [4] with larger datasets and model architectures. We ran additional OOD detection experiments using a pretrained CLIP ViT-B/16 with ImageNet-1K as ID and iNaturalist, Places365, SUN and Textures as OOD datasets. The complete table is in Appendix I.

[1] Training OOD Detectors in their Natural Habitats, ICML 2022

[2] Semi-supervised novelty detection using ensembles with regularized disagreement, UAI 2022

[3] VOS: Learning What You Don't Know by Virtual Outlier Synthesis, ICLR 2022

[4] Non-parametric Outlier Synthesis, ICLR 2023

[5] Dream the Impossible: Outlier Imagination with Diffusion Models, NeurIPS 2023

We wanted to follow up to see if the response and revisions address your concerns. We are open to discussion if you have any additional questions or concerns, and if not, we kindly ask you to reevaluate your score. Thank you again for your reviews which helped to improve our paper!

Dear Reviewer sVZ1,

Thank you for your response.

We agree our method is simple, but we view simplicity as a strength, especially in light of the strong empirical performance. Many impactful machine learning methods are valued for being straightforward, as they're easier to use and build upon.

The proof's practical value is to show that if some OOD examples are included in the uncertainty set, then filtering out ID examples from the uncertainty dataset (as done by OE) is unnecessary.

Finally, we also note that the ICLR 2024 call for papers lists societal considerations including "fairness, safety, privacy" as specifically relevant topics. Considering the critical reliability and safety risks of machine learning model errors, we feel that our work showing substantial mitigation of this problem is very relevant and will be of great interest to the ICLR community.

We appreciate your feedback and hope this clarifies our perspective.