A Data-Driven Measure of Relative Uncertainty for Misclassification Detection

We thank reviewer tQ2Y for their helpful feedback. In the following, we address each of their concerns individually.

Weaknesses:

• We thank the reviewer for bringing this to our attention. In order to address this point, we have added a dedicated section on conformal prediction in the related work section, including a short explanation in section A.3.7 in the appendix, and most importantly a new set of experiments in section A.6 (Performance of conformal prediction). We considered 2 modesl, 2 datasets and 3 possible training techniques for a total of 12 additional numerical results. The content related to these changes is reported in blue in the revised version of the paper.

Questions:

• The weights within the $D$ matrix can be interpreted as a measure of dissimilarity between the soft-prediction values for all pairs of classes $(y,y’)$ (higher values indicate higher uncertainty). However, it's crucial to note that the notion of similarity or dissimilarity is contextual and relative to the positive and negative examples within the validation data. Therefore, it is "relative" in the sense that it describes the observer in relation to this specific dataset.
• For every y, the function $p_{X|Y}$ is a probability density function, while $p_Y$ is a probability mass function. In order to avoid complicated expressions when referring to $p_{X,Y}$, we slightly abuse terminology and also call it a probability density function.

We believe that all the concerns and questions were adequately addressed, and kindly ask the reviewer to consider raising their scores accordingly. We are happy to provide any further clarification where needed.

We thank reviewer YaSF for their helpful feedback. In the following, we address each of their concerns individually.

Weaknesses:

• We thank the reviewer for the observations. We moved the related works section up and placed it after the introduction. We also structured it better, by highlighting important related domains, adding additional comments and references pointed out in the review process.

  The measure investigated in Rao (1982) is introduced in Eq. (3). We improved the phrasing, to highlight this fact.

• We invested significant efforts in standardizing our terminology. We refer the reviewer to the updated manuscript.

• We thank the reviewer for pointing out these concerns. We made the necessary changes, updating the terminology and refer the reviewer to the current version of the manuscript. The variable $\lambda$ is a real number in $[0,1]$. It is introduced in Definition 1, and the related eq. 4.

• We thank the reviewer for their suggestions. We added an example of positive and negative samples right after these terms are introduced and rephrased some text to eliminate any possible ambiguity between an iterative learning algorithm and the optimization of the objective function with Lagrangian multipliers.

Questions:

• We refer the reviewer to the newly introduced Algorithm 1 in the Appendix A.2 where we detailed how to prepare negative and positive data for the computation of the $D^*$ matrix.

• We investigate two sources of mismatch:

  1. Datasets with different label domains, which translates to experiments of CIFAR-10 vs CIFAR-100 and vice versa or CIFAR-10 vs SVHN for instance. \
  2. Perturbation of the feature space domain generated using popular distortion filters, which translates to running experiments on CIFAR-C datasets [A].

• We thank the reviewer for pointing out this typo. Actually, we meant AUROC, not just AUC. The evaluation metrics are indeed the same as in Granese et al. 2021.

We believe that all the concerns and questions were adequately addressed, and kindly ask the reviewer to consider raising their scores accordingly. We are happy to provide any further clarification where needed.

References

[A] Hendrycks, Dan, and Thomas Dietterich. "Benchmarking Neural Network Robustness to Common Corruptions and Perturbations." ICLR 2019. /abs/1903.12261.

We thank reviewer Pj5j for their helpful feedback. In the following, we address each of their concerns individually.

Weaknesses:

1. We added discussion on the baselines and objective functions. Please see Section A.3 in the appendix.
2. We thank the reviewer for pointing out the references [1-4]. They were added and properly cited in the discussion on Related Works.
3. We employ relative uncertainty to measure the overall uncertainty of a model, encompassing a mixture of both aleatoric and epistemic uncertainty components. On a principled basis, our method, like any approach solely utilizing the model output and validation samples, does not anticipate capturing specifically either aleatoric or epistemic uncertainty. This limitation arises from inherent constraints within the model architecture. We have added a clarification in the revised version of the paper.

Questions:

1. While Rel-U does not appear sensitive to changes in λ as shown in Figure 3, a reasonable choice would be $\lambda = N_+ / (N_+ + N_-)$ in order to balance the effect of positive and negative examples. We added a discussion in Appendix A.5.
2. We thank the reviewer for bringing this useful comment to our attention. We meant that strictly speaking, eq. 4 only needs positive and negative instances, which are task-specific. Of course in our case, the labels are implicitly needed to produce negative and positive instances. Thus, we removed the sentence from the paper.
3. In order to obtain the matrix D we utilize the validation set. Figure 1 illustrates that D can approximate relatively well the confusion matrix for the same samples that were used for computing it.
4. This might be addressed in part by tuning the $\lambda$ parameters as mentioned above. However, if the imbalance is too severe (e.g., near perfect classification), it is questionable whether misclassification detection is feasible (or even suitable) in such a setting. This would certainly require further analysis and could be a possible direction for future research.

We believe that all the concerns and questions were adequately addressed, and we are happy to provide any further clarification where needed.

References:

[1] Kotelevskii N. et al. Nonparametric Uncertainty Quantification for Single Deterministic Neural Network //Advances in Neural Information Processing Systems.

[2] Van Amersfoort J. et al. Uncertainty estimation using a single deep deterministic neural network //International conference on machine learning.

[3] Liu J. et al. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness //Advances in Neural Information Processing Systems.

[4] Mukhoti J. et al. Deep deterministic uncertainty: A simple baseline //arXiv preprint arXiv:2102.11582.

We thank reviewer 9edT for their helpful feedback. In the following, we address each of their concerns individually.

Weaknesses:

• We added an overview of our strategy in Algorithm 1 in the Appendix Section A.2 and improved the discussion on how the metric function is learned.
• We conducted ablation studies on all relevant parameters: $T$, $\epsilon$, and $\lambda$ (refer to Section 4.1). It's crucial to note that T is intrinsic to the network architecture and, therefore, must not be considered a hyperparameter for Rel-U. Additionally, the introduction of additive noise $\epsilon$ serves the purpose of ensuring a fair comparison with Doctor/ODIN, where noise was utilized to enhance detection performance. Nevertheless, as indicated by the results in the ablation study illustrated in Figure 3, $\epsilon=0$ seems to be close to optimal most of the time, thereby positioning Rel-U as an effective black-box algorithm. Furthermore, Rel-U exhibits a considerable degree of insensitivity to various values of $\lambda$, as evident from Figure 3. This suggests that a potential selection for λ could have been $\lambda=N_+/(N_+ + N_-)$, aiming to balance the ratio between positive ($N_+$) and negative examples ($N_-$). In such a scenario, there are no hyper-parameters at all. This clarification has been incorporated into the revised version of the paper, cf. Appendix A.5.

Questions:

• We added an overview and algorithm block in the appendix Section A.2 and expanded discussion on assumptions and requirements.

• Our metric doesn't directly incorporate uncertainty statistics obtained from the model, as relying on such statistics may create a misleading sense of the model's genuine, as opposed to self-assured, confidence in detecting its classification errors. Unlike metrics that depend on transformations applied to latent code, our approach restricts observation to black-box access of the model, where only the final layer statistics can be observed (with $\epsilon$ set to 0). Instead of adopting a strategy that involves learning a metric for uncertainty through an additional model (as seen in [1] for OOD), which might raise concerns about the uncertainty of this secondary model, our metric focuses on learning to recognize the signature of uncertainty for each pair of classes using the soft-prediction of the existent model. In essence, our observer '$d$', given a sufficient number of observed positive and negative samples, can effectively weigh the uncertainty metric based on the model's soft-probabilities in a manner specific to the class pair in question.

• Experiments with a MLP trained on the output logits of the validation set to predict mistakes were performed. Please see Table 3 in the Appendix for details. We added additional details on the hyper-parameters and the exact architecture used in the new Section A.3 in the Appendix.

• We compared to MCDropout [A] and Deep Ensembles [B], which are estimates of Bayesian Neural Networks, in Table 3 in the Appendix. We added further details on these baselines in Section A.3 in the appendix.

  In contrast to [2], we are given a pre-trained model and we want to gauge the uncertainty w.r.t. pairs of labels. This is slightly different from measuring the uncertainty on a class of models or on a given label for a family of models. It is worth mentioning that our focus is to develop a methodology that can be applied to any classifier that outputs soft-predictions, which includes, but is not limited to (Bayesian) neural networks.

We believe that all the concerns and questions were adequately addressed, and kindly ask the reviewer to consider raising their scores accordingly. We are happy to provide any further clarification where needed.

References:

[1] Dinari, O. & Freifeld, O.. (2022). Variational- and metric-based deep latent space for out-of-distribution detection. Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research 180:569-578 Available from https://proceedings.mlr.press/v180/dinari22a.html.

[2] Vadera, M., Li, J., Cobb, A., Jalaian, B., Abdelzaher, T., & Marlin, B. (2022). URSABench: A system for comprehensive benchmarking of Bayesian deep neural network models and inference methods. Proceedings of Machine Learning and Systems, 4, 217-237.

[A] Gal, Yarin, and Zoubin Ghahramani. "Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning." ICML 2016. /abs/1506.02142.

[B] Lakshminarayanan, Balaji, Alexander Pritzel, and Charles Blundell. "Simple and Scalable Predictive Uncertainty Estimation Using Deep Ensembles." NIPS 2017. /abs/1612.01474.

Thank you for carefully reviewing our answers and the revised manuscript. Next, we address the follow-up questions.

• To appropriately contextualize our contribution in the existing literature, initially, we adhered to the glossary provided by [1], which defines black-box methods as those relying solely on the final layer statistics. To address this valid concern, we revised section A.5 by eliminating the term "black-box" and rephrasing the corresponding sentence.
• Broadly, our paper aligns with the extensive literature that leverages the model's statistics to articulate the uncertainty of the model's decision (e.g., "logit/feature transformation on held-out set"). The novelty introduced by our method is fundamentally characterized by two main aspects:

  • As pointed out by the reviewer, our approach suggests a closed-form solution, that distinguishes it from [2], where an iterative algorithm is necessary to train a generative model (VAE) for estimating the likelihood of a sample. Notably, our method alleviates the need for practitioners to specify hyperparameters such as learning rate, number of layers, number of nodes, and number of epochs.

  • Instead of directly calculating an uncertainty score based on the model's statistics (see [1, 3]), our data-driven approach optimizes a positive definite matrix D for the purpose of identifying misclassifications. This optimization implicitly defines an inner product to gauge similarities between vectors of softmax outputs.

Summary of discussion and added comment in the paper

Following the suggestion of the reviewer, we summarized the content of our discussion and added to the manuscript (in blue) as follows: We added a reference to the algorithm that clarifies the description of the computation of D (see Section 4.2 and Section A.2). To summarize our discussion on the Bayesian NN (BNN), we added relative literature to the related works section, and we introduced the results for MC dropout, deep ensemble, and MLP in section 5.1, also referencing our extended results in Appendix A.6. We clarified the hyperparameter tuning procedure, and provided further details on the ablation studies in Appendix A.5.

References:

[1] Granese F., Romanelli M., Gorla D., Palamidessi C., and Piantanida P. “DOCTOR: A simple method for detecting misclassification errors”. NeurIPS, 2021
[2] Dinari, O. & Freifeld, O. (2022). Variational- and metric-based deep latent space for out-of-distribution detection. Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research 180:569-578
[3] Kimin Lee, Kibok Lee, Honglak Lee, Jinwoo Shin. A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks. NIPS, 2018.