Deep Discriminative to Kernel Density Graph for In- and Out-of-distribution Calibrated Inference

Thank you for your thoughtful comments. We are pleased that you recognize the efficacy of our proposed approach in providing an integrated solution for both ID and OOD calibration problems in traditional deep learning models and random forests. We believe we have addressed all your concerns in our responses below. If you find these responses satisfactory, we would greatly appreciate it if you could consider updating your score.

• The paper evaluates calibration using Maximum Calibration Error (MCE) for ID data, but does not justify the use of MCE over Expected Calibration Error (ECE) or Adaptive Calibration Error (ACE)...

Thanks for the reference and feedback. We noticed we actually evaluated ECE and called it MCE (please see kdg_code/kdg/utils.py line 36 in the provided codes and definition of ECE in Section 2.1 in [1]). We apologize for the mistake and have corrected it throughout the draft. Additionally, we have also reported ACE in the attached pdf. ECE and ACE provide nearly similar results when the number of classes is low. We noticed ACE gives better estimation of calibration compared to that of ECE when the number of classes is large (for example, cifar100 new results) [1]. This is due to the fact that ECE considers the calibration error only for the predicted class, whereas ACE considers all the classes. Our 45 datasets from OpenML CC-18 suites have (<=) 10 classes [4]. However, we will add the ACE curves for the Openml datasets in the appendix which is similar to the current ECE curves (because of having <=10 classes). We will add the above discussion to the paper and cite the provided reference.

• Additionally, the definition and justification for OCE (Out-of-distribution Calibration Error) would benefit from a similar comparison with ACE.

OCE measures OOD calibration and assumes that true class conditional priors for the datasets are known. On the contrary, ACE is used for measuring ID calibration and is a surrogate measure used when true posteriors are not known. For example, in Figure 1 where we know the true distribution, we have used Hellinger distance from the true posteriors instead of ACE. Please see the global response for the corrected definition of OCE. We will add the above clarification in the camera ready version.

• To emphasize the effectiveness of the proposed methods, a comparison of execution times would be beneficial, especially since practical applications like web Click-Through Rate (CTR) estimation place significant importance on runtime.

Please see the global response.

• The paper should clarify what the noise in Table 1 represents.

We sample noise samples of size $32 \times 32 \times 3$ according to a Uniform distribution with pixel values within range [0,1]. We will add this text in the draft.

• It would also be advantageous to include experiments on larger and more varied datasets…

We have added additional vision experiments using cifar100 (100 classes) and SVHN (10 classes and bigger training size) as ID datasets as suggested by reviewer fbQU. We emphasize that cifar10, cifar100 and SVHN are some of the hardest ID and OOD pairs according to various papers [2, 3], and hence they are adopted as the benchmarking datasets by many of the papers in the literature. Note that doing experiments with extremely large datasets like imagenet (14 million images) is computationally and storage-wise expensive using our current implementation within the rebuttal deadline. We will pursue extremely large datasets in future. Many relevant papers on OOD calibration use only small- and mid-sized datasets [5, 6, 7]. We acknowledge this limitation in the paper.

• An evaluation of the methods' performance when combined with in-training calibration methods, which are commonly used alongside post-hoc calibration methods.

We have used ACET as an in-training approach followed by our approach to do the proposed experiment using CIFAR100 as ID data. Note that ACET improves OOD calibration leaving less room for improvement for KDN, however KDN improves the ID calibration for ACET. ACET+KDN has same ECE and ACE as KDN and CIFAR-10, SVHN, Noise OCE as 0.12, 0.04, 0.04 respectively. In the end, KDN has nearly similar performance with or without in-training approaches. Moreover, ACET adds a significant computational burden to the whole process. The authors in [8] observed a similar phenomenon. We think this experiment provides important insights about our approach and we will add it to the appendix.

• Does the formulation of OCE assume a lack of usable features from the in-distribution domain? What practical and theoretical conditions are required for this assumption?

The formulation of OCE does not assume a lack of usable features from the ID domain. Our goal is defined in Eq. 1 and OCE measures the OOD calibration error, i.e., difference from the maximum of the class conditional priors in the OOD region according to Eq. 1. For calculating OCE, we need to know the true priors.

• This paper mentions computational complexity and limitations in practical applications, but lacks detailed experimental results to support these claims. Including such data would provide valuable insights for future research and implementation.

Please see the global response.

[1] https://arxiv.org/abs/1904.01685

[2] Nalisnick, Eric, et al. "Do deep generative models know what they don't know?."

[3] Fort, Stanislav. "Exploring the limits of out-of-distribution detection."

[4] Bischl, Bernd, et al. "Openml benchmarking suites." arXiv preprint arXiv:1708.03731 (2017).

[5] Gardner, Josh. "Benchmarking distribution shift in tabular data with tableshift."

[6] Borisov, Vadim, et al. "Deep neural networks and tabular data: A survey."

[7] Ulmer, Dennis. "Trust issues: Uncertainty estimation does not enable reliable ood detection on medical tabular data."

[8] Wang, Deng-Bao. "Rethinking calibration of deep neural networks: Do not be afraid of overconfidence."

We appreciate the reviewer's thoughtful comments. We are glad to see that you recognize the effectiveness of our approach in balancing ID and OOD calibration. We believe we have addressed all your concerns in our responses below. If you find these satisfactory, we would be very grateful if you could consider updating your score.

• The major concern is insufficient discussion over the research context of the paper, which renders it hard to precisely evaluate the contribution. The related work section is short. Section 2 shows that "OOD detection" is the closest area to this paper, but this keyword is totally absent from the introduction, where the research area is named "OOD confidence calibration". What is the relation between OOD detection and OOD confidence calibration?

We apologize for the confusion. Our work addresses both in- and out-of-distribution (ID and OOD) calibration. While traditional ID calibration methods like Isotonic and Sigmoid regression focus on achieving calibrated inference within the ID region, they do not address OOD calibration. Conversely, there is another group of literature which is primarily concerned about detecting OOD points. These approaches such as ACET, OE, and ODIN mainly focus on OOD detection rather than OOD calibration. Calibration is harder than detection, akin to how regression is harder than classification. To see that calibration is harder than detection, consider the fact that a well-calibrated model can perform detection, but a model capable of detecting OOD points may not be calibrated. OOD detection works as long as there are two distinguishable score sets for ID and OOD points, whereas calibration aims at estimating the true predictive uncertainty of these points. To our knowledge, only Meinke et al. [1] explicitly addressed OOD calibration (Section 3 Theorem 1 in their paper), but they do not consider ID calibration. Our work treats calibration problems as a continuum between the ID and OOD regions rather than addressing them separately. We will revise Section 2 to reflect this discussion.

[1] Meinke, Alexander, Julian Bitterwolf, and Matthias Hein. "Provably Robust Detection of Out-of-distribution Data (almost) for free." arXiv preprint arXiv:2106.04260 (2021).

• In Eq 3, how to ensure the non-negativity of the class-conditional density? The model of affine function might take negative values.

The class-conditional density is non-negative because of ReLU activation (please see Section 2 in [1], Figure 1,2,3,4 in [2] for the details). We will clarify in the camera ready version.

[1] Hein, Matthias, Maksym Andriushchenko, and Julian Bitterwolf. "Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.

[2] Xu, Haoyin, et al. "When are deep networks really better than decision forests at small sample sizes, and how?." arXiv preprint arXiv:2108.13637 (2021).

• The model of class-conditional density with gaussian kernels, referring to Eq 5, resembles Kernel Density Estimation. Could the author please discuss the relation between KDE and the proposed method?

This is an excellent point! We agree there are similarities between KDE and the proposed method. However, there is an indicator function in Eq. 5 which is implemented using the geodesic distance proposed in the draft (which is absent in KDE), which makes KDN, KDF scale better with higher dimensions. Moreover, the center and bandwidth of the Gaussians are estimated in a data-driven manner using the representation learned by the parent discriminative model. We will add the above clarification after Eq 5.

• In Fig 1, KDF shows a weaker advantage for low dimensional settings. Could the author please explain why?

In Fig 1, each Random Forest is an ensemble of 500 decision trees. An ensemble of uncalibrated learners has improved calibration over the individual uncalibrated learner [1]. This phenomenon leaves less room for improvement for KDF in low dimensional settings. On the contrary, the deep-net models are standalone learners with poor calibration which can be improved a lot by KDN.

[1] Stickland, Asa Cooper, and Iain Murray. "Diverse ensembles improve calibration." arXiv preprint arXiv:2007.04206 (2020).

• In Fig 2, why are OOD approaches absent? Fig 2 demonstrates the effectiveness of KDF compared to ID approaches in terms of OOD calibration. The readers could be more interested in the performance of OOD approaches.

ACET, ODIN, OE are tailor-made for vision problems, therefore we can not run them on tabular data using the author provided codes. . To the best of our knowledge, the tabular OOD method [1] that we found does OOD detection, not calibration. As it does not yield any posterior, we could not benchmark with the above method. We will add the above explanations to Section 5.1.2.

[1] Ren, Jie, et al. "Likelihood ratios for out-of-distribution detection." Advances in neural information processing systems 32 (2019).

• In Table 1, it seems OOD approaches including ACET and ODIN can't even beat the parent model under OOD settings. Could the author please explain why?

ACET and ODIN highly depend on the model architecture and the nature of the ID and OOD testsets. Moreover, they also depend on the OOD set used to train them. We used the OOD set used by the authors of the above algorithms. All these factors contribute to their inconsistent performance across datasets and model architecture. The authors of [1] found a qualitatively similar result.

[1] “Tajwar, Fahim, et al. "No true state-of-the-art? ood detection methods are inconsistent across datasets." arXiv preprint arXiv:2109.05554 (2021).”

• “This experimental setting essentially reduces the task to a binary one: whether to output normal confidence for an ID sample or just a prior for an OOD sample. ”

The reviewer is correct, in the experimental setting, as one moves further away from the ID setting, an OOD calibrated classifier will output the prior. However, close to the ID setting, the OOD calibrated classifier will output a probability that the sample is in any given class. This is in contrast to an OOD detector, which, regardless of how close or far the data are from the ID data, if is OOD, it always effectively outputs the prior.

• “A binary OOD detector can also immediately output a confidence based on the prior if a sample is identified as OOD.”

Yes, a binary OOD detector can output a confidence based on the prior. However, without calibration, there is no reason to expect that confidence to be….calibrated. This is precisely why the OOD calibration is more difficult, and more informative, than pure OOD detection.

• “Therefore, the proposed method is essentially an OOD detector, which implies the need to reconsider both the theoretical and empirical results within the broader literature of OOD detection.”

We would say that our method subsumes OOD detection, because it also includes ID calibration, and OOD calibration. To our knowledge, there are no other papers demonstrating any algorithm with all these properties.

We thank the reviewer for the intuitive comments. We are glad that the reviewer recognized the intuition behind our approach . We believe we have addressed all your concerns in the response below. If you think these responses are satisfactory, we would be very grateful if you can update your score.

• The main weakness I find is about the computational time of the method. The number of polytopes scales exponentially with the number of neurons, so I am concerned with the applicability of the method to large (or even medium-scale) neural networks. What is the computational cost of the method for the considered benchmarks, in terms of runtime?

Please see the global response.

• The toy simulations are unnecessarily tedious to grasp and take up a lot of space. I do not say that they are complex, but they hinder the reading flow and do not bring much to the presentation. I would advise putting some of them in the appendix to leave more space for other explanations. Indeed, Section 5 is difficult to read (many "chunk" paragraphs with mathematical notations) and would benefit from more structured writing and more flow.

This is an excellent suggestion! We will put rows four of the rows of Figure 1 to appendix. We will reorganize the chunk paragraphs and make Section 5 more concise in the camera ready version, and put the formal equations in the appendix as necessary.

• Section 4.4 w_{rs} How is estimated in that case?

We apologize for missing it. We have rewritten line 159-160 as “We estimate $w_{rs}$ by exponentiating the above kernel using Equation 15.“

• Table 1: what is the Parent approach exactly compared to KDN and KDF? The definition of Section 3.3 does not seem to describe a whole model but only the parent idea that is then declined with KDN and KDF.

Sorry for the confusion. In Table 1, by “Parent approach” we mean the original vision transformer [1] that was trained on CIFAR10. We will update the table accordingly.

[1] https://pytorch.org/vision/main/models/generated/torchvision.models.vit_b_16.html

• What would be the AUROC, FPR, i.e. metrics commonly used in OOD detection?

We have added AUROC and FPR in the updated table in the attached pdf. KDN has nearly similar AUROC and FPR to those of OOD detection approaches. Note that these scores are used for OOD detection. However, we are addressing both ID and OOD calibration (Eq. 1 in our paper). OOD detection works as long as there are two distinguishable score sets for ID and OOD points, whereas calibration aims at estimating the true predictive uncertainty of these points. That being said, a well-calibrated model can perform detection, but a model capable of detecting OOD points may not be calibrated. We will add the above discussion in Section 2.

• Figure 1. What is "dimensions" in x-axis?

We will replace it with “Number of dimension” which indicates the number of increasing dimensions from the Trunk simulation.

• l. 192 I do not get how OCE measures an error since it only includes estimated quantities, and I do not see the estimated confidence score

Thanks for this intuitive observation and catching the mistake. Please see the global response.

• In addition, are x_i OOD samples in that case? If yes it should be made explicit.

We have rewritten line 192 as: “Given n OOD samples {x_i}_{i=1}^n, we define OOD calibration error (OCE) to measure OOD performance for the benchmark datasets as:”

• l. 213 I disagree that normalizing ensures that distances up to 1 are ID and above are OOD. There can be "holes" in the distribution (as the circle dataset), or modes. What is the authors' opinion about that?

This is an excellent catch and we agree! We have rewritten line 213 as: “Therefore, ID samples are confined within distance 1. “