Dear reviewer rex2,

We thank you for the thorough review and your encouraging comments.

As highlighted in the review, our paper provides a comprehensive empirical evaluation across a wide range of architectures (CNNs, ResNets, ViTs, Swin) and post-hoc OOD detection methods. We want to highlight that the major contribution of our study lies in the empirical observations of a double descent phenomenon in post-hoc OOD detection. To the best of our knowledge, the double descent phenomenon has not been observed before in the OOD context. Our experiments depict how model complexity affects post-hoc OOD detection methods and exhibit, for the first time, a double descent phenomenon in OOD detection. Furthermore, we observe that post-hoc OOD detection methods may perform less effectively on overparameterized models compared to underparameterized ones. While the paper also includes a theoretical part, this part provides insights rather than an exhaustive analysis. In particular, this part depicts a first attempt to mathematically study the double descent phenomenon in post-hoc OOD detection methods based on confidence through a simplified model.

A.1 & A.2 Least-Squares Binary Classifier Definition (Q1 & Q2):

Our theoretical insights show that the expected OOD risk for least-squares binary classifiers on Gaussian covariate models under some assumptions exhibits an infinite peak when the number of parameters is close to the number of data, which we associate with the double descent phenomenon. To establish this result, we derive a lower bound $c(n,p,\sigma)$ for the expected OOD risk, and we show that $c(n,p,\sigma)=\infty$ when $n-1 \leq p \leq n+1$. $c(n, p, \sigma)$ describes $E_X[|| \hat w - w^\text{OOD}||^2]$.

The confusion surrounding questions Q1 and Q2 arises from a typo in the definition of the least-squares binary classifier in equation 4. As correctly noted in the review, the least-squares binary classifier minimizes the logit-based expected risk rather than the probability-based expected risk. Therefore, in the definition of $\hat w_\mathcal{T}$ in equation 4, $y$ should be replaced with $u=X w^\text{OOD}$, which is the vector of logits of the optimal classifier. Note that we considered the logit-based definition of $\hat w_\mathcal{T}$ in the proofs, e.g., equation 12 (l.139). While the least-binary classifier minimizes the logits-based expected risk, Theorem 1 focuses on the probability-based expected OOD risk. The objective of this theorem is to show that probability-based post-hoc OOD detection methods may exhibit a double descent behavior. Without loss of generality, note also that the Gaussian noise could be added to the logit vector $u$. Accordingly, we will update definitions 4 and 5. We sincerely thank you for bringing this element to our attention. Corrections will be made promptly to eliminate further misinterpretation.

We emphasize that this work focuses primarily on empirical investigation and practical analysis rather than theoretical derivation. Its core contribution lies in empirically analyzing the relationship between model complexity and post-hoc OOD detection methods. The major contribution stems from the empirical observation that post-hoc OOD detection methods also exhibit a double descent phenomenon. While the paper includes theoretical elements, these serve as insights with an illustrative example rather than exhaustive analysis. They depict a first attempt to mathematically study the double descent phenomenon in post-hoc OOD detection method through a simplified model. As mentioned in the review, although the theoretical framework is constrained to a particular model, the derivation of Theorem 1 is a non-trivial extension of Theorem 1 in [1]. We hope that future works could extend this analysis to more complex scenarios, such as linear models with random features, gradient descent-based algorithms, or sub-gaussian distributions. We believe these theoretical extensions fall outside our current scope, and we reserve these explorations for future research.

A.3 Correlations between ID Accuracy and OOD Detection (Q3):

We would like to clarify that we did not claim a direct link between NC and OOD quality. Rather, we explained lines 318-321 that we use NC as a tool to describe the structure of the final-layer features. This representation helped us gain a better understanding of the quality of OOD detection, as we showed empirically, lines 327-333. Also, while OOD detection performance and ID accuracy may be positively correlated, prior works show that this correlation is not universally consistent [1, 2]. For instance, Kim et al. [1] showed that ResNet-50 v1 outperforms its v2 counterpart on OOD benchmarks despite having lower in-distribution accuracy (76% vs. 80%). [2] and [3] observed a similar trend. These results align with our empirical findings. For example, ViT demonstrates lower ID accuracy in overparameterized regimes but achieves superior OOD detection performance using NECO and ViM methods in Figure 2. Furthermore, variations between ID accuracy and OOD detection performance may not show a consistent correlation. For instance, feature-based OOD methods do not exhibit a double descent phenomenon consistently, while the ID accuracy describes a double descent curve. To better illustrate this, the Table below compares OOD detection performance (for logit- and a hybrid-based methods) vs accuracy/NC ratios like in Table 1. The accuracy ratio is defined as the local maximum accuracy in the over-parameterized regime divided by the local maximum in the under-parameterized regime. It follows that value greater than 1 indicate improved accuracy with over-parametrization. We highlight in bold the ratio if its value correlates with the OOD detection performance improvement or degradation. As illustrated, the accuracy ratio fails to provide information on OOD detection performance in the under- and over-parameterized regimes.

A.4 Typos

Regarding other feedback:

• We will add the x-axis label in Figure 2.
• We will change the minima to maxima in Table 1.
• $z(x)$ depicts predictions rescaled within the range [-1, 1]. We believe this formulation helps in better understanding the expected OOD risk definition, particularly concerning Remark 4.1 and Remark 4.2.

We would like to thank you again for your thoughtful review.

References

[1] Kim, C., Singh, S., & Kolter, J. Z. Investigation of Out-of-Distribution Detection Across Various Models and Training Methodologies. Neural Networks, Volume 172, 2024, Pages 1–12.

[2] Mueller, J., Zhu, Y., & Liang, P. Mahalanobis++: Improving OOD Detection via Feature Normalization. Proceedings of the 42nd International Conference on Machine Learning (ICML), 2025.

[3] Yang, Jingkang, et al. "Openood: Benchmarking generalized out-of-distribution detection." Advances in Neural Information Processing Systems 35 (2022): 32598-32611.

[4] Ammar, M. B., Belkhir, N., Popescu, S., Manzanera, A., and Franchi, G. NECO: NEural Collapse Based Out-of-distribution detection. In The Twelfth International Conference on Learning Representations, 2024.

[5] Ming, Y., Sun, Y., Dia, O., and Li, Y. How to Exploit Hyperspherical Embeddings for Out-of-Distribution Detection. International Conference on Learning Representations,2023.

[6] Haas, J., Yolland, W., and Rabus, B. T. Linking Neural Collapse and L2 Normalization with Improved Out-of-Distribution Detection in Deep Neural Networks. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.

Dear reviewer 7SLJ,

We would like to thank you for your interest and for appreciating both the novelty and experimental findings of our study.

A.1 Expected OOD Risk (Q1):

We acknowledge that out-of-distribution (OOD) samples arise in many forms, including those that are far from the in-distribution (ID) support or decision boundary. Our definition of the expected OOD risk draws inspiration from the OpenOOD benchmark [1], which categorizes semantic shifts by assigning a binary score (1 for ID data, 0 for OOD data) through an unsupervised approach. Drawing inspiration from both this framework and traditional expected risk, we propose the expected OOD risk to evaluate whether confidence predictions from a binary classifier $f(\cdot)$ can be used to define an OOD scoring function. In particular, the expected OOD risk measures how far the predictions are from an idealized confidence function $f^\text{OOD}(\cdot)$, which satisfies: $f^{\text{OOD}} (x) \approx 0$ or $1$ and is aligned with the probabilities of a perfect classifier when $x$ is ID; $f^{\text{OOD}}(x) \approx 0.5$ when $x$ is OOD and depicts maximal uncertainty.

The greater the deviation from this ideal classifier, the higher the expected OOD risk. We emphasize that the expected OOD risk is not a direct OOD detection metric but evaluates whether a classifier's confidence predictions can be effectively used in post-hoc OOD detection methods, such as Maximum Softmax Probability (MSP) or MaxLogit approaches [2].

A.2 NC1 metric in Under- Over-parameterized Regimes (Q2):

We agree that reporting the NC1 metrics alongside their corresponding ratios would enhance clarity. While we plan to incorporate these values into the final version, we can already share their values in the table below. Note that we have also added values for the depth-wise experiments as recommended by Reviewer ekWE.

A.3 Overparameterization across Architectures (Q3):

Transformer architectures may indeed have more parameters than conventional neural networks due to their attention mechanisms. However, increasing attention parameters doesn't automatically lead to superior performance, and the effects of overparameterization in transformers remain poorly understood. For instance, recent studies have shown that these architectures may suffer from attention rank collapse, i.e., a phenomenon where all tokens converge to a single representation with wider or deeper architectures [3-6]. Other works observed that the norm of the residual stream grows exponentially along the layers over the forward pass of multiple transformer blocks [7-9]. [10] shows that such behavior may hurt post-hoc OOD detection.

In our study, we have decided to increase the model complexity of transformers by increasing the width of the MLP in the transformer block, following a similar strategy to that described in [11]. Note also that in our experiments, the number of parameters is not consistent and varies significantly across architectures. Additionally, the training procedure differs for each architecture, which may also impact OOD detection [12].

References

[1] Yang, Jingkang, et al. "Openood: Benchmarking generalized out-of-distribution detection." Advances in Neural Information Processing Systems 35 (2022): 32598-32611.

[2] Hendrycks, Dan, and Kevin Gimpel. "A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks." International Conference on Learning Representations. 2017.

[3] Dong, Yihe, Jean-Baptiste Cordonnier, and Andreas Loukas. "Attention is not all you need: Pure attention loses rank doubly exponentially with depth." International conference on machine learning. PMLR, 2021.

[4] Noci, Lorenzo, et al. "Signal propagation in transformers: Theoretical perspectives and the role of rank collapse." Advances in Neural Information Processing Systems 35 (2022): 27198-27211.

[5] Saada, Thiziri Nait, Alireza Naderi, and Jared Tanner. "Mind the Gap: a Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers." arXiv preprint arXiv:2410.07799 (2024).

[6] Geshkovski, Borjan, et al. "A mathematical perspective on transformers." Bulletin of the American Mathematical Society 62.3 (2025): 427-479.

[7] Wang, Haoqi, Tong Zhang, and Mathieu Salzmann. "Demystifying Singular Defects in Large Language Models." Forty-second International Conference on Machine Learning.

[8] Heimersheim, Stefan, and Alex Turner. "Residual stream norms grow exponentially over the forward pass." AI Alignment Forum. 2023.

[9] Merrill, William, et al. "Effects of Parameter Norm Growth During Transformer Training: Inductive Bias from Gradient Descent." Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.

[10] Mueller, J., Zhu, Y., & Liang, P. Mahalanobis++: Improving OOD Detection via Feature Normalization. Proceedings of the 42nd International Conference on Machine Learning (ICML), 2025.

[11] Nakkiran, Preetum, et al. "Deep double descent: Where bigger models and more data hurt." Journal of Statistical Mechanics: Theory and Experiment 2021.12 (2021): 124003.

[12] Kim, C., Singh, S., & Kolter, J. Z. Investigation of Out-of-Distribution Detection Across Various Models and Training Methodologies. Neural Networks, Volume 172, 2024, Pages 1–12.

Thank you for the reply and for appreciating the rebuttal.

We believe there may have been a misunderstanding, and we would like to offer some clarifications. There are two key aspects to address: the OOD criterion and the OOD evaluation metric.

First, regarding the OOD criterion, we follow standard post-hoc approaches commonly used in the literature, such as the Maximum Softmax Probability (MSP) [1]. In this setting, we would like to emphasize that our method does not deviate from established practices. In binary classification setups, for a given sample $x$, the MSP score is computed as $\text{MSP}(x)=\max(\hat f(x), 1-\hat f(x))$, where $\hat f(x)$ denotes the predicted probability of a trained classifier $\hat f(\cdot)$. The MSP method effectively performs under two conditions: $\hat f(\cdot)$ exhibits a high confidence for IN samples (yielding $\text{MSP}(x) \approx 1$, i.e., $\hat{f}(x) \approx 0$ or $\approx 1$), and $\hat f(\cdot)$ depicts low confidence on OOD samples (resulting in $\text{MSP}(x) \approx 0.5$, i.e., $\hat{f}(x) \approx 0.5$).

Second, concerning the OOD evaluation metrics, conventional experimental papers use measures such as AUROC, AUPR, and FPR@95. For our insights, the OOD evaluation metrics is the expected OOD risk as a first attempt to mathematically study the double descent phenomenon in post-hoc OOD detection method through a simplified model. Additionally, we would like to stress that the concept of expected OOD risk is not presented as a novel contribution in our work. Similar formulations have already been introduced in prior research (e.g., [2]) to support PAC-style generalization bounds for OOD detection.

Thus, both our OOD risk definition and our OOD criterion are aligned with existing literature and not introduced as new elements. While we acknowledge that multiple classes of OOD examples exist, we believe these variations should be addressed by the choice of OOD criterion itself and not the OOD risk definition.

We hope those clarifications have addressed the concerns on the definition of the expected OOD risk raised in the review.

[1] Hendrycks, Dan, and Kevin Gimpel. "A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks." International Conference on Learning Representations. 2017.

[2] Fang, Zhen, et al. "On the Learnability of Out-of-distribution Detection." Journal of Machine Learning Research 25.84 (2024): 1-83.

Dear reviewer ekWE,

We would like to thank you for your interest and for appreciating the novelty of our study.

As highlighted in the review, our paper provides a comprehensive empirical evaluation across a wide range of architectures (CNNs, ResNets, ViTs, Swin) and post-hoc OOD detection methods. We want to highlight that the major contribution of our study lies in the empirical observations of a double descent phenomenon in post-hoc OOD detection that had never been observed. Our experiments depict how model complexity affects post-hoc OOD detection methods and exhibit, for the first time, a double descent phenomenon in OOD detection. We would like to emphasize the challenges involved in conducting these experiments with real-world deep networks. Furthermore, we observe that post-hoc OOD detection methods may perform less effectively on overparameterized models compared to underparameterized ones. While the paper also includes a theoretical part, this part provides insights rather than an exhaustive analysis. In particular, this part depicts a first attempt to mathematically study the double descent phenomenon in post-hoc OOD detection methods based on confidence through a simplified model.

A.1 Assumptions on OOD distribution (Q1):

Our theoretical insights show that the expected OOD risk for least-squares binary classifiers on Gaussian covariate models under some assumptions exhibits an infinite peak when the number of parameters is close to the number of data, which we associate with the double descent phenomenon.

In Theorem 1, the influence of the OOD distribution is encapsulated within the matrix $\Sigma^\text{OOD}$. Assumptions made in the paper ensure that matrices $\Sigma^\text{OOD}$ and $\Sigma$ are positive-definite, for which we have $\lambda_{\min}(\Sigma)>0$ and $\lambda_{\min}(\Sigma^\text{OOD})>0$. The non-singularities of these matrices ensure that the expected OOD risk is lower bounded by $c(n, p, \sigma)$. Since the expected OOD risk is also upper bounded by $c(n, p, \sigma)$ and that $c(n, p, \sigma)=\infty$ when $n-1 \leq p \leq n+1$, we can deduce from the squeeze theorem that $E_X[R_\text{OOD}(\hat f)]=\infty$ when $n-1 \leq p \leq n+1$. Note that $c(n, p, \sigma)$ depicts the behavior of $E_X[|| \hat{w} - w^\text{OOD}||^2]$ and does not depend on the OOD distribution.

For $p < n-1$ or $p>n+1$, the degree of distributional shift between in- and out-of-distribution scenarios affects how tightly the bounds approximate the expected OOD risk. However, our primary objective was not to provide accurate proxies for the expected OOD risk, but rather to highlight the existence of an infinite peak when $n$ is close to $p$ that we associate with the double descent phenomenon.

In the experimental part, we focused on semantic shifts and considered both near and far OOD distributions, i.e., shifts that are close to or distant from the in-distribution (ID). Our results show that both types of semantic shift are similarly affected by the double descent phenomenon, following the same curve (see Figure 2 and Appendix Figures 4-9). Additionally, to the best of our knowledge, we think that quantifying the amount of shift between two shifted distributions is more relevant to covariate shift scenarios linked to domain adaptation as in [5]. In our setting, divergence-based assumptions are less applicable.

A.2 Depth versus Width (Q2):

Most works on the double descent phenomenon focus on varying the network width rather than depth [1-4]. A primary reason stems from challenges in modifying network depth across empirical and theoretical frameworks. From an empirical perspective, increasing depth often introduces training instabilities and necessitates hyperparameter adjustments (e.g., learning rates) that complicate comparative analyses. From a theoretical perspective, studies typically rely on frameworks like Random Matrix Theory (RMT) or statistical physics, which often model networks as linear systems with random features. Increasing the depth also increases nonlinear interactions and hierarchical dependencies, which may limit the applicability of these mathematical tools.

Nonetheless, we have recently investigated the influence of the model depth on the double descent phenomenon. In these experiments, we used ResNet-like models with fixed width and increased the model complexity by varying their depth. We used CIFAR10 as the ID dataset and reused the same OOD datasets from the width experiments. Note that we kept the same training hyperparameters for the study of the depth. Our results depict comparable double descent patterns in both ID generalization and post-hoc OOD detection performance (measured via AUC). We are open to including these figures in the paper. To the best of our knowledge, they represent the first empirical illustration of depth-wise double descent in realistic models and datasets. Since we can not add additional figures during the rebuttal, we include a table similar to Table 1 to describe our results. In particular, we report the accuracy results for the underparametrized optimum (depth = 4), the interpolation threshold (depth = 8), and overparametrized optimum (depth = 120). These results are averaged across five seeds.

Accuracy (Under / Interpolation / Over): 77.31% / 66.98% / 76.21%
NC1 Ratio (Under / Over): $\frac{1.53}{1.57} = 0.97$

A.3 Generalization of the Theoretical Results (Q3):

We emphasize that this work focuses primarily on empirical investigation and practical analysis rather than theoretical derivation. Its core contribution lies in empirically analyzing the relationship between model complexity and post-hoc OOD detection methods. The major contribution stems from the empirical observation that post-hoc OOD detection methods also exhibit a double descent phenomenon. While the paper includes theoretical elements, they serve as illustrative tools rather than exhaustive analysis. They depict a first attempt to mathematically study the double descent phenomenon in post-hoc OOD detection through a simplified model. Although the theoretical framework is constrained to a specific model architecture, the derivation of Theorem 1 is a non-trivial extension of Theorem 1 in [1]. Future works could extend this analysis to more complex scenarios, such as linear models with random features, gradient descent-based algorithms, or sub-Gaussian distributions. We believe these theoretical extensions fall outside our current scope, and we reserve these explorations for future research.

References

[1] Belkin, Mikhail, Daniel Hsu, and Ji Xu. "Two models of double descent for weak features." SIAM Journal on Mathematics of Data Science 2.4 (2020): 1167-1180.

[2] Couillet, Romain, and Zhenyu Liao. Random matrix methods for machine learning. Cambridge University Press, 2022.

[3] Bach, F. (2024). High-dimensional analysis of double descent for linear regression with random projections. SIAM Journal on Mathematics of Data Science, 6(1), 26-50.

[4] Nakkiran, Preetum, et al. "Deep double descent: Where bigger models and more data hurt." Journal of Statistical Mechanics: Theory and Experiment 2021.12 (2021): 124003.

[5] Ben-David, S., Blitzer, J., Crammer, K., & Pereira, F. (2007). "Analysis of Representations for Domain Adaptation." NeurIPS 2006.

Dear reviewer UoHZ,

We would like to thank you for your interest and for appreciating both the novelty and the experimental findings of our study.

As highlighted in the review, our paper provides a comprehensive empirical evaluation across a wide range of architectures (CNNs, ResNets, ViTs, Swin) and post-hoc OOD detection methods. We want to highlight that the major contribution of our study lies in the empirical observations of a double descent phenomenon in post-hoc OOD detection. To the best of our knowledge, the double descent phenomenon has not been observed before in the OOD context. Our experiments depict how model complexity affects post-hoc OOD detection methods and exhibit, for the first time, a double descent phenomenon in OOD detection. Furthermore, we observe that post-hoc OOD detection methods may perform less effectively on overparameterized models compared to underparameterized ones. The theoretical section in our paper should be seen as insights with an illustrative example of a toy distribution, and not as an exhaustive analysis. However, this part depicts a first attempt to mathematically study the double descent phenomenon in post-hoc OOD detection methods based on confidence through a simplified model.

A.1 Gaussian Constraints & Assumptions on OOD Distribution (Q1+Q2+Q3):

Our theoretical insights show that the expected OOD risk for least-squares binary classifiers on Gaussian covariate models under some assumptions exhibits an infinite peak when the number of parameters is close to the number of data, which we associate with the double descent phenomenon.

In this work, we adopt the conventional OOD framework, where we consider two distinct distributions: an IN distribution $P_{\mathcal{X}, \mathcal{Y}}$ from which training samples are drawn and a separate OOD distribution $P_{\mathcal{X}, \mathcal{Y}}^\text{OOD}$. In Theorem 1, the influence of the OOD distribution is encapsulated within the matrix $\Sigma^\text{OOD}$. Assumptions made in the paper ensure that matrices $\Sigma^\text{OOD}$ and $\Sigma$ are positive-definite, for which we have $\lambda_{\min}(\Sigma)>0$ and $\lambda_{\min}(\Sigma^\text{OOD})>0$. While the choice of the IN and OOD distributions may induce a manifold structure, we do not need to leverage the manifold structure to show the non-singularities of $\Sigma^\text{OOD}$ or $\Sigma$. Indeed, the matrix $\Sigma^\text{OOD}$ is positive-definite if we have $u^T\Sigma^\text{OOD}u>0$ for all non-zeros $u \in \mathbb{R}^d$. Let $u \in \mathbb{R}^d$. From the mean-value theorem and since $\phi(\cdot)$ is differentiable and positive, we have

$u^T \Sigma^\text{OOD} u=u^T E_{(x, \cdot) \sim P_{\mathcal{X}, \mathcal{Y}}^\text{OOD}} \biggl[\Bigl(\tfrac{\phi(x^T \hat w)-\phi(x^T w^\text{OOD})}{x^T \hat w-x^T w^\text{OOD}}\Bigr)^2 x x^T \biggr] u = \int_{\mathbb{R}^d}\Bigl(\tfrac{\phi(x^T \hat w)-\phi(x^T w^\text{OOD})}{x^T \hat w-x^T w^\text{OOD}}\Bigr)^2 (x^T u)^2 \mu^\text{OOD}(x) dx >0.$

The non-singularities of these matrices ensure that the expected OOD risk is lower bounded by $c(n, p, \sigma)$. Since the expected OOD risk is also upper bounded by $c(n, p, \sigma)$ and that $c(n, p, \sigma)=\infty$ when $n-1 \leq p \leq n+1$, we can deduce from the squeeze theorem that $E_X[R_\text{OOD}(\hat f)]=\infty$ when $n-1 \leq p \leq n+1$. Note that $c(n, p, \sigma)$ depicts the behavior of $E_X[|| \hat w - w^\text{OOD}||^2]$ and does not depend on the OOD distribution. As noted in the review, we derive $c(n, p, \sigma)=\infty$ by using the Gaussian assumption on $X$ and Wishart distribution properties. Note that the Gaussian constraint on $X$ is often mentioned in the paper and is explicitly defined in l.141-147.

For $p < n-1$ or $p>n+1$, the degree of distributional shift between in- and out-of-distribution scenarios affects how tightly the bounds approximate the expected OOD risk. However, our primary objective was not to provide accurate proxies for the expected OOD risk, but rather to highlight the existence of an infinite peak when $n$ is close to $p$ that we associate with the double descent phenomenon. This explains why we do not consider assumptions on divergences between IN/OOD distributions. We hope those clarifications have addressed the concerns raised in the review.

A.2 Ambiguity in CIFAR-10/100 Experiments (Q4):

As stated in the paper (l.40-43), our study considers post-hoc OOD detection methods. Post-hoc OOD detection methods operate on top of a pretrained classifier after the model’s training process is finished and leverage internal model properties (probabilities, logits, or features) to define a scoring function measuring the prediction confidence. All classifiers are trained exclusively on in-distribution (ID) data, i.e., without using OOD samples during training. The OOD samples are only used to measure the performance of post-hoc OOD detection methods with the AUC and FPR@95.

Dear reviewer RJrF,

We would like to thank you for your interest and feedback regarding our paper.

We would like to remind that overparametrization is known to benefit generalization (through the double descent phenomenon), but its effects on OOD detection still remain unclear. In this work, we investigate how the model complexity affects the effectiveness of post-hoc OOD detection methods. Post-hoc OOD detection methods operate on top of a pretrained classifier after the model’s training process is finished and leverage internal model properties (probabilities, logits, or features) to define a scoring function measuring the prediction confidence. The objective in this paper is to investigate and better understand the relationship between model complexity and post-hoc OOD detection methods.

We first focus experimentally to highlight that post-hoc OOD detection methods are sensitive to the model complexity. In particular, we observed that logit-based and hybrid methods describe a double descent phenomenon, while feature-based methods may not as depicted in Figure 2 and Table 1. Those results suggest that both logit-based and hybrid OOD methods performance improve as the model complexity increases in the overparametrized regime, while we may observe an opposite behavior for feature-based methods. Furthermore, we observed that some post-hoc OOD methods may demonstrate stronger performance in the under-parameterized regime than in the overparameterized regime for certain architectures, e.g. as shown by the CNN in Figure 2. Interestingly, we also found that Neural Collapse can be used to provide insights into whether an overparameterized or underparameterized model is more suitable for OOD detection.

In the theoretical insights, we show that the expected OOD risk for least-squares binary classifiers on Gaussian covariate models under some assumptions exhibits an infinite peak when the number of parameters is close to the number of data, which we associate with the double descent phenomenon. This result may suggest that confidence-based post-hoc OOD detection methods may exhibit a double descent phenomenon.

Overall, our study provides practical guidelines for selecting post-hoc OOD methods and choosing the appropriate model complexity for effective OOD detection.