Pursuing Feature Separation based on Neural Collapse for Out-of-Distribution Detection

W1-Dependency on NC Property: Thanks for your insightful comment. Your observation is indeed correct that the effectiveness of our method is contingent upon the presence of NC. For modern CNNs and language models, NC is a relatively common phenomenon [2][3] that happens when networks convergence on training dataset. The limitation proposed in [1] that NC property usually fail on the testing set refers to a relative degree compared to training set, but we could still use it to identify OOD samples. Furthermore, our key idea is that placing OOD features on space that orthogonal to principal dimensions of ID features; therefore, our requirement for ID property can be simplified from strictly NC to locating in low-dimensional subspace.

[1] Limitations of Neural Collapse for Understanding Generalization in Deep Learning, arXiv, 2022

[2] Neural Collapse Anchored Prompt Tuning for Generalizable Vision-Language Models, SIGKDD, 2024

[3] Neural Collapse in Deep Linear Networks: From Balanced to Imbalanced Data, ICML, 2023

W2-ViT Architecture: Thanks for your valuable suggestion. For ViT model, previous studies have verified that its feature also shows NC property [4][5], thus satisfying the NC requirement of our method.

To evaluate our generalizability, we conduct experiments with ViT-B-16 model on ImageNet benchmark. Training hyper-parameters are the same as our previous setting, e.g. learning rate 0.0001, epoch 5, and batch size 64, and we use pre-trained model in Pytorch as our initializing model. The results in Table 9 show that our method achieves lower FPR95 and higher AUROC compared with OE method on ViT model, although in a relatively small range. The potential reason is that: the feature dimension of ViT-B-16 is 784 while ImageNet dataset has 1000 categories, makes feature separation difficult because no redundant dimensions can be provided for OOD features. In this case, although our training loss still pursues larger discrepancy between ID and OOD data, aligning ID features with class weights and encouraging OOD features to differ, the optimization of the loss is difficult for large-scaled datasets and thus resulting in relatively slight improvement.

One technical solution is that inserting feature-ascending linear layers into the network to extend the available dimensions, but we haven't done experiments to verify that because of time limitation. Additionally, one possible idea is that computing our loss on shallow high-dimensional features since neural collapse phenomenon also occurs in the intermediate hidden layers [6][7], which is an interesting direction for our future work.

In light of your suggestion, we will ensure to add the experiment on ViT architecture, and importantly, discuss the requirement/limitation on the feature dimension and its impact on our performance in our final paper.

Table 9. Results on ViT-B-16 model on ImageNet benchmark. We report FPR95/AUROC.

[4] Understanding and Improving Transfer Learning of Deep Models via Neural Collapse, TMLR, 2024

[5] NECO: neural collapse based out-of-distribution detection, ICLR, 2024

[6] Feature Learning in Deep Classifiers through Intermediate Neural Collapse, ICML, 2023

[7] Neural Collapse in the Intermediate Hidden Layers of Classification Neural Networks, arXiv, 2023

W3-Near/Far OOD categorization: Thanks for your helpful suggestion. Your intuition is definitely accurate that our performance on Far-OOD is better than on Near-OOD. We have conducted experiments on hard detection setting, as shown in Table 12 in Appendix A.3, where our improvement is relatively lower compared to Far-OOD datasets like SVHN. Additionally, according to OpenOOD, we evaluate our method under Tiny-ImageNet (Near-OOD) as test OOD dataset on CIFAR benchmarks. The result is shown in Table 10, which indicates that we still has performance improvement on Near-OOD datasets.

In light of your suggestion, We ensure that OOD rows in our Table will be categorized according to the categories in OpenOOD.

Table 10. Results on CIFAR benchmarks with Tiny-ImageNet (Near-OOD) as test OOD dataset. We report the FPR95/AUROC.

Q1-Experiment on OpenOOD framework: Thanks for your valuable suggestion. We believe that OpenOOD provides a standardized and comprehensive benchmark for evaluating detection performance. In the following Table, we list the Near and Far OOD datasets for CIFAR10, CIFAR100, and ImageNet benchmarks in OpenOOD, and flag the datasets we have evaluated. It can be seen that our experiments contain both Near and Far OOD for evaluation, but indeed lack testing on some datasets, also testing on transformer based architectures. We are grateful for your recommendation and will use OpenOOD framework for standard evaluation in our future works.

Q2- NC impact: Thanks for your insightful question, which really provides new insight into our approach. For the case that NC is weakened or absent, our method can still work in a relatively relaxed condition, that is, ID features locate in a low-dimensional subspace. In this way, we can still find redundant dimensions to place OOD features to enlarge ID-OOD feature distance. In fact, NC property should always be present, just in different degree, because a good network would output the maximum probability at labeled class, which decides that the penultimate feature should be align most closely with weights of the corresponding class.

For imbalanced data distribution, the NC property declines to “Minority Collapse[8]”, where the classifiers for minority classes are squeezed into one direction. This phenomenon destroys the geometric structure of NC, but has relatively minimal influence on our effectiveness. Since our approach utilizes the alignment between ID features and model weights to pursuing feature discrepancy, we are not strict on the feature structure. Actually, as long as ID features have low-dimensional property that locates in a subspace, our dimension separation idea can be used. In the following, we conduct an experiment on imbalance data to verify our effectiveness.

According to settings in [9][10], we create an imbalanced version of CIFAR10 dataset, denote as CIFAR10-LT. Specifically, we consider a long-tailed imbalance with ratio $\rho=50$ (denote the ratio between sample sizes of the most frequent and least frequent class). Based on above imbalanced data, we firstly pretrain a WideResNet-40-2 model on CIFAR10-LT, and then finetune the model using auxiliary OOD datasets. The training setting is the same as our previous experiments. The final detection performance of our method and OE is shown in Table 11, which provides empirical evidence of our effectiveness under imbalanced data condition.

Table 11. Results on imbalanced CIFAR10 data. We report the FPR95/AUROC.

[8] All-around Neural Collapse for Imbalanced Classification, arXiv, 2024

[9] Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss, NeurIPS, 2019

[10] Balanced Contrastive Learning for Long-Tailed Visual Recognition, CVPR, 2022

W1-Discussion on Performance Difference: Thanks for your question. For OOD detection tasks, there are many factors that may influence the final result, such as the complexity of ID dataset, Near v.s. Far OOD data for evaluation, and model accuracy on ID data. In our ImageNet experiment, the potential reason may be that feature representations are more entangled for large-scaled datasets, thereby increasing the difficulty of feature separation. Since the model, ID dataset, and test unseen dataset is different from those on CIFAR benchmarks, it is kind of hard to quantify the extent of improvement and analysis the potential reason. Hope the above explanation address your question and look forward to further discussions.

W2-Dimension Exhaustion: Your foresight is quiet insightful! As ID features occupy a fixed space (whose dimension is the number of classes), the rest dimensions are provided for OOD features, which is expected to be large to ensure abundant space for diverse OOD data. When faced with large-scale datasets, the dimension required by both ID and OOD features become larger; therefore, dimension exhaustion may occur, which affects performance.

Nevertheless, even in the exhausted case, our method can still enlarge feature distance between ID and OOD data, aligning ID features with class weights and encouraging OOD features to differ. To verify that, we conducte experiments using WideResNet-40-1, whose feature dimension is 64, on the CIFAR100 benchmark. The results, as shown in Table 7, indicate that our method outperforms the basic OE loss, suggesting that our approach is effective even when feature dimensions are constrained.

In response to your valuable question, we will incorporate additional justification and experiments in our final paper.

Table 7. Results on WideResNet-40-1 model (feature dimension is 64) on CIFAR100 benchmark. We report FPR95/AUROC.

W3-Feature Distance in Table-9 of our paper: Following your suggestion, we evaluate the Euclidean distance and cosine similarity between features and their corresponding class mean vectors. The results are shown in Table 8. The absolute value is indeed different from results calculated based on model weights. In relative comparison, our method still shows better feature separation between ID and OOD data. Thanks for your valuable advice, and we will include the results in our final paper.

Table 8. Feature Distance based on class mean vectors. we report the average Euclidean distance/cosine similarity on the whole dataset.

Firstly, we are very grateful for your recognition. The discussion about the connection between NC and shaping the models’ embedding space is quiet interesting and thought-provoking. After reading these three papers, we find that they are all based on a distribution assumption on the features of ID samples, i.e., they model the feature distribution as a class-conditional vMF distribution, which is formulated as:

$z\sim Z_{d}(\kappa)\exp(\kappa \mu_{c}^Tz)$

where $\kappa$ is a constant, $Z_{d}(\kappa)$ is a normalization factor, $\mu_c^T$ is the class prototype of class $c$ with unit norm, and $z$ is the feature vector. From this equation, we can discover that its distance measurement is cosine similarity instead of Euclidean distance, which is consistent with the metric in NC phenomenon, i.e., NC also uses the angle between features and FC weights to measure their alignment degree.

In CIDER [1], they design two losses: compactness loss to encourage samples to be closely aligned with its corresponding class prototype, and dispersion loss to enlarge angular distances among different class prototypes. If we consider class prototype and FC weights to be equivalent, the compactness loss actually promotes alignment of features and weights, and the dispersion loss reaches its optimal value at a simplex ETF to which NC finally converges. Therefore, from a NC point of view, it can be said that CIDER method promotes the NC properties of ID features. But this method is motivated from the vMF distribution assumption of ID features, instead of the utilization of NC property.

Another paper PALM [2] claims that only using one prototype for each class to model ID features is not sufficient, which ignores the natural diversities within the data. Therefore, they propose to automatically and dynamically learn a mixture of prototypes for each class. In our view, it can be regarded as an extension of the NC property. Especially on large-scaled datasets, it would be more effective since the alignment to FC weights would not strictly hold.

The last paper DRL [3] claims that there is a gap between the distribution prior (vMF) and the true feature distribution. Therefore, they propose a method to mitigate this gap using evidence lower bound (ELBO, which explicitly measure the extent to which a training sample fits the assumed distribution) during training process. Since our previous discussion points out that the distance measurement of vMF distribution is consistent with the metric in NC property, maybe the DRL method promotes NC in some degree.

Overall, these three methods are all based on a vMF distribution assumption on ID features, and then design losses to pursue intra-class aggregation and inter-class dispersion. Their connection to NC is that all of them employ cosine similarity as distance measurement, and then the alignment loss and dispersion loss respectively corresponds to the feature-weight alignment and simplex geometry in NC properties. The main difference between our method and them is that we focus on the low-dimensional space induced by NC property, so as to use redundant dimensions to enlarge the ID-OOD discrepancy, while they are more concentrated on the distribution modeling of ID features, so as to design losses for intra-class aggregation and inter-class dispersion.

In light of this interesting question, we will include the discussion about the connection between NC and these methods in our final version.

[1] Ming et al. How to exploit hyperspherical embeddings for out-of-distribution detection? ICLR 2023.

[2] Lu et al. Learning with mixture of prototypes for out-of-distribution detection. ICLR 2024.

[3] Zhang et al. Learning to Shape In-distribution Feature Space for Out-of-distribution Detection. NeurIPS 2024.

W1-Feature Distribution in Figure2: Thanks for your question. Figure 2 is plotted on the test ID data and unseen OOD data, completely not on training samples. Sorry for the confusion, we have stated the experiment setting more clearly in our paper.

W2-Modest Improvement: Thanks for raising the performance question. We recognize the significance of substantial performance gains in the context of our research and have taken your feedback seriously. In response to this concern, we demonstrate our effectiveness as follows.

For CIFAR benchmarks, where the AUROC improvement is inherently in a small range, an increase exceeding 1% is considered significant. In recent paper DAL[1], their AUROC improvement is 0.64% on CIFAR100, while ours is 1.18%. On CIFAR10 benchmark, the AUROC is approaching 100% for most methods, thus the room for improvement is minimal. DAL showed 0.04% improvement in AUROC, which is negligible but understandable. Similarly, we acknowledge that our AUROC performance on CIFAR10 benchmark is relatively modest, but this does not mean that our approach is unproductive.

For ImageNet benchmark, our method shows 0.58% improvement in AUROC compared to DAL. A notable fact is that most approaches[2][3][4][5] did not report their results on ImageNet-1K, only evaluating on CIFAR datasets and a small subset of ImageNet-1K, e.g. ImageNet-100 that consists of 100 randomly sampled classes, which implies their difficulty to implement on large-scaled datasets, while our approach demonstrates its effectiveness over the large-scaled dataset.

We hope the above explanation demonstrates the performance of our approach and address your concern. Look forward to your feedback.

[1] Learning to Augment Distributions for Out-of-Distribution Detection, NeurIPS, 2023

[2] Energy-based Out-of-distribution Detection, NeurIPS, 2020

[3] POEM: Out-of-Distribution Detection with Posterior Sampling, ICML, 2022

[4] How to exploit hyperspherical embeddings for out-of-distribution detection, ICLR, 2023

[5] Dos: Diverse outlier sampling for out-of-distribution detection, ICLR, 2024

W3-Metric Discrepancy: Thank you for your meticulous review. Upon reviewing the experimental setup and results, we have identified several factors that may contribute to the differences. And we have found several works to prove the validity of our results.

The performance discrepancy may be mainly due to the randomness in experiments including:

1. Training Data Variance: While both studies utilize ImageNet-21K-P, the large scale of ImageNet-21K-P allows for considerably different subsets, potentially leading to variations in performance.
2. Test OOD Data Selection: Each test OOD dataset has huge amount of samples, thus, in practice researchers randomly select a subset of these datasets to evaluate their method. The selection of test OOD data subsets can introduce variability.
3. Model Parameters: Pytorch provides two version of pre-trained ResNet50 parameters, which can affect the detection performance.

In addition to analyzing the potential reasons for differences, we have re-checked our code and uploaded it to Supplementary Material to support the result reproduction.

More importantly, we note that results from other studies, such as DOE [6] and HB [7], also showed variations from the result reported in DAL[1] under the same setting. We list the reported result of DAL method in different papers in Table 5, where the reported result in HB[7] is consistent with ours. These results indicate that relative performance is more important than absolute numbers when making a fair comparison of methods.

Table 5. Reported results of DAL method in different papers of their tables. We record FPR95/AUROC.

We hope this clarification addresses your concerns and provides a satisfactory explanation for the observed differences.

[6] Out-of-distribution detection with implicit outlier transformation, ICLR, 2023

[7] Energy-based Hopfield Boosting for Out-of-Distribution Detection, arXiv, 2024

W4-Insufficient ImageNet Experiment: Thanks for your question. Based on our empirical results on CIFAR benchmarks, DAL method shows superior performance compared to other approaches, standing for SOTA baseline in our compared methods. Therefore, we choose it as one of baselines on ImageNet benchmark. More importantly, other approaches compared in our paper did not report their results on ImageNet-1K, only evaluating on CIFAR datasets and a small subset of ImageNet-1K, e.g. ImageNet-10 that consists of 10 randomly sampled classes, implying their difficulty to implement on large-scaled datasets. Since DAL has shown superior performance compared to other methods on ImageNet benchmark in its paper, we choose it as our comparison baseline. We believe that more comprehensive experiments can improve our paper and we will consider to compare with more approaches in our future work.

Besides, thank you for your scrupulous reading, we have uniformed the decimal places between Lines 381 and 382.

Q1-Difference between KNN: Thanks for your question. The key distinction between our method and KNN is that: we explicitly enlarge the feature discrepancy through optimization while KNN uses the natural difference in feature space between ID and unseen OOD data. Utilizing the distance in feature space to detect OOD samples is a common idea [8][9][10], their core is to design a detection metric based on feature representations. However, none of them considers enlarging the feature distance by designing a training loss with auxiliary OOD data. The main contribution of our paper is that proposing a NC-property based loss to effectively enlarge the difference between ID and OOD features, solving the problem in describing OOD feature distribution and offering new insights to this field, instead of proposing a new feature-based metric. The experiment results in Table 8 of our paper showed that even under the maximum output probability (MSP) metric, we still have outstanding performance. Hope the above explanation addresses your question.

[8] A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks, NeurIPS, 2018

[9] A simple fix to mahalanobis distance for improving near-ood detection, arXiv, 2021

[10] Understanding the Feature Norm for Out-of-Distribution Detection, ICCV, 2023

Q2-Requirement for Feature Dimension: Thanks for your interesting and insightful question.

Intuitively, the feature dimension larger than the output dimension is preferable for our method to facilitate ID-OOD feature separation by redundant dimensions. And this condition is commonly satisfied in modern CNNs, as the class number of common classification datasets ranging from tens to thousands.

Nevertheless, we also considered the scenario where feature dimensions are less than output dimensions. Actually, in this condition, our training loss still pursues larger discrepancy between ID and OOD features, aligning ID features with class weights and encouraging OOD features to differ. To verify our effectiveness, we conduct experiments using WideResNet-40-1, whose feature dimension is 64, on the CIFAR100 benchmark. The results, as shown in Table 6, indicate that our method outperforms the basic OE loss, suggesting that our approach is robust even when feature dimensions are constrained. In addition to using our loss directly, one possible idea is that computing our loss on shallow features since neural collapse phenomenon also occurs in the intermediate hidden layers [11][12], which is an interesting direction for our future work.

Table 6. Results on WideResNet-40-1 model (feature dimension is 64) on CIFAR100 benchmark. We report FPR95/AUROC.

Hope the above explanation and empirical results can answer your question. And we ensure to include the discussion about feature dimensions in our final paper.

[11] Feature Learning in Deep Classifiers through Intermediate Neural Collapse, ICML, 2023

[12] Neural Collapse in the Intermediate Hidden Layers of Classification Neural Networks, arXiv, 2023

Dear reviewer,

Thanks again for your valuable questions and suggestions. We noticed that the discussion period is extended, thus we could provide more experiments if necessary. We would greatly appreciate it if you could provide your feedback at your earliest convenience.

Explanation on substantial discrepancy: Thanks for your question. We are also curious about the performance variation for Places dataset.

First of all, we confirm that the performance in our paper is repeatable, for not only ours but all the methods reported. And all the methods are evaluated on the fixed and same subsets, the comparison is fair.

Second, we here report several performance from different papers, which uses Places dataset. (Because most auxiliary OOD data based methods did not conduct experiments on ImageNet-1K benchmarks, we only find DOE (which proposes a novel output-based loss) and HB (which proposes a data selection method) for result reference in the same setting as DAL. And the results of other papers may be under different model architectures and belong to different types of detection methods. )

Table III. Performance on Places OOD dataset under ImageNet-1K benchmark. We record FPR95/AUROC.

One could see that indeed the values are different, but most of them are within a range about 40%~60% on FPR95, while DAL shows distinct difference with others. Actually, we are confused about the result of DAL on ImageNet, because their performance under CIFAR100 benchmark on Places365 is FPR95=49.35%, AUROC=90.81% in their paper. As the dataset becomes larger-scaled, e.g. ImageNet-1K, OOD samples should be more difficult to detect. However, their performance dramatically improved into an outstanding level, with FPR95=5.83%, AUROC=99.09%.

Third, for the reason for such a dramatic difference on Places dataset, we would like to put down to different chose of the data.

The Places dataset is a subset of Places365 [7] curated by Mos [4] as “50 categories that are not present in ImageNet-1K”, which contains 9822 images from 50 environment classes and is widely used as test OOD datasets. However, there are some similar ones to the classes of ImageNet, including hayfield v.s. hay, cornfield v.s. corn, lagoon v.s. seashore and lakeshore, underwater v.s. coral reef and scuba diver, ocean v.s. seashore, etc. These are identified as Near-OOD data that are hard to detect.

Places is such a dataset containing Near-OOD data and therefore, the OOD detection is very challenging. We use Places dataset, however, DAL paper [8] used a subset of Places365 without claiming which subset. Since the whole Places365 dataset contains 10 million images (some images are totally different from ImageNet while some are similar to ImageNet), the subset choice would have a large influence on the detection performance. DAL did not publish their code on ImageNet benchmark, therefore, we cannot say what subset they test on Places365. From their outstanding result on Places365, it is reasonable to guess that they may have used a subset (or randomly selected from the whole Places365) that contains more Far OOD samples, instead of Places subset.

Overall, we think the dramatic difference comes from different dataset (Places vs Places365). For the same dataset, the OOD performance is stable and the comparison among different methods is fair.

[2] Out-of-distribution Detection with Implicit Outlier Transformation, ICLR, 2023

[3] Energy-based Hopfield Boosting for Out-of-Distribution Detection, arXiv, 2024

[4] Mos: Towards scaling out-ofdistribution detection for large semantic space, CVPR, 2021

[5] ReAct: Out-of-distribution Detection With Rectified Activations, NIPS, 2021

[6] Delving into out-of-distribution detection with vision-language representations, NIPS, 2022

[7] Places: A 10 million image database for scene recognition, TPAMI, 2017

[8] Learning to Augment Distributions for Out-of-Distribution Detection, NIPS, 2023

Multiple OOD subsets: Thanks for your suggestion. We evaluate our method with different seeds, which corresponds to different subset selection, on CIFAR100 benchmarks. In our code, we randomly select 2000 OOD samples for each OOD dataset, and the ID samples are the test set of CIFAR100 with 10000 images. Notably, the Places365 dataset is extremely large (about ten million images), and we download a subset of it (about ten thousand images) for CIFAR100 benchmark (following the setting in KNN [1] and download the subset from https://github.com/deeplearning-wisc/knn-ood). And then we randomly choose 2000 images from the subset for evaluation.

The result is shown in Table I, where the performance is stable across different subsets.

Table I. Evaluation results with different subsets (control by different random seeds) on CIFAR100 benchmarks.

For ImageNet benchmark, the subset choice may have a larger influence on performance because that an OOD dataset may contain both Near and Far OOD samples relative to the ImageNet dataset. Far OOD samples are easier to detect compared to Near OOD data. Therefore, if the subset contains more Far OOD samples, the detection performance on it will be better. And for CIFAR datasets, the OOD samples of test OOD datasets all belongs to Far-OOD data relative to CIFAR datasets, thus different subsets rarely influence the performance.

In our ImageNet experiments, we use the fixed subset of test OOD datasets provided in KNN paper [1] to evaluate all the compared methods and our approach, without special test data selection. Specifically, following the published code of KNN [1] (https://github.com/deeplearning-wisc/knn-ood?tab=readme-ov-file), we download the sampled 10000 images from their selected concepts for iNaturalist, SUN, and Places, which can be download via the following links:

wget http://pages.cs.wisc.edu/~huangrui/imagenet_ood_dataset/iNaturalist.tar.gz
wget http://pages.cs.wisc.edu/~huangrui/imagenet_ood_dataset/SUN.tar.gz
wget http://pages.cs.wisc.edu/~huangrui/imagenet_ood_dataset/Places.tar.gz

For Textures, following common setting, we use the entire dataset, which can be downloaded from their original website (https://www.robots.ox.ac.uk/~vgg/data/dtd/).

Because that the whole dataset of iNaturalist, SUN, and Places is large (as Table II shows), we actually do not download them entirely. But it should be fair to compare all the methods on the same subsets of test OOD datasets.

Table II. Size of different datasets. The unit is ten thousand.

[1] Out-of-distribution Detection with Deep Nearest Neighbors, ICML, 2022

W3-Insufficient ImageNet Experiment: Thanks for your question. Upon comparing with Table 2 and Table 3, we explain our experiment inconsistency on ImageNet as follows.

1. Comparison with other methods: Based on our empirical results on CIFAR benchmarks, DAL method shows superior performance compared to other approaches, standing for SOTA baseline in our compared methods. Therefore, we choose it as one of baselines on ImageNet benchmark. More importantly, among all the compared auxiliary OOD data based methods in our paper[2][3][4], only DAL reported their result on ImageNet-1K benchmark, while others only evaluated on CIFAR benchmarks and a small subset of ImageNet-1K, e.g. ImageNet-10 that consists of 10 randomly sampled classes, implying their difficulty to implement on large-scaled datasets. Since DAL has shown superior performance compared to other methods on ImageNet benchmark in its paper, we choose it as our comparison baseline.
2. Experiment on more architectures: We have tested our method on ImageNet benchmark with ResNet50 architecture. Following your suggestion, we further evaluate our method on transformer based network, e.g. ViT-B-16, on ImageNet benchmark. The training hyper-parameters are the same as our previous setting, e.g. learning rate 0.0001, epoch 5, and batch size 64, and we use pre-trained model in Pytorch as our initial model. The results are shown in Table 4, where our method achieves lower FPR95 and higher AUROC compared with OE, although in a relatively small range. The potential reason may be due to the limitation of feature dimensions, where ViT-B-16 has 784-dimensional feature space while ImageNet dataset has 1000 categories, makes feature separation difficult because no redundant dimensions can be provided for OOD features. The solution to the feature dimension insufficiency is an interesting direction for our future work, and we have some ideas like inserting feature-ascending linear layers into the network to extend the available dimensions or computing our loss on shallow high-dimensional features where neural collapse phenomenon also occurs[5][6].

Table 4. Results on ViT-B-16 model on ImageNet benchmark. We report the average FPR95/AUROC.

Hope the above explanation address your concern and we will conduct more comprehensive experiments in our future work. Also, thanks to your question, motivating us to discuss the requirement/limitation on the feature dimension and its impact on our performance in our final paper.

[2] POEM: Out-of-Distribution Detection with Posterior Sampling, ICML, 2022

[3] How to exploit hyperspherical embeddings for out-of-distribution detection, ICLR, 2023

[4] Energy-based Out-of-distribution Detection, NeurIPS, 2020

[5] Feature Learning in Deep Classifiers through Intermediate Neural Collapse, ICML, 2023

[6] Neural Collapse in the Intermediate Hidden Layers of Classification Neural Networks, arXiv, 2023

W4-Analysis of Feature Embedding: Thanks for your valuable suggestion. For embedding visualization, we provide the the 2D and 3D visualization of features of test ID data and unseen OOD data in Figure 2 of our paper. And the feature separability of vanilla, OE-trained, and our model gradually increases, especially, OOD features of our model mostly locate in the dimension that orthogonal to the principal subspace of ID features. Moreover, we evaluate the degree of feature separation in numerical in Table 9 of our paper. The results indicate that our method presents larger feature distance between ID and OOD data. Hope the feature visualization and numerical results meet your suggestion, strengthening the evaluation of our effectiveness.

W1-Training Strategy and Impact: Thanks for your question. Below, we address your questions in detail.

1. Training Strategy Choice: The choice between incorporating all loss terms directly or initially training with cross-entropy (CE) loss depends on the convergence state of the network. In most cases, the initial network for OOD fine-tuning has been well-trained on ID data, thereby directly using all loss terms is applicable. In our experiments, we choose all loss terms on ImageNet but initially training with CE on CIFAR10/CIFAR100 because that the initial network on CIFAR benchmarks are half-trained models (99 epoch v.s. standard 200 epoch). We assessed the feature variability supperssion metric (NC1)[1] for the initial model and the model after CE loss fine-tuning, as shown in Table 1. The results indicate that the degree of NC is significantly improved after CE loss fine-tuning, but the absolute value depends on the size of datasets. Therefore, we do not specific a threshold for the strategy guidance, instead, empirically choosing loss terms based on initial model is enough.

Table 1. Comparison of Feature variability supperssion metric (NC1, [1]) on initial and fine-tuned model.

2. Impact on Performance: On CIFAR10 benchmark, we study the influence of training epochs with only CE loss (Ep-CE) on performance to explore the impact of training strategies. From directly using all loss terms to gradually increasing Ep-CE with fixed epochs for training with all loss terms (Ep-All), we finetune the initial model and evaluate our detection performance on CIFAR10 benchmark. Experiment results are shown in Table 2, where our method is not sensitive to the strategy choice, but CE loss fine-tuning does enhance performance for half-trained models.

Table 2. Influence of Ep-CE on CIFAR10 benchmark. We report the average FPR95/AUROC and clean accuracy.

In light of your suggestion, we will ensure to add the discussion on strategy choice and influence in our final paper. Hope the above explanation address your question.

[1] Neural Collapse: A Review on Modelling Principles and Generalization, TMLR, 2023

W2-Role of Loss Terms: Thanks for your question. The analysis of Table 5 in our original paper can be listed as follows:

1. Main role of Sep loss: Comparing results of No.1 (only OE loss) and No.3 (OE loss + Sep loss), it shows that our Sep loss significantly improves detection performance.
2. Auxiliary role of Clu loss: Comparing results of No.1 (only OE loss) and No.2 (OE loss + Clu loss), it shows that purely using Clu loss does not work. In contrast, comparing results of No.3 (OE loss + Sep loss) and No.4 (OE loss + Sep loss + Clu loss), it shows that cooperating Clu loss with Sep loss can further improve our performance.
3. Effectiveness of Sep and Clu loss: In addition to Table 5 in our original paper, we conduct an experiment that only using Sep loss and Clu loss (without OE loss) to further demonstrate our effectiveness. The results are shown in Table 3, where our method shows superior performance compared to OE.

Table 5 in our original paper. Performance under different training losses.

Table 3. Comparison between OE loss and Sep+Clu loss.

Thank you for your response.

• The current model requires a manual selection of techniques depending on whether it is half-trained or fully trained, which is often impractical. Can the proposed method achieve strong performance through a unified process?

• All experiments indicate that Clu loss alone is not effective. It only shows utility when combined with Sep loss and performs well exclusively on the CIFAR-100 dataset. This raises questions about the claimed effectiveness of Clu loss. Could you provide an explanation for this?

• Can the visualizations be improved by using more realistic examples, such as data samples from the benchmark datasets used in the experiments, rather than the simplified two-class examples shown in Fig. 2?