What If the Input is Expanded in OOD Detection?

Thank you for your time devoted to reviewing this paper and your constructive suggestions. Here are our detailed replies to your questions.

W1, Q1: Although the proposed method can improve the performance of the SOTA methods, whether they face the same challenges and how does the proposed method improve each method are not clearly explained. Since the proposed method is a framework to improve the OOD performance, I wonder how does the proposed CoVer method improve each method? Why the proposed method is effective to improve different types of the method should be further explained.

Thanks for your constructive comments. First, we would like to state that almost all the SOTA methods considered in our work face the same challenge on excavating discriminative representation for OOD detection. Although the SOTA methods are designed with different advanced techniques (e.g., ReAct, DICE, and ASH integrates the activation regularization or reshaping to the forward path of a single input in DNNs; MCM, LoCoOp, CLIPN, and NegLabel explore the CLIP's representation of a single input to detect OOD samples in VLMs), we notice that the single input may implicitly constrain the representation dimension for detection as the discriminative features are not infinite to utilize.

Second, our CoVer introduce the extra dimensions with corruptions to reveal the intrinsic distinguishes of ID and OOD samples. In the multi-inputs space, ID data maintains an overall higher confidence expectation, whereas OOD data encounters notable changes in model confidence since the high-frequency features are altered as verified in our analysis. Leveraging the difference trends between ID and OOD samples by a simple but effective averaging operation, our CoVer can further improve the ID-OOD separability when combined with those SOTA methods (i.e., adding the corrupted input dimension for scores averaging).

W2, Q2: For the different methods, what types of the inputs can improve them effectively lacks of deeply analysis. For the proposed method, it mainly focuses on expanding the inputs. While for the different method, they usually focus on different knowledge. I wonder how to decide what types of the inputs that can improve the performance of different methods.

Thank you for your thoughtful question. For the types of corrupted inputs and their corresponding severity levels, we have conducted some related explorations (e.g., Tables 10 and 11, Figures 6 and 7) in the Appendix of our original submission for performance references. We have noticed that some specific corruptions (e.g., Brightness, Fog, Contrast, Motion Blur, Defocus Blur) can generally improve the OOD detection performance. It can be found those types provide corruptions on the non-semantic level of the input, instead of damaging the semantic feature too much like the other types. It is also empirically verified in our previous analysis about confidence mutation, are induced by the high-frequency feature changes. However, to further efficiently determine which corruption types to choose for expanding the input dimension, it is recommended to rely on an additional validation set (e.g., the SVHN dataset used in our exploration) as the selection basis, similar to the hyperparameter selection process in previous works.

Thank you for your time devoted to reviewing this paper and your constructive suggestions. Here are our detailed replies to your questions.

W1, Q1: Lack explaination and comparison. Different types of corruptions are used in the method, which is similar to the idea of Watermarking, where a generic watermark is trained to distinguish between ID and OOD data. The advantages are not deeply analyzed. Watermaking is a very popular and SOTA method for OOD detection. Compared with watermarking and so on, what are the advantages of using corrupted images to expand the input space. Since the corrupted images are the important data in the proposed method, it is necessary to present the advantages for OOD with corrupted images more clearly.

Thank you for your constructive comments. We have added the analysis about the Watermarking method with our CoVer in the following two aspects.

Conceptually, we have noticted that Watermarking utilizes a well-trained mask to help the original images be distinguishable from the OOD data. However, Watermarking is still trying to excavate the useful feature representation in a single-input perspective. In contrast, the critical distinguishable point and also the advantage of our CoVer method lies in input dimension expansion with the corrupt variants, which instead provide a extra dimension to explore the confidence mutation to better identify the OOD samples.

Experimentally, we have conducted the comparison and report the results in Table 9 in attached PDF. The results show that, on the one hand, training an optimized watermarking for effectively distinguishing between ID and OOD samples is a time-consuming process. On the other hand, CoVer achieves this by introducing corrupted inputs to capture the confidence variations between ID and OOD data during the test phase, which is simpler, faster, and more effective. We will add this part in our revision and discuss them in detail in our appendix.

W2, Q2: Less clarification. The CoVer approach is flexible and compatible and can be combined with different methods to achieve performance gains. The combination is not clearly presented for different methods. The proposed method can enhance most of the OOD methods and improve their performance. However, these methods usually designed with different techniques, such as fine-tuning and zero-shot methods. Hence, I wonder whether there exsit differences to improve these methods with CoVer and how to guarantee the effectiveness when combine CoVer with these methods.

Thank you for your thoughtful question. The fine-tuning methods designed for OOD detection focus on utilizing extra auxiliary outliers to regularize the model to be better aware of OOD data, while the zero-shot methods are focusing on scoring function design of excavating the original distinguisable feature that can better represent the differences. Although their focus are conceptually different, our CoVer provides the similar enhancement as the adapation is on the input side (i.e., adding the corrupted input dimension for score averaging). It sheds new light on leveraging raw input in an extra dimension, but is not bound to either fine-tuning or zero-shot framework that targets on utlizing or enhancing the single-input representativeness for OOD detection task. Thus, we have found limited significant differences when CoVer combined with these two methods, both in their high-level adapation and in our empirical results.

To guarantee the effectiveness of CoVer, we recommend to use an additional validation set (e.g., the SVHN dataset used in our exploration) as the selection basis for choosing the effective corruption types and severity level, similar to previous works like Watermarking to learn a optimal mask using the validation set.

Thank you for your time devoted to reviewing this paper and your constructive suggestions. Here are our detailed replies to your questions.

W1: Omission of Data Depths and Information Projections in Related Work

Thanks for your valuable suggestions and bringing us those insightful works of data depths and information projections. Here, we have concluded a related discussion about these studies in the following.

Computing OOD scores on the embedding output of the last layer of the encoder is not the best choice for textual OOD detection. To address this, [1] proposed aggregating OOD scores across all layers and introduced an extended text OOD classification benchmark, MILTOOD-C. In a similar vein, RainProof [2] introduced a relative information projection framework and a new benchmark called LOFTER on text generators, considering both OOD performance and task-specific metrics. Building on the idea of information projection, REFEREE [3] leveraged I-projection to extract relevant information from the softmax outputs of a network for black-box adversarial attack detection. On the other hand, APPROVED [4] proposed to compute a similarity score between an input sample and the training distribution using the statistical notion of data depth at the logit layer. HAMPER [5] introduced a method to detect adversarial examples by utilizing the concept of data depths, particularly the halfspace-mass (HM) depth, known for its attractive properties and non-differentiability. Furthermore, TRUSTED [6] relied on the information available across all hidden layers of a network, leveraging a novel similarity score based on the Integrate Rank-Weighted depth for textual OOD detection. LAROUSSE [7] employed a new anomaly score built on the HM depth to detect textual adversarial attacks in an unsupervised manner.

We will add the above to the related work section for providing a comprehensive overview, and also add the formal citations for reference in our revised version.

Q1: About data depths

Thanks for your question. We have discussed the related work on data depths in the previous response and will add it to the related work section in our revised version. Since the discussion of the related work in our original submission mainly refers to recent representative reviews of visual OOD detection, like [8] and [9], we conduct limited exploration on this direction. Although our CoVer has conceptual distinguishes with exploring data depths for OOD detection, it is notable that both provide new perspective for exploring discrimnative features between the ID and OOD samples beyond the raw inputs.

Q2: About Isolation Forest

Thanks for the question. We have noticed that Isolation Forest, though a classical method of anomaly detection (AD), is not a common baseline considered in previous literatures [8, 9] for OOD detection. In fact, AD is quite different from OOD detection, as AD treats ID samples as a whole [9]. This means that regardless of the number of classes (or statistical modalities) in ID data, AD does not require differentiation within the ID samples (while OOD detection does). OOD detection considers the knowledge for all the known classes, while AD mainly learns the normal patterns from the majority of data and identifies the anomalies. In our work, we select the baseline methods following previous well-recognized studies, such as DNN-based methods (e.g., ReAct, ASH, DICE) and VLM-based methods (e.g., MCM, CLIPN, NegLabel).

Q3: Comparison with data depths, information projections, and Isolation Forest

Thank you for the question. We have conducted comparison experiments between our CoVer and baselines as you mentioned, as detailed in Table 8 in attached PDF.

Due to the large scale of the ImageNet training set, we sampled 50 samples per class to construct a subset from the training data to represent the training distribution, as recommended by the similar work named NNGuide [10]. For data depths, we reimplemented APPROVED [4] for comparison. For information projections, we reproduced REFEREE [3] for comparison. For Isolation Forest, we use logits as the input to detect the anomaly logits in ID and OOD samples.

The results indicate that AD and textual OOD detection methods, such as Data Depth and Information Projection, may not suit for visual OOD detection tasks, a view also mentioned in related surveys [8, 9]. Similarly, classical ML methods for AD, such as Isolation Forest, seem to be failed to excavate discriminative representations when applied to image OOD detection. However, since these methods are insightful in distinguishing the outliers, we believe it is worth further efforts in the future to adopt the critical intuition into the OOD detection problem.

References:

[1] Darrin M, Staerman G, Gomes E D C, et al. Unsupervised Layer-wise Score Aggregation for Textual OOD Detection. In AAAI, 2024.

[2] Darrin M, Piantanida P, Colombo P. ainproof: An Umbrella To Shield Text Generators From Out-Of-Distribution Data. In Arxiv, 2022.

[3] Picot M, Noiry N, Piantanida P, et al. Adversarial Attack Detection Under Realistic Constraints. 2022.

[4] Picot M, Staerman G, Granese F, et al. A Simple Unsupervised Data Depth-based Method to Detect Adversarial Images. 2023.

[5] Colombo P, Picot M, Granese F, et al. A Halfspace-Mass Depth-Based Method for Adversarial Attack Detection. TMLR, 2023.

[6] Colombo P, Dadalto E, Staerman G, et al. Beyond Mahalanobis Distance for Textual OOD Detection. In NeurIPS, 2022.

[7] Colombo P, Picot M, Noiry N, et al. Toward stronger textual attack detectors. In Arxiv, 2023.

[8] Yang J, Wang P, Zou D, et al. Openood: Benchmarking generalized out-of-distribution detection. In NeurIPS, 2022.

[9] Yang J, Zhou K, Li Y, et al. Generalized out-of-distribution detection: A survey. In IJCV, 2024.

[10] Park J, Jung Y G, Teoh A B J. Nearest neighbor guidance for out-of-distribution detection. In ICCV, 2023.

Dear Reviewer HCvU,

Thanks very much for your time and valuable comments on our work.

In the rebuttal, we have tried our best to address the concerns, and provided detailed responses to all your comments and questions. Would you mind checking our responses and confirming if there is any unclear point so that we could further clarify?

Best regards,

Authors of Submission 1367

Dear Reviewer HCvU,

Thanks very much for your time and valuable comments.

Thanks again for your time and valuable comments. We have tried our best to address the concerns. Specifically, we

• conclude a related discussion about the provided studies and cite them in our manuscript. (W1)
• discuss data depths in our related work and add it to our manuscript. (Q1)
• discuss isolation forest and clarify the difference between anomaly detection and OOD detection. (Q2)
• verify the performance of CoVer with extra comparison experiments with the introduced baselines. (Q3)

Is there any unclear point so that we should/could further clarify?

Thanks for your attention and best regards,

Authors of Submission 1367

Thank you for your time devoted to reviewing this paper and your constructive suggestions. Here are our detailed replies to your questions.

W1: About Novelty

We would like to reclaim that our work is novel in introducing a new perspective, i.e., expanding the dimension of input with corrupted variants, for OOD detection. This has not been explored in previous methods [1, 2], and has both conceptual and techniqual distinguishes from them with significant knowledge advancement.

Conceptually, we are the first to expand the raw input with the corrupted ones and identify the phenomenon of confidence mutation, and reveal the underlying mechanism with high-frequency feature change. The multiple input dimensions introduced in our framework is the critical advance compared with previous ones focusing on single-input.

Techniqually, CoVer captures the intrinsic differences of ID and OOD samples using the averaging operation, as founded to be simple yet highly effective and evidenced by our experiments in Table 1,2,3 of the original submission. As our idea provides a different perspective, the combination with these SOTA methods aims to demonstrating that our CoVer reveals the additional enhancement not covered by the basis. Similar to previous methods (ASH, DICE, ReAct) originate from Softmax/Energy, we believe our work brings new insights on this problem.

W2: About "extra data"

Thank you for the question. We would like to clarify that it is our novelty to use extra corrupted inputs to expand the input space, and it is not unfair. While previous OOD detection methods have distinguished between ID and OOD data based on a single input, we are the first to identify the challenges in this paradigm and propose expanding it to multiple inputs, measuring their confidences respectively. With these novel insights, CoVer provides a new perspective for the OOD detection problem.

W3: About using VLMs/CLIP

Thank you for the insightful comments.

First, we have conducted various experiments of CoVer on the DNNs architecture (ResNet50). The results demonstrate performance improvements and exhibit the same trend as those on VLMs (refer to Table 1 and Table 3 of the original submission).

Second, we would like to explain that the zero-shot settings in our work follow the MCM [3], which is the pioneering work in zero-shot OOD detection with VLM. As in MCM, the in-distribution classes in zero-shot OOD detection are the classification task of interest, which is defined by a set of class labels/names $Y_{in}$ instead of the classes used in pre-training. Accordingly, OOD is defined w.r.t. the ID classes, not the data distribution during pre-training. Hence, we only utilize the powerful image encoder to extract visual features and the text encoder to extract textual embeddings as ID concept prototypes, without the prior knowledge for distinguishing ID and OOD classes. We will add this part of explanation to our paper.

W4, Q2: About which corruption types or severity levels to use

Thanks for your constructive comments.

First, in all experiments, we use the SVHN dataset as the validation set to select the most effective corruption types for each method. We will clearly explain the usage of the validation set in our main paper.

For the types of corrupted inputs and their corresponding severity levels, we have conducted related explorations (e.g., Tables 10 and 11, Figures 6 and 7 of our original submission) for performance references. Some specific corruptions (e.g., Brightness, Fog, Contrast, Motion Blur, Defocus Blur) can generally improve the OOD detection performance, as the corruptions are mainly on the non-semantic level of the input, instead of damaging the semantic features too much like the other types.

Empirically, refer to Table 3 in attached PDF, we can use the same type of corruption as the expanded input (e.g., here Brightness with severity 1 used) to perform better than the original version. This provides the verification of the previous intuition about the general guidance for choosing appropriate corruption types, and understanding dimension expansion for OOD detection.

W5, Q4: About runtime

Thank you for your valuable question. Runtime is indeed an issue we didn't consider. As you mentioned, if there are $N$ expanded dimensions, it will take $N$​ times the duration of a single input to implement CoVer. However, our CoVer is only applied in the inference phase of OOD detection, and it is generally fast. As shown in Table 4 in attached PDF, we report the inference time of each single input on ID and OOD datasets. We will clearly discuss the runtime issue in the revised version.

W6: About failure cases

Thank you for the constructive comment. We indeed identified some failure cases and reported them in Figure 7 of the original submission. When CoVer utilizes certain severe corruption types (e.g., Spatter, Elastic transform), its performance is worse than with single input. This is because these types are more severe compared to others, leading to excessive damages to semantic features. Effective corruption types are those only perturb non-semantic features, which generally exist at the high-frequency level, resulting in different confidence variations between ID and OOD data. We will incorporate these discussions into the revised version and make it clearer with our empirical results.

References:

[1] Yang J, Wang P, Zou D, et al. Openood: Benchmarking generalized out-of-distribution detection. In NeurIPS, 2022.

[2] Yang J, Zhou K, Li Y, et al. Generalized out-of-distribution detection: A survey. In IJCV, 2024.

[3] Ming Y, Cai Z, Gu J, et al. Delving into out-of-distribution detection with vision-language representations. In NeurIPS, 2022.

W7, Q5, W8, Q1, Q3

Due to the space limit, we place our answer in the general response.

Dear Reviewer LFoY,

Thanks very much for your time and valuable comments on our work.

In the rebuttal, we have tried our best to address the concerns, and provided detailed responses to all your comments and questions. Would you mind checking our responses and confirming if there is any unclear point so that we could further clarify?

Best regards,

Authors of Submission 1367

Dear Reviewer LFoY,

Thanks again for your time and valuable comments. We have tried our best to address the concerns. Specifically, we

• reclaim our novelty from both conceptual and technical perspectives (W1);
• clarify the fair comparison in using CoVer's novel expanded inputs. (W2);
• discuss the rationale for using VLM/CLIP models in OOD detection experimentally and definitionally. (W3);
• clarify the usage of the validation set and provide general guidance for choosing appropriate corruption types empirically. (W4, Q2, Q3)
• discuss the runtime issue with extra experimental results. (W5)
• further clarify the failure cases in practice. (W6)
• verify the performance of CoVer on NINCO dataset with extra experiments. (W7, Q5)
• verify the performance of CoVer with comparison with NNGuide and MaxLogit methods. (W8)
• explain the reason for combining only with ASH and demonstrate the effectiveness of combining other methods empirically. (Q1)

Is there any unclear point so that we should/could further clarify?

Thanks for your attention and best regards,

Authors of Submission 1367

I thank the authors for taking time to submit a rebuttal and providing some clarity.

General Response (NINCO, NNGuide and MaxLogit, ASH)

Thanks for the comparison with more recent data and methods.

Using SVHN as a validation set is an interesting choice because the data is much smaller (32x32) than ImageNet (ID set) or any of the OOD datasets.

Knowledge Advancement

My comment should have indicated that the advancement in knowledge is not as significant, rather than suggesting that the idea itself is weak. CoVer requires [1] (corruptions and severity levels) and must be applied to an existing OOD method. Other work [2] has utilized [1] in OOD detection, but simply as an attack. The contribution in terms of knowledge advancement or new perspective is using a normal image in conjunction with a corrupted image/s for post-hoc OOD detection enhancement by averages scores.

Utilizing CoVer would require applying it to every method for a fair comparison rather than using it as a standalone technique, as it involves more than a single input. This isn't necessarily bad or wrong, but it is an important consideration when determining its impact. Its application does not alter performance rankings, and one would generally expect performance to improve with the use of multiple inputs.

Extra data

I believe this should be listed as a limitation because, for CoVer to be effective, it requires more than one input compared to other methods. Similar to how OpenOOD distinguishes between "Training Methods" and "Training With Extra Data," CoVer would need to be categorized as "Post-hoc Methods With Extra Data." This is why I mentioned that Table 1 seemed a bit unfair, as it compares methods using one input vs. two or more (CoVer).

VLMs/CLIP

I am aware how MCM and OOD are defined, but my point is an image encoder coming from a VLM/CLIP model is of course going to be better because it has seen more data than one with only ImageNet-1K. It is an advantage that should be noted.

Corruption types / severity levels

The variability I mentioned arises because the experiments used different corruption types and/or severity levels for nearly every method. This makes comparisons less discernible. Also, it introduces variability depending on the architecture, dataset, and other factors.

The attached PDF demonstrates that using a single corruption type along with a normal input improves AUROC and FPR95 alone.

Runtime

Thank you for attaching the runtime table. My concern is that if one were to use CoVer and explore corruption types and severity levels across different architectures and their own data, it could become time-consuming due to the amount of variability involved.

Failure cases

This was my oversight, as I missed Figure 7. I was simply wondering in what situations the use of CoVer might not make sense and a single input would be preferred.

References:

[1] Hendrycks, D., & Dietterich, T. (2018). Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. In International Conference on Learning Representations.

[2] Chen, J., Li, Y., Wu, X., Liang, Y., & Jha, S. (2021). ATOM: Robustifying Out-of-distribution Detection Using Outlier Mining. In Machine Learning and Knowledge Discovery in Databases.