Out-of-Distribution Detection by Leveraging Between-Layer Transformation Smoothness

Thank you for your constructive and helpful comments. We are glad you took notice of the novelty of our work and the usefulness of the estimator we proposed. We will be addressing your comments below in the order they appear in the review:

1) CV experiments - Please see our General Response and Updates to the Paper.

2) More SOTA baselines - Please see our General Response and Updates to the Paper.

3) Explanation of difference in smoothness - On the operational level, BLOOD works because the learning algorithm learns the smooth transitions for the data it was trained on, while the rest of the representation space is either left largely unchanged (if the task was simple enough) or sharpened as the byproduct of trying to fit the more complex data, as we showed in the paper. We introduced the intuition and motivation for our method in subsection 3.2. Our intuition is based on the phenomenon known from the literature that different Transformer layers model features at different levels of abstraction. Since the model learns on the ID training data, it should learn the natural progression between the layers of abstractions inherent to the task, i.e., ID data. However, the unseen OOD data will likely have different features in a given level of abstraction than the ID data since the nature of the data is different, so the model is not familiar with them or their progression. Because the model has not learned how to handle the mismatch in the transition between the levels of abstraction, it exhibits sharper between-layer transformations of those representations, i.e., there is a more considerable difference between how OOD representations get mapped onto the next layer and how its infinitesimally small neighborhood gets mapped.

We hope you find our response helpful. If there are any other questions you would like to ask, we would be glad to answer them.

Thank you for showing interest in our paper with many thought-provoking questions. We are glad you enjoyed the thorough analysis of our method and our findings on smoothness in DNNs. We respond to your questions and comments in order of how they appear in the review:

1) There are scenarios in which BLOOD is outperformed by other OOD methods - It is true that in certain scenarios BLOOD is outperformed by other popular OOD detection methods, e.g., for simpler training datasets or semantic shifts. However, the performance boosts in the scenarios suitable for BLOOD are significant (e.g., 12.09 percentage points AUROC better than the second-best white-box method for BigPatent dataset with ELECTRA). That is precisely why we carried out an extensive analysis of when BLOOD outperforms other methods and why it outperforms them when it does. Researchers and engineers must be able to make an educated choice when deciding on the suitable OOD detection method for their problem, and that is why we believe it is essential to showcase both the strengths and limitations of our work.

2) Table 2 contradicts Tabe 1 - No, that is as expected. A lower CLES implies a higher AUROC because the learning algorithm has sharpened the OOD region of representation space along with the smoothening of the ID region instead of just smoothening the ID region. That emphasizes the difference in smoothness between ID and OOD data. And since the learning algorithm had to “invest more work” in the last layers (as seen in Table 1), it sharpened the OOD region of representation space as a byproduct, leading to lower CLES for the last layer and higher AUROC when using the last layer. The reasoning behind the CLES of lower vs. higher layers is parallel to the reasoning behind the CLES of simpler vs. more complex datasets. If a dataset is more complex, the learning algorithm changes the model more during training, which, instead of just focusing on the ID region of the representation space, also makes changes to the OOD region of the representation space as a byproduct. Similarly, in higher layers, which are more influenced by the learning algorithm, there are more changes in the OOD region of space.

Because of that, the only instances where CLES in the last layer is larger than the CLES_mean for RoBERTa is for theMG and NG datasets, which are exactly the datasets on which BLOOD_{mean} outperforms BLOOD_{L-1} on RoBERTa.

3) Relationship between BLOOD_mean and BLOOD_{L-1} - Please see our General Response and Updates to the Paper.

4) BLOOD in an open-box setting - While we focused on the white-box scenarios because of the wider useability of our method, we are thankful for this great suggestion. We experimented with extending BLOOD by using a small sample of OOD examples to learn the optimal parameters for OOD detection (as is done by many popular OOD methods), e.g., learn the optimal weights for the weighted average of scores in different layers. We used a validation set of 5% of original training sets and fitted a logistic regression to learn the optima weights for a weighted sum of BLOOD values in different layers. This approach generally outperforms the open-box versions of our method. We provide results (AUROC) of this experiment below. We will be adding the results in the Appendix of our paper:

Thank you for your time and effort in reviewing our paper. We appreciate you finding our paper well-written and our results strong. All other reviewers shared your suggestions, so we acknowledged them in our General Response and Updates to the paper. We provide the summary of our answers below, but please see General Response for more detailed answers:

1) CV experiments - BLOOD is competitive and all of the conclusions of our paper hold when applied to vision datasets paired with Vision Transformer. However, when paired with CNNs, BLOOD behaves differently, likely due to differences in how the two architectures process data.

2) More SOTA baselines - We expanded our experiments with the suggested methods (ASH and ReAct), however those methods did not retain their SOTA status on the text classification task.

3) Relationship between BLOOD_mean and BLOOD_{L-1} - BLOOD_mean introduces noise to the method because it includes BLOOD scores from the lower layer that did not change much during training and thus retain their BLOOD scores from the pre-trained models where there is no difference between ID and OOD data. And since BLOOD scores in the low layers have higher magnitudes, they take over the average. However, for datasets MG and NG on RoBERTa, BLOOD_mean works better than BLOOD_{L-1}. The reasoning behind that is the same (BLOOD scores are kept the same as before training in low layers), but in these scenarios, there exists a priori difference in BLOOD score between ID and OOD data, as seen from Figures 9 and 10.

We hope you find our answers helpful. If you have any other questions, we would be happy to hear them.

Thank you for your constructive comments and questions, we are glad you find our work important. We address your questions below, as introduced in your review:

1) Insights on why BLOOD works better on complex datasets - In subsection 4.4, we test the hypothesis that BLOOD works better on simpler datasets by simplifying the more complex datasets. In that section, we show that reducing the dataset's complexity leads to an increase in CLES of the change in representations. This implies that the model can fit the data by only focusing on the parts of the representation spaces populated by the ID training data. However, for more complex datasets, the model has to undergo more radical changes during training to fit the data, sharpening the OOD part of the representation space as the byproduct. Because BLOOD compares the smoothness of the ID region and OOD region of the representation space, sharpening the OOD region of the space, along with smoothening the ID region of the space, emphasizes the difference in smoothness.

2) Acknowledge the "Estimating example difficulty using variance of gradients (VoG)" - Thank you for drawing our attention to this paper. It is an exciting work with slight similarities to our work. However, there are also crucial differences. One conceptual difference is that VoG considers the gradient of the inputs w.r.t. the logit of the true/predicted class (measuring how much the model's predictions vary with small changes in the input), while BLOOD is analyzing the Jacobians of neighboring intermediate layers (quantifying the stability of the intermediate representations during inference). Another practical difference is that VoG calculates the variance of the gradients – for which it needs the gradients at different timesteps during training. VoG requiring access to the model's training process makes it an open-box method, while BLOOD is a post-hoc white-box method, allowing it to be more widely used.

3) CV experiments - Please see our General Response and Updates to the Paper.

We hope we have addressed all of your questions. If there is anything else you would like to know about our work, please do not hesitate to ask.

It can be seen from the results that our conclusions are retained when BLOOD is applied to Vision Transformer (ViT), BLOOD_{L-1} works better than BLOOD_mean and BLOOD performs better on more complex cifar100 than on simpler dataset cifar10. However, it seems that BLOOD does not retain its effectiveness when paired with CNNs, i.e., ResNet34 and ResNet50. Also, for CNNs, BLOOD_mean works fine for detecting OOD data, while BLOOD_{L-1} is not discriminative between OOD and ID data. For more details, please see Appendix D of our updated paper.

There are inherent differences in how different architectures process data, which impact the availability of different OOD detection methods. In our experiments and the experiments in [1], the best performing OOD methods for Transformer-based language models are distance-based methods (e.g., KNN and Mahalanobis distance), while methods that are SOTA on CNNs (ReAct, ASH, DICE) often underperform even against the simple baselines. In [1], DICE is by far the worst-performing OOD detection method for Transformer-based LLMs. Conversely, in the original paper of ASH [2], the authors did a comprehensive OOD detection experiment on CNNs, which showed Mahalanobis distance by far underperforming compared to other methods. In light of the finding that the choice of architecture plays a crucial role in the success of the OOD detection method, we updated our paper to reflect that Transformers are the scope of our work.

3) Relationship between BLOOD_mean and BLOOD_{L-1} - Reviewers R2 and R3 were interested in the reasoning behind BLOOD-mean being the worst choice most of the time, except in a few instances. Firstly, let us explain why BLOOD_mean underperforms (especially for RoBERTa). Figure 1 and similar figures in the Appendix for other datasets (Figures 4-10) show how BLOOD changes across layers. Comparing the subfigures it can be seen that since there is almost no change after training in the lower layers, the BLOOD in those layers stays the same as before training, where there was no difference between the ID and OOD data (because the model has not yet seen the ID data). However, in those lower layers for RoBERTa the BLOOD scores have larger magnitudes than in higher layers where the difference in BLOOD of ID and OOD data is prominent. Thus, by averaging the BLOOD scores of different layers the noisy lower layers dominate the informative higher layer and make BLOOD less discriminative between ID and OOD data. Since for ELECTRA the higher and more informative layers have higher magnitudes of BLOOD score, this effect is less noticeable. However, the reviewers have noticed that in some cases, e.g., on MG and NG datasets with RoBERTa, BLOOD_mean archives better results. We can look at Figures 9 and 10 to explain this effect. Looking at subfigures (c) it can be seen that there is a difference in BLOOD scores between ID and OOD data across layers even before fine-tuning on the ID data, likely due to inherent differences in ID and OOD data (we speculate it could be due to the length of documents, as the longest documents from our experiments come from MG and NG datasets). It can be seen that in low layers OOD data has higher BLOOD scores than ID data, and that difference is retained even after fine-tuning, as those layers are not changed much. Thus, including the lower layers helps discriminate between ID and OOD data. However, we would like to note that it is still better and more consistent to use BLOOD_{L-1} than BLOOD_mean, since this difference in the earlier layers could have gone the other way around and impact the performance of BLOOD_mean in a negative way.

We have addressed all other comments in individual responses to each reviewer. The reviews have significantly improved our paper, for which we are thankful.

*As abbreviations, we refer to reviewers Txyj as R1, Bnog as R2, woCv as R3, and ELpL as R4 respectively.

[1] Mateusz Baran, Joanna Baran, Mateusz Wójcik, Maciej Zięba, and Adam Gonczarek. 2023. Classical Out-of-Distribution Detection Methods Benchmark in Text Classification Tasks. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics

[2] Andrija Djurisic, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. Extremely simple activation shaping for out-of-distribution detection. In The Eleventh International Conference on Learning Representations, ICLR 2023