GROD: Enhancing Generalization of Transformer with Out-of-Distribution Detection

The experiments are insufficient to prove that GROD achieves SOTA performance. Since the authors leverage the OpenOOD benchmark, more far-OOD datasets, such as Textures, Places-365, and MNIST, can be tested to validate GROD's performance.

In fact, we have conducted experiments of Texture, as OOD, and we have not reported these results just because far-OOD is simple to detect for GROD and other competitive models, and shows similar performance for method comparison, so we only remain SVHN as a representation of far-OOD. Here we offer the results on other far-OOD datasets compared with SVHN in Table 2 and Table 3, respectively:

We are open to adding these experimental results to the appendix, if you recommend this change.

Besides, it is sorry that we have report PCA only condition when using CIFAR-100 as ID in Table 3, but $\rm min \{ |\hat{I}|,[\frac{2B}{K\cdot num}]\}$ can be $1$ in some cases, and we updated the results for PCA-LDA GROD: ID ACC=86.10%, and OOD detection metrics (%):

It achieves SOTA when CIFAR-100 as ID (Table 3) even only adds one class of inter-class OOD, and we will update the results in the revised version.

We hope that the improvements we have made sufficiently address your comments. If you feel the revised version meets your expectations, we would be truly grateful if this could be reflected in the updated score. Please let us know if there are any remaining questions or points you would like us to clarify further.

Yours sincerely,

Authors

Dear Reviewer PBKN,

Thank you for recognizing our theoretical contribution and experimental validation of GROD. Meanwhile, we understand the reviewer's concerns, which are also very important to us. Below we clarify.

Why do the maximum and minimum values projected by PCA and LDA are considered boundary points? Further analysis of the intrinsic mechanism of PCA and LDA is needed.

PCA aims to find the directions of maximum variance in the data by projecting the data onto principal components. LDA, on the other hand, seeks to maximize the separation between classes by projecting the data onto a lower-dimensional space where the class separability is maximized. The maximum and minimum projected values represent the extreme values in the dataset along those representative directions globally or inter-class, and the extension can be considered as boundary points because they lie at the extremes of the decision boundaries in the transformed space, and any point beyond these boundaries would be considered as an OOD sample.

As claimed in line 149, LDA is selected to guarantee the robustness of generated OOD, but it is only utilized when the number of ID classes is small as defined in Equation (4). Does this mean that the generated OOD samples are not robust with large-scale ID datasets?

LDA's instability is not due to the generation method itself, but rather because, when the dataset has many classes, the number of samples per class in each batch can be too small. In contrast, PCA is more stable since it considers the global data distribution. As a result, we use Equation (4) to control the selection between LDA and PCA methods. Furthermore, our experimental results in Table 2/3 show that even when PCA is used alone, the performance remains highly competitive. This demonstrates that PCA alone is robust and more effective for the simpler process, even in the absence of LDA, especially in large-scale datasets with many ID classes.z

The baseline NPOS adopts a similar OOD synthesis pipeline, which first identifies boundary ID samples and then generates OOD samples via Gaussian sampling. The superiority of GROD against NPOS should be explicitly stated and the generated OOD samples of the two methods can be statistically compared to further distinguish GROD.

To the best of our knowledge, the ``gold standard" for measuring the quality of synthetic OOD data has not been proposed. And compared to task performance in Table 2/3/4/5, GROD is superior. In Table 6 of our revised paper, we test changing the generating method to Gaussian and randomly uniform noise, which further shows the effect of GROD:

Add to Table 6 in the paper, ID: CIFAR-10 Gaussian outliers: ID ACC: 96.86%

Uniformly random outliers: ID ACC: 96.67%

Also, we have visualized the generated outliers together with ID features in Fig. 6 in Appendix K, which shows GROD can generate high-quality outliers with LDA-generated outliers rounded each ID class, consistent with our algorithm design.

The notations are confusing, e.g., line 144 indicates the feature space is $\mathbb{R}^{n \times s}$, however, line 168 defines another $n$.

Thanks for pointing out our mistakes in the details of the notation! We have updated this notation with the revised version. Specifically, we changed $n$ in line 144 to $\kappa$.

Dear authors,

Thanks for your timely responses. The responses don't sufficiently address my concerns. The theoretical insight of why the maximum value of each dimension in the projected space is not provided. The ablation study on $num$ displays that GROD begins to degrade when $num$ scales up, which may conversely suggest the way that boundary samples are identified is unreasonable. As for the LDA, did the authors try to cache the ID features so that there are enough samples for each class, just like what NPOS does? The empirical results remain insufficient as results on the baselines are not provided.

I also looked through your discussions with other reviewers and found that using a backbone supervised on ImageNet-1K serves as an unfair setting as ImageNet-1K contains subclasses of "dogs" or so.

As not all concerns are sufficiently addressed, I will keep my ratings.

Best,

Reviewer PBKN

Thank you for your engagement and insightful feedback regarding our manuscript. We appreciate the opportunity to further clarify the points you have raised about the theoretical aspects and empirical performance of the GROD algorithm. Please find below our responses to each of the comments:

The theoretical insight of why the maximum value of each dimension in the projected space is not provided.

In the GROD methodology, different from your understanding, we select the maximum value of each dimension in the projected space and control the number of projected dimensions and OOD clusters by $num$. It is a deliberate choice aimed at generating representative OOD data. These maxima, identified through LDA and PCA projections, are crucial for creating synthetic outliers that embody the extremities of the feature space. This method ensures that the generated OOD samples are not merely random but are characteristic of likely outliers, providing a robust challenge to the OOD detection capabilities of the model.

The ablation study on displays that GROD begins to degrade when scales up, which may conversely suggest the way that boundary samples are identified is unreasonable.

The degradation observed when the parameter num (number of synthetic OOD samples) scales up can be attributed to two primary factors:

• Increase in Projection Dimensionality: As the dimensionality of the projection space increases, the selection of dimensions for generating OOD data becomes more random, which could dilute the specificity of the OOD samples. This supports the importance of selecting specific dimensions for their representative value.

• Ratio of In-Distribution to Out-of-Distribution Data: The second factor is the relatively low ratio of in-distribution to out-of-distribution data. Prior research, referenced as [1], indicates that effective OOD detection can be achieved with only a small proportion of OOD data, aligning with our findings that too many OOD samples may not be necessary and could even be counterproductive.

As for the LDA, did the authors try to cache the ID features so that there are enough samples for each class, just like what NPOS does?

Unlike the NPOS method, GROD does not cache ID features to generate OOD samples for each class statically. Instead, we dynamically determine appropriate clusters for generating inter-class OOD data during training whenever $|I|>0$, allowing for a more flexible and responsive approach to OOD sample generation based on the data observed during training.

To further validate our approach, we have conducted ablation studies to display the effects of randomly selecting dimensions for generating OOD data:

OOD generated from extreme values ​​of one random dimension: ID ACC: 97.47%

The performance of this generation method is not as effective as the dimension reduction using PCA and LDA in GROD, but it is superior to generating OOD data using Gaussian and uniform distribution sampling (Table 6 "G", "U" in the revised version). This further demonstrates the efficacy of our approach. We are prepared to include these results in Table 6 of the revised version if you consider it beneficial for enhancing the paper's comprehensiveness.

I also looked through your discussions with other reviewers and found that using a backbone supervised on ImageNet-1K serves as an unfair setting as ImageNet-1K contains subclasses of "dogs" or so.

Fair Comparison Across CV Tasks: For CV tasks (Table 2/3/4), GROD and all other baseline methods utilize the same pre-trained models, allowing for a fair comparison of GROD's effectiveness. This demonstrates that the improved results in OOD detection are likely due to the novel GROD algorithm rather than the characteristics of the supervised pre-trained data alone.

Use of Unsupervised Pre-trained Models in NLP: For NLP tasks, both BERT and GPT-2 (Table 5 in the revised version), which were utilized in experiments, are unsupervised pre-trained models. These two pre-trained models fully meet the two conditions you mentioned: unsupervised training and the absence of features with similar semantics during the training phase. The strong performance of GROD even with unsupervised models reinforces the argument that GROD's methodology is effective across different pre-training paradigms.

We value your feedback as it substantially contributes to improving the quality and rigor of our research, and we hope these explanations address your concerns and clarify the methodologies and decisions underpinning our work with the GROD algorithm.

Reference:

[1] Fort S, Ren J, Lakshminarayanan B. Exploring the limits of out-of-distribution detection[J]. Advances in Neural Information Processing Systems, 2021, 34: 7068-7081.

Thank you for your insightful comments and continued engagement with our manuscript. We appreciate your thoughtful suggestions and have addressed them as outlined below.

Regarding the backbone network, I'll be more convinced if the empirical comparison between GROD and the baselines with unsupervised backbones or simply pretrained on the ID dataset (e.g., CIFAR-10) is conducted. However, as there is not much time left for the discussion period, I don't expect the authors to provide such empirical results.

As previously noted, we have conducted fair comparisons between GROD and other methods, including empirical experiments on BERT and GPT-2 pretrained on unsupervised datasets. We acknowledge the time constraints of the rebuttal period and understand your position regarding the provision of additional empirical results at this stage. However, we insist that our comparison results (Table 2/3/4/5 and Fig. 2) can show the advantage of GROD.

As for the way that the outliers are synthesized, it adopts a similar pipeline as VOS or NPOS, which may undermine the novelty of the proposed method for there is no statistical or theoretical analysis to distinguish GROD from prior works on the quality/utility of the synthesized outliers.

We recognize your concerns regarding the novelty of GROD compared to VOS and NPOS. To address this, we highlight several innovative aspects of GROD, including:

• Ensuring interpretability safeguards by developing OOD detection learning theory on transformers.

• The novel introduction of Mahalanobis distance filtering methods and loss function.

• Distinct methodologies for modeling OOD data.

• Enhanced performance on downstream tasks and accelerated processing speed. (VOS is already verified to be inferior compared withNPOS according to [1], and NPOS is already tested.)

Additionally, we have conducted experiments to show the performance of GROD and analyze the efficiency of our generating methods. As shown in Table 2/3/4/5 and Fig. 2, extensive and fair comparative experiments demonstrate GROD's effectiveness and computational costs compared with NPOS and other methods, including 5 ID datasets and 3 backbones. Furthermore, Table 6 provides a comprehensive ablation study of GROD's modules and the mechanisms for generating OOD data, and Fig. 6 in Appendix K shows the t-SNE visualization of the generated OOD data.

In the field of synthetic OOD data generation, there is no established ``gold standard". That is why no statistical or theoretical analysis exists in GROD or any other methods like VOS, or NPOS as well regarding the theoretical quality of outliers. The basic process is to verify the performance of downstream tasks and provide intuitive visualization, which has been completed in our manuscript. As such, each exploration and discovery contributes valuable insights to our collective understanding.

We sincerely hope that our responses have clarified the concerns raised, and thank you once again for your detailed review and consideration.

[1] Tao et al., "Non-Parametric Outlier Synthesis," ICLR 2023.

Dear Reviewer fp5B,

Thanks for your constructive and valuable comments on our paper! We are encouraged by your recognition of our theoretical contribution and the computational efficiency of GROD. To address your concern, below we show our explanations and supplementary experiments following your suggestion:

Baseline Comparison with Outlier Exposure (OE): For outlier exposure (OE) methods like [1, 2], our OOD detection is under the assumption that outliers can not be seen in the training phase, which is another dev training with extra data in the generalized OOD detection task and also has its real application scenarios [a]. To make our paper more clearly, we added this sentence into our paper in Section 3: It is notable that we focus on OOD detection without training with outlier datasets, which is another dev different from OE and also has its real application scenarios [a]. We test several OE methods as the table below, but we insist that comparing OOD detection methods with OE is not fair for the comparison of different information from inputs. For convenience and the standard benchmark guarantee, we choose a classical OE method [1] and a prevalent method MIXOE [b] for CV tasks, which are incorporated in the OpenOOD benchmark. NLP OE datasets for CLINC-150 and IMDB have not been well-defined. We will add these results in the main part of our paper if you should. Besides, we will introduce OE methods in our revised manuscript in Section 2 and Appendix A.

OE: ID: CIFAR-10, Auxiliary OOD: Tiny ImageNet-597, ID ACC=95.70%

ID: CIFAR-100, Auxiliary OOD: Tiny ImageNet-597, ID ACC=74.97%

ID: ImageNet-200, ID ACC=89.48%

MIXOE: ID: CIFAR-10, Auxiliary OOD: Tin597, ID ACC=96.47%

ID: CIFAR-100, Auxiliary OOD: Tin597, ID ACC=77.84%

ID: ImageNet-200, ID ACC=90.49%

Evaluation of Synthetic OoD Data: Thanks for your suggestion. We think it is meaningful to further explore the generation strategy of GROD, and have complemented experiments in the ablation study in Table 6 in the revised version. For detail, the results are as the following table:

Add to Table 6 in the paper, ID: CIFAR-10 Gaussian outliers: ID ACC: 96.86%

Uniformly random outliers: ID ACC: 96.67%

We add these results in Table 6 of the revised version.

References:

[a] Yang J, Zhou K, Li Y, et al. Generalized out-of-distribution detection: A survey[J]. International Journal of Computer Vision, 2024: 1-28.

[b] Zhang J, Inkawhich N, Linderman R, et al. Mixture outlier exposure: Towards out-of-distribution detection in fine-grained environments[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2023: 5531-5540.

We appreciate your willingness to increase the score, and we hope that the clarifications and improvements we have provided align with your expectations. Should you need any further information, please do not hesitate to reach out.

Yours sincerely,

Authors

Dear Reviewer Hak4,

We are grateful for your insightful comments and feedback. We appreciate that you agree with our theoretical insights and thorough experimental validation. To address your concerns, we have provided comments as follows.

The GROD algorithm involves several hyperparameters (e.g., the scaling parameter for Mahalanobis distance, LDA cluster dimensions) that require fine-tuning.

Our method introduces three hyperparameters $\alpha$, $num$, and $\gamma$. $num=1,2$ is empirically an optimal choice, which is consistent with the conclusion in [a] that even adding one or two OOD can raise the OOD detection performance of transformers. The ablation results regarding $\gamma$ in Fig. 3 show that $\gamma \in [0.1,0.3]$ benefits the task performance, which is also in line with the theoretical insights and the classification (learned by $\mathcal{L}_1$) and OOD detection (learned by $\mathcal{L}_2$) goal of the task. Therefore, $num$ and $\gamma$ have their optimal solution, which would be clearly stated in the camera-ready version. As to $\alpha$, if $LDA$ is used, we recommend $\alpha=10^{-3}$, otherwise a larger value should be taken. In Subsection 4.3, we have analyzed in detail why these parameters are set like this, and give an explanation from the perspective of OOD detection learning theory.

[a] Fort S, Ren J, Lakshminarayanan B. Exploring the limits of out-of-distribution detection[J]. Advances in Neural Information Processing Systems, 2021, 34: 7068-7081.

While GROD achieves a balance between computational efficiency and performance, its iterative processes, including OOD data synthesis and Mahalanobis distance calculation, may not scale well with significantly larger datasets (e.g., ImageNet) or models. This limitation could restrict its deployment in real-time applications where processing speed is crucial.

Firstly, we have shown the computational efficiency and performance when using the large-scale dataset ImageNet-200 for comparison (Table 4 and Fig. 2), in which GROD gains superior performance. With the design of $|I|$, we achieve a balance between the complexity of OOD generation and the class number of datasets. Meanwhile, the speed of GROD is only related to the dimensions of features in the last layer of transformers, which is independent of the total scale of models.

There are missing citations in the manuscript. For example, the paper introduces generative models, but generative-based methods [1, 2, 3] are missed without corresponding details in the bibliography.

Thank you for pointing out the missing citations in the manuscript. GROD is not a classical generation method for it only generates OOD in the embedding feature space, but for the completeness of our work, we will add generative-based methods into related works in the revised manuscript which can be seen in the updated PDF in Section 2 and Appendix A.

"Learnability" is repeatedly used without a concise definition in layman’s terms, which could be clarified for a broader audience.

We give Definition 1. (Strong learnability) in Appendix B, and with your advice, we will add a qualitative description for "Learnability in OOD detection" in Section 3: "A model is considered learnable for OOD detection if, when trained on sufficient ID data, it is capable of distinguishing OOD samples from ID samples without compromising its classification performance.'".

How do the authors interpret the performance of LDA-based inter-class OOD generation in enriching OOD representation? More specifically, what are the primary limitations observed when using PCA-only projections, and how might these affect model robustness?

PCA-only projections see ID data as one class, so outliers generated by the PCA-only method extract less information compared with LDA-based inter-class OOD. PCA-only is robust, instead, if using LDA when the LDA-based outliers are computed to be none, the generation would be unstable because the number of each class of ID is too little to generate stable rounded outliers. Besides, it is sorry that we have report PCA only condition when using CIFAR-100 as ID in Table 3, but $\rm min \{ |\hat{I}|,[\frac{2B}{K\cdot num}]\}$ can be $1$ in some cases, and we updated the results as GROD in the transition mode:

ID ACC=86.10%, and OOD detection metrics (%):

It achieves SOTA when CIFAR-100 as ID (Table 3) even only adds one class of inter-class OOD, and we will update the results in the revised version.

We sincerely appreciate your positive feedback and recognition of the novelty of our work in the OOD detection community! If any of our responses fail to address your concerns sufficiently, please inform us, and we will promptly follow up.

Yours sincerely,

Authors

Dear Reviewer PTbR,

Thanks for your constructive and valuable comments on our paper, and for recognizing the contributions of learning theory and generalization capabilities of GROD in our paper. Below, we provide a detailed response to your comments.

Weakness 1: The pretraining on ImageNet-1K is primarily aimed at providing a good initialization for backbone networks. This pretraining sets up a solid foundation for the model, which can then be fine-tuned for the specific task of OOD detection. The overlap between the classes in the ImageNet-1K dataset (used for pretraining) and the OOD test set (such as CIFAR) is not an issue, because we utilize the well-trained features but do not recognize them as ID and OOD at the same time. All methods we compared use pre-trained parameters so that it is fair for comparison. Results demonstrate that the GROD method, through fine-tuning, effectively tightens the decision boundaries between the ID and OOD samples.

Weakness 2: It is a helpful suggestion and we will add a discussion on potential adaptations", which would be added to our paper as follows: As defined in Definition 4. (Transformer hypothesis space for OOD detection), GROD gains theoretical guarantee on transformers with multiple transformer layers and a classifier for OOD detection and classification tasks. So GROD has compatibility for transformers extract features like CLS tokens and inputs into the classifier, which is appliable to almost all transformer models. The OOD detection task in our paper is based on classification, and to achieve this goal, most transformer-based models could output features for classification so that these features can be used in GROD.

Weakness 3: We have conducted experiments with two backbones ViT-B-16 and BERT. To address the concern, we added another backbone network GPT-2 and the results are shown in the following table. Part of the methods (Baseline, GEN) is compatible for both models with CLS tokens as features and without CLS tokens, as they only need logits for processing. Others are only compatible with models with CLS tokens as features, since they utilize features and logits together. We test two modes (with/without CLS token) for the former, denoted as Method-C (with CLS) and Method-L (without CLS), respectively.

ID: CLINC150 with Intents, OOD: CLINC150 with Unknown Intents

ID: IMDB, OOD: Yelp

Weakness 4: We have derived a thorough learning theory to guarantee the OOD detection learnability on transformers, which may not be applicable to MLP-based and CNN-based models, unless the same learning theory is established. So we focus on transformers to validate the efficiency of GROD, which is more theoretically explainable and connects closely to our theoretical results.

We would like to thank you again for this in-depth review. If our clarifications and improvements address your concerns, we would greatly appreciate it if you could reconsider your evaluation. Should you have any additional questions or suggestions, we would be happy to provide further details.

Yours sincerely,

Authors

Thanks for the rebuttal, the rebuttal does not effectively address my concerns. The use of a supervised pre-trained model, particularly when there is an overlap between the pre-trained data and the OOD test data, raises significant concerns regarding the validity of the study's findings. The network's ability to extract representations from OOD data that it has been previously exposed to during pre-training undermines the challenge of distinguishing between in-distribution and OOD samples. This overlap effectively simplifies the task and may lead to the reporting of artificially inflated performance metrics. While some recent studies, such as NPOS, have employed pre-trained models, they have utilized unsupervised pre-trained models like CLIP and have also reported results from models trained from scratch. This approach provides a more robust evaluation of the model's ability to generalize to OOD data. It is suspected that the improvements in OOD detection reported for GROD may be attributable to the specific characteristics of the supervised pre-trained data rather than the model's inherent ability to detect OOD samples.

Thanks for your reply. Further clarification on fair comparison in CV tasks is not solid, as there is insufficient motivation to use supervised pretrained models. Therefore, I decide to keep my rate.

Weakness (D): We have read all the papers referenced in your comments. Although they are really excellent methods for OOD detection tasks, several reasons make us not choose them for comparison experiments. For [1], different from your description, we find it is not an outlier synthesis method. For outlier exposure (OE) methods like [2, 6], they use outliers from other datasets but do not generate them. And our OOD detection is under the assumption that outliers can not be seen in the training phase, which is another dev training with extra data in the generalized OOD detection tasks and also has its real application scenarios [a]. To make our paper more clear, we added this sentence into our paper in Section 3: It is notable that we focus on OOD detection without training with outlier datasets, which is another dev different from OE and also has its real application scenarios [a]. [3] is already verified to be inferior compared with [4] according to [4], and [4] is already tested in Table 2/3/4 as the method "NPOS". [5] uses ResNet as the backbone and seems compatible with transformers. However, it is highlighted as a way to generate high-resolution outliers in the pixel space, which seems designed especially for CV tasks.

Above all, we test several OE methods as the table below, but comparing OOD detection methods with OE is not fair for comparison of different information from inputs. For convenience and the standard benchmark guarantee, we choose a classical OE method [b] and a prevalent method MIXOE [c] for CV tasks as shown in Table 2/3/4, which are incorporated in the OpenOOD benchmark. NLP OE datasets for CLINC-150 and IMDB have not been well-defined. We will add these results in the main part of our paper if you should. Besides, we will introduce, discuss and distinguish the related concepts of [1-8] in our revised manuscript in Section 2 and Appendix A.

OE: ID: CIFAR-10, Auxiliary OOD: Tin597, ID ACC=95.70%

ID: CIFAR-100, Auxiliary OOD: Tin597, ID ACC=74.97%

ID: ImageNet-200, ID ACC=89.48%

MIXOE: ID: CIFAR-10, Auxiliary OOD: Tin597, ID ACC=96.47%

ID: CIFAR-100, Auxiliary OOD: Tin597, ID ACC=77.84%

ID: ImageNet-200, ID ACC=90.49%

Weakness (E): Our paper is different from [7] for the following reasons. Firstly, [7] mainly solves the novelty detection (ND) task, not for OOD detection. Different goals: While the detected novel samples are usually prepared for future constructive procedures, such as more specialized analysis, or incremental learning of the model itself in ND, OOD detection aims to detect outliers for the safety and generalization capability of models. [7] emphasize the quality of outliers generated by diffusion models, but GROD only needs to synthesize them in a high-dimensional embedding space of transformers to strengthen decision boundaries, which is lightweight and suitable for OOD detection. Different pipelines: ND is only to distinguish novel samples from ID, while OOD detection serves classification performance. Different novelty: [7] verify that diffusion models can generate novel outlier samples and benefit ND, while GROD is for OOD detection and is inspired and guaranteed by the proposed OOD detection learning theory.

Weakness (F): GROD is a fine-tuning and post-processing method that leverages information extracted by pretraining, to get an initial performance for classification. By fine-tuning with GROD, we further strengthen classification and OOD detection capability, which has been verified in the ablation study in Table 6.

References:

[a] Yang J, Zhou K, Li Y, et al. Generalized out-of-distribution detection: A survey[J]. International Journal of Computer Vision, 2024: 1-28.

[b] Hendrycks D, Mazeika M, Dietterich T. Deep anomaly detection with outlier exposure[J]. arXiv preprint arXiv:1812.04606, 2018.

[c] Zhang J, Inkawhich N, Linderman R, et al. Mixture outlier exposure: Towards out-of-distribution detection in fine-grained environments[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2023: 5531-5540.

Question 1: Our paper first proposes the learning theory and a theoretical guaranteed method for transformer-based backbones. [8] uses real auxiliary outlier data for training, which is another topic in OOD detection as explained in weakness (D). Besides, [8] has not achieved competitive performance on CIFAR10 vs. CIFAR100, instead, it uses CIFAR-10 or CIFAR-100 as ID and far-ood datasets like SVHN as OOD, so results are not comparable. Validated mainly using DensNet, [8] has no evidence to show if it needs pre-trained parameters when using transformers.

Question 2: Please see comments for weaknesses (D) and (E).

Question 3: Please see comments for weakness (B).

Question 4: We follow the supported benchmarks provided by OpenOOD, which defines Tiny ImageNet as near-OOD for CIFAR-10 and CIFAR-100.

Thank you again for your valuable suggestions and constructive feedback! If we have adequately addressed your concerns, we kindly ask for your consideration in enhancing the score you've allotted to our submission. Otherwise, we would be glad to answer any additional questions you may have.

Yours sincerely,

Authors

Dear Reviewer zQzi,

Thanks for your constructive and valuable comments on our paper. Below, we provide a detailed response to your questions and comments. If any of our responses fail to address your concerns sufficiently, please inform us, and we will promptly follow up.

Weakness (A): We have derived a thorough learning theory to guarantee the OOD detection learnability on transformers, which may not be applicable to MLP-based and CNN-based models unless the same learning theory is established. So we focus on transformers to validate the efficiency of GROD, which is more theoretically explainable and connects closely to our theoretical results. It is a helpful suggestion to "provide a discussion on potential adaptations", which would be added to our paper as follows: As defined in Definition 4. (Transformer hypothesis space for OOD detection), GROD gains theoretical guarantee on transformers with multiple transformer layers and a classifier for OOD detection and classification tasks. So GROD has compatibility for transformers extract features like CLS tokens and inputs into the classifier, which is appliable to almost all transformer models.

Weakness (B): For experimental validation, we consider multiple factors including dataset types, dataset size, outlier types, and different PCA and LDA generation conditions, and finally make a decision on our benchmarks. However, as discussed in Weaknesses (A), to prove the versatility of GROD concerning transformers, we add a group of experiments using decoder-only transformer GPT-2 as the backbone network on NLP task to verify our method further. Part of the methods (Baseline, GEN) is compatible for both models with CLS tokens as features and without CLS tokens, as they only need logits for processing. Others are only compatible with models with CLS tokens as features, since they utilize features and logits together. We test two modes (with/without CLS token) for the former, denoted as Method-C (with CLS) and Method-L (without CLS), respectively. The comparison results are reported as follows:

ID: CLINC150 with Intents, OOD: CLINC150 with Unknown Intents

ID: IMDB, OOD: Yelp

Weakness (C): Firstly, our experiment includes a largescale dataset ImageNet-200 as ID as shown in Table 4. In fact, we have conducted experiments of Texture, as OOD, and we have not reported these results just because far-OOD is simple to detect for GROD and other competitive models, and shows similar performance for method comparison, so we only remain SVHN as a representation of far-OOD. Here we offer the results on other far-OOD datasets compared with SVHN in Table 2 and Table 3, respectively:

We are happy to add these experimental results to the appendix, should you recommend this approach.

Thank you for the detailed and extensive responses; I really appreciate them. For each considered benchmark, there exist methods that have significant performance results. Although I like that the paper has theoretical analysis, I believe that OOD detection is a practical task, and experimental soundness is more important. Since I found the results gap compared to existing methods to be minor, and the pipeline novelty incremental, I will maintain my score.