Improving Out-of-Distribution Detection via Dynamic Covariance Calibration

We appreciate the reviewer's positive recognition of our work, particularly regarding the novelty and reasonableness of the idea, strong performance, and clarity of supporting evidence. Below, we provide detailed, point-by-point responses addressing each comment.

Other Strengths And Weaknesses:

C1: Why the baseline methods provided in CIFAR benchmark and ImageNet benchmark.

C1-Ans: The reason that the compared methods are different in CIFAR and ImageNet benchmarks is that we can not access the hyperparameter settings in the absent methods. To address the author's concern, we reimplement and do hyperparameter search on VIM and NECO in CIFAR benchmark. In addition, we add a gradient-based method in this setting, namely GradeNorm [2].

C2: Can we make sure this discrepancy gets improved with projection, compared with feature norm without normalization.

C2-Ans: Thank you for the insightful question. We have added the discussion about the observation of NAN[1] in our experiment. In the following table, we show the results w and w/o projection to illustrate the effectiveness of the projection.

In addition, we use the norm of the projected feature as the OOD score to answer this question, shown in the following table. The results in the table imply that normalization can help improve the discrepancy. We present our explanation of this phenomenon in the following

The $L_2$ norm of the feature can also be interpreted as the distance from the feature to the zero point. If the features in different semantic distributions have significantly different scales and the center of ID features is far from the zero point, the norm of the features makes it hard to detect the OOD sample very well, since the zero point may also be an OOD point. However, the normalized features are distributed on the sphere with radius 1 and the zero point as center. As long as the class clusters are not distributed too close, the center of normalized features can be easily near the zero points, thus making the norm of the features have geometrical meaning for OOD detection. If the center of the features is the zero point, the center of the projected features is also the zero point. We assume this is the reason that the normalization can help improve the discrepancy and also strengthen the neural collapse phenomenon

[1] Park, Jaewoo, et al. "Understanding the feature norm for out-of-distribution detection." Proceedings of the IEEE/CVF international conference on computer vision. 2023.
[2] Huang, Rui, et al."On the importance of gradients for detecting distributional shifts in the wild.

We thank the reviewer for the constructive comments and valuable suggestions. We provide detailed responses to each comment in the following.

Claims And Evidence

C1 Detailed ablation study:

C1-Ans: We include AUROC results in the ablation study table. To show full combinatorial results and enhance the readability, we also reformatted our ablation study table. Note that RSP (Residual Space Projection) and DCM (Dynamic Covariance Modeling) must be built on top of the DME (Dynamic Matrix Estimation) module. DCM is introduced in Section 4.3.

As discussed in Section 5.5, the large between-class covariance distorts the geometry with which the data can not form a dense manifold. Thus, DCM is essential on some backbones, like DenseNet. We also discussed this in detail in Section C of the appendix. As shown in the table, the DME can solely improve the performance on CIFAR10-DenseNet and ViT. Also, the RSP is essential in ResNet50.

Methods And Evaluation Criteria:

C2 Experiments on near-OOD detection and iNaturelist:

C2-Ans:We have reported iNaturelist results in Tab. 9 of Appendix E. The near-OOD results can be viewed in Reviewer YqZR's C4

Experimental Designs Or Analyses:

C3 Include NECO and VIM in CIFAR benchmark (Tab. 1) and DINO (Tab. 3):

C3-Ans: We provide the experimental results of NECO and VIM on CIFAR benchmark and the results of Residual score on DINO in the following.

Since DINO does not have the linear classifier as the supervised pretrained model does, for the DINO experiment, we only evaluate Residual score and discard the energy score in VIM.

Relation To Broader Scientific Literature:

C4: Detailed differences and relevance of the proposed approach compared to VIM and NECO

C4-Ans: Both methods(VIM, NECO) only consider subspace information without direction-specific adjustments, potentially ignoring part of the ID information geometry in the distance-based score. Additionally, these methods do not adaptively utilize test-time information to correct for possible distortions in the training distribution, further limiting their sensitivity to novel or shifted feature directions. We will revise this discussion in Section 3 of the manuscript. The VIM and NECO can be interpreted as the matrix-induced distance-based scores with the formula $d_M(f) = \sqrt{(f-f_a)M(f-f_a)^\top }$. From this perspective, NECO and VIM can be explained as these methods to find better geometry of the distance for OOD detection. However, these methods ignore the fact that the matrix $M$ or geometry may be affected by the outliers in ID distribution (Details can be viewed in R2(YqZR)'s C6(Q2)). In our proposed approach, $M$ is related to real time features $f$, denoted as $\mathcal{M}(f)$. Specifically, $\mathcal{M}(f) = (Cov-r_f^\top r_f)^{-1}$, where $Cov$ is the covariance matrix from available features in the train set and $r_f$ is the projected real time features $f$ projected on residual space. For VIM and NECO, $M = Res(Cov)$ and $M=Pri(Cov)$, with $Res(\cdot)$ and $Pri(\cdot)$ being the residual and principal spaces, respectively. This indicates that $M$ remains the same for both methods, regardless of any real-time features. In addition, they may ignore part of the ID information geometry in the distance-based score.

Other Strengths And Weaknesses

C5: The name of sub-section 2.2

C5-Ans: We have updated the subtitle of section 2.2 to Post-hoc OOD detection methods.

C6: The parameters of the OOD criterion are fixed for any given test-sample.

C6-Ans: The dynamic component aims to mitigate covariance distortion caused by outliers in real-time features, specifically in Mahalanobis distance. Please view R2(YqZR)'s C6(Q2) for clearer motivation for our method. Our method relies on the initial computation of $\Sigma_R$ and $B$ as necessary initialization steps, while the dynamic component is provided by real-time recalibration of the covariance structure individually tailored for each test sample, significantly enhancing adaptability and robustness.

Other Comments Or Suggestions:

C7: Typos

C7-Ans: Thank you for pointing out the typos in our paper. The z in s(z) should be f. We have revised these typos

We appreciate the reviewer's recognition of our work's novelty, improved performance demonstrated through comprehensive experiments, clear presentation, and its significance for AI system reliability. Below, we address each comment in detail.

C1: gradient-based methods

C1-Ans: Since the code of GradNorm[3] is the only implementation available, we compare our method with this gradient-based method. We have added these three works in Section 2

Due to its computational cost, we only conduct the comparison on CIFAR benchmark. In the following, we present the results of GradNorm.

C2: additional discussion on limitations and potential failure modes of Mahalanobis distance

C2-Ans: Previous work has discussed the failure within the dimension corresponding to the smaller eigenvalues of ID covariance [1] and less effective performance when ID and OOD are similar [2]. The strong performance of our method aligns with and supports the findings in [1]. Our approach explicitly adjusts the covariance matrix within the residual space, thereby mitigating the aforementioned vulnerability regarding the discrepancy of Mahalanobis distance in residual space. In addition, our adjustment can also help improve the performance of Mahalanobis distance in near OOD detection. Details can be viewed in the answer of C4. We have added this discussion in our paper.

C3(Q1): Detailed discussion and comparison on computational complexity

C3(Q1)-Ans: In the following table, we provide the detection time cost comparison over 10,000 CIFAR10 instances with DenseNet as the backbone network.

We note that our method has a comparable time cost to Mahalanobis distance method. This is because in post-hoc OOD detection, the dominant computational cost lies in the feature extraction, which is shared across most methods. The time complexity we provided is only for the feature-level operation, which is insignificant as $n$ (feature dimension) is small. In addition, we compare with the time cost of GradNorm in the table

C4: Near-OOD on OpenOOD benchmark

C4-Ans: We conduct near OOD detection on OpenOOD benchmark with ResNet18. We report the experimental results in the following table. We also compare with the results of the recent distance score and subspace score available in OpenOOD, namely KNN and VIM. All the results are averaged over 3 runs. We also provide the results of RMDS [1] with our method. RMDS is a Mahalanobis distance tailored for near-OOD detection. As the table shows, our method can effectively improve the performance of Mahalanobis distance in near OOD detection. It can also improve the RMDS, reflecting the robustness of our method in different distances in near OOD detection.

C5: typos

C5-Ans: Thank you for pointing out this typo. We have revised it in our paper.

Questions

C6(Q2): Visualization of the phenomenon that Mahalanobis distance may become less sensitive to OOD samples that align with the directions of high variance in ID data caused by outliers

C6(Q2)-Ans: Visualiztion: https://i.imgur.com/U4rrTTt.png We visualise this phenomenon in 2D space. Note that, as the features are in the spaces with higher dimensions, the situation may be more complicated than our demonstration. Mahalanobis distance can be considered as a geodesic distance of normal distribution, thus we utilize the geodesic line within the randomly sampled normal distribution (ID data) to demonstrate this phenomenon. As the figure shows, the geodesic line may be made further from the center point by the outlier points. It is obvious that the OOD points, aligned with outlier points, can be closer to the dashed line than the bold line, which makes the distance between OOD points and ID center smaller.

[1] Ren, Jie, et al. A simple fix to Mahalanobis distance for improving near-ood detection. [2] Tajwar, Fahim, et al. No true state-of-the-art? OOD detection methods are inconsistent across datasets. [3] Huang, Rui, et al."On the importance of gradients for detecting distributional shifts in the wild.