Mahalanobis++: Improving OOD Detection via Feature Normalization

We thank the reviewer for the thoughtful comments and appreciate the positive feedback. Below we address the reviewers remarks:

• “The fix intends to alleviate the difference between the feature norms of samples from different classes”

  We would like to clarify that different feature norms for samples from different classes are not a problem per se. For instance, if the mean vectors of different classes were of different magnitude (which is typically not the case), the Gaussian assumption could still be satisfied. However, our analysis shows that the observed feature norm distribution and the one that we would expect under the Gaussian model are significantly different, for instance showing very heavy tails. We take this as indication that the Gaussian assumption is violated, and substantiate this with further analysis (QQ plots, etc).

• “nothing ensures that after the fix, which is just a normalization, the features follow a normal gaussian, and that their covariance matrices are equal.” and “QQ plots are here to show that the obtained features are closer to normal gaussian, but they might still be non gaussian.”

  We agree that we cannot guarantee that the features follow a normal distribution. In fact, we believe that there is no reason to believe that the features follow any particular distribution. However, we provide strong empirical evidence that modelling the feature distribution with a normal distribution with shared covariance is more appropriate after normalization. In particular, 1) the QQ plots are less skewed, 2) the shared covariance assumption is better satisfied, and 3) the feature norm does not act as a confounder for OOD detection anymore.

• "What is called a 'fix' might not be an actual 'fix'"

  In addition to the above, we are happy to rephrase from "fix" to e.g. "remedy"

• “Could you plot the distribution of normalized features in a plot similar to Figure 3?”

  The feature norms of the normalized features would show as a straight line at 1 with no deviation. Please let us know in case this does not clarify the question.

• We thank the reviewer for the remarks about the trace notation and for noting that $\Phi(X)$ has not been properly introduced. We will adjust the notation and clarify that $\Phi(X)$ is a random variable representing the feature distribution for input $X$.

• ”two very different covariance matrices could have a deviation of zero with this metric. “ (Eq. 5)

  We respectfully disagree with the reviewer. In particular, we can write $\mathbb{E}_u \left[ \left( \frac{u^T \hat{\Sigma}_i u}{u^T \hat{\Sigma} u} - 1 \right)^2 \right]=\mathbb{E}_u \left[ \left( \frac{u^T (\hat{\Sigma}_i-\hat{\Sigma}) u}{u^T \hat{\Sigma} u}\right)^2 \right]$ Since $\Sigma$ is pd, $u^T {\Sigma} u>0$, and the only way for the expectation to be zero is that $u^T (\hat{\Sigma}_i-\hat{\Sigma}) u=0$ for all $u$, wich is only the case when $\Sigma=\Sigma_i$.

• "expected squared variance deviation ... is not a standard metric"

  We agree that this metric is not commonly evaluated, but we argue that it is the right one to look at. In particular, the Mahalanobis distance performs a whitening by the variances: Deviations in a certain direction are measured relative to the sample variance in this direction. Small absolute deviations can thus result in large distances when they are along a direction of small variance. We therefore need a measure that can capture relative deviations instead of absolute deviations, since absolute deviations would be dominated by directions of large variance. Our proposed measure computes the relative deviation of the variance of $\Sigma_i$ from $\Sigma$ in every direction u and averages this deviation over all directions. This is a natural way to assess whether $\Sigma_i$ and $\Sigma$ are similar in all possible directions in the feature space. A similar measure that is commonly used to compare covariance matrices is the Riemannian metric (see e.g. [1,2]) $d(\Sigma_1, \Sigma_2) := \sqrt{\mathrm{tr}\left(\ln^2\left({\Sigma_1^{-0.5}}\Sigma_2{\Sigma_1^{-0.5}}\right)\right)}$. It is also possible to compute an appropriate measure with divergences like the KL divergence: $KL_{\text{normal}}(I_n,\Sigma_1^{-0.5}\Sigma_2\Sigma_1^{-0.5})$. We evaluate both, confirming that normalization aligns the covariance structure in a meaningful way (lower is better):

We are happy to discuss any of the points further!

[1] Förstner & Moonen. (2000). A Metric for Covariance Matrices. 10.1007/978-3-662-05296-9_31.

[2] Pennec, Fillard, & Ayache. A Riemannian Framework for Tensor Computing. Int J Comput Vision 66, 41–66 (2006).

We thank the reviewer for their positive feedback, and for appreciating our work. We address the remarks below:

1. "organize related work section" and "elaborate difference to existing similar methods"

   We will extend the discussion about related work, and emphasize the differences to previous work that used feature normalization [3,4] or the Mahalanobis distance, or both [1,2]. Most importantly, other works have investigated train-time methods that involve normalization. Either implicitly through contrastive losses (CIDER [1], SSD[2]), or explicitly to improve OOD detection [3,4] It is then natural to also apply normalization at inference time. For instance, CIDER applies KNN, and SSD performs k-means and then Mahalanobis. Those methods thus normalize their features for OOD detection because they also normalize during training. This is orthogonal to our work: The standard Mahalanobis method for OOD detection is a post-hoc method, where adjusting the pretraining scheme is not feasible. We show that in this setting, the Gaussian assumption underlying this method is often severely violated, and that normalizing the features better aligns with this assumption, consistently improving OOD detection across architectures and pretraining techniques. We will clarify this distinction and expand the discussion of other approaches in the paper (see the answer to reviwer jfEM for a more thorough discussion and quant. comparisons to SSD). If there is a specific reference the reviewer would like us to discuss, please let us know.

2. Lemma 3.1, not sufficient condition

   We will clarify that a concentrated feature norm is not a sufficient, but a necessary condition for a Gaussian distribution. Lemma 3.1 only shows that - under the assumption of a Gaussian distribution in feature space - we expect some concentration of the feature norm. To illustrate this, we sample from class-specific Gaussian distributions with the estimated means and shared covariance matrix (Figure 3-left), noting that in practice (Figure 3-right) the feature norms deviate strongly from the Gaussian model (e.g. via heavy tails). This suggests severe violations of the Gaussian assumption, which we substantiate by QQ plots and the variance alignment analysis. Our remedy - normalization - aligns the features better with the premise of normally distributed data with shared covariance matrix.

3. "elaborate on neligibility" (in QQ plot analysis)

   In QQ-plots, we compare empirical quantiles against a theoretical standard normal distribution. Since normalized and unnormalized features have different variances, their QQ-plots would have different slopes, making direct comparison difficult. To align comparisons, we divide both samples by their empirical standard deviation—this ensures both are evaluated against the same reference slope (black line in Figure 4). Dividing by the empirical variance technically transforms a normal distribution to a Student's t-distribution with n-1 degrees of freedom. As the reviewer pointed out correctly, this matters for small $n$. However, the t-distribution converges to a Gaussian as $n\to\infty$, and for $n>30$, the difference is typically negligible [5]. We use all ImageNet train features ($n>10^{6}$) in our QQ plots, making the t-distribution practically Gaussian, allowing for the analysis we performed in the paper. We would like to stress that all of this is only a technicality in the analysis of the features via QQ plots, and irrelevant for Maha++ as an OOD detection method.

4. concentration of feature norm is "key motivation"

   Our key motivation is not to concentrate the feature norm. Instead, feature norm concentration is a necessary condition IF the features were indeed normally distributed. As we find, the feature norms are, however, not concentrated, but for instance show extremely heavy tails. We take this as an indication that the Gaussian assumption is violated, and further validate it via QQ plots and our variance analysis. Regarding the reviewers question about concentrating even further: We are not sure we understand what the reviewer means by this. One could, in principle, normalize by a different norm (e.g. $\ell_1$ or $\ell_\infty$), but this would change the direction of the features. We therefore opted for $\ell_2$ normalization. Does this answer the question?

We are happy to clarify any of the points further!

[1] Ming et al. How to exploit hyperspherical embeddings for out-of-distribution detection? ICLR2023

[2] Sehwag et al. Ssd: A unified framework for self-supervised outlier detection, ICLR 2021

[3] Regmi et al. T2fnorm: Train-time feature normalization for ood detection in image classificatio, CVPR 2024 workshop

[4] Haas et al. Linking neural collapse and l2 normalization with improved out-of-distribution detection in deep neural network, TMLR 2023

[5] https://www.jmp.com/en/statistics-knowledge-portal/t-test/t-distribution

We thank the reviewer for carefully reading and evaluating our paper, and we are glad that the reviewer finds that our claims are “supported by clear and convincing evidence”, that our method is “well motivated”, that that they appreciate the “wide variety of model types, architectures and sizes” in our “thorough evaluation". We agree with the reviewer that the results about augreg ViTs stand out, and think that investigating the underlying reasons for the behaviour of those models (i.e. the why the augreg training scheme results in the favourable structure of the feature space) is an interesting direction for future research. We thank the reviewer for pointing out the incorrect ImageNet reference, which we will fix. For the rebuttal, we have included a more thorough discussion and comparison to SSD (see response to reviewer jfEM), an evaluation of a DinoV2 model (also in response to reviewer jfEM) and more variance deviation measures (see response to reviewer qhSY). If there is anything else the reviewer would like to see addressed, we would be happy to discuss this.

We thank the reviewer for their valuable comments and address the remarks below:

• "the features should be concentrated around $\sqrt{\mathrm{tr}(\Sigma)-\|{\mu}\|^2_2}$" (in Lemma 3.1)

  We thank the reviewer for checking our proof, but we strongly believe that the term $\sqrt{\mathrm{tr}(\Sigma)+\|{\mu}\|^2_2}$ is correct. The term stated by the reviewer can even become complex if the norm of the mean is large enough. As the variance goes to zero, the norm of the random variable should be concentrated around $\|\mu\|_2$, which is exactly what our term states. We are happy to answer any questions on a particular step of the proof to resolve any potential confusion.

• "the higher the dimension, the looser is the upper-bound" (in Lemma 3.1)

  This is expected, as the squared $\ell_2$-norm grows with dimension $d$. However, the deviation per dimension decreases: $\Pr\left(\frac{1}{d}\left|\|\Phi(X)\|^2_2 - \left(\mathrm{tr}(\Sigma) + \|\mu\|^2_2\right)\right| \geq \epsilon\right) \leq \frac{\mathrm{Var}\left(\|\Phi(X)\|_2^2\right)}{d^2 \epsilon^2}$ The right side decreases with $d$ as the variance grows linearly in $d$. Lemma 3.1 shows that under the Gaussian assumption, we should see some concentration of the feature norms. To illustrate this, we simulate it in Figure 3 (left) by sampling from class-specific Gaussian distributions with the estimated means and shared covariance matrix, noting that the actual feature norms (Fig. 3 right) deviate strongly from the Gaussian model (e.g. via heavy tails). This suggests severe violations of the Gaussian assumption, which we substantiate by QQ plots and the variance alignment analysis.

• "the sentence 'Adapting them to IN-scale setups ... has so far not been successful' ... is a bit of an overstatement ... as Mahalanobis ... has been successful on IN-1k"

  We agree, this statement only refers to Gaussian mixture models (GMMs), and not to the Mahalanobis distance. We will clarify this in the paper and explain the difference between GMMs and the Mahalanobis distance.

• "the sentence 'l2-normalization...is ... not used with the Mahalanobis score' ... is a bit of an overstatement ... as SSD ... has been successful on IN-1k"

  We agree that Mahalanobis has been applied to normalized features in other works like SSD[1] and CIDER[2], and we should have chosen our statement more carefully. However, these are train-time methods where normalization is implicitly part of their contrastive loss. Those methods thus normalize their features for OOD detection because they also normalize during training. This is orthogonal to our work: The standard Mahalanobis method for OOD detection is a post-hoc method, where adjusting the pretraining scheme is not feasible. We show that in this setting, the Gaussian assumption underlying this method is often severely violated, and that normalizing the features better aligns with this assumption, consistently improving OOD detection across architectures and pretraining techniques. We will clarify this distinction and expand the discussion of [1-4] in the paper (see below for SSD).

• "a broader discussion and comparison with SSD" and "which challenges are unaddressed by normalized approaches"

  SSD involves three steps:

  1. Training with a supervised (SSD+) or unsupervised (SSD) contrastive loss (implicitly normalizing features),
  2. Cluster estimation via k-means in the normalized feature space,
  3. Mahalanobis-based OOD detection using cluster labels instead of class labels.

  This setting differs fundamentally from ours, as SSD, like [1,3,4], is a train-time method. Methods like [1-4] cannot be directly applied to the pretrained checkpoints we evaluate. To demonstrate the advantages of post-hoc approaches, we evaluate SSD+ on NINCO using the ResNet50 from [5] (trained for 700 epochs). SSD+ is clearly outperformed by our top models, with FPR >3× higher. Those are obtained from various pretraining schemes, and retraining models with SSD or [1,3,4] on this scale is typically not feasible. | model | FPR| |-------------|-------| |SSD+ w. 100 clusters|66.0% | |SSD+ w. 500 clusters|65.7% | |SSD+ w. 1000 clusters|67.8% | |CnvNxtV2-L + Maha++ | 18.4%| |EVA02-L14 + Maha++| 18.6%|

• comparison with DINO

  We report the FPR of a DinoV2-S model (ft on IN1k) on NINCO. Maha++ outperforms Maha clearly.

[1]Ming et al. How to exploit hyperspherical embeddings for out-of-distribution detection? ICLR2023

[2]Sehwag et al. Ssd: A and unified framework for self-supervised outlier detection, ICLR 2021

[3]Regmi et al. T2fnorm: Train-time feature normalization for ood detection in image classificatio, CVPR 2024 workshop

[4]Haas et al T. Linking neural collapse and l2 normalization with improved out-of-distribution detection in deep neural network, TMLR 2023

[5]Sun et al. Out-of-distribution detection with deep nearest neighbors. ICML 2022