X-Mahalanobis: Transformer Feature Mixing for Reliable OOD Detection

[W1] Rephrase some of the sentences.

[A1] We will rephrase the relevant sentences to ensure accuracy. Additionally, we will incorporate discussions on works [1, 2] to better contextualize our contributions, thereby improving the paper.

[W2] Ways to fuse shallow features.

[A2] We did experiment with simply averaging layer-wise Mahalanobis scores, but found this approach highly unstable. While it performed moderately on a few datasets (slightly below X-Maha), it deteriorated significantly on most others. This instability arises because some shallow layers individually achieve AUROC scores below 50% on certain datasets, which dilutes the overall performance when averaged. In contrast, our adaptive weighting strategy down-weights such low-discriminative layers, ensuring robust performance across datasets. We have already provided the ablation experiments for this part in Tables 7, 8, and 9 of the appendix.

[W3] OpenOOD shows that Mahalanobis with only the final layer feature consistently outperforms Mahalanobis with multiple layers' features.

[A3] We appreciate the reviewer's insightful observation. We attribute this discrepancy to two key factors: model architecture differences (CNNs vs. Transformers) and fusion strategy nuances.

First, model architecture plays a critical role. OpenOOD's evaluations primarily use CNNs, where shallow layers tend to capture low-level features (e.g., edges, textures) that are less discriminative for high-level ID/OOD separation. In such cases, fusing multiple layers (as in MDSEns) may introduce noise from less informative shallow features, diluting the final-layer's strong discriminative signals. In contrast, Transformers (the focus of our work) learn hierarchical features across layers, where even shallow layers encode semantically meaningful patterns (e.g., local object parts) that complement deep layers' global semantics . Our experiments (Figure 5) show that Transformer layers retain varying but useful discriminative capacity, making selective fusion beneficial.

Second, the fusion strategy matters. MDSEns in OpenOOD likely uses simple averaging or fixed weights for layer-wise scores, which fails to downweight low-quality layers. In contrast, X-Maha employs adaptive weights based on feature variance, explicitly suppressing layers with poor discriminative capacity (e.g., those with AUROC < 50% on specific datasets) . This targeted fusion ensures that only useful shallow features are integrated, avoiding performance degradation.

Thus, the contradiction arises from architecture-specific feature properties and fusion strategies: CNNs may benefit less from multi-layer fusion due to less informative shallow features and suboptimal weighting, while Transformers, with richer hierarchical features and our adaptive fusion, gain from leveraging multiple layers. We will clarify this in the revised manuscript to highlight architecture and strategy dependencies.

We are very grateful to the reviewer for many excellent suggestions. We have responded to each of them one by one and will incorporate these revisions into the next version of the paper.

[W1] "... attention-based fusion module ..." (line 40, lines 318-319)

[A1] Sorry for the confusion. The use of "attention-based fusion module" refers to a data-driven adaptive weighting mechanism that prioritizes layers based on their discriminative capacity, not the standard query-key-value attention in Transformers. We agree that this term was misleading, and we will revise it to "adaptive fusion module" in the next version.

[W2] Missing lower index in layer feature representations (lines 126-128)

[A2] Thank you for pointing out this. It should be 'For an instance $\boldsymbol{x}_i$, its output from the $l$-th layer is denoted as $\boldsymbol{x}^{l}_i = \phi_l(\boldsymbol{x}_i)$'.

[W3] Eq. 2: Definition of $x^l$ in ViT architectures

[A3] In ViT architectures, an input image is split into patches, linearly projected, combined with positional embeddings, and concatenated with a class token, resulting in a sequence of length "number of patches + 1". In our work, $x^l$ specifically refers to the embedding of the class token from the $l$-th layer, which captures global semantic information and ensures consistent dimensions across layers for fusion . This aggregation relies on the compatibility of features from different layers, which is supported by residual connections in ViT, though this may not hold for all general architectures. We will clarify this definition in the revised manuscript.

[W4] Covariance vs. mean computation (line 149).

[A4] This is an error in writing. It should be 'where $\widetilde{{\boldsymbol{\Sigma}}}=\frac{1}{N} \sum_{c=1}^{C} \sum_{i: y_{i}=c}\left(\Phi(\boldsymbol{x}_i)-\widetilde{\boldsymbol{\mu}}_{c}\right)\left(\Phi(\boldsymbol{x}_i)-\widetilde{\boldsymbol{\mu}}_{c}\right)^{\top}$ and $\widetilde{\boldsymbol{\mu}}_{c}=\frac{1}{N_{c}} \sum_{i: y_{i}=c} \Phi(\boldsymbol{x}_i)$.'

[W5] Feature variability and discriminative capacity (line 155, lines 162-164)

[A5] The link between high feature variability (measured by the trace of the covariance matrix) and high discriminative capacity is rooted in the neural collapse phenomenon. During training, within-class covariance diminishes, making total feature variance dominated by inter-class separation. Thus, higher total variance indicates larger inter-class distances, a key property for distinguishing ID classes and, by extension, ID vs. OOD samples. Layers with low variability often fail to separate even ID classes (e.g., some shallow layers with AUROC < 50% on specific datasets), further justifying this relationship.

[W6] Novelty of feature mixing (lines 170-172)

[A6] While Trusted [7] and Mahalanobis [27] have explored feature mixing, our key novelty lies in the adaptive computation of mixing weights—specifically, the ability to downweight or ignore low-quality layers, as highlighted in lines 179–181. We will revise the manuscript to clarify this distinction, ensuring the focus is on our unique weighting mechanism to better emphasize the novelty of our approach.

[W7] Fig. 3: Model accuracy before fine-tuning

[A7] We will add the model accuracy before fine-tuning in Fig. 3. Additionally, the accuracy and OOD detection performance of the model without fine-tuning can be found in Tables 6, 15, and 16 in the appendix.

[W8] Model agnosticism (line 209)

[A8] The term "model-agnostic" here specifically refers to its applicability across transformer-based models, rather than all model architectures. Currently, our method is constrained to models where features from each layer have consistent dimensions, which prevents direct adaptation to convolutional networks like ResNet (where layer-wise feature dimensions vary) or models without residual connections. We acknowledge this limitation and plan to explore extensions to handle models with varying layer dimensions in future work. Additionally, we will clarify this scope in the revised manuscript.

[W9] Scoring function for vision-only models (line 217).

[A9] Yes, for vision-only models, the scoring function simplifies to $G\left(\boldsymbol{x}\right) = \max_{c\in [C]} M_{\text{X-Maha}}(\boldsymbol{x}; {\widetilde{\boldsymbol{\mu}}_c }, \widetilde{{\boldsymbol{\Sigma}}})$ (excluding the vision-language similarity term). This will be explicitly stated in the revised manuscript for clarity .

[W10] Typo in line 245 (".. our methoMSP, MSP ...").

[A10] This typo will be corrected to "our method with: MSP" in the revised manuscript to ensure accuracy.

[W11] Table 1: Highlighting Trusted's performance.

[A11] Trusted ties with X-Maha on Texture, LSUN, and Places365. We will bold Trusted's results on these datasets in Table 1 and apply the same correction to other tables for consistency.

[W12] Figure 4: "SFM" definition.

[A12] "SFM" is a typo and should be "X-Maha". This correction will be made in the revised figure to align with the method described in the text .

[W13] Table 2: Trusted's AUROC < 50%.

[A13] Yes, Trusted achieves AUROC < 50% on ImageNet with ViT. This is because uniform averaging dilutes deep-layer signals in Transformers (unlike CNNs), leading to worse-than-random OOD detection. We will note this architecture-specific behavior in the revised manuscript.

[W14] Computation of $\alpha^l$ (lines 296-298).

[A14] $\alpha^l$ is computed once on ID training data using feature covariance (trace of the covariance matrix). It does not depend on validation, deployment, or OOD data, ensuring generalization to new samples.

[W15] Not coincide with the reported X-Maha result.

[A15] These results in Figure 5 are actually derived from applying the formula in [A9], rather than the formula presented in line 217 of our manuscript. This discrepancy explains the observed inconsistencies with the values in Table 2. We will update Figure 5 in the next version to align with the core formula of our method (line 217) to avoid confusion.

[W16] Table 3: PM's AUROC vs. FPR95

[A16] Thank you for catching this error. The AUROC value for ImageNet CLIP PM in Table 3 is incorrect; it should be 81.03 (not 91.03), which is why it does not warrant bold highlighting. This lower AUROC aligns with its relatively poor FPR95 performance. Additionally, the corresponding average AUROC will be revised to 89.3 to reflect this correction. We apologize for the confusion and will update the table in the revised version.

[W17] Appendix A.3: Importance of $\alpha^l$.

[A17] While some fixed weights perform well on specific datasets, X-Maha's adaptive $\alpha^l$ ensures consistent robustness across all settings. Uniform weights over top layers underperform as they can't downweight noisy layers, making our weight computation crucial. This consistency validates the proposed method's necessity.

Thank you again for your valuable advice!

We are very grateful to the reviewers for their valuable suggestions, especially regarding some stronger comparison methods. We will cite these works and include experimental comparisons in the next version.

[W1] Theoretical insights or experimental evidence to support this proposal.

[A1] First, our use of feature covariance (specifically, the trace of the covariance matrix) to determine layer importance is motivated by the properties of neural representations during training. As described by the "neural collapse" phenomenon [38], deep networks converge such that within-class feature covariance approaches zero, while ID-class separation becomes maximized. In this context, the total variance of features in a layer—quantified by the trace of the covariance matrix ($Tr((A^l)^\top A^l)$)—directly reflects the layer's ability to distinguish between classes. A higher trace indicates greater spread in feature distributions, which corresponds to stronger inter-class discriminability—an essential property for distinguishing ID samples from OOD samples, which often lie outside the compact manifold of ID features.

Second, we validate this relationship through multiple experiments:

• Layer weight analysis: Figure 6 shows that layers with higher trace values (e.g., deeper layers) are assigned higher importance weights. These layers, consistent with neural collapse, exhibit stronger separation between ID classes, which translates to better OOD detection performance when fused .
• Ablation on layer fusion: Figure 5 demonstrates that fusing layers with non-negligible trace values (e.g., the last 6 layers) yields optimal OOD detection results, while including layers with low trace values (e.g., the first 6 layers) provides minimal gain. This directly supports that covariance-derived weights effectively prioritize discriminative layers.
• Comparison with alternative weighting strategies: Table 3 compares our covariance-based weighting against other fusion methods (e.g., uniform averaging, score aggregation). X-Maha consistently outperforms these alternatives, confirming that covariance-derived weights better capture discriminative capacity.

[W2] Different learning rates between FFT and PEFT.

[A2] In our experiments, we observed that FFT performance does improve with smaller learning rates (1e-5 or 1e-6), as noted. However, even with these optimized lower learning rates, FFT still lags behind PEFT when the latter is trained with a higher learning rate (1e-4).
As shown in the table, FFT performance improves with smaller learning rates (from 1e-4 to 1e-6). However, even at its optimized lower learning rates (1e-5 or 1e-6), FFT still lags behind PEFT when the latter is trained with a higher learning rate (1e-4). Additionally, PEFT maintains superior performance across the tested learning rate range (1e-6 to 1e-4).

[W3] Comparsion with prior works.

[A3] We appreciate the reviewer's suggestion to include recent zero-shot CLIP-based methods as baselines. Since NegLabel and CLIPScope requires external knowledge to facilitate negative label mining, it is unfair to directly compare them with our approach. However, we find that our approach can seemlessly combines with GL-MCM, NegLabel, and CLIPScope to further boost their performance. The following table reports the results when combinging X-Maha with GL-MCM, NegLabel, and CLIPScope across OOD datasets, using FPR95 (%) and AUROC (%) as metrics:

[W4] Comparison of the computational cost.

[A4] Regarding GPU memory consumption, our approach is designed to avoid significant overhead when utilizing shallow layer features. While we do store intermediate features from shallow layers, this is done in CPU memory on a per-batch basis during the forward pass. When computing the importance weights for each layer , we process these CPU-stored features in a sequential manner: we load features from one layer into GPU memory, compute its weight, and then unload them before loading the next layer’s features. This sequential processing ensures that only a single layer’s features occupy GPU memory at any time, preventing cumulative memory usage from multiple layers. As a result, the additional GPU memory required for weight computation remains minimal and does not scale with the number of layers.

For computational cost, as detailed in Section A.5 and Table 13 of the appendix, X-Maha increases total inference time by approximately 10% compared to baselines that use only final-layer features. This modest overhead is accompanied by an average improvement of 2.39% in AUROC and a 7.66% reduction in FPR95, demonstrating a favorable trade-off between efficiency and performance.

[W5] Generalizability to other architectures.

[A5] Thank you for your suggestion. Currently, our method assumes consistent feature dimensions across all layers, which aligns with Transformer architectures (e.g., ViT, CLIP ViT-B/16) where each layer outputs features of the same dimension. This constraint limits direct application to ResNet-50, as its layers have varying feature dimensions (e.g., increasing from 64 to 2048), making feature fusion across layers non-trivial.
We acknowledge this limitation and plan to address it in future work by exploring dimension-adaptive fusion strategies to extend the method to architectures with heterogeneous layer dimensions. However, in this paper, we focus on levearging the Transformer-based models to improve the OOD detection performance.

Thank you again for your precious advice!

Thank you sincerely for your continued feedback and patience as we refine our work. We appreciate your focus on experimental evidence and novelty, and we have restructured our response to align with your concerns, our core approach, and our advisor’s guidance.

1. Experimental Scope: Comprehensive Evaluations Across Diverse Settings

To address your request for clarity on our experimental rigor, we have conducted extensive evaluations across datasets, model architectures, and baseline methods, ensuring our findings are robust and generalizable:

• Datasets: We tested on both class-balanced and challenging real-world scenarios:

  • Class-balanced ID datasets: CIFAR-100 and ImageNet, standard benchmarks for OOD detection .
  • Long-tailed ID datasets: CIFAR-100-LT and ImageNet-LT, which reflect imbalanced real-world data distributions where minority classes are underrepresented .
  • Complex distribution shifts: OpenOODv1.5 benchmark, featuring diverse shifts like natural corruptions, style variations, and semantic drift—critical for testing robustness beyond standard settings .

• Model architectures: We evaluated across two key model types to validate generalizability:

  • Vision-only models: Pre-trained Vision Transformers (ViT) fine-tuned on ID data, where X-Maha leverages rich visual feature hierarchies (shallow to deep layers) .
  • Vision-language models: CLIP (ViT-B/16), adapted to work with X-Maha by integrating cross-modal features, despite CLIP’s inherent focus on language-visual alignment .

• Baseline comparisons: We benchmarked against 14+ methods, including:

  • Recent CLIP-based OOD detectors (GL-MCM, NegLabel, CLIPScope) to contextualize performance on language-aligned data .
  • Traditional OOD methods (Mahalanobis, Energy, Residual) to ground our approach in foundational techniques .
  • Methods leveraging middle layers [1, 2, 3] (as you noted) to explicitly highlight differences in design and performance .

2. Novelty: Key Contributions Distinguishing X-Maha

X-Maha introduces unique innovations that address limitations in existing methods, including the three works you referenced [1, 2, 3]. Our distinct contributions are:

• No reliance on validation data for weight tuning: Prior methods like [3] and Mahalanobis [27] require OOD validation data to optimize fusion weights, limiting practicality. X-Maha computes weights directly from ID training data, making it applicable to scenarios where OOD validation data is scarce or unavailable .

• Synergy with parameter-efficient fine-tuning (PEFT): We demonstrate that PEFT (e.g., LoRA, Adapter) outperforms full fine-tuning for OOD detection, especially on long-tailed data . X-Maha is uniquely designed to work with PEFT: by preserving pre-trained feature quality while adapting to downstream tasks, it avoids overfitting to imbalanced data—unlike [1, 2, 3], which focus on full fine-tuning and lack this integration .

3. Compatibility with CLIP: Addressing Inherent Trade-offs

We acknowledge CLIP’s strength in OOD detection via cross-modal alignment, but X-Maha’s design addresses its limitations:

• CLIP’s contrastive pre-training prioritizes language-visual alignment over fine-grained visual feature learning, leading to lower in-distribution classification accuracy compared to ViT . This limits X-Maha’s gains on CLIP, as our method relies on high-quality visual features to drive layer-wise fusion.
• Despite this, we adapted X-Maha to CLIP by integrating text-image similarity scores into our OOD scoring function , enabling compatibility with CLIP-derived methods (e.g., GL-MCM, NegLabel). While gains are modest on balanced data (due to CLIP’s strong baseline), X-Maha still enhances their robustness on long-tailed and shifted data .

We believe these clarifications highlight the breadth of our experiments, the uniqueness of X-Maha’s design, and our effort to align with CLIP-based methods where possible. Thank you again for pushing us to strengthen these points—your insights are critical to ensuring our work meets rigorous standards. We are committed to refining the manuscript further and hope this addresses your considerations.

[W1] Risk of overfitting on the validation dataset.

[A1] We appreciate the reviewer's insightful observation regarding the distinction between our approach and the Mahalanobis method [27] in terms of validation set dependence. We agree that such overfitting could lead to inflated performance estimates that do not generalize to unseen test data, especially when the validation set is small or not representative of the target distribution. To further mitigate the risk of overfitting and demonstrate generalizability, we conducted extensive experiments across diverse settings:

1. Diverse datasets: We evaluated on both class-balanced (CIFAR-100, ImageNet) and long-tailed (CIFAR-100-LT, ImageNet-LT) in-distribution datasets, with nine OOD datasets spanning various domains (e.g., Textures, SVHN, Tiny ImageNet) . Consistent performance gains across these datasets suggest our method is not tied to specific data distributions.

2. Multiple model architectures: We validated our approach on pre-trained Vision Transformers (ViT) and CLIP models, showing robust improvements across both vision-only and vision-language frameworks . This cross-model consistency indicates the method is not overfitting to idiosyncrasies of a single architecture.

3. Zero-shot settings: Critical to ruling out validation set overfitting, our zero-shot experiments (without fine-tuning) still demonstrated superior performance compared to baselines . Zero-shot settings inherently avoid any validation-based tuning, providing strong evidence that our weight calculation generalizes without relying on auxiliary data.

4. Minimal hyperparameter tuning: As noted, most hyperparameters (e.g., learning rate, batch size) followed the settings in LIFT [45], with minimal adjustments . This reduces the risk of over-optimizing for specific datasets and aligns with principles of reproducible research.

In summary, the breadth of our experimental validation—across datasets, models, and zero-shot/fine-tuned scenarios—along with limited hyperparameter tuning, strongly supports that our training-data-only weight calculation avoids the overfitting risks associated with validation set dependence.

[W2] Consistent formatting for result representation.

[A2] We will maintain consistent result formatting across all tables, which will enhance the readability of the manuscript and improve clarity in data presentation.

[W3] Limitations section.

[A3] To clarify, our statement in the Limitations section does not contradict the empirical validity of X-Maha but rather emphasizes a gap in theoretical understanding despite strong experimental support.

Our work thoroughly validates the effectiveness of mixing shallow and deep features through extensive empirical evaluations:

• Across diverse in-distribution datasets (class-balanced CIFAR-100/ImageNet and long-tailed CIFAR-100-LT/ImageNet-LT) ;
• Over multiple OOD benchmarks (e.g., Textures, Tiny ImageNet, OpenOODv1.5) ;
• With different model architectures (ViT, CLIP) and settings (fine-tuned, zero-shot) ;
• Under varied parameter-efficient fine-tuning strategies (Adapter, LoRA, VPT) .

These results consistently demonstrate that shallow features, when weighted adaptively, enhance OOD detection. However, our limitation lies in the theoretical underpinnings: we lack a formal proof explaining why shallow features improve performance in specific scenarios (e.g., why layers with higher variance are more discriminative for OOD) or under what conditions this gain holds. For example, while we observe that fusing the last 6 Transformer layers works best , we cannot yet theoretically characterize why this specific range is optimal across datasets.

This gap motivates future work to establish theoretical guarantees, which would strengthen the generalizability argument beyond empirical observations. Our current empirical rigor confirms practical effectiveness, but theoretical exploration remains a necessary next step to fully understand and extend the method.