A Contrastive Teacher-Student Framework for Novelty Detection under Style Shifts

W4: We kindly request the reviewer to refer to Table 18, where we have included a comparison with the most recent ND methods. For convenience, we have also included the table here to highlight the superior performance of our method.

In the experiments, the authors should compare more state-of-the-art methods.

The testing datasets are somewhat small. The authors should verify their method on more dataset.

Our experiments utilize a diverse set of large-scale datasets, encompassing both real-world and synthetic distribution shifts, as detailed in Appendix J. Specifically, we include datasets from various domains such as autonomous driving (Cityscapes, GTA5), medical imaging (Camelyon17, Brain Tumor, Blindness Detection, Skin Disease, Chest CT-Scan, White Blood Cells), and industrial anomaly detection (MVTecAD, VisA). We are also open to evaluating our method on any dataset the reviewer suggests.

Besides, the authors should give some feature-level visualization analysis, which is better to understand the proposed method.

As our method operates in the image domain, we have included several images to provide intuition about its functioning. Additionally, to address the reviewer's concerns, we have provided feature space visualizations in Appendix P of the revision. Any additional details or suggestions regarding the visualizations would be greatly appreciated to help us create more informative and insightful plots.

W2-b:

Why This Helps:

For Shift Problem 1:

The teacher model, trained on both ID and crafted OOD samples with various style augmentations, learns to focus on core features and ignore style variations. The student model, by aligning with the teacher on ID samples regardless of style shifts, also becomes invariant to these shifts. For Shift Problem 2:

By forcing divergence on OOD samples that share style features with ID samples, the student model learns that style features are not sufficient for predicting a sample as ID.

The student model is encouraged to rely on core features for making predictions, reducing reliance on spurious correlations.

4. Causal Viewpoint and Intervention

Breaking Unwanted Correlations:

From a causal perspective, style features act as confounders that can mislead the model. By intervening on the core features (altering them to create OOD samples) and varying style features (through augmentations) without changing labels, we weaken the causal link between style features and the label. This intervention helps the model focus on the true causal factors (core features) that determine whether a sample is ID or OOD.

Why This Helps:

For Both Shift Problems: By disrupting the spurious causal pathways that link style features to the label, the model is less likely to rely on these features. The model becomes robust to style shifts and focuses on the features that are truly indicative of the sample's class.

W3:

We understand the reviewers' concerns regarding the modules in our pipeline, which build upon foundational concepts in the field, such as data-centric approaches and contrastive learning. However, we believe there are several points that highlight the novelty of our study:

• While the mentioned principles are well-established, they remain essential components in Novelty Detection research, as evidenced by recent works such as ReContrast (NeurIPS 2023) and General AD (ECCV 2024).

• Our proposed method makes a distinct contribution by addressing a critical and underexplored challenge in Novelty Detection—achieving robustness under style shifts.

• Our framework employs a causal approach with an effective strategy to craft auxiliary OOD data. Although alternative strategies exist for crafting OOD samples, we differentiate ourselves with a data-efficient approach. Many existing methods rely on large generators (e.g., Stable Diffusion [1,2]) or extensive datasets (e.g., LAION-5B [1,2]); on the other hand, our approach does not require additional datasets. Moreover, our framework is underpinned by a theoretical foundation that validates its design and demonstrates its effectiveness, setting it apart from most existing ND approaches.

• Furthermore, our task-based knowledge distillation strategy goes beyond simply reusing established techniques. By introducing a novel loss function and defining an auxiliary task, the teacher model is first adapted to the ID set and subsequently used to train the student model. This approach ensures alignment on ID samples while encouraging divergence on crafted OOD samples.

• Our approach enhances performance, achieving superior results in both clean and robust evaluations, even on challenging real-world datasets, without relying on metadata or additional datasets.

[1] Du et al, Dream the OOD: Diffusion Models, Neurips 2023

[2] RODEO: Robust Outlier via ICML 2024

W2-a:

In Introduction Section, it is better to show a figure to analyze the corresponding problems of existing methods:

We kindly request the reviewer to refer to Figure 1, provided immediately after the Introduction, where we analyze the limitations of existing methods in detail.

Besides, I am not clear why the proposed method could solve these two shift problems. The authors should give more interpretations.

Shift Problem 1: Performance drops when test data exhibits style shifts (e.g., environmental variations).

Shift Problem 2: Unwanted correlation between style features and labels due to training data limitations (only ID samples available).

Understanding the Shift Problems

Shift Problem 1: Traditional ND methods assume that the training and test data come from the same environment and share similar style features (like lighting, texture, or color). However, in real-world applications, test data often have style variations not present in the training data. For example, images taken under different lighting conditions or with different camera settings. These style shifts can cause existing models to misclassify ID samples as OOD because the models have learned to associate specific style features with the ID class.

Shift Problem 2: Since only ID samples are available during training, models can inadvertently learn to rely on style features present in the ID data as cues for classification. This leads to a spurious correlation where the model associates the presence of certain style features with the ID label. Consequently, if an OOD sample shares these style features, it might be incorrectly classified as ID, and if an ID sample has different style features, it might be misclassified as OOD.

How Our Proposed Method Addresses These Shift Problems

1. Crafting an Auxiliary OOD Set by Distorting Core Features

Identifying Core Features:

• We use feature attribution methods like Grad-CAM to generate saliency maps for ID samples. These maps highlight the core regions of the image that the model considers important for making predictions.

• By applying light augmentations (e.g., slight changes in brightness or contrast) and generating saliency maps for both the original and augmented images, we create a final saliency map that is less sensitive to style features and more focused on core features.

Distorting Core Features:

• We apply hard transformations (e.g., elastic transformations, cutouts) specifically to the core regions identified in the saliency maps.

• This process alters the essential parts of the image that are critical for determining whether it's ID or OOD, effectively creating synthetic OOD samples.

• By doing so, we generate OOD samples that share the same style features as the ID samples but differ in core features.

Why This Helps:

For Shift Problem 1:

By maintaining the style features while altering the core features, we teach the model that style features are not reliable indicators for prediction. The model learns to focus on the core features that truly distinguish ID from OOD, making it more robust to style variations in the test data.

For Shift Problem 2:

• The introduction of OOD samples that share style features with ID samples but differ in core features breaks the spurious correlation between style features and the ID label.

• The model learns that style features alone are insufficient for accurate classification, reducing the reliance on these features.

2. Applying Style Augmentations to Both ID and Crafted OOD Samples

Light Augmentations:

• We apply light augmentations (e.g., color jitter, slight rotations) to both the ID and crafted OOD samples during training. These augmentations simulate various style shifts that might occur in real-world scenarios. Why This Helps:

For Shift Problem 1:

Exposing the model to a wide range of style variations during training encourages it to become invariant to such shifts. The model learns that the label (ID or OOD) remains the same despite changes in style features, reinforcing the focus on core features. For Shift Problem 2:

By varying style features while keeping labels consistent, we further weaken any spurious correlations between style features and labels. The model is trained to disregard style variations when making predictions. 3. Task-Based Knowledge Distillation Framework Teacher-Student Model:

We use a pre-trained teacher model (with a trainable binary classification layer) and a student model trained from scratch. The teacher is trained to classify the crafted ID and OOD samples, updating only its binary classification layer. Novel Loss Function:

We introduce a novel objective function that encourages the student model to align its outputs with the teacher's outputs for ID samples and to diverge for OOD samples. This loss function is contrastive, meaning it pulls together representations of similar samples (ID) and pushes apart representations of dissimilar samples (ID vs. OOD).

W1:

The motivation of this paper is not clear

We kindly hope that reviewing the problem setup and the highlighted real-world examples provides clarity on why the robust ND problem serves as a strong motivation for our work.

This paper aims to address novelty detection under style shifts, which involves two shift problems, i.e., covariate shift and semantic shift. However, this paper does not sufficiently analyze why existing methods could not solve these two shifts simultaneously....

We believe that one of the key limitations of existing ND methods in achieving robustness stems from their reliance on strong inductive biases tailored to specific datasets. SOTA methods for these datasets are often designed with assumptions that align closely with the dataset's unique characteristics. For instance, PatchCore [1], which achieves an impressive 99.6% AUROC on the MVTecAD dataset, relies heavily on patch-based feature extraction. While this approach performs exceptionally well on datasets like MVTecAD, which primarily feature texture-based novelty samples (e.g., a broken screw versus an intact screw), it tends to degrade in performance on datasets emphasizing semantic novelty detection (e.g., distinguishing a dog as a novel concept when a cat is considered inlier).

This limitation is also evident in the standard deviation of performance metrics, where our method consistently demonstrates lower variance compared to existing approaches (Please refer to Table 18.). This suggests that our method is inherently more robust across varying novelty detection datasets.

..To the best of my knowledge, there exist some methods that aim to leverage large-scale models, e.g., CLIP, to solve this challenge. The authors should introduce these methods and make an analysis.

There are numerous studies that explore the robustness of CLIP models [2,3,4]; however, we did not find specific works demonstrating the inherent robustness of CLIP for the ND problem.

Although we compared our method to several existing ND approaches, to address the reviewers' concerns, we also compared our method against existing CLIP-based ND methods. The results are summarized in the table below:

The results clearly indicate that our method outperforms these competing approaches across all datasets.

[1] Karsten Roth, Towards Total Recall in Industrial Anomaly Detection

[2] Mitigating Spurious Correlations in Multi-modal Models during Fine-tuning

[3] Fairness and Bias in Multimodal AI: A Survey

[4] MM-SpuBench: Towards Better Understanding of Spurious Biases in Multimodal LLMs

[5] Exposing Outlier Exposure: What Can Be Learned From Few, One, and Zero Outlier Images 2022

[6] WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentation 2023

[7] ANOMALYCLIP: OBJECT-AGNOSTIC PROMPT LEARNING FOR ZERO-SHOT ANOMALY DETECTION 2024

Dear Reviewer iu4w,

Thanks for your constructive and valuable comments on our paper. Below, we provide a detailed response to your questions and comments. If any of our responses fail to address your concerns sufficiently, please inform us, and we will promptly follow up.

W1-a:

As the reviewer noted that the motivation is unclear, we have provided a brief description of the problem setup for the task and clarified our motivation.

Novelty Detection

We would like to clarify potential misunderstandings regarding the problem we addressed and the proposed method. To provide clarity, we first review the setup of our problem.

The Novelty Detection (ND) problem involves a scenario where one class is designated as the in-distribution (ID) semantic, while all other semantics are considered as out-of-distribution (OOD). The primary goal in novelty detection is to develop a detector model, denoted as $f$, to distinguish between ID and OOD concepts.

To formalize this task, a detector model $f$ and a dataset $D$ are considered, where one class, such as $X$, is designated as the ID class, while the remaining classes, $D \setminus X$, constitute the OOD samples. A superior detector is one that more effectively distinguishes ID (i.e., $X$) from OOD (i.e., $D \setminus X$).

The difference between $X$ and $D \setminus X$ is based on a set of features, which we categorize into two types: core features and style feature. Core features capture the essential semantic differences between ID and OOD, while style features represent non-essential differences that do not generalize well across datasets. In this study, we aim to focus on learning core features and disregard style features.

For example, consider a specific subset of the colored MNIST dataset, $D$, where $X$ consists of green images of the digit 0'' (ID), and $D \setminus X$ consists of red images of the digit 1'' (OOD). The primary core feature here is the distinction between the digits 0'' and 1.'' However, there is also a style feature difference---namely, the color of the digits. A detector could distinguish ID from OOD based on either core features (digit identity) or style features (color).

In this study, our objective is to learn core features, as they are more robust and transferable to new datasets. For instance, consider a new dataset $D'$ during inference, where $X'$ consists of blue images of the digit 0'' (ID), and $D' \setminus X'$ consists of blue images of the digit 1'' (OOD). Here, the style feature (color) remains the same, and a model that relied on style features for distinguishing ID from OOD would perform no better than random guessing. In contrast, a model that learned core features would still perform effectively, as the core feature (digit identity) remains consistent.

First, note that as mentioned in the introduction lines 53 and 54, existing methods mostly require accessing either class labels, or else environment labels (also known as group labels in the spurious correlation mitigation setup) to work. Unfortunately, this information is unavailable in the novelty detection setup, where only the unlabeled samples from the normal class are given. To clarify, in our setup, we assume that the features consist of two parts, "core" denoted by $x_c$, and "environment related" denoted as $x_e$. We assume that $x_c$ would only causally affect the label, i.e. normal and anomaly (see fig 2a). Therefore, we tested against whether an ND algorithm could successfully classify normal vs. anomaly when $x_e$, or its distribution, changes at the test time. Many existing methods rely on two sets of augmentations ($T^+$ and $T^-$ known as light and hard augmentations) to learn a new representation that is invariant against natural variations within the normal class, but be sensitive to the changes that make a normal sample anomalous. These methods suffer from the fact that the augmentation is applied to the entire feature set, i.e. both $x_e$ and $x_c$. We argue that it is essential to only apply the hard augmentation to the core part $x_c$, as mentioned in lines 297 and 298. The reason is that the training data is often associated with a spurious correlation between $x_e$ and $y$, therefore, the learning algorithm may wrongly capture the variations in $x_e$ as signs of anomaly. To break this correlation, we suggested to only apply the hard augmentation on $x_c$ to obtain synthetic OODs while keeping $x_e$ unchanged. This helps in breaking the spurious correlation between $x_e$ and $y$, and preventing the algorithm to learn this spurious connection. All this analysis is provided in Sec. 4 Causal Viewpoint. Therefore, we believe that making a distinction between $x_c$ and $x_e$ in applying hard augmentations to make synthetic OODs is the major distinction between our method and prior algorithms, and potentially the reason why our algorithm works in our setup. The results also backup this hypothesis: looking at the table 3 in the ablation studies, and comparing setup D, in which "core estimation" is replaced with some random selection of the image patch to be augmented, against setup E, which is our proposed method, we see a big jump in the robust AUROCs.

Q2&W3:

Several methods based on knowledge distillation have been proposed for novelty detection tasks, and this remains an active area of research, as evidenced by works cited in references [1-7]. While our results on multiple challenging real-world datasets underscore the effectiveness of our approach, we believe our method introduces critical innovations that set it apart from existing knowledge distillation-based approaches:

Task-Based Knowledge Distillation:

We propose a novel task-driven knowledge distillation pipeline that classifies samples into two categories—'inliers' and 'outliers'—to learn the inlier distribution while incorporating additional information to selectively update parts of the teacher model’s weights. This approach diverges from conventional methods, which rely solely on the pretrained teacher's weights and are prone to biases originating from the dataset used for initial training. In contrast, our method integrates a task-specific objective, overcoming the limitations of existing approaches whose objective functions merely aim to mimic the teacher's weights without leveraging any auxiliary task.

Enhanced Loss Function for Robustness:

We introduce a new loss function designed not only to encourage the student model to mimic inlier features but also to actively diverge from outlier features. This augmentation significantly improves the robustness of the pipeline in handling distribution shifts—an aspect not addressed in prior knowledge distillation-based methods.

State-of-the-Art Robustness: Unlike previous works that primarily focus on traditional novelty detection benchmarks, our study highlights how distribution shifts can degrade the performance of existing methods. Our proposed model not only achieves state-of-the-art performance on standard benchmarks but also demonstrates superior robustness to distribution shifts. Please kindly refer to Table 1 for a comparison of average performance under clean and robust setups.

Theoretical Insights via Causal Perspective:

Our work also offers theoretical insights by adopting a causal viewpoint to identify and mitigate unwanted correlations between style features and labels—issues that often undermine the effectiveness of novelty detection methods. This theoretical framework supports the design of our task-based knowledge distillation strategy, enabling the model to focus on core features and enhancing robustness against style variations without the need for auxiliary datasets or metadata.

Extensive Ablation Studies:

We conducted extensive ablation studies on prior teacher-student methods, analyzing their loss functions (Appendix H) and pipelines (Section 7). The results demonstrate the significant superiority of our method over previous approaches.

References:

[1] Bergmann et al. Uninformed Students: Student–Teacher Anomaly Detection with Discriminative Latent Embeddings. CVPR 2020.

[2] Salehi et al. Multiresolution Knowledge Distillation for Anomaly Detection. CVPR 2021.

[3] Deng et al. Anomaly Detection via Reverse Distillation from One-Class Embedding. CVPR 2022.

[4] Cohen et al. Transformaly - Two (Feature Spaces) Are Better Than One. CVPR 2022.

[5] Guo et al. ReContrast: Domain-Specific Anomaly Detection via Contrastive Reconstruction. NeurIPS 2023.

[6] Cao et al. Anomaly Detection under Distribution Shift. ICCV 2023.

[7] Wang et al. Student-Teacher Feature Pyramid Matching for Anomaly Detection. 2022.

W2:

We appreciate the referee's feedback regarding the complexity of style variations in real-world scenarios and their impact on OOD detection, a point with which we fully agree. We would like to emphasize that the style shifts introduced during the training phase of our method are intentionally limited and minor. However, our proposed pipeline is evaluated extensively on complex, unseen style shifts during testing, highlighting its effectiveness and robustness.

To address the referee's concerns, we would like to further clarify that the test datasets analyzed in our paper encompass a broad spectrum of both synthetic and natural style variations, as detailed in Section J of the Appendix:

Synthetic Style Variations

• In datasets such as DiagViB-MNIST, we introduced artificial changes, including variations in texture, brightness, saturation, and spatial placement. These synthetic alterations are designed to emulate diverse stylistic shifts that can occur in real-world data, providing a controlled environment to rigorously evaluate the model’s robustness.

Natural Style Variations

• Autonomous Driving: We utilized large-scale datasets such as Brain Tumor, Chest CT-Scan, Cityscapes, and GTA5, which exhibit significant stylistic differences. For instance, Cityscapes captures German streets, while GTA5 represents synthetic U.S. streets, introducing natural variations in lighting conditions, road markings, and atmospheric effects. These datasets test the model's adaptability to a wide range of environmental and stylistic shifts. Camelyon-17: This dataset presents inter-hospital variations, arising from differences in imaging equipment, protocols, and lighting conditions. These variations realistically simulate the style shifts commonly encountered in medical imaging tasks, challenging the model to generalize features across diverse sources. By considering these datasets, we ensure that the model is rigorously tested across a wide spectrum of stylistic variations, ranging from synthetic alterations to realistic environmental and procedural changes. Moreover, our methodological framework is designed to capture core features that are invariant to style shifts, irrespective of their complexity.

While we acknowledge that real-world style variations can extend beyond those examined in our study, the datasets we selected provide a comprehensive and diverse set of challenging conditions to evaluate the robustness of our proposed method.

Finally, we wish to emphasize that our study does not claim to have fully resolved the robust ND problem. Instead, our goal is to highlight the critical importance of this issue in real-world scenarios and propose a theoretically grounded method that achieves superior performance compared to existing approaches, as demonstrated in both clean and robust settings. We view our work as the beginning of a longer research trajectory, not the conclusion of the journey.

Dear Reviewer LwHG,

Thank you for your review and useful comments. Specific comments are answered below:

W1&Q1:

We appreciate the reviewer’s observation regarding potential computational and time costs. Below, we address this concern with detailed points supported by empirical data.

1. Saliency Detection Occurs Only During Training

The saliency detection process is limited to the training phase, where it is used to craft auxiliary OOD samples. This means there is no computational overhead during inference, where real-time performance is critical.

Once the model is trained, saliency detection is no longer needed, ensuring that the inference process remains as efficient as standard methods.

2. Efficiency of Grad-CAM

Our method employs Grad-CAM, a lightweight and computationally efficient approach for saliency detection. Compared to other saliency methods, Grad-CAM introduces minimal overhead while generating effective saliency maps.

3. One-Time Computation Per Sample

Saliency maps are computed once per training sample and reused throughout the augmentation pipeline. This one-time computation avoids repetitive processing, keeping the overall training cost manageable.

4. Empirical Evidence of Training Overhead

Based on time complexity analysis of our method across various datasets in below table. Grad-CAM computation time is negligible and insignificant compared to the training and testing times, making it efficient for real-world applications.

The table shows that the additional computational cost during training is insignificant. Considering the performance improvements demonstrated in our work, this overhead is justifiable for practical applications.

We believe these clarifications address the reviewer’s concern. By emphasizing that saliency detection is confined to the training phase, leveraging the lightweight Grad-CAM approach, and demonstrating modest computational overhead with empirical data, we establish that the proposed method remains practical and efficient for real-world use cases.

Dear Reviewer LwHG,

thank you again for your review. We wanted to check in to see if there are any further clarifications we can provide. We hope that our updated PDF, including new experiments and explanations, effectively addresses your concerns.

Best, the authors

W2:

As is common in the general knowledge distillation literature, where information from different layers is utilized, we were inspired by this approach and aimed to adopt a similar strategy. However, we understand the reviewer’s concerns and provide the following responses:

Early layers in neural networks typically capture low-level features (e.g., edges, shapes) , while deeper layers focus on high-level semantic information. Our framework, however, is based on style features and core features, and while the intuition regarding layer functionality makes sense, it is important to note that low-level features are not equivalent to style features, nor are high-level features directly equivalent to core features.

For instance, in medical images, ID and OOD differences often revolve around tumor regions, which we consider core features in our study. These features are shape-based and may sometimes appear in shallow layers. Therefore, ignoring shallow layers and relying solely on deeper layers does not seem entirely acceptable. That said, we acknowledge that deeper layers provide meaningful representations and could serve as a good approximation of all layers in certain cases.

To further address the reviewer’s concerns, we conducted additional experiments where we dropped some shallow layers and compared the results with the default settings reported in the manuscript. For these experiments, we now consider four layers and evaluate two configurations:

Config 1: Using only the last two layers.

Config 2: Using the last three layers.

These results demonstrate that incorporating shallow layers, even minimally, contributes to performance improvements. However, they also indicate that their role is not pivotal.

Dear Reviewer pgZK,

Thank you very much for your feedback on our paper. Please find our responses below:

W1:

We acknowledge the concern regarding the extent of differences between crafted OOD samples and ID data. However, it is important to note that OOD samples can generally be categorized into two groups: (1) texture-level OOD samples, and (2) semantic-based OOD samples. In the first category, OOD samples exhibit significant differences from ID samples at the texture level while retaining some semantic similarity. In the second category, OOD samples are entirely different from ID samples in terms of semantics.

Notably, many challenging novelty detection benchmarks focus on detecting texture-level OOD samples, as they are more representative of real-world tasks. For example, in industrial production lines, detecting broken devices requires identifying subtle texture changes (e.g., the MVTeCad and Visa datasets). Similarly, in medical imaging tasks, ID and OOD differences often arise due to tumors or regional distortions (e.g., Brain Tumor or Chest CT-Scan datasets). For instance, while a brain image with a tumor may share similar global features with a healthy brain image, it is considered OOD due to its specific regional abnormalities.

In our study, we specifically aimed to craft texture-level OOD samples instead of semantic-based OOD samples for the following reasons:

Data Efficiency: Crafting texture-level OOD samples is more data-efficient. Generating new semantic-based OOD samples often requires very large generative models or extensive datasets, as demonstrated by previous methods such as Dream-OOD (Du et al., NeurIPS 2023).

Practical Usefulness of Near OOD: Studies have shown that near-OOD samples are more useful than far-OOD samples for many applications (ATOM, Chen et al., 2021; POEM, Ming et al., ICML 2022; VOS, Du et al., ICLR 2022). As discussed in our manuscript, near-OOD samples act as placeholders for inlier boundaries provide effective information for learning robust decision boundary . Although the definition of "near OOD" is still evolving, it is generally agreed that these samples share visual appearance similarities with ID samples but do not belong to the ID class.

As a result, we focused on crafting texture-level OOD samples. By distorting a significant portion of an image (e.g., 25% of an important region), we can shift it from ID to OOD while maintaining some similarity to ID samples. This makes them near-OOD samples, which, according to the above definition, are more useful for practical applications.

Finally, to fully address the reviewer concern, we conducted an additional experiment across multiple datasets. In this experiment, we focused on distorting all core regions of the images by identifying contours using Grad-CAM maps and applying distortions to all regions within these contours. The results from this experiment will be included in the revised manuscript, demonstrating how varying distortion levels impact the model's performance and its ability to distinguish between ID and OOD samples. This decrease in performance might be attributed to crafting OOD samples that are too far removed from the ID data. For instance, instead of distorting specific brain regions, we applied distortions to the entire brain region to generate OOD samples.

By replacing our default generation strategy with this approach while keeping all other components fixed, we obtained the following results:

Fair enough. No futher questions on my side. I personally find this paper interesting so I raised my score.

Dear Reviewer pgZK,

Thank you for your valuable review and positive feedback! We are delighted that you found our work interesting.

Sincerely, The Authors.

Dear Reviewer 2JvA, Thank you for taking the time to review our paper and for the insightful comments. Here is our detailed response:

W1:

There are no more intuitive experiments to demonstrate that the unwanted correlation between style features and labels has been weakened.

we evaluated our method on both synthetic and real-world datasets to rigorously test its ability to focus on core features while being agnostic to style features. For instance, in DiagViB-MNIST, style variations such as brightness, texture, and spatial placement were introduced, and the performance improvements shown in Table 1 demonstrate the model’s robustness in isolating core features. Similarly, the significant performance gains on real-world datasets like Cityscapes and GTA5, which feature large-scale environmental and stylistic shifts, further support our claim. These datasets inherently challenge the model to generalize under varying style conditions, and our method consistently outperforms existing approaches in robust performance metrics (see Tables 1 and 18).

An additional intuitive experiment showcasing our method's contribution to weakening the correlation between style features and labels is the ablation study (Table 3). Here, we extend the experimental setups to analyze the impact of including or excluding crafted OOD data during training. The results, summarized below, underscore the importance of our approach across diverse dataset, demonstrating notable improvements when OOD data is incorporated.

These results showcases the effectiveness of our method in reducing reliance on style features.

W2:

The OOD generation strategy heavily borrows from methods in self-supervised learning that preserve and do not preserve semantics

Using transformations such positive augmentation for style shift is a minor aspect of our method, intentionally employed because such augmentations are well-known to preserve semantics.

W2:

there are not many improvements to the knowledge distillation model, which makes the innovation somewhat lacking.Improve the knowledge transfer mechanism in the distillation process, or develop new technologies to more effectively extract knowledge from the teacher model.

Several knowledge distillation-based methods have been proposed for novelty detection tasks, highlighting this as an active area of research, as demonstrated by references [1-7]. While our results on multiple challenging real-world datasets emphasize the effectiveness of our approach, particularly when compared to knowledge distillation methods such as ReContrast, we believe our method introduces critical innovations that distinguish it from existing knowledge distillation-based techniques. Task-Based Knowledge Distillation:

We propose a novel task-driven knowledge distillation pipeline that classifies samples into two categories—'inliers' and 'outliers'—to learn the inlier distribution while incorporating additional information to selectively update parts of the teacher model’s weights. This approach diverges from conventional methods, which rely solely on the pretrained teacher's weights and are prone to biases originating from the dataset used for initial training. In contrast, our method integrates a task-specific objective, overcoming the limitations of existing approaches whose objective functions merely aim to mimic the teacher's weights without leveraging any auxiliary task.

Enhanced Loss Function for Robustness:

We introduce a new loss function designed not only to encourage the student model to mimic inlier features but also to actively diverge from outlier features. This augmentation significantly improves the robustness of the pipeline in handling distribution shifts—an aspect not addressed in prior knowledge distillation-based methods.

State-of-the-Art Robustness: Unlike previous works that primarily focus on traditional novelty detection benchmarks, our study highlights how distribution shifts can degrade the performance of existing methods. Our proposed model not only achieves state-of-the-art performance on standard benchmarks but also demonstrates superior robustness to distribution shifts. Please kindly refer to Table 1 for a comparison of average performance under clean and robust setups.

Theoretical Insights via Causal Perspective:

Our work also offers theoretical insights by adopting a causal viewpoint to identify and mitigate unwanted correlations between style features and labels—issues that often undermine the effectiveness of novelty detection methods. This theoretical framework supports the design of our task-based knowledge distillation strategy, enabling the model to focus on core features and enhancing robustness against style variations without the need for auxiliary datasets or metadata.

Extensive Ablation Studies:

We conducted extensive ablation studies on prior teacher-student methods, analyzing their loss functions (Appendix H) and pipelines (Section 7). The results demonstrate the significant superiority of our method over previous approaches.

References:

[1] Bergmann et al. Uninformed Students: Student–Teacher Anomaly Detection with Discriminative Latent Embeddings. CVPR 2020.

[2] Salehi et al. Multiresolution Knowledge Distillation for Anomaly Detection. CVPR 2021.

[3] Deng et al. Anomaly Detection via Reverse Distillation from One-Class Embedding. CVPR 2022.

[4] Cohen et al. Transformaly - Two (Feature Spaces) Are Better Than One. CVPR 2022.

[5] Guo et al. ReContrast: Domain-Specific Anomaly Detection via Contrastive Reconstruction. NeurIPS 2023.

[6] Cao et al. Anomaly Detection under Distribution Shift. ICCV 2023.

[7] Wang et al. Student-Teacher Feature Pyramid Matching for Anomaly Detection. 2022.

Q1:

Near-OOD samples, which share similar appearances with ID samples but differ in their core features, are more useful and informative compared to far-OOD samples. These samples provide challenging examples that force the model better delineate the decision boundary between ID and OOD. The mask size parameter $\alpha$ controls the proportion of the image to be distorted, directly impacting the crafting of OOD samples.

As discussed in the manuscript and supported by prior work, selecting an appropriate range for $\alpha$ ensures that the distorted core regions shift the sample away from the ID class while leaving some regions intact to resemble ID samples. This balance is essential to maintain the near-OOD property, allowing the model to focus on meaningful differences while learning robust boundaries. Furthermore, randomizing $\alpha$ increases the diversity of the crafted OOD samples, which enhances robustness by exposing the model to a broader range of scenarios.

The results in Table 8 (which also provided below) demonstrate the impact of different mask sizes on performance across various datasets.

From Table it is evident that $\alpha$ influences performance. For instance, using a very high mask ratio (e.g., 80%-100%) results in far-OOD samples that differ greatly from ID, leading to reduced performance.

Finally, the method demonstrates robustness to minor variations in $\alpha$, as shown by the relatively stable performance across different settings. This indicates the reliability and flexibility of our approach in crafting effective OOD samples.

Q2:

In Theorem 1, we demonstrated that improving detection performance between ID and real-OOD requires the distribution of synthetic OOD samples to closely resemble that of real-OOD samples. Notably, in many real-world scenarios, ID data is naturally similar to OOD. Thus, achieving this proximity between synthetic and real anomalies can be effectively accomplished through minor distortions of ID data, which is precisely the approach we adopted in our method.

Dear Reviewer 2JvA,

we have aimed to thoroughly address your comments. If you have any further concerns, we would be most grateful if you could bring them to our attention, and we would be pleased to discuss them.

Sincerely, The Authors

Dear 2JvA,

Apologies for reaching out again, but we are keen to hear your feedback before the rebuttal period concludes. Your insights have been incredibly valuable, and we hope the new experiments and revisions to the manuscript allow you to reevaluate your score.

Thank you!

Dear Reviewer 2JvA,

This message serves as a friendly reminder that the discussion period is nearing its conclusion. We have submitted our rebuttal addressing your comments and are hopeful for your feedback. Please let us know if we have successfully addressed your concerns. Should there be any outstanding issues, we are more than willing to continue the dialogue.

Sincerely, The Authors

Q1:

As we mentioned in the manuscript, ID and OOD samples differ in core features across all considered datasets, and our study aims to develop a robust ND method that learns core features while being agnostic to style features. As noted by the reviewer, core features represent content and semantics. We kindly ask the reviewer’s attention to Figure 4, which includes visualizations illustrating these differences.

For example:

• Brain Tumor Dataset: ID and OOD samples differ based on the presence of a tumor in the brain region.

• Waterbirds Dataset: The distinction between ID and OOD samples lies in the semantics; land birds and water birds are categorized based on their species and context.

• MVTecAD Dataset: ID and OOD samples are differentiated by their condition, where ID samples are intact instances, and OOD samples represent broken devices.

• Colored MNIST Dataset: ID and OOD samples are differentiated similarly based on content and semantics.

These examples, along with the visualizations in Figure 4, demonstrate the relevance of core features and their critical role in robust ND methods.

Q2:

We believe that many of the datasets we considered exhibit the mentioned attribute. For instance, in the MVTecAD and VisA datasets, all style features are identical between the ID and OOD samples, while the core features differ and is unrelated to the style. In the mentioned dataset, the core feature is distortion: images without distortion are categorized as ID, whereas those with distortion are categorized as OOD.

W1:

We understand the reviewers' concern that previous ND methods often aim to improve performance on specific datasets without considering robustness against shifted tasks, leading to unfair comparisons. However, we have corresponding responses to address this concern:

Comparison with Existing Methods

Several methods, such as GNL and RedPanda, have aimed to develop robust ND methods targeting problems similar to ours. However, these methods rely on additional supervision to achieve robustness. In contrast, as shown in the results (please refer to Table 1), our method outperforms these approaches.

General Robustness and Performance

Our primary focus is on developing a robust ND method rather than merely improving performance on specific datasets. Nonetheless, our method achieves:

• Higher average performance across multiple datasets compared to existing ND methods.
• Low standard deviation (STD) across various datasets, demonstrating reliability when applied to new datasets.
• Significantly higher robust performance with a notable margin.

Please refer to Table 1 and Table 18 for detailed results.

Strengths and Novelty of Our Approach

We acknowledge that part of our method’s robustness stems from incorporating style shifts during training to mitigate the impact of nuisance features during testing. However, our method also introduces several novel and effective components, including:

1. A novel OOD crafting strategy
2. A task-based teacher-student pipeline

These modules collectively enable our model to be robust against a wide variety of natural and synthetic unseen style shifts. It is important to emphasize that the style shifts introduced during training are limited and general (e.g., color jitter). This highlights that our method is not tailored to specific observed style shifts but achieves robustness more broadly.

Motivation

Why Developing a Robust ND Method is Important?

Robust ND is critical in various real-world scenarios where environmental variations can lead to style shifts. Below are some examples:

1. Autonomous Driving

• Scenario: An ND model trained on images of roads in one city (e.g., Berlin) might encounter roads in another city (e.g., Los Angeles) with different lighting conditions, weather, or architectural styles.
• Importance: The system must reliably distinguish unusual objects like pedestrians or animals on the road (novelties) regardless of the style differences, such as sunny versus rainy conditions.

2. Medical Imaging

• Scenario: Medical images such as MRI or CT scans might be captured using different equipment or imaging protocols across hospitals, leading to style shifts in the data.
• Importance: Detecting anomalies like tumors or lesions should rely on core pathological features rather than stylistic variations introduced by imaging devices or techniques.

3. Industrial Quality Control

• Scenario: Automated inspection systems in factories may analyze products under different lighting conditions or camera settings.
• Importance: The system must detect defective products or anomalies regardless of changes in visual style caused by environmental or equipment variations.

4. Video Surveillance

• Scenario: Surveillance systems deployed across different locations or times of day may face variations in background, lighting, or weather conditions.
• Importance: Detecting suspicious activities or objects should remain unaffected by these style shifts, ensuring consistent performance in diverse settings.

5. Wildlife Monitoring

• Scenario: Cameras deployed in different ecosystems or under varying weather conditions may capture images with substantial stylistic differences.
• Importance: Identifying new species or unusual animal behavior requires robustness to such style shifts.

6. Retail and E-commerce

• Scenario: ND models used to monitor inventory might encounter different lighting, packaging designs, or shelf arrangements across stores.
• Importance: Detecting misplaced or counterfeit items should not depend on these stylistic changes.

7. Satellite and Aerial Imaging

• Scenario: Satellite images of the same location might appear different due to atmospheric conditions, seasons, or times of day.
• Importance: Detecting deforestation, urban development, or natural disasters requires focusing on core changes rather than irrelevant stylistic variations.

8. Cybersecurity

• Scenario: Network traffic data might vary in structure due to changes in protocols or encryption methods.
• Importance: Robust ND is essential to detect novel cyber-attacks while ignoring benign variations in network activity style.

Dear Reviewer nLRW,

Thanks for your valuable comments on our paper. Below, we provide a detailed response to your questions and comments:

Novelty Detection

We would like to clarify potential misunderstandings regarding the problem we addressed and the proposed method. To provide clarity, we first review the setup of our problem.

The Novelty Detection (ND) problem involves a scenario where one class is designated as the in-distribution (ID) semantic, while all other semantics are considered as out-of-distribution (OOD). The primary goal in novelty detection is to develop a detector model, denoted as $f$, to distinguish between ID and OOD concepts.

To formalize this task, a detector model $f$ and a dataset $D$ are considered, where one class, such as $X$, is designated as the ID class, while the remaining classes, $D \setminus X$, constitute the OOD samples. A superior detector is one that more effectively distinguishes ID (i.e., $X$) from OOD (i.e., $D \setminus X$).

The difference between $X$ and $D \setminus X$ is based on a set of features, which we categorize into two types: core features and style feature. Core features capture the essential semantic differences between ID and OOD, while style features represent non-essential differences that do not generalize well across datasets. In this study, we aim to focus on learning core features and disregard style features.

For example, consider a specific subset of the colored MNIST dataset, $D$, where $X$ consists of green images of the digit 0'' (ID), and $D \setminus X$ consists of red images of the digit 1'' (OOD). The primary core feature here is the distinction between the digits 0'' and 1.'' However, there is also a style feature difference---namely, the color of the digits. A detector could distinguish ID from OOD based on either core features (digit identity) or style features (color).

In this study, our objective is to learn core features, as they are more robust and transferable to new datasets. For instance, consider a new dataset $D'$ during inference, where $X'$ consists of blue images of the digit 0'' (ID), and $D' \setminus X'$ consists of blue images of the digit 1'' (OOD). Here, the style feature (color) remains the same, and a model that relied on style features for distinguishing ID from OOD would perform no better than random guessing. In contrast, a model that learned core features would still perform effectively, as the core feature (digit identity) remains consistent.

Dear Reviewer nLRW,

We sincerely appreciate the thoughtful feedback you’ve offered on our manuscript. We have carefully reviewed each of your comments and have made efforts to address them thoroughly. We kindly ask you to review our responses and share any additional thoughts you may have on the paper or our rebuttal. We would be more than happy to accept all your criticisms and incorporate them into the paper.

Sincerely, The Authors