Anomaly Detection by Context Contrasting

Dear reviewer yEuR,

We would like to thank you again for your detailed review. Below, we answer your questions and address your concerns.

Since there are existing contrastive anomaly detection methods as described in the experiments, I think the proposed approach's novelty is not outstanding. Also, the differences between context and other ordinary transformations are not clearly explained. For example, can we use context transformations in the same way as ordinary transformations? (The same applies in reverse). I want to know the key differences between context augmentations and ordinary transformations.

The proposed Con2 contains two parts. In the first part, the samples are projected into different contexts using context augmentations. In the second part, the samples are structured according to their content given a context. For contexts, augmentations should fulfill the two properties, distinctiveness and alignment. For content augmentations, any augmentation typically used for instance-discrimination can be leveraged. The main difference between context and content augmentations lies in their role within our loss function. Content augmentations are designed to capture sample invariances, while context augmentations define a new representation space where all normal samples remain consistent with one another (alignment) but appear distinct from their original forms (distinctiveness).

There are many anomaly detection approaches other than the contrasting-based approach described in Section 2. Thus, this paper would be improved by showing the effectiveness of the proposed method through experimental comparisons with these methods.

Contrastive learning-based approaches are indeed only one of the multiple approaches to anomaly detection. However, contrastive learning approaches have continuously outperformed other self-supervised learning approaches, e.g., [1, 2], when applied to the contamination-free AD setting. Our proposed method itself is also a contrastive learning method. These are the two reasons we only compare it to other contrastive learning-based anomaly detection methods.

In Figure 4, the significant difference between the proposed and existing methods is unclear.

We clearly outperform even the best baseline approaches for 6 out of 10 different normal classes (airplane, car, cat, dog, frog, truck) with at least one of the proposed context augmentations. The flip augmentation also outperforms all baseline methods when comparing their average performance across all classes. Additionally, we highlight Con2's clear advantages on specialized medical datasets, where the approach of generating synthetic anomalous samples, like in CSI and UniCON, is more challenging.

The problem settings only have unlabelled or normal data. How are the hyperparameters of the proposed method practically determined?

We follow best practices in this work and choose the hyperparameters based on the validation set, which can contain only normal or a combination of normal samples and anomalies. For our algorithm, we chose the default parameters from previous works, e.g., the augmentation strengths of the SimCLR augmentations for the content objective. We only anneal the alpha value that weights the different loss functions because of the different scaling between the content and the context loss (see more details in response to Reviewer h61g).

Can the proposed method be applied to domains other than the image domain?

Yes, it can. The proposed Con2 objective is general and can be applied broadly across multiple modalities. However, we would need to define an objective for the content loss either by leveraging natural invariances in the data, e.g., time-series data, or designing meaningful augmentations. In addition, we also need to define the context object with augmentations that reflect distinctiveness and alignment. We view this as a promising direction for future work and plan to explore it in follow-up studies.

Thank you again for your feedback. We welcome any additional suggestions, questions, or requests for information and encourage further discussions.

Dear reviewer h61g,

We would like to thank you again for your detailed review. Below, we answer your questions and address your concerns.

The novelty of the proposed method is limited.

We respectfully disagree with the reviewer’s opinion on this point. We introduce a novel contrastive learning objective that can leverage different contexts, defined by distinctiveness and alignment, to improve anomaly detection performance. Unlike previous works, which try to simulate anomalous samples, the proposed work is based on modeling properties of the normal samples, which is much simpler.

The advantage of the proposed method is unclear compared with the existing anomaly detection methods with data augmentation by flipping, inverting, and equalizing. What are the advantages of the proposed method compared with the existing anomaly detection methods with data augmentation?

The primary advantage of our method is its independence from prior knowledge about anomalies, unlike previous approaches that rely on simulating anomalous samples [1,2]. Instead, we leverage prior knowledge about the normal data available in the training set, making our method particularly suitable for specialized domains such as medicine.

Further, our assumptions of distinctiveness and alignment of context-augmented samples are directly incorporated into the two loss terms of Con2. These assumptions enable us to learn compact and informative representation spaces while avoiding challenges like hypersphere collapse encountered in previous methods [4].

Preparing contexts might require prior knowledge for the anomaly. Can any context that fullfills distinctiveness and alignment be useful in the proposed method?

The design of context augmentations only requires knowledge about the normal class and not anomalies. State-of-the-art works like UniCon and CSI try to model anomalies, which relies on having additional knowledge about anomalous data. The proposed method instead creates context augmentations based on knowledge about normal samples, which we have access to during training and, hence, is much simpler. In principle, any augmentation that satisfies the two properties can be utilized with Con2. However, it is important to note that no single augmentation typically fulfills both assumptions for every sample in a dataset. As a result, we cannot guarantee that all feasible augmentations will perform identically.

How can we tune alpha?

We did not experience the need for tuning the hyperparameter $\alpha$ extensively. We only anneal $\alpha$ from 0 to 1 because of the different ranges between context and content loss. This difference arises because the context loss incorporates multiple positive samples, which results in different lower bounds for the two loss terms. Without annealing, the content loss would dominate the Con2 loss, causing the context loss to be neglected for a significant portion of the training. This imbalance can result in suboptimal local minima. However, in our opinion, there is no need to further tune this hyperparameter as the described setting works well across all our datasets.

How to chose LH or NND?

The NND score is prominent among representation learning-based approaches, e.g., [1, 2, 5, 6]. It works well (see the reported results, incl. the updated results for the Melanoma dataset). However, computing the NND for the full set can be computationally expensive, especially for larger training datasets. Therefore, we propose a simple and lightweight likelihood-based score that works almost as well and is suitable for resource-constrained settings.

The proposed method depends on the context transformation to be used (Equalize, Invert, Flip). How can we choose appropriate transformation for given applications?

The new method, Con2, highlights that augmentations – if they fulfill the properties of distinctiveness and alignment – can be used to map samples into different context spaces. Equalize, Flip, and Invert are example augmentations that we found to work well in the image domain, both on natural and medical images. However, there might be additional augmentation classes that work equally well. Different augmentations might be needed for other modalities, and we will continue exploring additional modalities in future work. Designing context augmentations is about finding transformations such that the transformed and original sample follow the properties of distinctiveness and alignment (see also section 3.1 in the original manuscript). We are not sure what you mean by “for given applications”? We would appreciate a clarification of the question.

Dear reviewer ENmc,

We would like to thank you again for your detailed review. Below, we answer your questions and address your concerns.

There is somewhat misleading about covering pre-trained based methods as "outlier detection". There is a vast number of papers showing that pre-trained features, while reliant on seeing images not in the training set, are much more generalizable than outlier exposure. Namely, strong visual features go beyond purely adding prior knowledge about specific anomalies [1][2]. Only one none-SSL method is compared, and not the most standard one

We focus on SSL methods due to their superior performance in AD, when we only have access to normal samples. Our focus is on leveraging AD for the medical domain, where – in general – we do not have access to large, internet-scale datasets that would allow the pretraining of methods. Exploring pre-trained models in more specialized areas is an interesting direction for future work.

The main results table, justifying the focus on SSL, shows only two datasets. Table 1 shows the results of the experiments in the medical domain, which we see as the most targeted application for the proposed Con2 objective. However, we present additional convincing results on general benchmark datasets such as CIFAR 10, CIFAR 100, Imagenet 30, Cats vs. Dogs, and Muffin vs. Chihuahua.

The paper claims (line 46) "without assuming prior knowledge about anomalies". I find that misleading, prior are always needed in order to generalize anything to an unseen test. Even more so in unsupervised settings. In this case, the priors are the inductive biases induced by the augmentations among other things.

We agree with the reviewer that the referred statement can be misleading, and we always leverage prior knowledge to some extent. We want to emphasize that we only leverage prior knowledge about normal samples when designing the objective function. The same applies to the design of context augmentations, which model the distinctiveness and alignment of normal samples. This differs from previous works that try to model or incorporate knowledge about anomalies in their objective functions. We updated the manuscript in the following way:

Unlike previous works, which focus on prior knowledge about anomalies, the proposed Con2 models properties of normal samples, which is particularly useful in more specialized data, such as in the medical domain, which we demonstrate in our experiments.

I found Figure.1 more confusing than helpful.

Thanks for this comment. We always seek to improve the individual parts of our submission(s). However, we would appreciate some more specific details about what could be improved in Figure 1.

Please compare to more pre-trained-based methods. E.g., (Cohen & Avidan, 2022; Reiss & Hoshen, 2023; Li et al., 2023) cited by your paper. Please extend this comparison to more medical datasets. E.g., ChestX-ray14, HAM10000.

Adding more datasets and benchmark results is always valuable. However, we strongly believe that the chosen set of baseline methods and datasets is valuable and shows the advantages of the proposed method. In addition, implementing and running more experiments is not feasible in the short amount of time during the rebuttal, given our computing resources.

Thank you again for your feedback. We welcome any additional suggestions, questions, or requests for information and encourage further discussions.

I thank the authors for their response.

“Exploring pre-trained models in more specialized areas is an interesting direction for future work.”

I understand that’s not the focus of your work, but (i) I think the manuscript is still misleading in classifying all methods that use pre-training as "Outlier Exposure" (ii) I am still not convinced that “pure SSL” method outperform other methods, even on the evaluated dataset (they might, but the paper does not show it convincingly in my opinion)

“We want to emphasize that we only leverage prior knowledge about normal samples when designing the objective function”

Respectfully, that's still not accurate. Prior knowledge also includes inductive bias - some assumptions regarding unseen samples. Prior knowledge must be used even in supervised classification (e.g. “No Free Lunch” theorems), and even more so in unsupervised settings. I do agree that the wording in the revised version is not as problematic.

“We always seek to improve the individual parts of our submission(s). However, we would appreciate some more specific details about what could be improved in Figure 1.”

A few options are: (i) Color the arrows in more distinct color (ii) Remove the figure (iii) Keep it if other readers find it useful

In any case, that’s not a major concern for me. Just trying to help :-)

“we strongly believe that the chosen set of baseline methods and datasets is valuable and shows the advantages of the proposed method.”

I respectfully disagree that it shows that. A main claim by the paper (the advantage on SSL methods over pretrained ones in specific cases) is supported with the evaluation of a single pretrained method, on only two dataset

Therefore, I remain with my initial score.

Thank you for your thoughtful feedback. We respectfully disagree with some of the comments made and provide additional clarifications below.

Outlier Exposure and Pretrained Models

As defined in [9], Outlier Exposure (OE) involves external datasets and pretrained models inherently fall under this category since they are trained on such datasets. Additionally, the superiority of contrastive SSL AD methods over other SSL approaches is well-documented in [1,2,5].

Prior Knowledge and Assumptions

We do not introduce additional assumptions about anomalies beyond those already made for normal samples. Could the reviewer clarify where our reasoning may diverge from this?

Main Claims of the Paper

Our main claim is not about comparing SSL methods to pretrained models but rather about the advantage of leveraging prior knowledge of normal samples over simulating anomalies, which is the current state-of-the-art for contrastive SSL-based AD [1,2]. We believe the evaluations included are sufficient to support this claim.

We hope these clarifications address your concerns and thank you again for your feedback.

[9] Hendrycks, Dan, Mantas Mazeika, and Thomas Dietterich. "Deep anomaly detection with outlier exposure." arXiv preprint arXiv:1812.04606 (2018).

Outlier Exposure and Pretrained Models

While both Pretrained Models and other Outlier Exposure use pre-trained data, they succeed or fail in different situations. Therefore, I believe that while the terminology may differ between papers, the distinction is important. I am not sure to which paper list the numbers [1,2,5] refer to, but to the best of my knowledge, pretrain models outperform in most cases.

Prior Knowledge and Assumptions

The claim about the use of prior knowledge is not specific to your method. All methods that aim to generalize must rely on some prior assumptions. The original paper's language was confusing; I agree the new language is not as problematic.

Main Claims of the Paper

1. I think that if that’s the paper's main claim, the focus on SSL should be better motivated.
2. Comparing a single non-pure-SSL dataset on only two datasets to claim it outperforms is even more problematic than stating you choose to focus on SSL for another reason.

Dear reviewer tEZB,

We would like to thank you again for your detailed review. Below, we answer your questions and address your concerns.

The method is sensitive to context augmentation methods, which is demonstrated in evaluation part as well. However, there is no systematic method about how to choose proper augmentation method. If use wrong augmentation, this method may perform worse than baselines. (see Table 1). Suggestion: you may consider using validation data to automatically choose augmentations, or if you could provide guidelines for selecting augmentations based on dataset characteristics.

We thank the reviewer for their remark. The two properties introduced in our manuscript, distinctiveness and alignment, serve as guidelines for selecting augmentations tailored to the characteristics of the dataset. Further, we can compare context augmentations by monitoring the validation loss on a separate set of normal samples and choosing the augmentation that results in the lowest validation loss.

Even consider optimal augmentation method only, the performance gain to baselines is marginal in many cases. For example, I cannot tell Con2 is better than baselines in Figure. 4. For instance in Figure 4, UniCon-HA outperform Con2 on Bird, Deer, Horse, and Ship, 4 out of 11 classes. I would suggest statistical tests to demonstrate the significance of their performance gains over baselines, such as t-Test. It would also be helpful to measure the epistemic uncertainty by runing multiple times with different random seed, so that you will know if the the 7 out 11 performance gain is the worst case or best case.

In our experiments with natural image datasets, we demonstrate that our approach matches or surpasses the performance of previous contrastive methods on these benchmarks. Additionally, we highlight its clear advantages on specialized medical datasets, where generating synthetic anomalous samples is more challenging. Considering the large body of excellent research already conducted in anomaly detection, these results are very promising.

Figure 1 is not referred at the beginning of the paper but it is put to page 2, and it is hard to understand. The loss function is introduced on page 5, section 3.2, but it is mentioned here without any explanation or reference. To address this, the author may refer to figure 1 in the later part of introduction, when introducing the proposed solutions. Please add a cross reference to the definition of loss function in the caption of Figure 1.

Figure 2 and Figure 3 are not on the page they are refered to. Please nsure all figures are properly placed and referenced in the final version.

Thank you for spotting this. We added more cross-references to the figures and the loss functions. However, we believe it to overly strict to force all tables and figures to be on the same page as where they are mentioned for the first time.

The second anomaly score is questionable, start from line 306, page 6. With augmentation, we should obtain different representations of the input data, some of them maybe aligned with existing normal data well. Therefore, we should use some metric like maximal score to highlight the matching, instead of averaging out the potential matching (might be sparse). Suggestion: you might compare the current approach with a max-based score and discuss the trade-offs between the two approaches.

Please clarify why the average is used in the second anomaly score? From the definition, I think a maximal value based metric makes more sense. Please add this comparision in section 4, or around the ablation study

Can you please clarify what the "o" means in page 6 line 311?

Can you please explain the second metric for anomaly score? Why take the average 1/A and why sum between 1~A/2 interval?

We follow previous work [1, 2] by using the average score over test set augmentations, and note that this is generally the standard when using test-time augmentations [3].

We take the sum over $A/2$ values as there are two sums inside the brackets and both sum over $A/2$ values, which in total sums over $A$ terms. Hence, we also scale the loss with $1/A$.

The symbol $\\circ$ defines a composition of functions, see Wikipedia, Function Composition or [8], and $t_i \\circ t_{\\mathcal{C}}= t_i(t_{\\mathcal{C}}(\\cdot))$ .