Self-Tuning Self-Supervised Anomaly Detection

We thank the reviewer for the constructive and detailed feedback. We provide our answers to the weaknesses and questions pointed out by the reviewer. Some of our answers point to the common responses.

W1. The authors evaluate MV Tech AD with the data being collected according to given abnormal patterns. I wonder if the proposed framework can simultaneously deal with multiple different augmentations, since this case is more general in real-world anomaly detection scenarios and is more likely to avoid overfitting to the optimal a. It will be helpful to show the results with UNsplit original classes in MV Tech AD.

Our current approach can be subpar on test data that contains anomalies of multiple types, for which tuning the discrete choice of augmentation functions becomes a new challenge. Our work presents a framework to tune the continuous hyperparameters of an augmentation function, provided with the discrete choice. Please refer to C2 of our common responses for possible hyperparameter optimization (HPO) approaches we laid out as future work.

W2. The assumption that the validation and test distributions are consistent may be too strong for the setting of anomaly detection task (e.g. new anomaly pattern or even mixing anomalies makes a not optimal). It may not be reasonable to omit test images drawn from unseen distributions in real scenarios.

We believe that transductive learning is common in real-world anomaly detection tasks. Please refer to C1 of our common responses for an extended discussion on transductive vs. inductive learning in the context of anomaly detection. Nevertheless, we also conducted additional experiments dividing unlabeled test data into validation and (new) test data, and then using only the validation data for computing the alignment loss. Please refer to C3 #2 of our common responses for the results.

W3. The author believes that the heuristic function $S$ will be high (higher variance of anomaly scores) only if augmentation parameters are initialized better (more separable distributions). However, this may not always hold, as the optimization of a mismatched strong/hard augmentation (as long as the decision boundary divides testing distribution into any two parts) may also lead to high variance of anomaly scores. I wonder if there are any theoretical or intuitive explanations about this issue.

We remark that this heuristic is employed only after optimizing for the alignment first. In other words, we use it only to choose from among models that are already optimized to align with the test data as well as possible. Since a decision boundary dividing the test data would not yield a low validation loss in the first place, the heuristic is not likely to be exposed to such scenarios in practice.

W4. The authors use CutDiff and Rot for non-semantic and semantic shift detection respectively. But I would like to know could ST-SSAD learn zero-angle Rot for non-semantic defects and zero-size CutDiff for semantic shifts in end-to-end optimization when using both augmentations together.

As discussed in detail in C2 of our common responses, our current model is designed to tune the continuous hyperparameters of a given augmentation function. Given a set of augmentation functions, making a discrete choice between the different functions or optimizing a mixture of functions together poses a new challenge. For example, with both CutDiff and Rotate included, the ratio of augmented samples to generate with the two functions becomes another hyperparameter. Being the first solution to the automatic model selection for SSAD, we expect our work will trigger several follow-up works.

W5. The term “Ratio invariance” may be imprecise. With the augmentation quantity changed, $L_\mathrm{val}$ could be changed together because the included $\sqrt{N} / \| Z^C \|_F$ is different (I notice that this have no negative impact on optimization target).

That is correct. The total distance normalization is affected by the size of augmented data, and thus our loss function is not exactly invariant to it. However, the effect is negligible, since that mainly changes the scale of embeddings, rather than their distributions. For preciseness, we have changed the term into "ratio robustness" in the revised manuscript.

W6. There are some writing issues, e.g. superscript $^n$ should be $^a$ in Lemma 2.

Thank you for finding out the typo. We have updated the revised manuscript accordingly.

We thank the reviewer for the constructive and detailed feedback. We provide our answers to the weaknesses and questions pointed out by the reviewer. Some of our answers point to the common responses.

W1. During evaluation, each anomaly type is separated. The proposed ST-SSAD seems to assume only one anomaly type exists in the test set. That is, the user might need to use ST-SSAD for each anomaly type. How to handle multiple anomaly types in the same test set is not clear.

Q1. In the experiments, anomaly types are separately evaluated. That seems to mean that the anomaly type is known, and the augmentation parameters are learned to match the anomaly type. This seems to be assumed in Equation 3 because the second term calculates the mean of the augmented instances. That is, Eq. 3 seems to be finding an augmentation and its parameters to match the anomaly type. If that is correct, how can multiple types of anomalies in the same test set be handled?

As the reviewer accurately pointed out, our approach does not currently address test data that contain anomalies of multiple types, for which tuning the discrete choice of augmentation functions becomes a new challenge. Our work presents a framework to tune the continuous hyperparameters of an augmentation function, provided with the discrete choice. We refer to C2 of our common responses for possible hyperparameter optimization (HPO) approaches we laid out as future work toward tackling a mixture of anomalies in test data.

Q2. While the augmentation parameters are learnable, the augmentation types such as cut and rotation need to be specified. Also, to be detected, seemingly an anomaly type in the test set needs to match one of the augmentation types. Could the matching be relaxed? That is, the user potentially does not need to know what the anomaly types are.

As we show in Section 4.4, our approach succeeds if the augmentation function is aligned with the true anomalies at the functional level. As discussed in C2 of our common responses, the discrete choice of the augmentation from a set of possible functional forms can be addressed via black-box hyperparameter optimization (HPO). Our work presents a framework to tune the continuous hyperparameters provided with the discrete choice. We expect several follow-up works, not only proposing new differentiable augmentation functions but also more elaborate HPO techniques to select from this set, as laid out in C2.

Q3. What is the size of $D_{aug}$?

We set the size of $D_{aug}$ to be the same as that of $D_{trn}$ in all experiments. This was to avoid introducing an additional hyperparameter. To show robustness, we conducted additional experiments varying the ratio of augmented data over the size of training data. Please refer to C3 #1 of our common responses for the results.

Q4. Eq 2, equation on the right for $z_i^c$: should $z$ be $z_i$ and the summation is over another index such as $j$ to not confuse with $i$?

That’s correct. We have changed Eq. (2) accordingly in the revised manuscript. Thank you for pointing it out.

Q5. Eq 3: mean(.) seems to be similar to $1/N \sum_{i=1}^N z_i$ in Eq 2. If so, using mean(.) in Eq 2 would make the presentation consistent. If not, what is the difference?

Both expressions are almost the same: mean(.) in Eq. (3) and $1/N \sum_{i=1}^N z_i$ in Eq. (2). The only difference is that mean(.) in Eq. (3) is an operator on a set of vectors, while in Eq. (2) we computed the mean of rows in a matrix. It would make them consistent if we treat both as sets or both as matrices, but we believe the current way is more intuitive and easier to understand.

Q1. Can you provide results using a validation set that is disjoint from the test set?

Yes, we have conducted additional experiments dividing unlabeled test data into validation and (new) test data, and then using only the validation data for computing the alignment loss. Please refer to C3. #2 of our common responses for the results.

Q2. I guess the proposed method may be quite sensitive to the proportion of abnormal samples in the test set. Can you provide experimental results or a discussion addressing this issue? And please provide the ratio between normal and abnormal samples in the presented results of the current manuscript.

Our datasets already vary in the proportion of anomalies they contain in the test data. The following table shows the percentage of anomalies across different tasks in the MVTec dataset, where the minimum and maximum are 0.147 and 0.404, respectively. Similarly in the SVHN dataset, the fraction is between 0.279 and 0.722, since the different classes contain different numbers of examples. Thus, we believe that our approach is not sensitive to the ratio of anomalies in test data, as long as the anomalies are separable from the normal data and thus the alignment can be measured by our validation loss.

Q3. Is it possible for the model to learn effectively in a scenario where abnormal samples are inherently present in the training set but remain unlabeled?

Yes, this refers to the setting where training data is not “clean” (i.e. anomaly-free) but is contaminated with some anomalies. We have conducted additional experiments by varying the degree of contamination in the training data. Please refer to C3 #3 of our common responses for the results.

We thank the reviewer for the constructive and detailed feedback. We provide our answers to the weaknesses and questions pointed out by the reviewer. Some of our answers point to the common responses.

W1. A major concern is that ST-SSAD replies on the entire test dataset during training and tuning. While the authors mention transductive learning, this approach is not quite realistic, particularly in the context of anomaly detection. The tuning result will be overly sensitive to the specific anomaly types in the test data and may not generalize well. The paper lacks clarification, experimental results, or in-depth discussion on this issue, which significantly limits the applicability and advantages of the proposed method.

We believe that transductive learning is common in real-world anomaly detection tasks, whereas generalization is a matter for inductive learning. Please refer to C1 of our common responses.

W2. The mean distance loss is proposed for ratio invariance with theoretical properties, but no experimental result validating this invariance is provided.

We have conducted additional experiments varying the ratio of augmented data over the size of training data. Please refer to C3 #1 of our common responses for the result.

W3. The method still requires prior knowledge about anomalies and heavily depends on it. For example, the augmentation functions of either local (CutDiff) or global (rotation) augmentations are considered and therefore, it works well only when anomalies closely resemble specific shapes that these functions can reflect. The method will also fail in the case where rotated samples are considered normal.

As we discussed in Section 4.4, our approach succeeds if the augmentation function is aligned with the true anomalies at the functional level. Nevertheless, as the first systematic approach to the automatic model selection for self-supervised anomaly detection, we believe our work can lead to numerous future works that extend our approach to new differentiable augmentations that capture other types of real-world anomalies. Please refer to C2 of our common responses for more discussion on the problem setup and suggested directions of future works.

W4. The authors state that 'we focus on the performance of each anomaly type rather than overall accuracy.' However, in real-world scenarios, it is common to encounter various types of anomalies. Therefore, it will be more crucial to investigate such practical scenarios.

Our current approach does not address test data that contain anomalies of multiple types, for which tuning the discrete choice of augmentation functions becomes a new challenge. Our work presents a framework to tune the continuous hyperparameters of an augmentation function, provided with the discrete choice. As we answered to W3, please refer to C2 of our common responses for possible hyperparameter optimization (HPO) approaches we laid out as future work toward tackling a mixture of anomalies in test data.

W5. It was mentioned that "there are no direct competitors on end-to-end augmentation hyperparameter tuning..."; however, it is essential to include performance comparison with the latest models that clearly distinguish train and test data. The results in Tables 1 and 2 appear to be more like an ablation study, so it is difficult to assess whether the proposed method truly outperforms the latest models in a meaningful way.

Please note that it is difficult to conduct a fair comparison between our approach and other works on anomaly detection, since the extent of “fair” hyperparameter tuning is hard to define between different lines of works. A vast majority of work on unsupervised anomaly detection does not address the unsupervised hyperparameter (HP) tuning problem at all.

Our results show significant performance boost over two most prominent unsupervised approaches; DeepSVDD (one-class) and DeepAE (reconstruction-based). On the self-supervised side, we show that tuning augmentation significantly outperforms arbitrarily choosing the augmentation, including the CutOut and CutPaste approaches (CO and CP in Table 1) proposed by Devries & Taylor (2017) and Li et al. (2021), respectively. Since we used the detector network and the score function used in (Li et al., 2021) throughout all experiments, we can consider the RS-CP and RD-CP baselines in Table 1 as "untuned" versions of the CutPaste approach (Li et al., 2021).

C3. Additional experiments

In light of reviews, we performed 3 additional experiments: #1) varying augmentation data sizes, #2) validation-test split, and #3) contaminated training data. The results are also given in Appendix E of our revised manuscript.

All experiments were done in the MVTec dataset, specifically the Cable object and the 8 anomaly types associated with it, since we didn’t have time to run experiments for all cases. The numbers in parentheses represent the ranks, and we aggregate the results over different tasks by the average accuracy and the average rank.

#1) Varying augmentation data sizes. Our validation loss is carefully designed to be robust to $|D_{aug}| / |D_{trn}|$, since the ratio of true anomalies in test data is unknown at training time. Consequently, we simply set $|D_{aug}| = |D_{trn}|$ in all experiments, such that we perform one augmentation per training example. Specifically, instead of using all training data in every computation of the validation loss, we randomly sample 256 training samples (and 256 augmentation samples) at each computation for efficiency. The number 256 is chosen large enough to estimate the distribution of all training data.

In this additional experiment, we vary the number of augmentation samples that are used in the computation of the validation loss following { 64, 128, 256, 512 }, where 256 is the choice in the original experiments. The performance is not sensitive to the size of augmented data, and we obtain similar results across eight different tasks.

#2) Validation-test split. Although we focus on transductive learning, where unlabeled test data are given at training time, we perform an additional experiment where we split the original test data into two disjoint sets of validation and (new) test data, with the size ratio 1:1, and use only the validation data in the computation of our alignment loss. This is to create an evaluation setting where the evaluation data (i.e., new test data) are completely separated from the data observed in training time.

The following table shows the result. We find a negligible difference between the two settings, demonstrating that our approach succeeds with the separated test data.

#3) Contaminated training data. We assumed a problem setting where training data consist of only normal samples, as done in many previous works on anomaly detection. However, one may consider this setting unrealistic, since acquiring pure clean data is often hard in real-world scenarios. Thus, we conduct experiments with contaminated training data, which contain a small fraction of anomalies in it. We increase the ratio of contamination up to 2%, since the number of anomalies in the MVTec dataset is not large enough to increase more.

The following table shows the result. The overall performance is similar in the three settings, showing that our approach is robust to the small fraction of anomalies in training data.