Rotation Has Two Sides: Evaluating Data Augmentation for Deep One-class Classification

We sincerely thank Reviewer jwSM for providing insightful suggestions. Below, we address your constructive comments individually.

W1: The reason for the contrastive pre-training in the first stage.

Contrastive pre-training enables the model to extract non-trivial image representations, allowing us to measure the distribution discrepancy in the representation space instead of low-level raw pixel space. The first stage of contrastive pre-training is crucial to make the entire pipeline unsupervised, although the method is not limited to self-supervised feature extractors. Alternatively, using a more powerful feature extractor pre-trained on large-scale datasets in a supervised manner yields improved recall and precision scores. See Q2 for experimental details.

W2: Large-scale dataset evaluation.

We conduct experiments on the TinyImageNet which has 100,000 images of 200 classes. As the two-stage pipeline in our manuscript, the first stage pre-trains the encoder by using SimCLR and the second stage learns rotation distribution. Likewise, we use the prediction (Eq.(2)) to identify RAI and non-RAI. Given the partition of RAI and non-RAI, we again pre-train the model by using SimCLR, following the practice [5] to deal with RAIs and non-RAIs, i.e., RAIs and their rotations are postive while non-RAIs and their rotations are negative. For fair comparison, we keep the same hyper-parameters in the pre-training except for the different separations of RAIs and non-RAIs. We adopt liear probing and report the top-1 classification accuracy. We improve the baseline by +2.59% and we attribute the gain to the enhanced distinction between RAI and non-RAI.

W3: Missing quantitative evaluation of the RAI predictions.

For quantitative evaluation, binary labels of RAI and non-RAI for the CIFAR-10 training set are manually annotated. The rotation prediction model [3] is initially employed to identify RAI samples incorrectly predicted at 0 degrees, which are then corrected manually as ground-truth RAIs. The remaining samples are designated as non-RAI. The prediction precision and recall of RAI, organized by class, are reported in the table. The table shows that our method can accurately identify RAIs in the training set.

We sincerely thank Reviewer Yjs1 for providing insightful suggestions. Below, we address your constructive comments individually.

W1: Its applicability remains limited to rotation-related datasets and methods.

Our work extends beyond rotation-related datasets and methods. We additionally conduct experiments on translation distribution prediction in the SVHN dataset (See the Section 4.3 in the updated manuscript). Our experimental evaluation on MvTec-AD, adopting patch representations of 32x32 built upon UniCon-HA, demonstrates our method's superiority compared to counterparts that incorporate rotation augmentation. Though our method lags behind state-of-the-art anomaly detectors, we underline the need for tailored pretext tasks [1] or leveraging pre-trained models [2] in the context of industrial anomaly detection.

The reported Image/pixel-level AUROC scores are as follows:

W2: The improvements (Table 1) are somewhat modest.

Our approach, altering loss functions to handle semantics-preserving/-shifting samples separately, consistently achieves improvements on powerful baselines. While the average improvement over ten classes on one-class CIFAR-10 (Table 1) is not significantly high, it is noteworthy that different classes contain varying RAI ratios. Larger improvements are achieved for classes with a higher proportion of RAIs. For example, for the cat class, which has the largest number of RAIs (see Figure 4(c)), we obtain +1.0% based on CSI and +1.2% based on UniCon.

W3: Unclear reasons for improvement.

Based on CSI, we present separate results for RAI and non-RAI on one-class CIFAR-10. The table below displays the results for the classes which include a relatively high proportion of RAI samples. Our method significantly enhances OCC performance on RAI, especially for classes with a larger proportion of RAI samples, such as plane and cat.

W4: Paper structure.

Figure 1 and Figure 2 illustrate the correlation between rotation prediction and OCC. Figures 4-7 provide detailed analyses of the effects of RAIs and non-RAIs. We reorganize the structure in our final version.

W5: Use the complete version of UniCon.

Built upon the complete version of UniCon-HA, which is a very strong OCC detector, it is noted that we also achieve the consistent improvement (+0.2%) on one-class CIFAR-10.

[1] Li et al. Cutpaste: Self-supervised learning for anomaly detection and localization. In CVPR, 2021.

[2] Paul et al. Uninformed students: Student-teacher anomaly detection with discriminative latent embeddings. In CVPR, 2020.

[3] Sohn et al. Learning and evaluating representations for deep one-class classification. In ICLR, 2021.

[4] Wang et al. Unilaterally aggregated contrastive learning with hierarchical augmentation for anomaly detection, In ICCV, 2023.

We sincerely thank Reviewer vKuc for providing insightful suggestions. Below, we address your constructive comments individually.

W1(Q1): Quantitative accuracy in RAI vs. non-RAI determination.

For quantitative evaluation, binary labels of RAI and non-RAI for the CIFAR-10 training set are manually annotated. The rotation prediction model [3] is initially employed to identify RAI samples incorrectly predicted at 0 degrees, which are then corrected manually as ground-truth RAIs. The remaining samples are designated as non-RAI. The prediction precision and recall of RAI, organized by class, are reported in the table. The table shows that our method can accurately identify RAIs. The results can be further improved by leveraging a more powerful pre-trained feature extractor, as demonstrated in W2.

Trainig set:

W2(Q2): Failure modes.

In fact, the last table shows that our model has misclassified images and these incorrect predictions stem from imperfect representations learned in contrastive pre-training, which are contextually biased towards spurious scene correlations [1, 2]. To address this, we suggest using a more powerful feature extractor pre-trained on large-scale datasets. For instance, leveraging an encoder pre-trained on ImageNet-1k has resulted in improved outcomes in the identification of both RAI and non-RAI. The table shows the results on the training set of CIFAR-10.

W3: Transformation distribution vs. OCC.

While rotation prediction as a pretext task has been explored in the self-supervised literature and facilitates bundles of downstream tasks, its significance becomes more pronounced in OCC, where the effectiveness relies solely on the design of the pretext task to capture normal patterns. Furthermore, our empirical experiments reveal a strong linear relationship between the accuracy of rotation prediction and the performance of OCC. This observation motivates us to delve into investigating how the separation between RAI and non-RAI impacts OCC. While we have achieved improved results in multi-class classification on TinyImageNet, evaluated by linear probing. We have revised the manuscript to provide additional clarity on this aspect.

Q1: Determination of RAI and non-RAI. Yes. Images with a prediction of 0 are deemed non-RAI, i.e., $argmax_r{p_r(x))}$ $r \in$ {0,90,180,270} equals 0, while those with any other predictions are considered RAI.

Q3: Determination of a semantically-shifting sample. Which $r$ is used?

Semantically-shifting samples are determined by comparing the original image x and its rotated version $R_r(x)$ ($R_r$ denotes rotating $x$ by $r$ degrees, $r$ belongs to {0, 90, 180, 270}). An image $R_r(x)$ satisfying $p(R_r(x)) < p(x)$ is considered semantically-shifting. The determination is conducted by considering all four rotation angles. Note that for a RAI sample, at least one rotated sample is semantically-preserving.

[1] Mo et al. Object-aware Contrastive Learning for Debiased Scene Representation. In NeurIPS 2021.

[2] Purushwalkam et al. Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases. In NeurIPS, 2020.

[3] Miyai et al. Rethinking rotation in self-supervised contrastive learning: Adaptive positive or negative data augmentation. In WACV, 2023.

We sincerely appreciate your recognition of the novelty and motivation behind our work. Below, we address your constructive comments individually.

W1: Lack of empirical validation on datasets other than CIFAR-10.

We additionally conduct experiments on the TinyImageNet which has 100,000 images of 200 classes. As the two-stage pipeline in our manuscript, the first stage pre-trains the encoder by using SimCLR and the second stage learns rotation distribution. Likewise, we use the prediction (Eq.(2)) to identify RAI and non-RAI. Given the partition of RAI and non-RAI, we again pre-train the model by using SimCLR, following the practice [5] to deal with RAIs and non-RAIs, i.e., RAIs and their rotations are postive while non-RAIs and their rotations are negative. For fair comparison, we keep the same hyper-parameters in the pre-training except for the different separations of RAIs and non-RAIs. We adopt liear probing and report the top-1 classification accuracy. We improve the baseline by +2.59% and we attribute the gain to the enhanced distinction between RAI and non-RAI.

For the experiments on SVHN, refer to responses to W2 for details.

W2 (Q1 and Q2): Other transformations.

To address the need for exploring other transformations, we have extended our evaluation to include translation on the Street View House Numbers dataset (SVHN). The goal is to classify the central digit in each image into one of the categories (0 through 9). We consider five translation directions (original, up, down, left, right) with an 8-pixel shift. Notably, our approach is not confined to self-supervised pre-training. The first stage trains the ResNet-18 from scratch in a supervised way, achieving 96.24% accuracy on the test set. The second stage predicts the 5-way translation distribution. As shown in the updated manuscript, it showcases the model’s ability to correctly identify off-centered digits. See Section 4.3 in the updated manuscript for more discussion on the experiments on SVHN.

W3: Experiments across other models.

We have explored ViT-based models, specifically ViT-B/16, for rotation distribution prediction in CIFAR-10. Additionally, we considered variations in architecture types and pre-training strategies (self-supervised vs. supervised). For quantitative evaluation, binary labels of RAI and non-RAI for the CIFAR-10 training set are manually annotated. The provided table demonstrates the architecture-agnostic nature of our method, with ResNet-18 leading to a larger improvement than ViT-B/16. This is attributed to the lack of inductive bias in ViT, which struggles to learn local relations well with a small amount of data, such as CIFAR-10. The results also highlight our method's preference for the supervised training strategy, emphasizing the significance of well-trained image representations, as indicated by average recall and precision values across the 10 classes in CIFAR-10.

[1] Miyai et al. Rethinking rotation in self-supervised contrastive learning: Adaptive positive or negative data augmentation. In WACV, 2023.

Thank you for appreciating and acknowledging our work. We address your constructive comments below:

W1: Related works.

We have incorporated more discussion on relevant OCC literature in the revised manuscript. Anomaly detection (AD) and OCC share similarities, aiming to identify instances deviating from normal training data and providing a binary classifier to determine normality or anomaly. The key distinction lies in the training data characteristics. AD considers both normal and anomalous instances during training, while OCC is confined to only normal instances in training. Therefore, OCC can be regarded as a special case of AD. When all training data belongs to one class, AD and OCC methods are expected to yield similar results.

W2(Q2): Missing citations.

We appreciate the reviewer's suggestion for additional references. We previously noted that [1] uses the similar idea with [2] to identify RAI and non-RAI by training a binary classifier on the noisy dataset while [2] improves the idea by leveraging an extra validation set to approximate the epoch just before over-fitting. We have included it and discussed its relevance in the revised version.

[1] Fent et al. Self-Supervised Representation Learning by Rotation Feature Decoupling. In CVPR, 2019.

[2] Miyai et al. Rethinking rotation in self-supervised contrastive learning: Adaptive positive or negative data augmentation. In WACV, 2023.