Continual Supervised Anomaly Detection

Thank you for your valuable review and suggestions.

• On tabular datasets such as UNSW, bank, and credit, the model does not significantly outperform the other baselines. In many cases, the performance is similar to that of the binary classifier, suggesting that the performance is due to the binary classifier included in the method rather than the proposed method.

As you mentioned, the difference between the performances of our method and the binary classifier is very small in Table 1. However, the performance of the binary classifier should decrease when the number of anomalies available for training becomes small because the binary classifiers are not suitable for detecting unseen anomalies. To confirm this, we conducted some experiments when the number of anomalous instances available for training was reduced to 1/5. The following are the results.

We can see that while the performance of the binary classifier has dropped significantly, the performance of the proposed method has not dropped that much. The difference is especially obvious for the credit dataset, which is tabular data.

• Many anomaly detection methods in the image domain are not designed for continual learning, but since EWC and A-GEM can be applied, it would be meaningful if the proposed method outperforms in this setting.

To compare our method with the latest supervised anomaly detection method, we have implemented BGAD[3], mentioned by Reviewer QxJr, and conducted experiments. The experimental results are shown below. [3] Yao X, Li R, Zhang J, et al. Explicit Boundary Guided Semi-Push-Pull Contrastive Learning for Supervised Anomaly Detection. CVPR, 2023.

Although the method uses the trained backbone model to extract the feature maps from the inputs, it was removed in the experiment above because it does not fit our paper's problem settings. This is because one of the goals of this paper is how to continuously train models from scratch from multiple tasks consisting of a small number of data, and the availability of models trained on large data sets in advance, such as backbone models, is inconsistent with this goal. The hyperparameters for BGAD basically followed its paper and GitHub implementation. We have added the details of the experiments to our manuscript. From the above results, we can see that our method outperforms the latest supervised anomaly detection methods.

• In the image domain, the proposed method shows better performance than other baselines, but it seems that experiments on larger datasets are needed to show the practicality of the proposed method. The method was only tested on FMNIST and MNIST with MLP structure, but it would be useful to test it on larger datasets such as CIFAR10 and CelebA.

As we mentioned in the replay to Reviewer vozw, our method is not domain-specific and is applicable to other domains, including tabular data. We conducted experiments on MNIST and FMNIST to show that the proposed method can detect anomalies even in image domains, and we believe these results are helpful enough to show that. Of course, using the proposed method on difficult image datasets would also be possible by using powerful generative models (VQ-VAE, GAN, diffusion models, etc.) instead of VAE. A more in-depth investigation of proposed methods for specific domains is a topic for future work.

• Similar to other continuous-learning papers, it would be nice to be able to see how performance changes with additional training on each task.

Thank you for your advice. We will add to the paper the performance changes with additional training on each task.

We thank you for reading our paper and your review comments.

• The novelty of the proposed framework is limited.

As you mentioned, our method consists of a binary classifier and a VAE, which are very simple and basic modules. However, this does not mean that the novelty of the proposed method is limited. In fact, it is not obvious how to combine VAE with a binary classifier and how to perform generative replay from them. Our method's novelty is not in using VAEs or binary classifiers but in finding a way to use them to solve the problem of continual supervised anomaly detection. For example, in Eqs. (7)-(9), the formulation of generating anomalies is naturally derived by appropriately combining VAE and a binary classifier. In addition, its simplicity has the advantage that it can be easily extended if necessary (e.g., by changing the generative model or the classifier).

• This paper does not include a comparison with some of the latest supervised anomaly detection methods such as DRA[1], PRN[2], BGAD[3], which might be relevant for a more comprehensive evaluation.

Thank you for your comment. For a more comprehensive evaluation, we have conducted the experiments with BGAD[3]. The following are the results.

Although the methods you mentioned [1-3] rely on the trained backbone model to extract the feature maps from the inputs, it was removed in the experiment above because it does not fit our paper's problem settings. This is because one of the goals of this paper is how to continually train models from scratch using multiple tasks consisting of a small number of data, and the availability of models trained on large datasets in advance, such as backbone models, is inconsistent with this goal. The hyperparameters for BGAD basically followed its paper and GitHub implementation. We have added the details of the experiments to our manuscript. From the above results, we can see that our method outperforms the latest supervised anomaly detection methods.

• Is there a more detailed explanation regarding the impact of the number of seen anomalous samples on the experimental results?

Thank you for your question. To investigate the impact of the number of seen anomalous instances to be used for training, we conducted some experiments when the number of seen anomalous instances available for training was reduced to 1/5. The following are the results.

We show only the results for the supervised anomaly detection methods because the performances of the unsupervised anomaly detection methods (i.e., AE and VAE) do not depend on it. The complete table has been added to the manuscript. Note that in the experiment, only in the credit dataset, the performance of the proposed method is not at the top, but the difference between the performances of the top methods and ours is still small enough. The table shows that our method shows stable performance regardless of the number of anomalous instances for training.

We sincerely appreciate your elaborate reading of our paper and your insightful comments.

• It represents a straightforward combination of ideas.

As you mentioned, our method looks very simple, but it does not mean the lack of novelty of our method. For example, the formulation for generating anomaly instances from VAEs is not a mere heuristic, but is derived naturally from the structure of our model. This was obtained by appropriately combining VAEs and binary classifiers to create our model. In fact, our method outperforms the conventional methods.

• The argument that one cannot keep data due to privacy reasons also applies for resampling data from an autoencoder. If it is a very good reconstruction, it would equally cause privacy issues.

Yes, the generator could cause privacy issues if the generated instances are of high quality. However, the risk should be reduced compared to keeping the data directly. For example, the webpage of SyntheticData4ML workshop says that "Synthetic data is regarded as one potential way to promote privacy. The 2019 NeurIPS Competition "Synthetic data hide and seek challenge" demonstrates the difficulty in performing privacy attacks on synthetic data." Further investigation into this issue is planned for future work. https://www.syntheticdata4ml.vanderschaar-lab.com/

• The important and relevant case of slow distribution shift is only partially addressed, via the credit data. Doing so in more and more controllable settings would be of interest.

Many previous studies on continual learning focused on mitigating catastrophic forgetting, and thus, they are more interested in quickly changing data distributions than in slowly changing ones. This is because catastrophic forgetting is unlikely to occur when the shift in distribution is small. However, as you mentioned, it is important to understand how well the proposed method works in a problem setting where the distribution changes gradually because such situations can occur in real-world applications. In this paper, we investigate such a situation on a credit dataset and confirm that our method is superior to the others. We believe that the superiority of the proposed method will probably remain unchanged in such situations where the distribution change is small.

• A comparison against reusing past training data would be of interest - in particular from tasks a few epochs ago. Reason being that a slow shift of parameters would also affect sampling of data from not so recent past tasks.

Thank you for your suggestion. The situation in which data from previous tasks can be used differs from the problem settings in this paper and the usual continual learning studies, making the problem much easier. Such a problem setting would be interesting, but may be outside the scope of this paper.

• MNIST and FMNIST might be too simple as problem. A more complex image dataset is missing.

Please note that our proposed method is not domain-specific, although its application to more challenging image datasets is important. Our approach can be applied to non-image datasets such as tabular data. We have included experimental results on MNIST and FMNIST to confirm that our proposed method can also be used in the image domain, and we believe that they played a sufficient role in this regard. Of course, it would also be possible to use the proposed method on difficult image datasets by using powerful generative models (VQ-VAE, GAN, diffusion models, etc.) instead of VAE. Such considerations are the subject of future work.

The reviewer recognizes this result. However with 32x32 one has the same limited spatial complexity as for mnist/cifar-10/100 . with 32x32 the complexity of it is largely gone.