Deep Positive-Unlabeled Anomaly Detection for Contaminated Unlabeled Data

Thank you for your feedback, which we shall address below.

Weakness 1: The setting and motivation do not appear to be novel in the context of anomaly detection. For example, the noisy-AD (SoftPatch) setting in industrial anomaly detection has been previously explored. Therefore, the authors should provide a detailed comparison and discussion with such related work.

We would like to emphasize that the problem setting we address is a well-established one in the field of semi-supervised anomaly detection. Our contribution lies in proposing a novel framework that integrates PU learning with deep anomaly detectors under this setting.

Thank you very much for pointing us to SoftPatch. Upon reviewing the paper, we found that SoftPatch [1] is an unsupervised anomaly detection method, and therefore not directly comparable to our semi-supervised approach. Nevertheless, we agree that it is a relevant and interesting work, and we will include it in the related work section of the revised paper to provide a more comprehensive discussion.

[1] Jiang, Xi, et al. "Softpatch: Unsupervised anomaly detection with noisy data." Advances in Neural Information Processing Systems 35 (2022): 15433–15445.

Weakness 2: The proposed method seems to be a combination of Positive-Unlabeled (PU) learning and existing anomaly detectors, such as Variational Autoencoders (VAE). As a result, it lacks significant novelty.

First, we would like to clarify a minor misunderstanding: our method is based on Autoencoders (AE), not Variational Autoencoders (VAE).

More importantly, existing anomaly detectors such as AE or DeepSVDD cannot be directly combined with PU learning, as PU learning is formulated for binary classification, while anomaly detectors typically follow a one-class formulation.

Our contribution lies in bridging this methodological gap. As detailed in Section 3, we design a tailored risk estimator and training objective to effectively integrate PU learning with deep anomaly detection. We believe this design represents a key aspect of our novelty.

Weakness 3: The writing of the paper requires substantial improvement. For instance, the explanations of unsupervised anomaly detection and semi-supervised anomaly detection are unclear and need to be more precise and comprehensive.

We provided explanations of the problem setting in Section 2.1, unsupervised anomaly detection in Section 2.2, and semi-supervised anomaly detection in Section 2.3.

We sincerely apologize if any parts were unclear. To help us improve the clarity of the revised version, could you kindly let us know which aspects were difficult to follow? We would greatly appreciate your feedback and will revise the manuscript accordingly.

Weakness 4: The experiments are insufficient. The paper only reports the performance using two classical anomaly detectors: VAE and SVDD. Moreover, the chosen baselines may not represent the state-of-the-art (SOTA) methods in this field. A more extensive evaluation with additional SOTA methods would strengthen the validation of the proposed approach.
Methods And Evaluation Criteria: The compared methods are not up-to-date, with the most advanced method SOEL published in 2023. In other anomaly detection fields like industrial anomaly detection, the author may find more advanced methods for comparisons.

We would like to clarify that SOEL (Li et al., 2023) is currently one of the state-of-the-art methods for semi-supervised anomaly detection under label contamination. SOEL is based on DeepSVDD (Ruff et al., 2018), a representative unsupervised anomaly detector. Accordingly, we include DeepSVDD and its extensions as baselines, including DeepSAD (Ruff et al., 2019), LOE (Qiu et al., 2022), and SOEL itself.

In addition, AE (Hinton & Salakhutdinov, 2006) remains a widely used baseline in the literature, and we also compare it with its semi-supervised extension, ABC (Yamanaka et al., 2019). For completeness, we further include Isolation Forest (Liu et al., 2008) as a representative shallow detector, along with a standard PU learning-based binary classifier (Kiryo et al., 2017).

We believe that our evaluation is both up-to-date and comprehensive, covering state-of-the-art methods and their fundamental models. Our framework is compatible with various base detectors as long as their loss functions are non-negative and differentiable, and has shown empirical improvements when applied to AE and DeepSVDD. If there are particular methods you believe should be included in our comparison, we would be happy to take them into consideration.

Thank you for your feedback, which we shall address below.

Experimental Designs Or Analyses: However, the paper lacks sufficient details in the experimental settings, such as data preprocessing steps, specific parameters of the model architecture, and hyperparameter settings during training. These details are crucial for other researchers to reproduce the experiments and validate the results. It is recommended to provide more detailed experimental settings in the paper to ensure the reproducibility of the results and facilitate further research.

Thank you for the suggestion. While the current version includes many experimental details such as network architecture (following Ruff et al., 2018), optimizer settings, training schedule, and dataset splits, we agree that providing further clarification would improve reproducibility. We will revise the paper to include more complete descriptions of experimental settings, possibly in tabular form to enhance readability.

Question 1: The paper's core idea is to use PU learning to handle unlabeled data contaminated with anomalies, but can this method adapt to the data characteristics and anomaly detection needs of different fields in practical applications? For example, can the framework remain robust when there's a certain distribution shift in the overall data?

Similar to other semi-supervised anomaly detection approaches such as SOEL, our current approach is not designed to explicitly handle distribution shifts. We agree that distribution shift is a critical real-world challenge, and we plan to address it in future work. Fortunately, several approaches have been presented for adapting PU learning under distribution shift [1, 2], and we will explore incorporating such approaches into our semi-supervised anomaly detection framework.

[1] Hammoudeh, Zayd, and Daniel Lowd. "Learning from positive and unlabeled data with arbitrary positive shift." NeurIPS 2020.
[2] Kumagai, Atsutoshi, et al. "AUC Maximization under Positive Distribution Shift." NeurIPS 2024.

Question 2: Has the paper fully considered the issue of unknown anomaly detection? Ideologically, can it further expand the modeling and recognition strategies for unknown anomalies to enhance the model's generalization when facing new anomalies?

Although it is important to provide theoretical guarantees for detecting unseen anomalies, our current approach offers only empirical and qualitative support for detecting unseen anomalies.

What our approach guarantees is that, when labeled anomalies are available, the anomaly detector can be trained robustly even if similar anomalies are also present in the unlabeled data. As for detecting unseen anomalies, the performance mainly depends on the base anomaly detector. Anomaly detectors can treat data far from normal data as anomalies. Therefore, as long as unseen anomalies lie far from the normal data, they can be detected.

Empirically, we have also observed in Appendix C that we may be able to improve the detection performance for unseen anomalies by using seen anomalies. Providing theoretical guarantees for this is an important direction for our future work.

Thank you for your feedback, which we shall address below.

Question 1: The compared methods are relatively outdated. Both the semi-supervised and unsupervised approaches used for comparison are from earlier works. Many new models may outperform those based on AE and DeepSVDD, so it is necessary to include newer baselines.

We would like to clarify that our experimental setup includes SOEL (Li et al., 2023), which is currently one of the state-of-the-art methods for semi-supervised anomaly detection under label contamination. SOEL is based on DeepSVDD (Ruff et al., 2018), a representative unsupervised anomaly detector. Accordingly, we include DeepSVDD and its extensions as baselines, including DeepSAD (Ruff et al., 2019), LOE (Qiu et al., 2022), and SOEL itself.

In addition, AE (Hinton & Salakhutdinov, 2006) remains a widely used baseline in the literature, and we also compare it with its semi-supervised extension, ABC (Yamanaka et al., 2019). For completeness, we further include Isolation Forest (Liu et al., 2008) as a representative shallow detector, along with a standard PU learning-based binary classifier (Kiryo et al., 2017).

We believe that our evaluation is both up-to-date and comprehensive, covering state-of-the-art methods and their fundamental models. Our framework is compatible with various base detectors as long as their loss functions are non-negative and differentiable, and has shown empirical improvements when applied to AE and DeepSVDD. If there are particular methods you believe should be included in our comparison, we would be happy to take them into consideration.

Question 2: The generalization of the method has not been validated on commonly used industrial anomaly detection datasets, such as MVTec and VisA datasets.

As practical benchmarks, we chose PathMNIST, OCTMNIST, and TissueMNIST from the MedMNIST datasets (https://medmnist.com/), which was also used in the SOEL paper. Although the name "MNIST" may be misleading, we emphasize that these datasets are real medical datasets.

Our intention was to provide a fair comparison under the same conditions as SOEL, and our approach consistently outperforms SOEL on these datasets. In future work, we plan to include experiments on MVTec and VisA.

Question 3: The paper only evaluates AUROC at the image level. However, pixel-level AUROC and other evaluation metrics, such as PR-AUC, have not been validated.

Our approach is designed for image-level anomaly detection, where the goal is to determine whether an entire image contains anomalies, rather than localizing them at the pixel level. Therefore, pixel-level AUROC is not applicable to our method, as it does not produce pixel-wise anomaly scores.

We followed the SOEL paper in using image-level AUROC, which we believe is the most appropriate metric for evaluating this setting. In future work, we will consider including additional evaluation metrics suitable for image-level detection, such as image-level PR-AUC.

Question 4: What happens if fewer labeled anomalies are available? In many real-world tasks, labeled anomalies are extremely scarce. The authors do not analyze how performance degrades when only a few labeled anomalies are available.

In our experiments, we used 250 labeled anomalies. For example, even when this number is reduced to 50, the superiority of our proposed method remains. When evaluated on MNIST, our PUSVDD achieved an AUROC of 0.994, while SOEL obtained 0.963.

Thank you for your feedback, which we shall address below.

Weakness 1: ... conducting experiments on real datasets ...

We provide a qualitative evaluation using a toy dataset in Figure 1, and quantitative evaluations using real datasets such as SVHN, CIFAR10/100, and Path/OCT/Tissue-MNIST in Section 5. Although the name "MNIST" may be misleading, these are real medical datasets, and were also used in the SOEL paper, which represents the state-of-the-art in this field.

To better clarify behavior on real data, we will include qualitative evaluations on these datasets in the revised paper.

Weakness 1: Furthermore, why does PU learning fail to detect unseen anomalies, while integrating PU learning with deep anomaly detectors enables the detection of such anomalies?
Weakness 2: As a key contribution, why does integrating positive-unlabeled learning with deep anomaly detectors help the detection model effectively estimate anomaly scores for test data and detect anomalies that are unseen during training? ...

As shown in Figure 1a, since conventional PU learning is designed for binary classification, its decision boundary lies between the seen anomalies and normal data. As a result, even if unseen anomalies are far from the normal data, they cannot be detected.

In contrast, our approach integrates PU learning with deep anomaly detectors, combining
(1) PU learning’s ability to approximate the normal distribution using unlabeled and anomaly data, and
(2) the anomaly detector’s ability to treat data far from this normal data distribution as anomalies.
As a result, our approach can detect unseen anomalies as long as they deviate from normal data distribution. Tables 4 and 6 demonstrate that our approach outperforms conventional PU learning in detecting unseen anomalies.

Weakness 3: ... unlabeled anomalies and labeled anomalies are likely to originate from different distributions ...

When labeled and unlabeled anomalies originate from different distributions, our approach treats all unlabeled data as normal, similarly to ABC and DeepSAD. We emphasize that even in such cases, if the unlabeled anomalies are few in number and lie far from the unlabeled normal data, they can be detected due to the nature of the base anomaly detector.

Our target scenario assumes human-in-the-loop feedback, where labeling a few clear anomalies in unlabeled data is feasible. Our goal is to use such limited supervision to improve anomaly detection performance in this practical and cost-effective setting.

To handle stronger distribution shifts of anomalies, we plan to incorporate PU methods designed for selection bias [1, 2].

[1] Kato, Masahiro, Takeshi Teshima, and Junya Honda. "Learning from positive and unlabeled data with a selection bias." International Conference on Learning Representations. 2019.
[2] Wang, Xutao, et al. "PUE: Biased positive-unlabeled learning enhancement by causal inference." Advances in Neural Information Processing Systems 36 (2023): 19783–19798.

Weakness 4: The assumption that hyperparameter $\alpha$ is known during training is not appropriate ...

We assume that the labeled and unlabeled anomalies come from the same distribution. Under this assumption, as described in Section 3.1, $\alpha$ can be estimated using existing PU learning methods (e.g., Menon et al., 2015).

In practice, $\alpha$ reflects the proportion of unlabeled data that resembles the labeled anomalies. Therefore, in the scenario described in Weakness 3, where labeled and unlabeled anomalies come from different distributions, the estimated value of $\alpha$ would be small. In such cases, our approach behaves similarly to existing semi-supervised approaches such as ABC and DeepSAD.

Moreover, even if $\alpha$ is not estimated accurately, our approach is empirically more robust to its value than existing baselines, as demonstrated in Figure 2.

Weakness 5: The authors mention that the proposed can be applied to self-supervised methods, how does it perform?

Our main contribution lies in using PU learning to approximate the loss for normal data using only unlabeled data and labeled anomalies. Hence, as long as the loss function for normal data is defined, our approach can be applied to any anomaly detection models, as described in Section 3.3. For example, it can be applied to self-supervised methods such as MHRot (Hendrycks et al., 2019), NTL (Qiu et al., 2021), and ICL (Shenkar & Wolf, 2021), which are mentioned in the LOE paper (Qiu et al., 2022). We plan to evaluate the performance of our approach when applied to self-supervised anomaly detection and include the results in the revised paper.

Weakness 6: Can the proposed method be applied to tabular data? How is the performance?

Yes. We have confirmed that our approach performs well on KDD99, a well-known tabular dataset for network anomaly detection. We will include additional tabular experiments in the revised version.