Unsupervised Anomaly Detection in The Presence of Missing Values

We are grateful for your reviews and recognition of our work. Our responses to your questions are as follows.

Response to Weakness 1 and Question 1:

The imputer is an MLP network, in which its backbone is designed as follows: $input \rightarrow 512 \rightarrow 128 \rightarrow 128 \rightarrow 512 \rightarrow output$, and we use LeakyReLU as the activation function and don't use bias term. As depicted in Figure 4 of our manuscript, the missing values in incomplete data are filled with zeros before being fed into the network.

Response to Question 2:

As shown in the table, we have latent dimension $d$, learning rate $\eta$, and loss parameters $\alpha, \beta, \lambda$ to tune. We have to say that hyperparameter tuning for unsupervised learning is a very challenging task and so far there is no good solution.

• Since different dataset usually have different ambient dimension, we need to use different latent dimension. Therefore, in the experiments, for low-dimensional data, we let $d=4$; for moderate-dimensional data, we let $d=32$; for high-dimensional data, we let $d=128$. These values were selected casually.

• For the learning rate $\eta$, we just let it be $0.0001$, which works well in most cases, as shown in the table. However, due to the high diversity of the datasets, we need to use other values for some datasets to ensure the convergence speed.

• For $\alpha, \beta, \lambda$, we set all values to 1, which works well in most cases. However, due to the high diversity of the datasets, in some cases, we need to tune them slightly according to the test performance. This strategy is very common in unsupervised learning such as anomaly detection. Almost all papers on unsupervised anomaly detection used this strategy.

It should be emphasized that to ensure fairness, we have sufficiently tuned the hyperparameters of all compared methods in the experiments.

Regarding overfitting, since our neural network is not very complex, the number of data points in each dataset is relatively large, and we fixed the number of optimization epochs to a small number, the overfitting seems not happened; otherwise, the AUROC score of our method will not be high.

Response to Weakness 2 and Question 3:

According to our experimental settings and empirical results, we have the following findings:

• Depending on the experimental settings and results of our manuscript, the detection performance of our proposed method outperforms all baselines in most cases, which indicates our method is more effective compared to "impute-then-detect" baselines under unsupervised settings.
• Moreover, according to some new experimental results (See Figure 1 in the attached PDF), the effect of the imputation bias from normal data on the ``impute-then-detect'' methods gradually diminishes as the missing rate increases. Therefore, our proposed method will be more effective when the missing rate is relatively small, say less than 50%.
• The generated pseudo-abnormal samples are critical to our method and our proposed method will be more effective when they overlap more significantly with real abnormal samples.

Response to Question 4:

This is a very practical and valuable question. In real scenarios, we generally don't know the missing mechanism of incomplete data. In such a situation, our method chooses the simplest missing mechanism (MCAR) based on Occam's Razor. For your second question, we have designed two experiments to answer it.

1. Observing the detection performance on a real incomplete dataset when using different missing mechanisms to generate pseudo-abnormal samples. In this situation, we don't know what the real missing mechanism is for the incomplete data.

2. Observing the detection performance on synthetic incomplete datasets when using different missing mechanisms to generate incomplete data and pseudo-abnormal samples. In this situation, we know the missing mechanism of the incomplete data.

The experimental results are provided in the following table.

According to the experimental results, on real incomplete data, our method is robust to the setting of missing mechanisms of the mask for the generated pseudo-abnormal samples and has better overall performance on MCAR. Therefore, based on the Occam's Razor and empirical results, we recommend using MCAR as the missing mechanism of generated pseudo-abnormal samples when the real missing mechanism is unknown. Moreover, on synthetic incomplete data, detection performance degrades when using different missing mechanisms generating incomplete normal data and pseudo-abnormal data.

Response to Question 5:

In fact, our work covered the real scenarios in Section 4 of our manuscript. In Table 1 of our manuscript, we introduced four real incomplete datasets, including Titanic (pattern recognition), MovieLens1M (recommendation system), Bladder, and Seq2-Heart (cell analysis), with missing rates of 10.79%, 82.41%, 86.93%, and 88.51%, respectively. The related experimental results are reported in Table 3 of our manuscript.

Thank you again for the comments.

We are grateful for your reviews and recognition of our work. Regarding your concern about computational time, we supplement comparisons of theoretical time complexity and experimental time cost between our proposed method and the baselines in the following Table 1 and Table 2.

The notations used in the complexity analysis (Table 1) are explained as follows:

• $n, m$ denote the number of samples of the training phase and inference phase, respectively.

• Missforest is a well-known data imputation algorithm based on random forest ($\mathcal{O}(t_1 \cdot v \cdot n\log{n})$) where $t_1$ denotes the number of trees, $v$ denotes the number of attributes.

• $T, T_g, T_d, T_{ae}, T_{oc}$ denote the iterations of corresponding methods.

• $\bar{L}$ and $\bar{d}$ denote the number of layers of the neural network and the maximum width of the layers of the corresponding models, respectively.

• $t_2$ denotes the number of trees of I-Forest and $t$ is the maximum iterations of the Sinkhorn algorithm.

• $p, \psi, K$ denote the key parameter of the corresponding methods.

Table 1: The time complexity of training and inference phase.

We have utilized the Speech and Usoskin datasets to benchmark the time cost of all methods, including ours and baselines. The two datasets exemplify the two distinct categories of tabular datasets used in our experiments. The Speech dataset contains 3,686 instances with 400 attributes and Usoskin dataset contains 610 instances with 25,334 attributes. ALL experiments were conducted on 20 Cores Intel(R) Xeon(R) Gold 6248 CPU with one NVIDIA Tesla V100 GPU, CUDA 12.0. The related results are provided in Table 2.

Table 2: The time cost (second) on Speech and Usoskin datasets.

Based on the theoretical analysis of time complexity in Table 1 and empirical results in Table 2, the "impute-then-detect'' pipelines have high time costs for the dataset like Usoskin with a large number of attributes. In contrast, our proposed method shows significant efficient advantages in the inference phase for both Speech-like and Usoskin-like datasets.

We are grateful for your reviews and suggestions of our work. Our responses to your questions are as follows.

Response to Weakness 1 and Question 2:

Since we are studying unsupervised anomaly detection, there is no validation set during the training stage. As shown in Table 8 of our paper, we have the latent dimension $d$, learning rate $\eta$, and loss hyperparameters $\alpha,\beta,\lambda$ to tune. Since we did not want to tune these hyperparameters, initially, we just let $d=4, 32, 128$ for low-dimensional, moderate-dimensional, and high-dimensional data casually, let $\eta=0.0001$, $\alpha=\beta=\lambda=1$. This setting works well in most cases. However, due to the high-diversity of the datasets (they have different sizes, are from different fields, and are with different missing rates and patterns), we have to tune the hyperparameters w.r.t. the test performance slightly. This is actually the convention of unsupervised anomaly detection. This strategy is commonly used in unsupervised learning such as clustering, novelty detection, and representation learning. Note that we have used grid search w.r.t. the testing set to find the optimal hyperparameters for all baselines to ensure fair comparisons. Tuning hyperparameters for unsupervised learning remains an open problem [1], although automated machine learning has made considerable progress for supervised learning.

[1] Fan et al. A simple approach to automated spectral clustering. NeurIPS 2022.

In Appendix I.1, we describe in detail the data sources of all the used datasets as well as the settings for normal and abnormal samples, where the arrhythmia and Speech datasets are from ODDs (Outlier Detection DataSets), with inherent settings for normal and abnormal samples. For the split of the training and test sets, we first set the ratio of normal and abnormal samples in the test set to 1:1, and then use all the remaining normal samples as the training set.

Response to Weakness 2 and Question 3:

1. In fact, besides the datasets with synthetic missing values (missing rate=20% and 50%), our work included real datasets with inherent missing values of which the missing rates are about 10% and 80%. Please refer to Table 1 and Table 3 in our manuscript.

2. The main experiments in (Yoon et al., 2018) are conducted with 20% missing rate only. The paper only tested one dataset with missing rates ranging from 10% to 80% in the ablation study.

3. In this rebuttal, we added related experiments on Speech dataset with missing rate $\text{mr} \in \\{0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8\\}$ and the results are shown in Figure 1 of the attached PDF, where the detection performance of "impute-then-detect" methods does not degrade and some of them even improve with the increasing of missing rate from 0.1 to 0.8. Moreover, our proposed method (ImAD) outperforms all baselines in almost all cases.

Response to Question 1:

This is a very practical question. Yes, they are the same in our experiments, because such a setting is consistent with existing data imputation works [Yoon et al., 2018; Muzellec et al., 2020] and can also facilitate the comparison among baselines.

However, the issue you mentioned is meaningful and worth exploring in depth for practical imputation scenarios. We conduct related experiments on the Speech dataset. In these experiments, we keep the missing rate $\text{mr}=0.5$ on the training set and change the missing rate from 0.2 to 0.8 on the test set. We visualize the experimental results in Figure 2 of the attached PDF. With such an experimental setup, the performance of the methods based on Mean-Filling fluctuates little, and the performance of the methods based on MissForest and GAIN fluctuates significantly. Our proposed method also shows some degree of performance fluctuation.

Response to Question 4:

Thanks for your question. It is hard to answer this question according to the current results accurately. Since our work is anomaly detection with missing data, our imputer is specially designed for the detector and the overall anomaly detection method is an end-to-end method, very different from the ``impute-then-detect'' pipelines. Therefore, we have to say that the learned imputer in our method is only effective for the task of anomaly detection, currently.

Response to Question 5:

The architectures of each module of our method are provided as follows:

• Imputer: $input \rightarrow 512 \rightarrow 128 \rightarrow 128 \rightarrow 512 \rightarrow output$
• Projector: $input \rightarrow 512 \rightarrow 256 \rightarrow 128 \rightarrow output$
• Reconstructor: $input \rightarrow 128 \rightarrow 512 \rightarrow output$

In all three modules, we used LeakyReLU as the activation function and didn't use bias terms.

Based on the empirical results, the architecture designed exhibits superior performance in comparison to all baselines. Therefore, we did not further explore the effects of network architecture and this is also not a key contribution to our work.

Response to Question 6:

Thanks for your question. Anomaly detection and data imputation are ubiquitous tasks across various data types. In this work, we primarily focus on incomplete tabular data since data missing is quite common on tabular data. Anomaly detection with missing data on tabular data has many practical applications such as identifying abnormal users in recommendation systems and discovering abnormal cells in bioinformatics. However, anomaly detection with missing data on other data types such as images may encounter some new questions and challenges, such as in what scenarios we need the method and how to define a meaningful missing pattern for images. Therefore, we are not sure whether our method can be directly applied to non-tabular data with missing values. But it is worth studying in the future.

We are very pleased and honored to receive your positive evaluation of our work. Our responses to your questions are as follows.

Response to Question 1:

        As you are concerned, we have explored the influences of constrained radii $r_1, r_2$ for detection performance, and the related results are reported in Appendix G of our manuscript. According to Proposition A.1 in our manuscript, we have $r= \sigma\sqrt{F ^ {-1} _ d (p)}$, where $\sigma$ and $d$ are the variance and dimension of the target distribution, respectively, and $p$ denotes the sampling probability. We set the target distribution as $\mathcal{D} _ {\mathbf{z}} \sim \mathcal{N}(\mathbf{0}, 0.5 ^ 2 \cdot \mathbf{I} _ d), \mathcal{D} _ {\tilde{\mathbf{z}}} \sim \mathcal{N}(\mathbf{0}, \mathbf{I} _ d)$, and $p=0.9$. We adjust the dimension $d \in \\{ 4, 8, 16, 32, 64, 128, 256, 512\\}$ of the target space and obtain the corresponding $r_1, r_2$ (see following table).

According to the results (Appendix G of our manuscript), our method is not very sensitive to the changes in the radii $r_1$ and $r_2$, but its performance degrades with the reduction in the latent dimension. This is reasonable since a smaller latent dimension results in more information loss.

Response to Question 2:

        In the proposed method ImAD, we utilize a projector $\mathcal{P}:\mathbb{R} ^ m\rightarrow\mathbb{R} ^ d$ to transform $\mathcal{D} _ {{\mathbf{x}}}$ (normal data distribution) and $\mathcal{D} _ {\tilde{\mathbf{x}}}$ (pseudo-abnormal data distribution) into $\mathcal{D} _ {{\mathbf{z}}}$ and $\mathcal{D} _ {\tilde{\mathbf{z}}}$ respectively. In this process, the samples from data distribution ($\mathcal{D} _ {{\mathbf{x}}}$ or $\mathcal{D} _ {\tilde{\mathbf{x}}}$) and target distribution ($\mathcal{D} _ {{\mathbf{z}}}$ or $\mathcal{D} _ {\tilde{\mathbf{z}}}$) do not have pairwise relationship and we need to measure the discrepancy between $\mathcal{P}(\mathcal{D} _ {{\mathbf{x}}})$ and $\mathcal{D} _ {{\mathbf{z}}}$ using their finite samples. Thus, we use Sinkhorn divergence to cover this situation.

Response to limitation:

        The key hyperparameters of our proposed method include latent dimension $d$, constrained radii $r_1, r_2$, learning rate and the trade-off coefficients $\alpha, \beta, \lambda$ in optimization objective. According to the results in Appendix G of our manuscript, our method is not very sensitive to changes in the radii $r_1$ and $r_2$, but its performance degrades with the reduction in the latent dimension $d$. The choice of latent dimension $d$ is based on the data dimension. Typically, high data dimensions correspond to high latent dimensions $d$. According to the ablation study on $\alpha, \beta, \lambda$ in Appendix H and hyperparameters selected via grid search in Appendix I.5, the changes of the three coefficients have a non-trivial impact on detection performance, which indicates that each module corresponding to $\alpha, \beta$, or $\lambda$ is indispensable for the proposed method. Moreover, while setting $\alpha, \beta, \lambda = 1$ (the same coefficient with Sinkhorn divergence), ImAD achieves good performance in most cases based on the empirical results. The learning rate is set to 0.0001 on most datasets and other choices mainly aim to make the optimization converge faster.