Towards a Unified Framework of Clustering-based Anomaly Detection

The ablation study on the hyperparameters 𝑘 and 𝑙 is insufficient. The authors only present results from a single dataset, satimage-2, where their method achieves an almost perfect score. It would be more informative to perform ablation studies across all 30 datasets or at least a subset where the model also shows lower performance. This broader analysis would demonstrate how these hyperparameters affect the average ranking of the method, similar to the results reported in the paper's table.

Thank you for your valuable feedback. Regarding the ablation study on the hyperparameters (k) and (l), we have tested various values for these parameters and found that the method exhibits a degree of robustness to their specific settings within a reasonable range. As stated in L243, our method utilizes a fixed set of parameters (k=10, l=1%) to ensure a fair comparison. We found that for most datasets, the method performs well with these settings.

Specifically, we conducted a grid search over the following coarse-grained parameter space and compared them across 30 datasets against 17 baseline methods. Indeed, for specific datasets, further tuning of hyperparameters can enhance performance. The detailed results, which represent the average ranking of the method in terms of AUC-ROC, are as follows:

Additionally, we have explored providing guidelines for selecting these hyperparameters. While methods like the elbow method and silhouette coefficient were considered to find the optimal cluster number, they proved time-consuming and not strongly correlated with anomaly detection performance. Instead, an ensemble learning approach, involving random searches of (k) values and aggregating anomaly scores, improved performance on certain datasets and model robustness. We plan to continue this research in future studies.

The authors introduce a 𝑔(Θ) term to prevent shortcut solutions, mentioning it in Equation 15. However, they do not discuss its importance or impact on performance after its introduction. Key questions remain unanswered, such as how the 𝑔(Θ) term affects the model's performance, what happens if it is removed, and how the autoencoder is implemented. These details are crucial, as the regularization term may significantly influence the results.

Thank you for your valuable feedback regarding the introduction of the term $g(\Theta)$ in our model. We appreciate your insights and would like to address your concerns as follows:

1. Importance and Impact of g(Θ)

The constraint term $g(\Theta)$ is indeed a fundamental and important part of the model. Using only $J(\Theta, \Phi)$ can lead to shortcut solutions, causing the loss to quickly become infinite, thereby preventing effective optimization of the deep network parameters. As shown in the ablation experiments in Table 3, we aim to demonstrate that having only $g(\Theta)$ or only $J(\Theta, \Phi)$ is insufficient; the combination of $g(\Theta) + J(\Theta, \Phi)$ significantly enhances the model's performance.

2. Implementation of the autoencoder

Our autoencoder consists of two main components: the encoder and the decoder. The encoder compresses the input data of dimension to a lower-dimensional representation of size. The decoder then converts this compressed representation back to the original data space of dimension. Both the encoder and decoder are implemented using fully connected neural networks with ReLU activation functions. The autoencoder is trained using a mean squared error loss function, which measures the difference between the original input and the reconstructed output.

The comparison with DeepSVDD and DIF is excellent, but I think the paper also needs to be compared with other Deep anomaly detection methods. For example, DROCC $1$ . $1$ Goyal, Sachin, et al. "DROCC: Deep robust one-class classification. "International conference on machine learning. PMLR, 2020.

We are grateful for your suggestion to include comparisons with other deep anomaly detection methods, particularly DROCC.

• Although our initial classification included only two NN-based methods, several other methods in different categories also employ representation learning, such as DAGMM and DCFOD. We will revise this classification to avoid any potential misunderstandings.

• We have included an additional NN-based comparison method (DROCC) in our experiments. As shown in our global response, our model significantly outperforms it. However, we appreciate your recommendation and will include and compare this method in the revised version.

• Furthermore, as shown in Appendix E, in the experiments on graph data, we compared our method with 13 deep anomaly detection approaches based on representation learning.

if the degrees of freedom is fixed to 1, the benefit of the t-distribution disappears. Why not learn the degrees of freedom as well?

Thank you for your insightful comment regarding the fixed degrees of freedom in our model.

Inspired by existing works $1~2$ , the cross-validating of v on the validation set or learning it is optional, especially in the unsupervised setting. However, in our work, we chose to fix it to 1 to simplify the model. This decision was made to reduce complexity and computational overhead while still maintaining robust performance.

I think \tilde{F}_{ik} 𝑖𝑠 𝑎 𝑠𝑐𝑎𝑙𝑎𝑟 𝑎𝑛𝑑 𝑡ℎ𝑒 𝑛𝑜𝑟𝑚 𝑜𝑓 \tiled{r}_{ik} is 1, so I don't see the difference between Eq. (9) and Eq. (8).

We appreciate your attention to detail regarding the mathematical formulation. To clarify, Eq. (8) represents a scalar sum, which is a constant addition, while Eq. (9) involves a vector sum where we take the norm of the vector.

The key difference lies in whether the direction of the sample and the cluster in the representation space is considered. Clusters in different directions are considered contradictory when estimating the anomaly degree of a sample and will cancel each other out. In Appendix C.1, we also provide an example for intuitive understanding.

As shown in Figure 2, I think the proposed method strongly depends on the hyperparameters 𝑙 and 𝑘. Is there any criteria for setting these values?

Thank you for your suggestion.

In our experiments, we have tested various values for these parameters and found that the method exhibits a degree of insensitivity to their specific settings within a reasonable range. As stated in L243, our method utilizes a fixed set of parameters to ensure a fair comparison (k=10, l=1%). We found that for most datasets, the method performs well with these settings.

We have also explored several criteria for selecting these hyperparameters, including the elbow method, silhouette coefficient, and ensemble of random search. We found that they are time-consuming and not strongly correlated with anomaly detection performance.

In the future, we will further explore strategies such as Outlier Detection Thresholding $3$ in future studies.

The method in this paper seems incremental. (Especially until 3.1.) I think the main novelty is the gravity-based anomaly score described in 3.2. To what extent does this anomaly score improve performance compared to the regular anomaly score? Also, can you give a theoretical explanation?

We appreciate your observation regarding the novelty of the gravity-based anomaly score.

The gravity-based anomaly score represents a significant advancement over traditional anomaly scores due to its ability to leverage the complex relationships among samples and clusters. Unlike conventional scores, which often rely on heuristic designs, our approach is grounded in a theoretical framework that connects clustering and anomaly detection through posterior probabilities and likelihood estimations. This theoretical underpinning allows our score to more effectively capture the nuances of data distributions, leading to improved anomaly detection performance.

In our comparative experiments presented in Table 1, we discuss the impact of different scoring methods. The results indicate that our gravity-based anomaly score ranks higher on average compared to traditional scores across 30 datasets, demonstrating its versatility and effectiveness in diverse scenarios.

Additionally, we provide an intuitive explanation of the gravity-based anomaly score in Appendix C.1, where we illustrate its advantages through a toy example. This example highlights how our score can better identify group anomalies, which are often challenging for traditional methods to detect.

References:

$1$ Laurens Van Der Maaten. Learning a parametric embedding by preserving local structure. In Artificial intelligence and statistics, pages 384–391. PMLR, 2009.

$2$ Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In International conference on machine learning, pages 478–487. PMLR, 2016.

$3$ Perini L, Bürkner P C, Klami A. Estimating the contamination factor’s distribution in unsupervised anomaly detection $C$ //International Conference on Machine Learning. PMLR, 2023: 27668-27679.

My concerns have been addressed to some extent, especially with the additional experiments involving DROCC. While the method itself appears to be incremental, its performance on tabular data is excellent. I will raise my score.

Thank you for the feedback. We will address each of the weaknesses and suggestions you mentioned.

The connection between force analysis and anomaly detection, particularly between Equations 7 and 8 in Section 3.2.1, could benefit from further justification. While the analogy is interesting, it may not be immediately intuitive for all readers.

Thank you for your suggestion! We share similar concerns and, due to space limitations, we have provided an intuitive explanation of the advantages of this analogy in Appendix C, along with a toy example to illustrate it more vividly. In this example, we offer a detailed explanation of the differences between scalar and vector concepts.

The iterative optimization process may pose scalability issues for large datasets. An in-depth analysis and discussion of this would further strengthen the quality of this research.

Thank you for your valuable feedback regarding the iterative optimization process and its potential scalability issues for large datasets. We have analyzed the computational complexity in Appendix D.4. The time complexity for t iterations is O(tN(logN + Td(D + K))). According to our complexity and run-time analysis, our method is scalable to large datasets.

In the future, we will explore two techniques that can help reduce the computational burden when processing large datasets: 1. Training on multiple manageable-sized data subsets and combining their scores using ensemble methods. 2. The Mini-Batch EM algorithm, which uses only a small batch of the dataset in each iteration.

Although the model maps data to a low-dimensional representation space, the effectiveness of this mapping for very high-dimensional datasets could be explored further.

Thank you for your suggestion. We agree that this is an important consideration. In fact, we have addressed this issue from two perspectives:

• On one hand, we can directly map high-dimensional data to a lower dimension using deep neural networks, which helps alleviate the problem of high dimensionality.

• Additionally, as shown in Appendix E, we can also easily extend to higher-dimensional graph data, such as the Weibo dataset with 400 feature dimensions, by simply replacing the corresponding backbone, demonstrating competitive performance.

We appreciate the reviewers' valuable suggestions for our work. We hope the following responses will clarify any doubts and enhance the quality of our paper.

The clustering part is similar to a typical Gaussian mixture model for clustering via EM, except for t-distribution instead of Gaussian and the scaling factor.

We acknowledge that the clustering part seems similar to a typical Gaussian Mixture Model (GMM). However, the primary reason we use the Student's t-distribution instead of the Gaussian distribution is its robustness to outliers. As mentioned in Section 3.1.1, the introduction of the Student's t-distribution allows the model to perform better when dealing with high-variance data, especially in the presence of outliers. Our ablation study (Table 3) also demonstrates that using the Student's t Mixture Model (SMM) significantly improves the average performance of the method compared to GMM.

Two neural-network-based approaches were compared. As UniCAD utilizes representation learning, comparing with more approaches that utilize representation learning would be significant. Approaches without representation learning have an inherent disadvantage.

Thank you for your valuable suggestion. We also place great importance on comparing our method with various anomaly detection approaches that utilize representation learning.

• We have included an additional NN-based comparison method (DROCC) in our experiments. As shown in our general response, our model significantly outperforms it.

• Furthermore, on graph data, we compared our method with 13 deep anomaly detection approaches based on representation learning.

Eq 11: What is the motivation for the scale factor u_ik, used in Eq 13 and 14?

Thank you for your insightful question.

The primary motivation for introducing the scale factor $u_{ik}$ is to downweight the influence of outliers in the data when estimating the parameters of the mixture model. This approach is also grounded in the optimization derivation presented in the paper $1$ .

In the M-step of the EM algorithm, the estimates for the component means $\mu_i^{(k+1)}$ and covariance matrices $\Sigma_i^{(k+1)}$ are computed using weighted averages, where the weights are given by $u_{ij}^{(k)}$. This allows the model to give less importance to observations that are likely to be outliers, thereby improving the robustness of the parameter estimates.

How is K, number of clusters, determined?

The selection of cluster numbers, in the absence of prior knowledge, is indeed a topic worthy of in-depth discussion. However, according to our hyperparameter analysis, we found that k=10 generally adapts well to most evaluated datasets.

Furthermore, we have considered the searching strategy such as elbow method, silhouette coefficient $2$ and ensemble of random searches in our works, but we found that they are time-consuming and not strongly correlated with anomaly detection performance.

We will explore this active direction in future studies.

Since the method has representation learning, how does the method prevent the trivial solution of having most/all instances in one cluster?

Thank you for highlight this critical question in deep clustering. Indeed, we have some careful design for avoiding this problem.

The key to our model's ability to avoid this issue lies in its unified optimization objectives for representation learning and clustering. Using the maximum likelihood of a mixture model as the objective inherently prevents all samples from being assigned to a single cluster, as this would significantly reduce the overall likelihood. Optimizing the maximum likelihood objective leads to better results with a mixture model compared to a single cluster. Consequently, the model will naturally distribute samples across multiple clusters.

Sec 3.2.1: ... in anomaly detection, that might not be desirable, particularly, when the two clusters are far away. Any insights?

Thank you for raising this insightful question. In fact, the case you mentioned is precisely what we consider to be the most anomalous situation. Since our anomaly score is inversely proportional to the resultant force—when the resultant force is minimal, the anomaly score is maximized. As a result, the instance in the situation you described will be recognized as an anomaly with the highest score.

Furthermore, the toy example in Figure 3 illustrates a similar case, showing that the model can effectively detect anomalies situated at the intersection of multiple clusters.

What is the "schedule" for updating phi and theta? Updating one to convergence before updating the other to convergence?If one "round" is updating phi to convergence and then theta to convergence? Are there multiple rounds? If so, what is the overall stopping criterion?

We appreciate your insightful comments. In practice, we iteratively optimize $\Phi$ and $\Theta$ using tolerance λ and iterations t. Due to space constraints in the main text, more detailed optimization procedures can be found in Algorithm 1 in Appendix A.

• For the parameters $\Phi$ of the mixture model, we adopt a log-likelihood convergence strategy. The algorithm stops when the increase in log-likelihood between iterations falls below a predefined threshold.

• For the parameters $\Theta$ of the deep model, we use a maximum iterations strategy. The algorithm stops after a predefined number of iterations is reached.

References:

$1$ David Peel and Geoffrey J McLachlan. Robust mixture modelling using the t distribution. Statistics and computing, 10:339–348, 2000.

$2$ Shi C, Wei B, Wei S, et al. A quantitative discriminant method of elbow point for the optimal number of clusters in clustering algorithm $J$ . EURASIP journal on wireless communications and networking, 2021, 2021: 1-16.