Towards a Unified Framework of Clustering-based Anomaly Detection

Framework (theory): The authors did not provide a solid theoretical foundation behind the propose framework. For example, is Algorithm 1 guaranteed to converge? Is it possible that the algorithm will indefinitely oscillate, e.g., considering points 1, 2, 3 outliers in one iteration, then considering 4, 5, 6 in the next iteration, then considering 1, 2, 3 outliers again, then considering 4, 5, 6 outliers, etc. Moreover, Equation 1, while making intuitive sense, does not appear to correspond to any coherent statistical model. E.g., in a statistical model, one would expect that any point has a probability of being an outlier, even if it's not in the top l by score. Also, does the equation imply that for outliers p(x | params)=1? Given that a point is an outlier, what is p(x | params, x is outlier)?

Framework (novelty): There has been a long history of robust model fitting, including the idea of ignoring potential outliers (e.g., in RANSAC). The proposed framework is an instantiation of this idea. The paper has some novelty, but the novelty is not ground breaking.

Physics-inspired design: Vector sum is an aspect that I found quite interesting, but also most worrying. Thank you for including a deep dive example in Appendix Figure 4. However, what if the central cluster was just a regularly sized cluster? In this case, would a point in the centre of the central cluster receive a very high outlier score (because all "forces" neutralize)? But intuitively, this would be the least abnormal point in the data. I know that in the experiments using the vector sum appeared to increase performance, but I would have liked to see a deep dive from some of these datasets. I don't understand why using vector sum is not harming performance. Also, in Figure 2, you cannot compare the score relative to (mu_1, mu_2) with the score relative to (mu_3, mu_4), because these will be probabilities conditioned on different parameters. That is, you cannot compare outlier scores from different model estimates.

Evaluation (overfitting): Thank you for providing hyperparameter analysis. However, it does show that selecting the right values is important. This raises the concern of whether the parameters were overfit to the datasets. Even though the same setting were used for all datasets, I'm concerned whether the result was overfitted (overfit to the mega-dataset consisting of 30 datasets).

Evaluation (stat. significance): I'm concerned whether critical difference diagrams in Appendix Figure 5 imply that the winning method is not statistically significantly different from the 2nd, 3rd, etc. This is based on horizontal lines crossing several methods.

Presentation: There has been some minor issues with writing

• There is a typesetting issue with citations. Citations should be in brackets. E.g., according to Smith (Smith et al., 2010) or according to Smith [3]
• Some parts of the paper mention 17 baselines, while other parts mention 15
• Items 1, 2, 3, ... in the Introduction are mentioned before Figure 1 is cited
• Why are some arrows crossed in Figure 1?
• The paper mentions "Based on the aforementioned advantages of MMs, ... ", but there were no discussion of MM. Also, acronym MM is not defined
• In Section 3.3.1 (M-step), J is not defined
• In Section 4.4, what does "omitting the likelihood maximization loss" mean?
• Color coding in Table 1 is very confusing. To me "green" is something good, and I though "green" denoted the winner

The t-distribution-based clustering approach is established. The gravity-based, likelihood-based, and neighbor averaging techniques are also well-known in the field, raising concerns about the paper's novelty.

Additionally, the paper appears to mischaracterize previous research. The use of mixture models for anomaly detection has been documented for at least 25 years (e.g., Yamanishi et al., KDD 2000). Latent subspace-based formulations are also known, such as in Pesevski et al. (2018).

@inproceedings{yamanishi2000line, title={On-line unsupervised outlier detection using finite mixtures with discounting learning algorithms}, author={Yamanishi, Kenji and Takeuchi, Jun-Ichi and Williams, Graham and Milne, Peter}, booktitle={Proceedings of the sixth ACM SIGKDD international conference on Knowledge discovery and data mining}, pages={320--324}, year={2000} }

@article{pesevski2018subspace, title={Subspace clustering with the multivariate-t distribution}, author={Pesevski, Angelina and Franczak, Brian C and McNicholas, Paul D}, journal={Pattern Recognition Letters}, volume={112}, pages={297--302}, year={2018}, publisher={Elsevier} }

The clustering approach is similar to typical learning of Gaussian Mixture Models (GMMs) with EM, except with a t-distribution instead of a Gausian distribution.

Only two of the 10 existing alogrithms involve representation learning. Without representation learning, an algorithm is in an inherent disadvantage when compared to UniCAD.

1. Comparison with gravitation is a stretch and should be left out. While in some sense using vector addition might appear intuitive, mathematically, the underlying data generative model under a Mixture of Models assumption is probabilistic which does not correspond to any 'force' based model; probability at any point does not have a direction. Therefore, the reasoning behind the use of vectors can only be taken as a heuristic with little formal rigorous justification. Despite this weakness, the paper shows a potential for effectiveness and is quite novel.

1. The model jointly learns a mixture model similar to DAGMM and incorporates a clustering objective as a constraint. While the techniques used in this combination are relatively common in AD, this is not necessarily an issue if the combination demonstrates a unique synergy. The significance of the paper could be improved if the authors provided more insight into why this specific combination is more advantageous than others. For instance, how would performance differ if hypersphere-based constraints like Deep SVDD were combined with mixture models, or if other SSL methods were paired with the clustering objective? I look forward to other reviewers' opinions on the significance of this technical contribution.

2. Some recent baselines are missing. There are several comprehensive surveys on AD available through Google Scholar, and benchmark papers such as ADBench from NeurIPS 2022 could serve as a useful reference point.

3. The default neural network architecture is fairly basic. It might be worth exploring stronger backbones, as these could have a significant impact on the evaluation results.

4. A diagram of the proposed framework would greatly enhance clarity and help readers better understand the model's structure and components.