PTAD: Prototype-Oriented Tabular Anomaly Detection via Mask Modeling

Response to W3 about the code and hyperparameters: Thanks for your comments. Our learning rate is selected by the convergence speed of training loss, following the same selection criteria as NPT-AD[2]. Furthermore, we have provided the full set of hyperparameters for all datasets in Appendix D. Besides, to facilitate the reproduction of our results, we have provided an anonymized link (https://anonymous.4open.science/r/PTAD-D76D/), where the pre-trained models can be directly downloaded for validation. This allows researchers to verify our results without requiring extensive training efforts.

Table 8: Datasets hyperparameters. The batch size -1 refers to the input of the entire training set.

Response to Question-typos-1 about the reference [2]:

Thanks for your suggestions. What we originally meant to say in Line 150 ('Thimonier et al. (2024) introduces NPT and showcases its effectiveness and superiority in tabular AD') is in accordance with you. This expression may lead to a misunderstanding. Therefore, we revised the sentence to 'Motivated by NPT, Thimonier et al. (2024) introduce NPT-AD by incorporating both sample-sample and feature-feature dependencies in tabular AD, which showcases its effectiveness and superiority for tabular data.' Additionally, following your suggestion, we have included the work proposed by Thimonier et al. (2024) in the related work section.

Response to Question-typos-2 about the Projection-Space Mask Generation:

Thanks. Following your suggestion, we have clarified the paragraph about Projection-Space Mask Generation.

In our formulation, the H-dimensional vector $z_n$ denotes its representation of the-nth sample, where $z_{nh}$ is the $h$-th element of $z_n$. Besides, basis vector $\beta^k$ is also $H$-dimensional and $\beta^k_h$ denotes its $h$-th element. We element-wise compute the distance between $z_n$ and the basis vectors $\beta^k$ in the projection space, which is used to generate the projection-space mask. Specifically, each mask value $M_{nh}^{k}$ is computed by the Euclidean distance between the $h$-th value of the $n$-th representation, i.e., $z_{nh}$, and the $h$-th value of the $k$-th basis vector, i.e., $\beta^k_h$. This distance-based computation helps the model identify relevant patterns in the latent space and determine which features should be masked as suspicious anomalies.

Response to W1 about the challenges when introducing our method:

Thanks for your insightful advice for improving our paper. Firstly, we hope to emphasize that our model is problem-oriented and elaborately designed for tabular data. As the reviewer mentioned, it is challenging to design an appropriate method for tabular AD, which is reflected in the following aspects:

1. Developing a masking strategy is challenging due to the intricate, heterogeneous, and unstructured nature of tabular data. Deriving commonly utilized random masking may leak a large amount of abnormal information into the reconstruction of tabular data, which is highly unstructured and entangled, leading to irregular reconstruction. Thus, we introduce the data-adaptive masks to find suspicious anomaly locations and reconstruct them with normal information, thus resulting in small or large deviations for instructing normal and anomalies, respectively.

2. How to select data-adaptive suspicious abnormal masks? The straightforward way is to compare the representations with normal ones. However, the learned representations of existing deep tabular methods are usually entangled and thus cannot support finding the suspicious abnormality. Therefore, we encourage learning the disentangled representation within the projection space. To end this, we introduce a group of learnable orthogonal basis vectors. Furthermore, to capture various data characteristics and learn the diverse inherent relationships, we introduce a multiple mask strategy and only conduct it in the decoder to save computational consumption. Please note that introducing the data-adaptive strategy in not only observation space but also latent space is novel, which is non-trivial.

3. For compensation to the feature-level patterns as mentioned in 1) and 2), we also investigate the correlation patterns between features to facilitate the modeling of tabular normal patterns and detecting anomalies by introducing a novel association prototype learning. It is beneficial for fitting the heterogeneous and complex characteristics of tabular data.

To sum up, our model aims to improve the tabular AD from two perspectives: one is designing the adaptive mask and the other is learning the correlation patterns across features based on the adaptive masks. It is the interaction and complementarity between these two parts that make our method effective for tabular anomaly detection.

Respons to W2:

Sorry for any confusion caused, we have revised our paper to make the descriptions clearer and the logic easier to understand.

a) $N$ is the number of samples, and $n$ is a typo here, revised to $N$. $Z = \Phi_E(\hat{X}; \theta_E) \in \mathbb{R}^{N \times H}$ is the encoded representation of $N$ samples. Here, $\Phi_E$ is the NPT encoder layer composed of an ABD and an ABA layer, which is parameterized by $\theta_E$. $\hat{X}$ is the $N$ masked input obtained from Eq.(3).

b) Thanks for your insightful advice. As mentioned in Weakness-2), we introduce the masks before the projection space learning, as the latter serves as a support for the former, which selects data-adaptive suspicious abnormal masks by learning a disentangled projection space. We have revised the paper to make the logic clearer and easier to understand.

c) $\theta_E$ refers to the parameters of the encoder. $\delta_{\beta^k}$ is the $k^{th}$ basis vectors of the discrete distribution. $\delta_{\beta^k}$ is Dirac function that places a unit point mass at $\beta^k$, and $Q(\mathcal{B}) = \frac{1}{K}\sum_{k=1}^K \delta_{\beta^k}$ means a K-dimensional uniform discrete distribution. This is a commonly used notation in typical OT problems [1][2][3].

d) $P(\pi)$ represents the discrete uniform distribution over $N$ association vectors. The logic of section 4.2 can be stated as follows: i) we first extract the correlations between features in each input data as $\pi_n$, denoted as association vectors. ii) we record the normal correlations as $M$ association prototypes, which need to be learned. iii) The OT-based optimization objective in Eq.(7) is to minimize the distance between the association vectors to the corresponding nearest association prototype. This objective is utilized to learn the association prototypes to record normal features. The intuition behind this is normal patterns tend to approach one of the association prototypes rather than its fusions to alleviate the potential collapse.

Furthermore, for easier understanding of the information flow and logic, we have included our algorithm and training pipeline in Appendix A and Appendix B, respectively.

Response to the question about anomaly leakage:

Thanks for your question. 'Anomaly leakage' is a typical issue in the anomaly detection field [4,5,6]. It has various alternative names or variants, such as 'identical shortcut' and 'identical reconstruction'. Reconstruction-based methods detect anomalies by assuming that a well-trained model (only with normal samples) always produces normal samples regardless of whether the anomalies are within the inputs. In this way, there will be large reconstruction errors for anomalous samples, making them distinguishable from the normal ones. However, with the entire data as input, the reconstruction model can sometimes well reconstruct both the normal and anomalous regions, which appears as returning a good copy of the input disregarding its content. As a result, the anomalous samples can be well recovered with the learned model and hence become hard to detect. This will lead to lower anomaly scores in abnormal regions and thus failure of anomaly detection. Especially when encountering complex data distribution, such as heterogeneous unstructured tabular data, reconstruction methods tend to suffer from this 'Anomaly leakage' problem. It requires the model to work extremely hard to learn the complex data distribution. From this perspective, learning an “identical shortcut” appears as a far easier solution. Moreover, the model trained by this kind of data is more likely to be overfitting. Introducing our masking strategy could block the anomalous information from the origin and prevent the model from reconstructing anomalous, as those suspicious anomalous regions are already masked. To this end, the anomalous samples cannot be well recovered with the normal information and hence become easier to detect.

[1] Gabriel Peyr´e, Marco Cuturi, et al. Computational optimal transport: With applications to data science. Foundations and Trends® in Machine Learning, 11(5-6):355–607, 2019.

[2] Pavel Dvurechensky, Alexander Gasnikov, and Alexey Kroshnin. Computational optimal transport: Complexity by accelerated gradient descent is better than by sinkhorn’s algorithm. In International conference on machine learning, pp. 1367–1376. PMLR, 2018.

[3] Lenaic Chizat, Gabriel Peyr´e, Bernhard Schmitzer, and Franc¸ois-Xavier Vialard. Scaling algorithms for unbalanced optimal transport problems. Mathematics of Computation, 87(314):2563–2609, 2018.

[4] Perera, P.; Nallapati, R.; and Xiang, B. 2019. OCGAN: one- class novelty detection using gans with constrained latent representations, in CVPR.

[5] Zhiyuan You, Lei Cui, Yujun Shen, Kai Yang, Xin Lu, Yu Zheng, and Xinyi Le. A unified model for multi-class anomaly detection. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 4571–4584. Curran Associates, Inc., 2022.

[6] Yao, Xincheng, et al. One-for-all: Proposal masked cross-class anomaly detection. Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 4. 2023.