DRL: Decomposed Representation Learning for Tabular Anomaly Detection

Q2: The authors should consider addressing concern E.1, to at least experimentally verify T.2 by extend the experiment results of Figure 1 in original paper.

R to Q2: Thanks for your valuable suggestion! Following your suggestion, we extend the experiment results of Figure 1, and provide the extensive visualizations in Fig.14 and Fig.15 of Appendix 10 in revised version. We can observe that ignoring observation entanglement in tabular anomaly detection under the one-class classification setting can lead to diminished discriminative power between learned normal and anomalous patterns, as the overlap between normal and anomalous representations within the latent space of deep models may obscure the distinction between them. Our proposed method can amplify the discrepancy between the two (inlier and outlier) patterns.

Q3: Unclear description of Section 5.2.

R to Q3: Thanks for your insightful suggestion! We apologize for any confusion and we have made improvements to the clarity of Section 5.2 in the revised version for better illustration. Following your suggestion, we merge the ablation study and the comparison of DRL and variants in the observation space into the same section. Due to the limited space, we summarize the results over 40 datasets in Table 2 in the revised version. We also provide the full results with the reference of data in Table 11 to Table 14 in Appendix 10 of revised version. We also make the description of variant B (currently named variant E) clear in Section 5.2 as mentioned in E.2.

Q4: Code needs to be released.

R to Q4: Thanks for your valuable feedback! Due to time constraints at the time of submission, the code was not available. However, we have now released the source code to facilitate reproducibility. We hope this addresses your concern.

Additional remarks:

R1: P-values are not a scalar metric.

R to R1: Thanks for your valuable suggestion! We revised the Fig.4 in the revised paper.

R2: Considering using a multiple comparison test.

R to R2: Thanks for your insightful suggestion! We would use a multiple comparison, for example, the Conover-Iman test, in the final revision.

Dear Reviewer TJDb,

Thank you very much again for your time and efforts in reviewing our paper.

We kindly remind that our discussion period is closing soon.

We just wonder whether there is any further concern and hope to have a chance to respond before the discussion phase ends.

Many thanks, Authors

W6: Can the proposed method be applied to other data types (e.g., image or time series data)?

R to W6: Thank you for your insightful question!

As classical methods struggle to capture complex patterns and relationships in high-dimensional spaces [1], recent studies have prompted a shift toward deep learning methodologies for tabular data. Although fruitful progress has been made in the last several years, capturing the comprehensive normal patterns that are distinct from anomalous patterns for tabular data remains a challenging task, as real-world data may exhibit data entanglement between normal and anomalous samples. Ignoring observation entanglement in tabular anomaly detection under the one-class classification setting can lead to diminished discriminative power between learned normal and anomalous patterns. We attribute this challenge to the intrinsic heterogeneity of features in tabular data, which aligns with recent findings [2] indicating that neural networks struggle to distinguish regular and irregular patterns, particularly when confronted with numerous uninformative features present in tabular data. Our method re-maps observations into a tailor-designed constrained latent space, where normal and anomalous patterns are more effectively distinguished, thereby alleviating the entanglement problem.

In addition, when considering perceptual data (e.g., image and text), many methods have demonstrated significant success by leveraging the structure of the input data. For example, images can be rotated, and the ability to distinguish between different rotations varies between anomalies and normal samples. However, tabular data lacks such prior structural properties, and our DRL does not rely on prior knowledge of the data structure, making it especially effective for tabular data.

We believe that applying the insights from DRL to other data types is meaningful and it will be an important direction of our future work.

[1] Deep learning for anomaly detection: A review. [ACM Computing Surveys 2021]

[2] Why do tree-based models still outperform deep learning on typical tabular data? [NeurIPS 2022]

W7: This is a minor point, but the paper lacks a discussion on limitations and future work.

R to W7: Thank you for your valuable suggestion! We agree that discussing limitations and future work is important. Although our method can be used to different data types for the anomaly detection task by introducing representation decomposition, we may need to design the specific architecture for the weight learner and alignment learner due to the difference between data types. In the future, we plan to explore its application to other data types, where incorporating prior structural knowledge from these data types might be a possible solution. We have added the discussion on the limitations and future work in Appendix 11 in our revision.

W4: The sensitivity analysis of vector number is good. I wonder if the vector number is somehow related to the feature number of input data.

R to W4: Thank you for your valuable suggestion! We agree that the number of basis vectors may have some relationship with the number of features in the input data. Considering the feature number when constructing the set of basis vectors is indeed a meaningful idea, and we see this as an important direction for our future work.

In our DRL, we remap the sample from raw space to the latent space and enforce the representation of each normal sample in the latent space to be decomposed into a weighted linear combination of randomly generated orthogonal basis vectors. The hidden dimension for representation is fixed across all datasets, thus it is reasonable to use a default number of basis vectors for each dataset. In our experiments, we set the default number of basis vectors to 5 for all datasets. We also provide a sensitivity analysis in Fig. 9 of Appendix 6 in the original paper, where we show that setting the number of basis vectors to 5 is sufficient to achieve high performance.

W5: I appreciate the theoretical support for the separation loss. However, as the separation loss is applied between samples, is it sensitive to the batch size? Additionally, if the training normal samples are highly similar to each other, does this loss face convergence challenges?

R to W5: Thanks for your valuable suggestion!

(1) Following your suggestion, we include the sensitivity analysis w.r.t. batch size on AUC-PR, which is added into Fig. 9 (e) of Appendix (A.6) and summarized in table below for your convenience. The results demonstrate that the performance remains robust across different batch sizes.

(2) Regarding loss convergence, the primary objective of DRL is to minimize the decomposition loss (Eq. 3 in the original paper), while the separation loss (Eq. 5 in the original paper) serves as an additional constraint. The separation loss is applied to the normal weights using cosine distance, ensuring that the values remain bounded within a small range. Moreover, the loss weight for the separation loss is set to 0.06 by default, which further constrains its range. Additionally, the weights for basis vectors belong to a probability simplex, which prevents cases where all weights become zero during updates. These mechanisms collectively contribute to stable loss convergence.
To verify this, we have added the experimental results in Fig. 18 and 19 in Appendix 10 in the revised version, illustrating the effect of the separation constraint on loss convergence. The results confirm that the separation loss does not negatively impact convergence.

W2: The authors may need to explain the rationale for leveraging orthogonal basis vectors to learn such representations. Could vectors with other relational properties achieve the same effect? What are the specific advantages of using orthogonal basis vectors for decomposed representation learning?

R to W2: Thanks for your valuable suggestion! We are sorry for any confusion and will explain the advantages in detail.

(1) To accurately capture the statistical characteristics of normal samples while distinguishing them from anomalies, it is crucial that the shared basis vectors are sufficiently diverse to encapsulate the global structure of the normal data. To this end, we eliminate the dependencies among basis vectors by leveraging a set of orthogonal vectors as basis vectors.

(2) Given a subspace defined by a set of orthogonal basis vectors, each representation in this space can be expressed as $\mathbf{w}\mathcal{B}$, where $\mathbf{w}$ is the weight vector (denoting the coordinates) and $\mathcal{B}$ represents the fixed set of basis vectors.
To increase the discrepancy between different representations, as discussed in Sec. 3.2.2, we only need to enforce the separation in the weight vectors $\mathbf{w}$ due to the shared basis vectors, i.e., the separation loss (Eq.5 in original paper). It is easier than directly increasing the discrepancy between the representations since we already exclude the shared information for all representations and the dimension of weight vector is extremely low (only 5 for all datasets).

(3) Additionally, we have provided a comparison between using random basis vectors and orthogonal basis vectors in Table 1 of the original paper. From our experiments, we observe that the orthogonalization of basis vectors is crucial for effectively capturing normal patterns that distinguish them from anomalies.

W3: What is the performance when using cosine distance for the decomposition loss? Additionally, how does the model perform if only alignment losses are used as anomaly scores during inference?

R to W3: Thanks for your insightful suggestion! Following your suggestion, we include experiment results as follow to verify the performance of using cosine distance for the decomposition loss and using alignment loss as anomaly score for inference. We have also included these results in Table 2, Table 3 in main text and Table 11 to 14 in Appendix 10 in revised version. We could observe that using cosine distance for the decomposition loss achieves comparable performance to using L2 distance (ours). During inference, DRL only uses decomposition loss as anomaly score as detailed in Section 3.2.3 in original paper. When using alignment loss as the anomaly score for inference, we observed a degradation in performance. This is because the alignment loss is calculated in the observation space, which is prone to the observation entanglement issue discussed in Section 3.1 of the original paper. As a result, it might be inefficient to use alignment loss to distinguish the anomalous samples from normal ones.

Dear Reviewer hbkx,

Thank you very much again for your time and efforts in reviewing our paper.

We kindly remind that our discussion period is closing soon.

We just wonder whether there is any further concern and hope to have a chance to respond before the discussion phase ends.

Many thanks, Authors

W2: The reviewer is concerned about the efficiency of the proposed method. Although authors have shown the runtime in A.5, more comparison with other baselines in terms of runtime will improve the persuasiveness of this aspect.

R to W2: Thanks for your valuable suggestion!

We agree that we need to compute a unique weight vector for each sample. However, we optimize a shared weight learner to calculate the weight vectors, rather than directly optimizing a unique weight vector for each sample individually. This significantly reduces the computational cost.

Besides, we kindly remind you that the default number of basis vectors (i.e., the dimension of the weight vector) is set to 5 across all datasets. We also provide a sensitivity analysis in Fig. 9 of Appendix 6 in the original paper, where the results show that setting the number of basis vectors to 5 is sufficient to achieve high performance. Therefore, this number of basis vectors does not significantly increase the model complexity. Additionally, the weight learner is implemented as a simple two-layer fully connected MLP with Leaky ReLU activation, which does not require a large number of parameters.

Following your suggestion, we now include runtime comparisons with other baselines, which is summarized into Table 8 in Appendix (A.5) in our revision. The results show that our proposed DRL method is computationally efficient. For example, among recent baselines, MCM requires the generation of multiple learnable mask matrices, which increases training costs. NPT-AD, on the other hand, involves a high computational cost in terms of memory and time, due to its reliance on the training set during inference.

W3: Code needs to be released.

R to W3: Thanks for your valuable feedback! Due to time constraints at the time of submission, the code was not available. However, we have now released the source code to facilitate reproducibility. We hope this addresses your concern.

Dear Reviewer Sgt1,

Thank you very much again for your time and efforts in reviewing our paper.

We kindly remind that our discussion period is closing soon.

We just wonder whether there is any further concern and hope to have a chance to respond before the discussion phase ends.

Many thanks, Authors

Q4: How sensitive is DRL's performance to hyperparameter choices, such as the number of orthogonal basis vectors or weights in loss terms? Are there specific recommendations or guidelines for tuning these parameters across different types of tabular datasets?

R to Q4: Thanks for your insightful comment! We apologize for any confusion. Actually, as mentioned in the last paragraph of the section about the experiments in the original version, we had provided the sensitivity analysis results in the Fig. 9 in Appendix (A.6) due to space limitations, including the sensitivity analysis for the number of basis vectors, the hyper-parameters about loss weight, the number of training epochs and the number of batch size. Below, we will give a more detailed explanation of the sensitivity analysis.

As the very beginning, the performance increases rapidly as the number of orthogonal basis vectors increases and then the performance is stable. Thus, it is sufficient to set the number of orthogonal basis vectors to 5. For the loss weight $\lambda_1$ associated with separation loss, we find that the performance is generally robust across different values, but lower values of $\lambda_1$ tend to yield relatively better results. Therefore, a lower $\lambda_1$ is more suitable. Similarly, for the loss weight $\lambda_2$ associated with alignment loss, we observe that lower values of $\lambda_2$ lead to better AUC-ROC performance. Thus, a lower $\lambda_2$ is more suitable. Regarding the number of training iterations, we found that the performance stabilizes after 200 iterations, so it is sufficient to set the total training iterations to 200. The results also demonstrate that the performance remains robust across different batch sizes. As noted in Line 372 of the original paper, the hyperparameters were kept consistent across all datasets.

Q5: What are the main limitations of DRL as identified by the authors?

R to Q5: The primary limitation of the current DRL method is that it is designed specifically for tabular anomaly detection. Although our method can be used to different data types for the anomaly detection task by introducing representation decomposition, we may need to design the specific architecture for the weight learner and alignment learner due to the difference between data types. In the future, we plan to explore its application to other data types, where incorporating prior structural knowledge from these data types might be a possible solution. We have added the discussion on the limitations and future work in Appendix 11 in our revision.

Q2: How performance might change with L1 distance, and clarify why cosine distance was chosen specifically over both L2 and L1.

R to Q2: Thank you for your thoughtful comment! We have added results using the L1 distance metric for comparison as follows. We also included the results in Table 3 in main text, Table 13 and Table 14 in Appendix 10 of revised version. The results show that L1, L2, and cosine distances all yield promising performance, with cosine distance demonstrating relatively superior and more stable results. In the original paper, we used cosine distance for both separation and alignment loss. We now provide further clarification on why cosine distance was chosen over both L1 and L2 distances.

Our primary objective is to minimize the decomposition loss, which also serves as the anomaly score during inference. Separation loss acts as a constraint to enhance the discriminative power between normal and anomalous patterns within the latent space, thereby helping to better capture the distinct normal patterns through decomposition loss. Since separation loss is implemented by enforcing separation among the weights of normal training samples, it has the potential to affect the convergence of the training process. To mitigate this, we use cosine distance for separation loss, as it ensures the values are bounded within a smaller range compared to both L1 and L2 distances, which helps avoid convergence issues during training.

As for alignment loss, considering the potential entanglement between normal and anomalous samples in the observed data space, unlike previous methods minimizing the L2 distance between $\mathbf{x}$ and its reconstructed version $\tilde{\mathbf{x}}$ to accurately maintain the observation information, our strategy focuses on maximizing the cosine similarity between $\mathbf{x}$ and $\tilde{\mathbf{x}}$ to align the information of $\mathbf{h}$ with the intrinsic feature correlation of $\mathbf{x}$, while avoiding excessive retention of observational details that may contain entanglement patterns.

Q3: Proposition 1 proposes that maximizing variance of weights attached to basis vectors in normal latents enhances separation from abnormal latents. Are there trade-offs in terms of model stability or convergence when applying this approach?

R to Q3: Thanks for your valuable comment! Regarding loss convergence, the primary objective of DRL is to minimize the decomposition loss (Eq. 3 in the original paper), while the separation loss (Eq. 5 in the original paper) serves as an additional constraint. The separation loss is applied to the normal weights using cosine distance, ensuring that the values remain bounded within a small range. Moreover, the loss weight for the separation loss is set to 0.06 by default, which further constrains its range. Additionally, the weights belong to a probability simplex, which prevents cases where all weights become zero during updates. These mechanisms collectively contribute to stable loss convergence.
To verify this, we have included experimental results in Fig. 18 and 19 in Appendix 10 in the revised version, illustrating the effect of the separation constraint on loss convergence. The results confirm that the separation constraint does not negatively impact convergence.

W3: Unclear description of Section 2 and MCM limitations.

R to W3: Thanks for your insightful suggestion! We apologize for any confusion and we have made improvements to the clarity of Section 2 in revised version for better illustration.

(1) Standard reconstruction-based approaches consider learning a mapping $A(\cdot;\Theta):\text{R}^D \to \text{R}^D$ to minimize the reconstruction loss within $\mathcal{D}_{train}$, where $D$ is the number of input features. Typically, $A(\cdot;\Theta)$ first maps the sample from observation space to latent space, and then maps it back to the observation space to obtain the reconstruction of the sample. The parameters $\Theta$ are optimized by minimizing the reconstruction loss on normal training samples.

(2) Based on reconstruction, MCM employs a learnable masking strategy to the input and aims to reconstruct normal samples well with access to only unmasked entries of the input, where how to produce effective masks is challenging in this field. NPT-AD incorporates both feature-feature and sample-sample dependencies to reconstruct masked features of normal samples by utilizing the whole training set. Therefore, NPT-AD involves a high computational cost in terms of memory and time, due to its reliance on the training set during inference. Despite the effectiveness of MCM and NPT-AD, both of them design the reconstruction strategies in the observed data space, as illustrated in Section 3.1 in original paper, which usually suffer from the potential data entanglement in observed data space, where we showed the phenomenon on more real world tabular datasets in Fig. 7 of Appendix A.1. Ignoring the issue in anomaly detection can lead to diminished discriminative power between normal and anomalous patterns. To solve the potential data entanglement issue, our proposed DRL introduces representation decomposition in the constrained latent space, where the normal and anomalous patterns are more discriminative. Additionally, DRL exhibits lower computational costs compared to both MCM and NPT-AD as illustrated in Table 8 in Appendix 5 of revised version.

Q1: What would happen if the basis vectors were set as unit vectors, for example, [1,0,0,0],[0,1,0,0],[0,0,1,0],[0,0,0,1]?

R to Q1: Thanks for your comment! Following your suggestion, we include experiments as follows to evaluate the impact of using unit vectors as basis vectors in DRL.
The results indicate that initializing with unit vectors does not perform as well as the default basis vector generation method in DRL. In our approach, the reconstructed representation is computed as a linear combination of orthogonal basis vectors: $\tilde{\mathbf{h}} = \sum_{k=1}^Kw^k\beta_k$, where the basis vectors are defined by $\mathcal{B} = \\{\beta_k\\}_{k=1}^K \in \text{R}^{K\times E}$ with $K < E$. By default, the $K$ and $E$ is set to 5 and 128 respectively. If the basis vectors were set as unit vectors, the reconstructed representation $\tilde{\mathbf{h}}$ would become highly sparse, containing $E-K$ zero entries in its $E$-dimensional space. The main objective of DRL is to minimize the decomposition loss $d(\mathbf{h}, \tilde{\mathbf{h}})$ (Eq.3), where $d$ is the distance measurement and $\mathbf{h}$ is the representation extracted by feature extractor $f$. However, this configuration could lead to inefficient optimization, as $\mathbf{h}$ would need to be overly sparse to match $\tilde{\mathbf{h}}$.