AnoLLM: Large Language Models for Tabular Anomaly Detection

Thank you for your review. We sincerely appreciate the time you took to read our paper and are grateful for your feedback. Our responses are provided below.

Regarding the simplicity of proposed methods: We consider simplicity a key advantage of our method, as it makes implementation more accessible for practitioners and facilitates easier debugging. A major contribution of our work is demonstrating that LLMs can effectively handle tabular data, despite its sequential structure and the challenges of numerical reasoning. By employing techniques such as random permutation and number normalization, LLMs can be adapted for tabular anomaly detection while leveraging their strengths in text modeling. We view this as a solid starting point, leaving additional modifications and exploration of more advanced techniques to future research.

Addressing claims of pioneering the use of LLMs for tabular anomaly detection: We would like to highlight that AnoLLM fundamentally differs from the works you mentioned. Biester et al. (2024) and Park (2024) treat LLMs as agents for generating domain-specific contexts or formatting data, relying on additional modules—and in some cases, human intervention—to process the LLMs’ outputs. Li et al. (2024), on the other hand, focuses on zero-shot performance. In contrast, our approach directly fine-tunes LLMs on the target data and uses the LLMs’ outputs as anomaly scores, making it a more straightforward and self-contained method.

Choice of LLM backbones: We experimented with Qwen, a larger architecture with 500M parameters in its smallest variant, and other multi-billion scale models. However, we found this task was better suited to lower-capacity models fine-tuned (FT) for anomaly detection (AD), as AD typically requires high throughput and low latency, making high-capacity, multi-billion parameter models impractical. As shown in Table 4, scaling the size of LLMs did not yield performance improvements. To optimize efficiency, we focused on smaller LLMs as backbones. SmolLM, the state-of-the-art open-weight model at the time, was selected for our experiments, and initial trials with Qwen-0.5B showed comparable performance, reinforcing our preference for smaller models.

Distinction between Contributions 3 and 4: Contributions 3 and 4 focus on different aspects of experimental results across various datasets. Contribution 3 highlights AnoLLM's strength with datasets containing mixed-type features, where it consistently outperforms all other methods. In contrast, Contribution 4 addresses the ODDS benchmark, which is dominated by numerical features (over 98.5%). Despite this, AnoLLMs perform comparably to the best methods. We distinguish these two contributions to clearly highlight this difference.

Consideration of additional datasets containing textual features: We note that there are two datasets containing textual features, fake job posts and 20 newsgroups. Although we have explored other tabular datasets with textual features, they either lack a legal license for our use or are not publicly available. Additionally, we would like to emphasize that AnoLLM performs well on tabular datasets with mixed-type attributes, including both categorical and numerical features. We identify a total of six datasets that meet this criterion.

Impact of random column permutations: We conducted an ablation study on the effect of random permutations, detailed in Section C. The results indicate that random permutation is a critical component of AnoLLM, and its absence can lead to a significant decline in performance.

Case study of AnoLLM: One failure case of AnoLLM is that the negative log-likelihood assigns equal importance to all features, which can be problematic when certain features are more critical than others. For instance, in the wine dataset, we observed that a single feature, Proline, plays a key role in distinguishing anomalies. AnoLLM performs worse on this dataset because it aggregates anomaly scores across all features, diluting the influence of Proline. In contrast, methods like KNN can better identify the importance of Proline, as its significantly larger values dominate the anomaly scores, leading to more accurate predictions.

Formatting of line 249 (Eqn.) Thanks for your suggestion. We have reformatted the Eqn.6.

Thank you for the constructive feedback. We sincerely appreciate your time in reading the paper and we are grateful for your reviews! Our responses are given below.

Inconsistency of baseline results with previous work (ICL): We identified a bug in the DeepOD implementation of ICL. To address this, we adapted the original author’s code and reran the method, updating the results in the current version. Our findings align with your prior observations: ICL performs similarly to KNN. Accordingly, we have revised our claim, stating that AnoLLM performs on par with the best-performing baselines on the ODDS benchmark, rather than outperforming deep learning methods.

Missing diffusion time estimation (DTE) as baselines: We have included the results of DTE in the current draft. The updated results show that DTE performs worse than AnoLLM on the mixed-type benchmark but is on par with ICL, KNN, and AnoLLM on the ODDS benchmark.

Column permutation as ensemble strategies: We note that column permutation is not inherently applicable to other approaches, as they typically process a single vector as input and do not rely on order dependencies. For these methods, permuting columns is equivalent to permuting dimensions within the vectors and can be interpreted as a naive ensemble strategy. Furthermore, we implement ensemble strategies whenever specified in the original papers. Specifically, 5 out of the 11 baselines (Iforest, RCA, SLAD, ICL, and REPEN) already employ ensemble strategies, and their reported results are aggregated across multiple models.

Other evaluation metrics: We have included other evaluation metrics such as AUC-PR and F1 scores in the appendix. We observe similar trends as in AUC-ROC.

Distinction between unsupervised and semi-supervised methods: Thanks for pointing this out. We have clarified it in Section 2.1. For the contaminated unsupervised setting, we expect naively adapting AnoLLM pipeline to it may not perform well as LLMs might overfit to the contaminated samples. To address this, an outlier-robust variant of AnoLLMs could be explored. For instance, one might use efficient techniques like KNN to filter outliers from the training data prior to training AnoLLMs. Another potential approach is to filter out high-loss training samples during training, similar to the robust collaborative autoencoders (RCA) method. Investigating AnoLLM’s performance in a fully unsupervised setting would be an intriguing direction for future work.

Hyperparameters selection for baselines: For each method, we picked the best-performing set of hyperparameters given in their original paper. For others not specified, we use the default hyperparameters as suggested by DeepOD and PyOD toolkit. While it is possible to use a held-out set for hyperparameter tuning in experiments, this approach is impractical in the unsupervised setting. Therefore, to ensure a fair comparison, we adopted the same hyperparameter selection approach as used in ADBench.

Additionally, we would like to also emphasize that we do not manually tune hyperparameters for AnoLLM, as its performance stabilizes once the training loss converges. Thus, users can select the batch size and decide whether to use a LoRA adapter based on GPU memory constraints. Afterward, the learning rate and number of training steps can be chosen to minimize the training error. This process does not rely on labeled data and follows standard practices for fine-tuning LLMs.

Computation costs of AnoLLMs: A runtime comparison experiment is presented in Section E of the Appendix. It is important to note that we selected the smallest LLM backbone, containing only 135M parameters, which allows it to be fine-tuned on a single GPU with 24GB of memory. The use of 7 A100 GPUs was solely to expedite the development process. The total computational cost of AnoLLM-135M is approximately 90 GPU-hours on a single RTX-A6000 GPU for the whole experiment. This is also included in Section E.

Reproducibility and Code release: We recognize that open-source code plays a crucial role in facilitating academic research and ensuring reproducibility. However, as part of an industry lab, we must comply with company policies. The code will be made available on the company's official GitHub once it receives approval from the legal department.

Regarding your minor comments: Thanks for pointing out the typos and citation errors. We have corrected typos and replaced arxiv references with conference or journal citations where available.

Thank you once again for your thorough review. We are pleased that most of your concerns have been addressed. We hope the following clarifications address your remaining concerns:

Incorporating references into the main text: We appreciate your feedback on this matter. We have added a new reference to the Evaluation Protocols paragraph in Section 3. Additionally, we have included an overview of the Appendix in Section A. The tables in Appendix H have also been updated to reflect the standard errors for AUC-ROC, F1, and AUC-PR.

Performance of AnoLLMs without pretrained weights: We apologize that we did not have sufficient time to run the experiment in our previous response. Following your suggestion, we evaluated the performance of randomly initialized transformers on the ODDS benchmark, which predominantly comprises 98.5% numerical features. To ensure a fair comparison, we used the same model architecture, SmolLM-135M, with identical hyperparameters. The results are summarized in the table below. As shown, AnoLLM with pretrained weights slightly outperforms its randomly initialized counterpart in terms of overall average performance. It achieves better performance on 16 out of 30 datasets and matches performance on 4 datasets. Additionally, a visual inspection of the training curves reveals that AnoLLM with pretrained weights converges approximately twice as fast on most datasets. This faster convergence can be attributed to the pretrained LLM providing a better initialization for fine-tuning. In contrast, AnoLLM without pretrained weights not only converges more slowly but is also more susceptible to overfitting, as evidenced by its significantly lower training loss. While overfitting may not be a major concern in uncontaminated, unsupervised settings, it could present challenges in contaminated scenarios, where the model risks memorizing anomalous samples. As one of the objectives of this paper is to demonstrate that LLMs can be applied to tabular anomaly detection, an interesting direction for future work would be exploring the trade-off between efficiency and accuracy. Based on this ablation study, for datasets dominated by numerical attributes in uncontaminated and unsupervised settings, AnoLLM without pretrained weights may serve as a more efficient alternative when smaller pretrained models are unavailable.

Thanks for your encouraging words and constructive comments. Your questions are answered below.

(W1 & W2) Effects of number of decimal numbers: Designing controlled experiments to analyze the effect of the number of digits on raw numbers presents significant challenges. One key difficulty lies in managing the variation in leading zeros, even when the number of significant digits is carefully controlled. For instance, as observed in the ODDS benchmark, the numerical features can range from 10^5 to 10−7. This wide range makes it impractical to control the number of digits in raw data without applying normalization. Therefore, we use normalization so that most numbers can be represented using 2 or 3 digits. In our early experiments, we also tried 10 and 100 bins for equal-width binning but found no significant differences in the outcomes.

(W3) Effect of not permuting column names We conducted an ablation study on the effect of random permutations, detailed in Section D. The results indicate that random permutation is a critical component of AnoLLM, and its absence can lead to a significant decline in performance.

(W4) Failure cases of AnoLLM: One failure case of AnoLLM is that the negative log-likelihood assigns equal importance to all features, which can be problematic when certain features are more critical than others. For instance, in the wine dataset, we observed that a single feature, Proline, plays a key role in distinguishing anomalies. AnoLLM performs worse on this dataset because it aggregates anomaly scores across all features, diluting the influence of Proline. In contrast, methods like KNN can better identify the importance of Proline, as its significantly larger values dominate the anomaly scores, leading to more accurate predictions.

Thank you for taking the time to read and review our paper! We are grateful for your feedback. Please see our responses below.

An example for anomaly detection: Thank you for your suggestion. We have updated Figure 1, making the anomalous example more clearly visible.

Statistical testing for clearer comparison: We provide critical difference plots in Section H of the supplementary material to compare all models on the mixed datasets and the ODDS benchmark. Additionally, we include standard error bars for the ODDS benchmark in Figure 2.

Results of other performance metrics: We have included other evaluation metrics such as AUC-PR and F1 scores in the supplementary. We observe the same trends in these metrics. The averaged ranking can also be seen in the critical difference diagrams.

Tradeoff between runtime and performance: We provide a compute efficiency analysis in Section F of the Appendix, comparing the runtime of various methods. While AnoLLM is slower due to its large backbone LLM, its performance on datasets with mixed-type data is significantly superior, often outperforming other approaches by a considerable margin. This highlights an interesting trade-off for practitioners to consider: balancing computational efficiency with model efficacy. Future work could explore improving AnoLLM's efficiency through techniques like model distillation or by leveraging it as a feature extractor for classical anomaly detection methods.

Regarding evaluation protocols: Our experiments are conducted in an uncontaminated unsupervised setting, where the training set contains only normal samples. Since the datasets lack a predefined train-test split, we randomly partition them to measure performance. Specifically, “50% of normal samples for training” indicates that we randomly select 50% of the normal samples for the training set, while the remaining normal samples, along with all anomalies, are included in the test set. We have clarified it in the evaluation protocol section.

Other methods for numerical feature modeling: One of the primary goals of this paper is to demonstrate that LLMs can effectively perform tabular anomaly detection. To achieve this, we use the original pre-processing techniques for numerical features and find that a simple normalization approach performs well. While integrating a separate, specialized numerical encoder (e.g. FT-transformer) is a plausible direction, it would likely demand extensive training data to fine-tune LLMs and enable them to interpret the encoder's outputs. Additionally, adapting the next-token prediction loss to handle continuous outputs, such as numbers, would be necessary. We consider this an intriguing avenue for future research and a potential first step toward developing a foundation model for tabular anomaly detection.

Other methods for categorical feature modeling: Since our task is unsupervised anomaly detection, using a target encoder is not suitable as it requires labeled data. For AnoLLMs, one advantage of leveraging large language models is their ability to interpret raw text directly, so we believe using raw features is the most effective approach. For baseline methods, high-cardinality categorical features can pose challenges for certain approaches. To address this, we group rare categories (those appearing in less than 1% of samples) to reduce the number of classes.

Pretrained models for anomaly detection: Thank you for your suggestion. We have incorporated a discussion on the impact of our AnoLLM in the revised conclusion section.

Thank to the authors for making detailed revision on the manuscript. The authors have addressed the points that were made and changed the score accordingly.