Towards Anomaly Detection on Text-Attributed Graphs

We appreciate your thoughtful comments and constructive feedback. We are pleased to address any questions and clarify our manuscript accordingly.

W1: The experimental evaluation appeared insufficient. Only three datasets are used, and it is unclear what kind of textual information is associated with each node. All anomalies are synthetically generated. Evaluation on more realistic datasets would be desirable.

A1: We conduct experiments using the real-world Yelp review dataset under unsupervised settings. We use the review comments as the text attribute of the nodes. The AUC results are as follows. For the baseline methods, we use the BOW feature here. We can observe that our method significantly outperforms other baselines in this real-world dataset. We are conducting further experiments using LM-based features and under few-shot conditions, and will include those results in the revised version.

W2: The proposed method involves many hyperparameters ($\alpha$, $\beta$, $\lambda$, etc.), but it is unclear how these are selected in a fully unsupervised (zero-shot) setting. Appendix C.3 mentions the use of early stopping, but in an unsupervised setting where no labeled data is available, what criterion was used to determine the stopping?

A2: For the selection of hyperparameters, we perform a grid search over predefined ranges of values. The combination of hyperparameters is then chosen based on the loss function. The determination of early-stopping also relies on the loss function. If the loss function shows no improvement for many epochs, the model stops training.

W3: From a methodological standpoint, the proposed method seems an ad hoc combination of several existing techniques (e.g., reconstruction error, contrastive learning, etc.). The theoretical motivation behind this combination is unclear. For example, it is not well explained why both the BoW representations and embeddings from LMs are used to represent the node text.

A3: We would like to emphasize that, to the best of our knowledge, this is the first work to explore the anomaly detection problem on the text-attributed graphs. Our method is not an ad hoc combination of existing techniques, but rather a novel framework that leverages the text features in graphs.

In the global module, we explore a two-stage algorithm towards the anomaly detection on text attributed graph: aligning the text embedding and graph structure first, and then using reconstruction loss to detect the global anomaly nodes. In the local module, we introduce a subgraph comparison mechanism to compute the local anomaly scores by the difference between the text graph and the ego graph.

Regarding the combination of BoW and LM embeddings: BOW and LMs represent the different aspects of texts. BOW represents text as word frequency vectors, emphasizing global contextual statistics, without considering word order or semantics. In contrast, LMs capture word order and semantics but may ignore global word distribution patterns. Therefore, they are complementary and represent different aspects of the text.

We thank you for taking the time to review our paper and for your valuable comments. We are happy to clarify our manuscript in response to the reviewer's questions.

W1: More design choices should be discussed, including the way to combine the text and structure, the model used to encode the text.

A1: In Section 5.3, we have already conducted experiments using alternative language models for text encoding, such as DeBERTa-large, e5-v2-base, and e5-v2-large, showing the robustness of our method across different pretrained models. We will explore other encoder-based language models in the revised version, such as ALBERT [1], LUKE [2].

For the way to combine the text and structure, we will conduct experiments using other GNN models, including GAT [3], GAE [4], and GraphSAGE [5], to assess the different graph encoders.

W2 & Q4: The intuition behind the computing global anomaly score and the local score should be conveyed intuitively. Please explain intuitively that the differences between two subgraphs can serve the local anomaly.

A2: Thank you for the valuable suggestion. We will add a figure in the revised version that visually illustrates the computation process.

Intuitively, inspired by [6], the patterns of global anomaly nodes deviate significantly from the majority and cannot be accurately reconstructed. Therefore, after alignment of the BOW space and the LM space, the reconstructed procedure can detect these anomaly nodes.

For a local anomaly node, its features are typically different within its neighborhood. Therefore, in the text graph constructed by semantic similarity, it has few edges with its neighbors, leading to a notable difference between the text graph and the ego graph. Accordingly, the local anomaly score is computed by measuring the discrepancy between the text graph and the ego graph.

Q1: There are no significant differences between text feature and other attributes, if specific encoders convert them into the embedding. What is the key point if you say the text-attributed graph?

A3: The key distinction is the rich information of the text feature. Text features contain various content that can capture both global context patterns and local semantic relationships. In contrast, numeric or categorical attributes are often sparse and lack the compositional structure of text. Thus, text-attributed graphs require specialized models [7] that leverage the semantic hierarchy and contextual nuance of language.

Q2: The paper claim that it is the first work on the anomaly detection in the text-enhanced graph. It is somewhat over claimed. The experimental studies are performed on graph Cora, Arxiv, Pubmed, which are widely used before.

A4: To the best of our knowledge, this work is indeed the first to focus on anomaly detection in text-attributed graphs. While there are existing models designed for graph anomaly detection (GAD), they do not specifically address text-attributed graphs, which have unique characteristics and challenges. Although the datasets are widely used in the literature, they are primarily for node classification tasks, rather than graph anomaly detection. We inject anomalies into these datasets for graph anomaly detection tasks, distinguishing our approach from previous research.

Q3: What is the rationale behind the contrastive loss on the hidden state and the result of the text encoder? The hidden state preserves the structure and the content similarity, while the output of the text encoder only handles the content similarity. Do it means that your method mainly handles the homophilic graphs.

A5: The contrastive loss aims to find the global anomaly nodes that deviate from the major distribution. For the global anomaly nodes, the features will be changed a lot due the the GNN message passing, and thus hard to align. In the heterophilic graphs, where the neighbors of anomaly nodes are mainly normal nodes, the features will be changed larger, thus easier to find the anomaly nodes.

Q5: The interaction between the global and local score are not clear. The score of the global anomaly score is at the graph-level. How does this score aid to compute the node-level anomaly. I cannot find the relationship between two parts in the main figures. There are also two algorithms in the appendix. It is not encouraged to studied two subproblems in one paper.

A6: Although the global GAD module learns from the whole graph structure, the output is still node-level anomaly scores. Each node’s global anomaly score is computed by the alignment loss between its graph-based representation and its semantic feature.

The global and local modules are not independent, but rather interdependent. In the local module, the representation in the ego graph is computed by the alignment procedure in the global module. This ensures that the ego graph combines the global embedding, rather than ignoring the graph structure. In the final anomaly score, the global and local anomaly scores are combined complementarily. We can also observe from the ablation studies that removing either module will significantly reduce the performance of the model.

Q6: The experimental results in Table 3 show that the larger model can produce worse results in accuracy. Have you tried recent LLM? Please discuss the results more.

A7: We clarify that the differences between large and small pretrained language models are quite similar, typically within 1% in most cases. This suggests that the model size does not strongly affect performance, and these small differences may be attributed to dataset-specific noise. In our work, the language model is used for encoding the text. However, most recent LLMs, such as GPT-4 and Gemini, are based on decoder-only transformer architectures, which are not directly compatible with our method.

[1] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. ArXiv, 2019, 1909.11942.

[2] Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, Yuji Matsumoto. LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2020.

[3] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, Yoshua Bengio. Graph Attention Networks. International Conference on Learning Representations, 2018.

[4] Will Hamilton, Zhitao Ying, Jure Leskovec. Inductive Representation Learning on Large Graphs. Advances in neural information processing systems, 2017, 30.

[5] Thomas N. Kipf, Max Welling. Variational Graph Auto-Encoders. ArXiv, 2016, 1611.07308.

[6] Kaize Ding, Jundong Li, Rohit Bhanushali, Huan Liu. Deep anomaly detection on attributed networks. Proceedings of the 2019 SIAM international conference on data mining, 2019, 594-602.

[7] Hao Yan, Chaozhuo Li, Ruosong Long, Chao Yan, Jianan Zhao, Wenwen Zhuang, Jun Yin, Peiyan Zhang, Weihao Han, Hao Sun, Weiwei Deng, Qi Zhang, Lichao Sun, Xing Xie, Senzhang Wang. A Comprehensive Study on Text-attributed Graphs: Benchmarking and Rethinking. Advances in Neural Information Processing Systems, 2023, 17238-17264.

I appreciate the authors' detailed feedback. The basic idea of the proposed method is conveyed more clearly now. I will raise the Clarity score.

I am still not convinced by the feedback to Q1 and Q2. As you said, Text-attributed graph requires specialized model. However, the model itself is not your major contribution.

We thank you for taking the time to review our paper and for your valuable comments. We are happy to clarify our manuscript in response to the reviewer's questions.

W1: The global module feeds Bag-of-Words vectors into a Graph Convolutional Network (GCN) and forces that output to match a frozen language-model space. Because the two spaces differ in distribution and dimensionality, I am afraid that training may suffer from unstable gradients and scaling issues, thus leading to suboptimal performance.

A1: We have considered the two spaces to be different in distribution. To address this, we first align the BOW embedding space and the language model embedding space. This ensures that gradient updates are stable and not influenced by other objectives. After the two spaces are aligned, we introduce the reconstruction loss function to detect anomaly nodes more effectively. We illustrate this with the following training loss values on the Cora dataset, which show smooth convergence.

W2 & Q3: When the anomaly rate is not low, subtracting the global mean embedding may dampen true signals because the mean itself drifts toward anomalous directions. How does subtracting the mean embedding affect detection when the anomaly proportion exceeds ten percent?

A2: As established in prior work [1], [2], anomaly rates are generally low in the graph anomaly detection problem. In the real-world datasets in the benchmark [3], among the ten real datasets, only one exhibits a related high anomaly rate (21.8%), while two have anomaly rates slightly above 10%. The anomaly rates of the other seven datasets are all below 10%. Therefore, in most practical scenarios, the impact of anomaly nodes on the global mean embedding is minimal.

However, in rare cases with a high anomaly ratio, the mean embedding may shift toward anomalous features, potentially degrading the model's performance. We will include a discussion of this limitation in the revised version.

W3: The tables give mean Area-under-ROC but omit variances or confidence intervals, even though the appendix states that error bars were produced. This hides run-to-run instability.

A3: We have conducted experiments on the Cora dataset to demonstrate the variances of our models and other baseline models under zero-shot settings with 95% confidence intervals. The results are shown in the following table. These results demonstrate that our method achieves high performance with low variance. We are conducting further experiments under few-shot conditions with other datasets and will include those results in the revised version.

Q1: Can you provide standard deviations for the reported AUC scores and add precision-recall or top-K precision so readers can judge practical screening performance?

A4: We conduct experiments on the Cora dataset using the BOW embedding to evaluate the performance of our model on other metrics, F1-macro, and Rec@k. The results are as follows. We can observe that TAGAD emerges as the winner in other metrics. We are conducting additional experiments using LM-based features and under few-shot conditions with other datasets, and will include those results in the revised version.

Q2: Could the mixing weight $\lambda$ be learned, perhaps by optimizing a small validation set during training, to remove the need for manual tuning?

A5: We treat $\lambda \in [0, 1]$ as a tunable hyperparameter and determine its value via grid search. Specifically, we select the value $\lambda$ that minimizes the cross entropy loss on a small valid set. This approach allows us to adjust the hyperparameter $\lambda$ automatically, rather than manual tuning. We will add the details of hyperparameter tuning in the revised version.

Q4: Please show qualitative failure cases and explain which component (global or local) is responsible, so the community can understand the limits of the approach.

A6: We agree that providing failure cases is crucial. In the revised version, we will add a case study discussing specific failure scenarios on the real-world Yelp dataset, providing insights into the limitations and future work for improvement.

Q5 & Concerns: In line 26, “lecture” instead of “literature”? Table 3 spills over the margin in the provided PDF and is captioned at the bottom.

A7: Thank you for pointing out these errors. We have corrected them in the revised version.

[1] Yang Liu, Xiang Ao, Zidi Qin, Jianfeng Chi, Jinghua Feng, Hao Yang, Qing He. Pick and Choose: A GNN-based Imbalanced Learning Approach for Fraud Detection. Proceedings of the web conference, 2021, 3168-3177.

[2] Ge Zhang, Jia Wu, Jian Yang, Amin Beheshti, Shan Xue, Chuan Zhou and Quan Z. Sheng. FRAUDRE: Fraud Detection Dual-Resistant to Graph Inconsistency and Imbalance. 2021 IEEE international conference on data mining, 2021, 867-876.

[3] Jianheng Tang, Fengrui Hua, Ziqi Gao, Peilin Zhao, Jia Li. GADBench: Revisiting and Benchmarking Supervised Graph Anomaly Detection. Advances in Neural Information Processing Systems, 2023, 29628-29653.

We appreciate the reviewer's positive feedback. We also greatly appreciate your recognition and support for our work! We are happy to clarify our manuscript in response to the reviewer's questions.

W1: The paper presents experimental results only on synthetic datasets. However, real-world data distributions may differ significantly from those of synthetic datasets, which could impact the effectiveness and generalizability of the proposed method.

A1: We conduct experiments using the real-world Yelp review dataset under unsupervised settings. We use the review comments as the text attribute of the nodes. The AUC results are as follows. For the baseline methods, we use the BOW feature here. We can observe that our method significantly outperforms other baselines in this dataset. We are conducting further experiments using LM-based features and under few-shot conditions, and will include those results in the revised version.

W2: The proposed method is implemented with DeBERTa-base. In the current era of large language models, integrating more advanced or larger language models could further enhance the completeness of the experiments.

A2: In our work, the language model is used for encoding the text. However, recent large language models are generally based on decoder-only transformers, which cannot be directly integrated into our method.

W3: For Table 3, it appears that not only fine-tuned models but also larger language models result in worse performance. This outcome is quite counterintuitive, yet the authors offer only a brief explanation, stating that "the pretrained LM has already learned rich semantic representations." A deeper exploration and more thorough analysis are needed to provide reliable and explainable results.

A3: First, we clarify that the differences between large and small pretrained language models are quite similar, typically within 1% in most cases. This suggests that the model size does not strongly affect performance and these small differences may be attributed to dataset-specific noise.

Regarding the drop in performance for fine-tuned models, our analysis indicates that this issue is caused by the mismatch between the pretrained LM and the randomly initialized GNN during early training. Since the LM already encodes rich semantic information, introducing noise from the under-trained GNN during joint optimization can cause the LM’s representation quality to degrade. This mismatch leads to a decrease in overall performance compared to keeping the LM frozen.

Concerns: Line 26, I think you mean "In the GAD literature". The caption for Table 3 should be above instead of below the table.

A4: Thank you for pointing out these errors. We have corrected them.