Unifying Unsupervised Graph-Level Anomaly Detection and Out-of-Distribution Detection: A Benchmark

W1: While the paper provides extensive comparisons across methods, it could benefit from a more in-depth discussion of why certain methods fail in specific scenarios. For instance, understanding the root causes of poor performance under noisy training data or near-OOD conditions.

Answer W1: Thank you for your valuable suggestion. While the primary goal of our benchmark is to provide a robust framework and dataset setting for GLAD/GLOD research, we agree that a deeper discussion of method performance in specific scenarios, such as near-OOD detection and noisy training data, would enhance the paper. Below is a more detailed analysis based on our observations:

• Near-OOD Samples Are More Difficult to Detect than Far-OOD Samples. Figure 4 demonstrates that models generally struggle more with near-OOD samples compared to far-OOD ones. End-to-end methods like GLocalKD, OCGTL, and SIGNET consistently achieve higher AUROC values for far-OOD scenarios, especially on datasets such as AIDS, BZR, and ENZYMES. The key challenge with near-OOD samples lies in their high similarity to ID data, which requires models to capture subtle deviations that are often overlooked. This highlights the importance of developing methods capable of fine-grained feature discrimination.

• Limited Generalizability in Specialized Scenarios. Significant performance gaps emerge between near-OOD and far-OOD conditions, particularly in specialized scenarios. For instance, methods such as GOOD-D and GraphDE show clear limitations when dealing with near-OOD samples, where their detection capabilities are compromised due to the overlap with ID-like features. Similarly, GLAD methods often falter in size-based anomaly detection tasks, indicating that their design may not fully account for certain domain-specific characteristics or structural shifts.

• Dataset-Specific Sensitivity Variations. Our findings also reveal that the sensitivity of methods to anomalies varies significantly across datasets. For example, models like GLocalKD and OCGTL excel on the AIDS dataset but perform inconsistently on others, such as ENZYMES, where dataset-specific complexities reduce their effectiveness. This variability underscores the need for benchmarks like ours to evaluate methods across a diverse set of datasets, exposing strengths and weaknesses in different contexts.

These observations emphasize the critical need for more robust and adaptable models that can effectively handle near-OOD detection challenges, noisy training data, and diverse dataset characteristics. While the main manuscript focused on presenting the benchmark framework, these nuanced findings underline its value in highlighting gaps in current methods and guiding future research. We hope this deeper analysis provides the clarity and context you suggested. Thank you again for your insightful feedback.

W2: While the paper provides extensive comparisons across methods, it could benefit from a more in-depth discussion of why certain methods fail in specific scenarios. For instance, understanding the root causes of poor performance under noisy training data or near-OOD conditions.

Answer W2: Thank you for your valuable suggestion. We recognize the importance of providing concise summaries to enhance the readability and accessibility of our findings. In response, we have added summary sections at the end of each results subsection to highlight the most important insights from our multi-dimensional analyses. These summaries are presented in a bulleted list format to allow readers to quickly grasp the key takeaways.

We appreciate Reviewer yW1o for the positive review and constructive comments. We provide our responses as follows.

W1: In GLAD, anomalies can be considered as out-of-distribution (OOD) test samples for type 1 and type 2, but further clarification is needed in the paper on how the in-distribution (ID) test samples are constructed.

Answer W1: This is an excellent question, and we appreciate your attention to this detail. We have supplemented the description of in-distribution (ID) test samples in Appendix B: Detailed Description of Datasets in UB-GOLD. Below is a summary of how ID samples are constructed for the two dataset types:

• Type I: Datasets with Intrinsic Anomaly. These datasets inherently include both ID and OOD samples. For the test set, ID samples are selected proportionally from different labels, ensuring diversity in the in-distribution data. For four of these datasets, the ratio of test_{ID} to train_{ID} varies significantly. This design choice allows models to be tested under various conditions, ensuring their robustness in distinguishing ID and OOD samples.
• Type II: Datasets with Class-Based Anomaly. In this case, the ratio of test_{ID} to train_{ID} is fixed at 1:4. We employ 5-fold cross-validation during the experimental process, ensuring that all data points are used for both training and testing in a balanced manner.

W2：In an unsupervised setting, does this mean the test set can also be used as the training set? The in-distribution samples in the test set could potentially help improve performance.

Answer W2: Thank you for raising this important question. In our benchmark, the test set samples are strictly separated from the training set and are not used during training to ensure unbiased evaluation. Allowing test set samples to be included in the training data could lead to data leakage, artificially inflating performance and compromising the integrity of the benchmark.

In our setup, the training set consists solely of in-distribution (ID) samples, enabling the model to learn the characteristics of the ID data without being influenced by the test set. The test set, on the other hand, includes both ID and OOD samples, providing a robust evaluation of the model’s detection capabilities. This separation reflects real-world scenarios where unseen data must be detected independently and adheres to best practices in OOD detection and anomaly detection tasks. We believe this approach ensures that the reported performance metrics are reliable and reflective of the model's true generalization ability.

W3:From the several observations, we found there are large overlaps between OOD samples, in addition to the experimental explanation, more specific analysis is needed on this point for particular datasets.

Answer W3: Thank you for your question. We strictly ensured that the dataset partitions are non-overlapping, meaning there is no overlap between OOD samples and ID samples, or between training and test sets. For each dataset, we carefully curated the splits to maintain a clear separation, ensuring that OOD samples are entirely distinct from ID data and that this separation is consistently upheld across all experimental setups.

We sincerely apologize for any misunderstanding caused by our phrasing in the manuscript. To address this, we will add a specific statement in the main text(page 15) explicitly clarifying that there is no overlap in the dataset partitions. This clarification ensures that readers are fully aware of our rigorous dataset preparation process.

By maintaining strict non-overlapping partitions, we eliminate any potential data leakage and provide results that reliably reflect the model’s ability to generalize to truly unseen OOD samples. We appreciate your observation and hope this explanation resolves your concern.

We are grateful to Reviewer 2xER for providing insightful feedback. The detailed responses are provided below.

W1: Unification and Benchmarking of GLAD and GLOD Tasks. We sincerely appreciate the reviewer’s detailed feedback on the unification and benchmarking of GLAD and GLOD tasks. Below, we address each point raised and provide clarifications and proposed enhancements to improve the manuscript:

W1.1: Formal Definitions, providing these definitions upfront would establish clarity and context for readers unfamiliar with these tasks.

Answer W1.1: We acknowledge the importance of providing formal definitions for GLAD and GLOD before introducing the unified framework of "Unsupervised generalized graph-level OOD detection." These definitions will help readers unfamiliar with these tasks understand their foundations and provide necessary context for the unification effort.

In the revised manuscript (pages 3 and 19), we have included detailed formal definitions of GLAD and GLOD using mathematical symbols. This step ensures that the foundational concepts of GLAD and GLOD are clearly established before introducing the unified task of unsupervised generalized graph-level OOD detection, providing readers with a structured and logical progression of ideas. These updated sections are highlighted in blue for easy reference.

W1.2: Relationship Between GLAD and GLOD.

Answer W1.2: To clarify the relationship between GLAD and GLOD, we focus on their objectives and shared underlying principles rather than framing one as broader than the other. While the two tasks may have conceptual differences, their goals are aligned in practice: identifying graphs that deviate from the expected norm.

Both tasks ultimately aim to separate graphs into two categories: graphs that align with the expected norm and graphs that do not. This shared objective forms the foundation for our unified task of generalized graph-level OOD detection, which treats both anomalies and OOD graphs as generalized OOD samples. By unifying these tasks, we simplify their conceptual overlap, enabling methods and evaluations to address both scenarios under a single framework.

W2: Experimental Details and Completeness.

W2.1: Definition and Generation of Anomalies/OOD Samples.

Answer W2.1: We have added detailed explanations regarding the definition and generation of anomalies and OOD samples in Appendix B: Detailed Description of Datasets in UB-GOLD. This section includes comprehensive descriptions for each type dataset used in the benchmark.

W2.2: Completeness of Results in Figures.

Answer W2.2: We sincerely thank you for raising such a detailed and thoughtful question. Below, we provide explanations for the omissions in Figures 4 and 5 and outline the steps we’ve taken to address this concern:

• Omission of GK-D Method: The GK-D method differs from other approaches as it requires incorporating the test set into the training process to achieve meaningful results. This design makes it incompatible with tasks such as near-far OOD detection and perturbation analysis, which involve finer-grained partitioning of the test or training set. As this method's behavior does not align with the experimental setups for these tasks, we did not include GK-D in Figures 4 and 5.

• Omission of SSL-D Method in Figure 5: The SSL-D method exhibited relatively poor performance in the experiments. Including its results in Figure 5 would make them visually indistinguishable and potentially reduce the clarity of the figure.

• Complete Results for Figure 5 Added to the Appendix: We have supplemented the full results for Figure 5 in the appendix (Figures 8–10). These results ensure that readers have access to all experimental outcomes for transparency and completeness.

W2.3:Metrics and Additional Results:

Answer W2.3: We sincerely thank you for highlighting this important aspect. In the main text, we used the AUROC metric to analyze the results presented in Figures 4 and 5, as it is a widely recognized standard for evaluating model performance. To further clarify this point, we have updated the captions for Figures 4 and 5 to explicitly state that the results are presented in terms of AUROC. In the original manuscript, we included results for two additional metrics, AUPRC and FPR95, for Figure 4 in Table 7 of the appendix. To address the reviewer’s suggestion, we have supplemented the revised manuscript with complete results for all metrics (AUROC, AUPRC, and FPR95) related to Figure 5 and included them in the appendix as Figures 8–10. Additionally, we have added brief analyses in the appendix to provide a detailed comparison of methods across all metrics, offering readers a more comprehensive understanding of the methods’ performances. We hope these updates enhance the clarity, completeness, and value of the manuscript as a benchmark resource.

W1 and Q1: The main findings of the paper are mostly expected. How do the main findings of the paper contribute to future research on GLAD and/or GLOD?

Answer W1 and Q1: Thank you for your insightful comment. While some of our findings may align with existing expectations, the main contribution of our work lies in systematically validating these assumptions through a unified benchmark and uncovering critical insights that will guide future research. Below, we elaborate on how our findings contribute to the progression of GLAD and GLOD research:

Empirical Validation Across a Wide Range of Datasets: Although certain results, such as the superior performance of GNN-based methods, may seem anticipated, our comprehensive evaluation across 35 datasets provides a robust empirical foundation. This cross-domain validation confirms that these insights are not limited to specific scenarios but are consistent across various types of anomaly and OOD detection tasks. This extensive testing helps solidify the reliability of current methods and offers a benchmark for refining and developing new approaches.

Current Limitations and Future Directions: One of the most valuable contributions of our work is the identification of significant limitations in existing methods. These limitations are rarely discussed by most existing studies befofre. In this case, our identification of these limitations present clear avenues for future research:

• Challenges in Detecting Near-OOD Samples: We found that methods struggle more with near-OOD samples compared to far-OOD samples. Near-OOD samples are inherently more difficult to detect due to their subtle differences from ID data, which leads to challenges in distinguishing them from the majority class. This issue is a core part of what we now describe as the OOD sensitivity spectrum. Future research should focus on improving models’ sensitivity to these subtle shifts by developing more nuanced representations and decision boundaries that can distinguish near-OOD samples from ID data more effectively.
• Vulnerability to Noisy Training Data: Another limitation observed is the sensitivity of many methods to noisy or contaminated training data. In real-world applications, the presence of noise or mislabeled samples can severely impact performance. Our findings point to the need for more robust methods that are resilient to noisy training data, ensuring reliable performance in practical, less-than-ideal conditions.
• Difficulty in Handling Complex Distribution Shifts: Existing methods also show limitations when dealing with complex, multi-dimensional distribution shifts, particularly in specialized tasks like size-based or scaffold-based OOD detection. This highlights the need for more flexible models that can adapt to varying types of OOD shifts and better differentiate between different OOD scenarios. Developing models that are more adaptive to these shifts could greatly improve the robustness of OOD detection methods.

Bridging GLAD and GLOD: Our findings emphasize the overlap between GLAD and GLOD tasks, showing that techniques developed for one can often be applied to the other. This unification encourages the transfer of techniques between the two domains, fostering innovation and accelerating progress in both fields. By bridging these two tasks, our work lays the foundation for more generalizable solutions in graph-level anomaly and OOD detection.

UB-GOLD as a Foundational Resource for Future Research: By providing a unified and open-source benchmark, UB-GOLD serves as a foundational resource for researchers to test new hypotheses and validate methods under consistent experimental conditions. The challenges identified in this work, such as near-OOD detection and robustness to noisy data, present clear directions for future research, helping the community tackle these problems in a structured and meaningful way.

In summary, while some of the findings reaffirm existing knowledge, the systematic validation of these findings through the unified benchmark and the identification of critical method limitations provide actionable insights for future research. Addressing these limitations—particularly in detecting near-OOD samples, improving robustness to noisy data, and handling complex distribution shifts—will be key to advancing GLAD and GLOD methods. This work not only contributes a useful benchmark but also sets a clear path for further developments in the field.

W2 and Q2: It is not clear why the same methods should solve anomaly detection and out-of-distribution detection. What is the motivation for unifying GLAD and GLOD?

Answer W2 and Q2: We appreciate the reviewer’s question about the rationale for unifying GLAD and GLOD under a single benchmark. Our motivation stems from the significant conceptual overlap between these tasks and the practical benefits of exploring them together. Below, we outline the reasoning behind this unification:

Shared Objective with Practical Implications: Both GLAD and GLOD aim to identify unusual graphs—GLAD focuses on detecting irregularities within a known distribution, while GLOD identifies samples from unseen distributions. In real-world scenarios, these distinctions are often blurred:

• In drug discovery, a structurally unique molecule (anomaly) may come from a new chemical distribution (OOD).
• In social network analysis, detecting unusual user behaviors (GLAD) may involve recognizing emerging patterns from new or unknown groups of users (GLOD).

By unifying the tasks, we create a framework that more accurately reflects these real-world complexities, enabling methods to address both types of deviations effectively.

Overlapping Methodologies and Potential for Cross-Pollination: The existing solutions for GLAD and GLOD share common foundational techniques, such as contrastive learning, reconstruction, and one-class classification. The methodological commonality enables techniques suitable for one task to be adapted or extended for the other with minimal adjustments. Therefore, unifying these tasks under a single benchmark encourages researchers to adapt and extend methods for both tasks, accelerating progress and innovation across domains.

Efficiency and Broader Insights from a Unified Framework: A unified benchmark streamlines evaluation by providing a common set of datasets, metrics, and experimental setups. This reduces redundancy, enabling researchers to compare performance across tasks more effectively. Furthermore, by analyzing results across GLAD and GLOD jointly, we can uncover broader insights, such as the generalizability of methods across different detection challenges.

In summary, the unification of GLAD and GLOD is motivated by their conceptual overlap, shared methodologies, and practical relevance in real-world applications. The unified benchmark fosters efficiency, innovation, and deeper understanding, laying a foundation for more versatile and impactful detection methods.