Dissect Black Box: Interpreting for Rule-Based Explanations in Unsupervised Anomaly Detection

Thank you for your valuable feedback. We appreciate your insights and suggestions, which have helped us to improve our paper.

Comment 1: “The writing should be enhanced as some technical concepts are unclear. For example, a running example could be provided to explain the SCD tree and GBD algorithms.”

Response: We agree that a running example would greatly clarify the technical concepts of the SCD tree and GBD algorithms. We have added a detailed running example in Section 4.2, “Methodology Overview,” and pseudocode in Appendix A.2 where we walk through the entire process of constructing an SCD tree and applying the GBD algorithm using a sample dataset. This example includes:

1. Initial Data Segmentation (SCD-Tree)

   • The SCD-Tree uses the outputs of the anomaly detection model to segment the data.
   • The tree structure is built by recursively splitting the data based on the model's outputs.

2. Boundary Delineation (GBD)

   • Within each segment, the GBD algorithm uses Gaussian Processes to define decision boundaries.
   • For instance, in a segment with features A and B, the algorithm determines the boundary where the likelihood of being normal is highest.
   • These boundaries are then refined to capture the nuances of the data distribution.

3. Rule Extraction

   • The refined boundaries are translated into interpretable rules.
   • For the example dataset, a rule might be: "If A > 5 and B < 10, then the data point is normal."

Comment 2.1: “The work targets the structured, tabular data which should be clear at the beginning of the paper.”

Response: We apologize for the lack of clarity regarding the data type our method targets. We have revised the introduction to explicitly state that our approach is designed for structured, tabular data. Additionally, while our method is designed for structured, tabular data, we acknowledge the importance of extending interpretability methods to unstructured data (e.g., text, images). Potential strategies for applying the SCD-Tree to unstructured data include:

• Feature Engineering: Transforming unstructured data into structured formats using techniques such as text vectorization or image feature extraction.
• Hierarchical Clustering: Using hierarchical clustering to segment unstructured data before applying the SCD-Tree.

We will discuss these potential extensions further in the future work. Thank you for your carefully detailed suggestions, which helped us improve the comprehensibility of the article!

Comment 2.2: “ How to build such an SCD tree for unstructured data is not explored.”

Response: In future work we envisage Building SCD Tree for Unstructured Data. To extend the SCD tree for unstructured data, such as images, we propose the following steps:

1. Image Data Feature Engineering. Use convolutional neural networks (CNNs) or other feature extraction techniques to convert images into feature vectors. Pretrained models (e.g., VGG16, ResNet) can be utilized to extract high-level features from images.

2. Data Segmentation (SCD-Tree). Once the unstructured data is transformed into a structured format, the SCD-Tree can be applied in the same manner as for structured data. The tree uses the extracted features(chhannel or patch granularity) to segment the data based on the anomaly detection model's outputs. Subsequent GBDs and Rule Extractions will work in the same way as the original.

Comment 3: “What is the typical running efficiency?”

Response: We appreciate the reviewer's interest in the running efficiency of our model. We have conducted a thorough evaluation of both the training and inference times of our method. Detailed results are presented in Appendix A.4.1, where we provide comprehensive performance metrics for various datasets.

Specifically, Figure 2c in the main manuscript illustrates the typical running efficiency. This figure shows the training and inference times for our proposed method across different data dimensions, highlighting its scalability and efficiency. The results indicate that while the training time scales with the number of features and data points, the inference time remains consistently low due to the rule-based nature of our method. These findings demonstrate that our method is both efficient and scalable, making it suitable for deployment in high-stakes environments where timely decision-making is crucial.

Comment 4: “Considering the data drifts and perturbations, does the naive Gaussian-based assumption work for more heterogeneous drifting data?”

Response: The naive Gaussian-based assumptions can be effectively applied to more heterogeneous drift data.

• The probabilistic nature of Gaussian Processes provides a measure of uncertainty in the predictions. This is particularly useful for heterogeneous drift data, as it allows the model to quantify the confidence in its predictions.
• Gaussian Processes are inherently flexible and capable of modeling complex, non-linear relationships in data, allowing GPs to adapt to local patterns and variations within the data. And GPs can dynamically adjust their parameters to capture the nuances of different segments, ensuring accurate representation of the underlying distributions.
• In our methodology, the SCD-Tree initially segments the data into more homogeneous regions. Within each segment, the GBD algorithm applies the Gaussian-based assumptions which ensures that GP are applied within localized regions where they are more likely to hold true, even if the overall data distribution is heterogeneous.

The experiments results in Table 3 of Section 6.3 show that our method maintains high fidelity and robustness, indicating that the Gaussian assumptions are sufficiently flexible to handle diverse and drifting data patterns.

Thank you for your constructive review. We hope these changes meet your expectations and look forward to any further comments you may have.

Thank you! Your feedback has been invaluable in enhancing the clarity and impact of our work.

Comment 1:The proposed method primarily builds upon some existing techniques. ... While the integration and adaptation of these techniques are innovative to some extent, the foundation lacks substantial originality.

Response: While it is true that our method integrates existing techniques like decision trees and Gaussian Processes, the novelty lies in how these techniques are combined and applied to the problem of interpreting black-box anomaly detection models in high-risk fields like cybersecurity.

We observed that traditional methods struggle with rule fitting for high-dimensional data due to their reliance on direct Euclidean distance calculations in feature space. To address this, we developed the SCD-Tree, which integrates anomaly detection model outputs directly into its splitting criteria. Unlike traditional decision trees, our approach leverages the decision-making results of black-box models to capture complex data distributions effectively and this unsupervised calculation method subverts traditional entropy-based approaches.

In addition, we recognized that outliers often exist within normal data in anomaly detection scenarios. Traditional methods typically have rigid decision boundaries, leading to reduced robustness and the potential for false alarms and is a significant concern in security field [1]. To mitigate this, we introduced the GBD algorithm, providing flexible boundaries that better accommodate data variability. By integrating GBD with the SCD-Tree, our method offers an interpretable, and robust framework for anomaly detection that maintains high fidelity to the original black-box models while reducing the incidence of false alarms.

Comment 2: The clarity of the paper is compromised by the disorganized structure of the related work section. ...To improve clarity, the sub-sections should be organized more coherently.

Response: We appreciate the advice on the related work. We have reorganized the related work section into clearer sub-sections, each focusing on a specific aspect of related research: Unsupervised Anomaly Detection Techniques, Interpretability in Anomaly Detection, Issues inside Existing interpretation Approaches in Anomaly Detection Models. Due to word limitations, we will provide you with the revised full related work section in the final version during the discussion phase.

Thank you very much for your advice. This reorganization helps to show the logical flow more clearly.

Comment 3(i): The experiments use AE, VAE, OC-SVM as black-box models. ... it would be beneficial to include the latest state-of-the-art deep anomaly detection models.

Response: We would have liked to demonstrate the fitness of our model with as classical a model as possible, and the experimental results bear this out. We acknowledge the importance of using state-of-the-art deep anomaly detection models. To address this, we have extended our experimental evaluation to include recent advanced models such as VRAE and DAGMM. The results are shown in Table 3 of the global response. These models represent the latest advancements in deep anomaly detection and provide a more comprehensive validation of our method's applicability.

Comment 3(ii): The curse of dimensionality is highlighted as a challenge in the paper. However, the highest dimensionality of the datasets used in the experiments is only 80.

Response: We appreciate your concern regarding the dimensionality of our datasets. In anomaly detection, the curse of dimensionality is particularly challenging due to the inherent sparsity and complexity of high-dimensional data. Although 80 features might appear moderate in other fields, they are considered high-dimensional within anomaly detection. We have carried out an exhaustive study of datasets in the field of anomaly detection and have added a summary table to the attachment of global response. Our chosen datasets, such as CIC-IDS2017, and TON-IoT, are standard benchmarks in this domain, typically ranging from 10 to 80 features. These dimensions capture the real-world complexity, each feature signifys a specific aspect of network traffic.

In practice, datasets with 80 features are sufficient for effective anomaly detection, as demonstrated by our method's high fidelity and robustness across these benchmarks. We have also added to the global response the results of our model's experiments on data of different dimensions, demonstrating its ability to handle high-dimensional data. Research has shown that more features do help in security tasks, so building datasets with a higher number of features is really the trend nowadays. We are also in the process of collecting higher dimensional datasets about network packets, and in future work, we will also try to use our method to apply it to unstructured data to test its effectiveness in higher dimensional data.

Comment 4:Are there any theoretical advancements or unique aspects of these methods distinguishing them from similar techniques?

Response: Yes, our method introduces several theoretical advancements and unique aspects that distinguish it from similar techniques:

• The SCD-Tree uses the outputs of anomaly detection models directly in its splitting criteria, which is a novel unsupervised approach to enhance the tree's ability to capture complex data distributions.
• The GBD algorithm refines the decision boundaries within each segment using Gaussian Processes, providing a probabilistic framework that quantifies the uncertainty in boundary definitions, enhancing robustness against data drift.

We have also restated the necessity and innovation in global response, and thank you for your professional comments and suggestions.

Reference:

[1] Hassan, Wajih Ul, et al. "Nodoze: Combatting threat alert fatigue with automated provenance triage." network and distributed systems security symposium. 2019.

Thank you for your thorough review and insightful feedback on our paper.

Comment 1: Lines 106-110: "In summary ... their operational logic." The paper has not systematically addressed attributes such as interpretability, non-reliance on oversimplified surrogate models.

Comment 6:Line 282 -The more relevant interpretability measures in [49] are 'Number of rules' and 'Average rule length'.

Response: Thank you for pointing out this oversight. We had intended to use robustness to show that our model explains the black-model well. We agree that we should provide concrete evidence for interpretability and non-reliance on oversimplified surrogate models.

In revised paper, we incorporate these metrics ('Number of rules' and 'Average rule length' ) into our evaluation. Additionally, We add ablation experiments to demonstrate the interpretability. The results of the experiment are attached to the Table 4 and Table 1 of global response.

Comment 2:Section 4: The current paper should discuss the differences with prior literature and whether any of those earlier techniques can be utilized here.

Response: We appreciate the references to earlier work. Our approach differs from prior literature primarily in the integration of Segmentation Clustering Decision Tree with Gaussian Boundary Delineation.

To clarify these distinctions, we include a more comprehensive related work section that discusses the differences between our method and previous techniques [33, 34, 35, 36], specifically highlighting the novelty and advantages of combining SCD-Tree with GBD for rule extraction. Additionally, we explore the potential applicability of earlier techniques to our methodology and discuss their comparative performance.

Comment 3 & Questions: Section 5: A justification for using Gaussian Processes is not presented. Why not use some other model such as KDE?

Response: We use Gaussian Processes (GPs) for their probabilistic framework, which quantifies uncertainty at each prediction point through variance. This capability is crucial for ensuring robustness in boundary delineation by allowing us to assess confidence levels in decision boundaries. KDE is a non-parametric way to estimate the probability density function of a random variable. While KDE is effective in density estimation, it does not inherently provide a measure of uncertainty. This lack of uncertainty quantification limit the robustness of boundary delineation, especially in high-stakes anomaly detection tasks.

The probabilistic nature of GPs allows us to define decision boundaries that take into account both the mean and variance of the predictions. By setting thresholds on the mean and variance, we can delineate boundaries that are not only accurate but also resilient to variations and perturbations in the data. KDE can estimate density contours, but without an inherent measure of uncertainty, it may not delineate boundaries as effectively in terms of robustness.

Comment 4:Section 5: There should be an ablation experiment to show the benefits with boundary estimation vs without.

Response: We agree that ablation study is necessary to demonstrate the benefits of boundary estimation. We conduct ablation experiment that compares our method with and without the GBD step. The results of are presented in global rebuttal and Appendix A.4.5 of the final version which show that rules-based approach GBD improves the accuracy by up to 0.12.

Comment 5:Lines 267-268: "only normal instances from these datasets are utilized." -This contradicts the claim that the algorithm is unsupervised as this statement implies that labeled normal instances are available.

Response: We apologize for the confusion. Our approach is indeed unsupervised; however, for the calibration of hyperparameters, we utilized a small portion of one-class data presumed to be normal based on domain knowledge, not labeled instances. Instead, domain knowledge is leveraged to identify normal data for hyperparameter calibration. This ensures that the unsupervised nature of our anomaly detection approach is maintained.

While using purely normal data is the ideal state, in real-world scenarios, some attack data may inevitably be present. We conducted experiments using CIC-IDS and TON-IoT datasets to assess the efficacy of our method under varying percentages of “noisy” data. Our results demonstrate that the model maintain high levels of fidelity and robustness even as noise levels increase.

Comment 7:Line 312: 'data drift' is a different concept -- it means that over time the inherent data characteristics change permanently. What probably is being implied here is sample variance.

Response: Thank you for bringing this blunder to our attention. We acknowledge the misuse of the term 'data drift'. What we intended to convey is the model's ability to handle sample variance and minor perturbations. We will correct this terminology to 'Data Variability' in the final version and provide a more accurate description of our method's resilience to sample variance.

Our model's ability to handle sample variance is achieved through the dynamic nature of the Gaussian Processes used in the boundary delineation step. GPs provide a probabilistic framework that can adapt to changes in the data distribution by continuously updating the mean and variance estimates based on new data points. Ablation experiments demonstrated the effectiveness of the method in dealing with sample variance.

We sincerely thank you for your insightful feedback.

Question 1: While the method shows robustness across several datasets, it might not perform equally well in all types of data or anomaly detection scenarios.

Response: We acknowledge the concern regarding the generalizability of our method to vastly different types of data and anomaly detection scenarios. To address this, we have conducted additional experiments on datasets from domains not covered in our initial evaluation. The results, as shown in the attachment of global response Table 2, indicate that our method maintains high levels of robustness and interpretability across datasets in various fields.

However, we also recognize the inherent limitations in any single method's universal applicability. Therefore, we propose the following solutions to further enhance the generalizability of our approach and will implement in the future:

• Adaptive Thresholding. Implementing adaptive thresholding mechanisms that dynamically adjust based on the characteristics of the input data can improve performance across diverse scenarios.
• Domain-Specific Fine-Tuning. Developing domain-specific fine-tuning protocols to adjust the SCD-Tree and GBD algorithms based on the unique properties of different datasets.

Question 2: The effectiveness of the interpretative rules depends heavily on the quality of the initial black-box model.

Response: You hit the nail on the head, and that's what we're trying to focus on as we do our work! We appreciate your insight regarding the dependency of our interpretative rules on the initial black-box model's quality. Indeed, the primary motivation behind our research is to enhance the interpretability of black-box models in anomaly detection. Our goal is to interpret the black-box model into understandable rules by using a rule-based approach, which is essential for high-risk domains. The aim is not only to explain the decision-making processes of these models but also to improve their transparency and trustworthiness.

However, we would like to highlight several key aspects of our methodology that ensure its effectiveness, even when the black-box model is relatively small or less complex:

• We have unified the output standards of the black-box models by using metrics such as Mean Squared Error (MSE) and threshold values to represent data normality. These standardized metrics serve as inputs for the SCD-Tree, ensuring consistent and reliable decision-making regardless of the black-box model’s complexity.
• The SCD-Tree effectively segments the data into smaller, more manageable clusters. This segmentation reduces the dependency on the black-box model's complexity by isolating simpler patterns within each cluster.
• We conducted extensive experiments using various black-box models (e.g., AE, VAE, Isolated Forests, One-Class SVM) of different sizes and complexities. The results in section 6.3 demonstrate that our method maintains high interpretability and accuracy on the currently dominant neural network models in anomaly detection field, even with smaller models. For instance, our experiments with a simple autoencoder model yielded interpretative rules with high fidelity and robustness, as evidenced by the performance metrics provided in Table 3 and Table 4(1) .

Question 3: What are the limitations of the SCD-Tree and GBD algorithms when dealing with extremely large datasets or real-time data streams?

Response: The SCD-Tree and GBD algorithms, while effective, have certain limitations when handling extremely large datasets or real-time data streams. Here are the key challenges :

1. The time complexity of Gaussian Processes (GP) is relatively high ( $N^3$ ), which can lead to longer processing times when dealing with high-dimensional data. However, we have used the Segmentation Clustering Decision Tree (SCD-Tree) for spatial partitioning, so this issue only arises when there is a large amount of data within a particular subspace. By partitioning the data, we localize the high complexity to smaller regions, thus making the overall process more manageable.
2. Our hierarchical model involves several steps: running the black-box model, applying the SCD-Tree, performing Gaussian Process delineation, and boundary acquisition, which make the process lengthy. However, once the data has been partitioned by SCD-Tree , each subspace can be processed in parallel, which reduces the training time.

However, the aforementioned challenges primarily occur during the training phase. In real-world scenarios, once the model has been trained and the rules have been inferred, anomaly detection only requires simply examines whether the sample meets these rules. The rules-based interpretation remains efficient and scalable for real-time applications and is well compatible with high-performance applications (like P4 [1], reaching up to 100Gbps throughput by integrating rule-based models [2]), which allows for the implementation of data plane processing with high efficiency and low latency.

And our model is designed to handle new data continuously and achieve incremental rule updates. Therefore, even when encountering new types of data, there is no need to re-train the entire model. Instead, we can incrementally update the rules to adapt to new data, ensuring the model remains accurate and up-to-date without extensive re-training. This is certainly efficient in real life use.

Thank you for your constructive feedback, which has been invaluable in guiding these enhancements.

References: [1] P. Bosshart, D. Daly, G. Gibb, M. Izzard, N. McKeown, J. Rexford, C. Schlesinger, D. Talayco, A. Vahdat, G. Varghese, and D. Walker, “P4: programming protocol-independent packet processors,” ACM SIGCOMM Computer Communication Review, 44(3):87–95, 2014

[2] R. Li, Q. Li, Y. Zhang, D. Zhao, X. Xiao, and Y. Jiang, “Genos: General in-network unsupervised intrusion detection by rule extraction,”