LEMMA-RCA: A Large Multi-modal Multi-domain Dataset for Root Cause Analysis

Dear reviewer BP9D:

Thank you for your invaluable feedback. We would like to address your primary concerns and provide corresponding responses below.

Response to: What do "IT" and "OT" refer to? Are "Prometheus" and "ElasticSearch" tools or companies? Figure 2(a) would benefit from a more detailed explanation in its caption. In Figure 3, what is the KPI being referenced? How were the root causes for each system fault labeled? The authors may detail their labeling methodology.

A: Thank you for your detailed suggestions regarding terminology clarification, system KPI definitions, figure descriptions, and the labeling methodology. We appreciate the opportunity to improve the accessibility and clarity of our work. Below, we outline the updates made to address your feedback:

1. Terminologies and System KPI Clarifications:

• IT and OT:
  • IT (Information Technology) refers to technology systems used for managing data, computing, and communications in an organizational context.
  • OT (Operational Technology) refers to systems that control physical devices, processes, and infrastructure in industries such as manufacturing, utilities, and transportation.
  • We have clarified these terms in the abstract and the main text where they first appear.
• Prometheus and ElasticSearch:
  • Prometheus [1] is an open-source, metrics-based monitoring service commonly used in cloud platforms to track system performance.
  • ElasticSearch [2] is an open-source big data processing tool designed for text search and analysis. Its efficiency in managing log data has led to its widespread use in system monitoring [3].
  • These explanations have been added to the revised draft.
• System KPI and Figures:
  • The system KPI referenced in Figure 3 represents system latency (response time), measured in seconds.
  • The y-axis in Figure 3 has been clarified to explicitly indicate it is the system KPI (latency).

2. Root Cause Labeling Methodology:

To address your question on labeling, we have added detailed descriptions of the labeling process and validation to the revised manuscript (Appendix H).

• Root Cause Labeling Process:
  • For each system fault, controlled fault scenarios were designed to mimic realistic fault patterns, such as external storage failure or database overload.
  • During these fault scenarios, system behaviors—including metrics and logs—were monitored to identify the exact root cause of the fault.
  • The ground truth root cause was labeled based on the specific fault induced, ensuring high accuracy through direct observation of the fault’s impact.
• Label Validation:
  • To validate the labels, system behaviors during and after the fault were analyzed. Observed anomalies in metrics and logs were cross-checked against the expected outcomes of the fault scenario.
  • Multiple domain experts reviewed the labeled faults to confirm their correctness and consistency.

3. Figure Improvements:

• Figure 2(a): The caption has been expanded to include a more detailed explanation of the figure, including the context and relevance of the data presented.
• Figure 3: The y-axis has been clarified as representing system latency (seconds), and the KPI reference has been explicitly tied to this metric.

We hope these updates address your concerns. Please let us know if there are additional clarifications or enhancements you would like to see.

[1] https://www.prometheusbook.com/

[2] https://www.elastic.co/elasticsearch

[3] Vlad-Andrei Zamfir, Mihai Carabas, Costin Carabas, and Nicolae Tapus. Systems monitoring and big data analysis using the elasticsearch system. In 2019 22nd International Conference on Control Systems and Computer Science (CSCS), pp. 188–193. IEEE, 2019.

Response to: While the authors view missing data as a limitation, I consider it a realistic aspect of real-world data, which poses a meaningful challenge rather than a flaw.

A: Thank you for your insightful comment regarding missing data. We agree that missing data is a realistic aspect of real-world systems and can present meaningful challenges for root cause analysis, rather than being viewed as a flaw.

In light of your feedback, we have updated the manuscript to reflect this perspective. Specifically:

• We have removed the mention of missing data as a limitation in the updated version.
• We have reframed missing data as an inherent characteristic of real-world datasets, highlighting its potential to challenge and improve the robustness of RCA methods.

We appreciate your perspective and believe it adds value to the interpretation of our dataset's characteristics. Please let us know if there are further improvements you would like to see.

Thank you to the authors for their response and for addressing my questions. I appreciate the efforts made. Here are my revised comments:

1. Issue of Overstatement: The statement “We evaluate the quality of LEMMA-RCA by testing the performance of eight baselines” seems to overstate the implications of the baseline tests. While I acknowledge the dataset’s quality through expert inspection and validation, I don’t think the baseline comparisons alone can determine the relative quality of datasets. I recommend revising similar statements.
2. Baseline Implementation Details: I suggest including the key parameters used for the baselines in the appendix, as these are crucial for replicating the results and understanding the methods. It would be informative to know if parameter tuning was performed for each baseline. Additionally, incorporating a discussion on the sensitivity to hyper-parameter tuning in a dedicated column could enhance the clarity and completeness of the comparison.

Dear reviewer UCbX:

Thank you for your invaluable feedback. We would like to address your primary concerns and provide corresponding responses below.

Response to: Should provide more valuable data rather than simply assembling data. The quality of the data is not clarified and the data collection period is short, difficult to see if the dataset captures a wide range of fault patterns...

A: Thank you for your feedback. We would like to clarify the following points regarding the representativeness and quality of our datasets, as well as the duration of the data collection period.

1. Data Collection Period and Representativeness:

• For the Product Review dataset, each system fault is collected over a period of approximately 49 hours, resulting in a total collection period of 8 days for the four cases. We have clarified this in the manuscript to avoid confusion.
• In real-world scenarios, this 49-hour period is sufficient to comprehensively capture both normal and malfunction patterns for each system fault.
• Compared to existing publicly available datasets, such as NeZha and PetShop, our datasets are significantly larger in scale and temporal coverage. For instance:
  • The train ticket dataset in NeZha contains only 920 timestamps for 28 system faults, whereas our dataset includes over 130,000 timestamps for each system fault. This richer dataset allows for a more detailed representation of system behaviors.

2. Not Simply Assembling Data:

• Our approach goes beyond merely assembling data. To ensure the quality and representativeness of the collected data:
  • We collect data for one system fault at a time, isolating it from potential interference with other faults.
  • After collecting data for a specific fault, we rebuild the microservice system to capture a fresh and independent dataset for the next fault.
• During the data collection phase, we mimic the patterns of real-world system faults (e.g., external storage failure by filling up the external storage disk connected to the Database (DB) pod). Importantly, we do not directly simulate system metrics or logs, differentiating our data from simulated datasets like the train ticket dataset in NeZha, where malfunctionities are injected artificially.

3. Significance of Our Dataset:

• The data collection and fault injection methodology ensure that the dataset is both realistic and comprehensive. This distinguishes it from other datasets with limited coverage or artificially simulated faults.
• By capturing a large number of timestamps and diverse system fault patterns, our datasets enable more thorough evaluations of root cause analysis methods in real-world-like environments.

We hope this clarifies the quality, representativeness, and comprehensiveness of our datasets. If further clarification or additional analysis is required, we are happy to provide more details.

Response to: The experiments designed by the authors do not seem sufficient to demonstrate the value of the dataset. More results and analysis of widely recognized RCA methods are needed.

A: Thank you for your valuable suggestions.

We have included a variety of recent and widely recognized RCA methods in our experiments to evaluate the value of the dataset. These include REASON, RCD, ϵ-Diagnosis, CIRCA, as well as BARO and PCMCI (as suggested by Reviewer E7LX).

If there are specific RCA methods you believe should be included in the evaluation to further demonstrate the dataset’s value, we would greatly appreciate your recommendations. We are committed to making our experiments as comprehensive as possible and welcome your input to enhance our analysis.

Response to: Pay attention to the readability of the figures and the normalization of the experimental results.

A: Thank you for raising the issues regarding figure readability and consistency in reporting experimental results. We have made the following updates to address your concerns:

1. Improved Figure Readability:

• We have updated the figures to a vectorized format for better clarity and resolution.
• Figure 2(a)(b) and Figure 4(a)(b) have been enhanced to improve their legibility.
• To further ensure readability, we have included these figures in a single-column layout in Appendix G, making them easier to interpret.

2. Consistency in Reporting Experimental Results:

• We have reviewed and updated the experimental tables (Table 3, 4, 5, and 6) to ensure consistency in the number of decimal places across all reported results. All results are now presented to three decimal places for uniformity.
• Regarding standard deviation, it is generally not reported in root cause analysis tasks as the results focus on deterministic evaluations of methods rather than stochastic variability.

We appreciate your constructive feedback and hope these updates address your concerns effectively. Please let us know if there are additional areas where further improvements can be made.

Response to: Your analysis indicates that your task data is time-series data; however, algorithms such as PC, Notears, and CORAL are not designed for time-series data. You should consider incorporating more algorithms specifically designed for time-series data.

A: Thank you for raising this concern. We would like to clarify that most of the baseline methods we employ, including CORAL, Dynotears, REASON, MULAN, $\epsilon$-Diagnosis, Nezha, CIRCA, RCD, Baro, and PCMCI, are specifically designed for analyzing time-series data, which ensures that the majority of our comparative analysis is meaningful and relevant to the task. Regarding PC and Notears, we acknowledge that these methods were not originally designed for time-series data. However, they have been widely utilized in recent works to detect root causes in time-series datasets similar to ours, as evidenced by [1], [2], [3], and [4]. We selected these methods as baselines to maintain consistency with the current literature and provide a point of comparison, as they have demonstrated competitive performance in similar applications. We appreciate your suggestion and believe that this combination of existing literature and time-series-specific baselines strikes a balance between tradition and task-specific relevance. Please let us know if there are specific time-series algorithms you would recommend for inclusion in future work.

[1] Wang, Lu, Chaoyun Zhang, Ruomeng Ding, Yong Xu, Qihang Chen, Wentao Zou, Qingjun Chen et al. "Root cause analysis for microservice systems via hierarchical reinforcement learning from human feedback." In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 5116-5125. 2023.

[2] Ikram, Azam, Sarthak Chakraborty, Subrata Mitra, Shiv Saini, Saurabh Bagchi, and Murat Kocaoglu. "Root cause analysis of failures in microservices through causal discovery." Advances in Neural Information Processing Systems 35 (2022): 31158-31170.

[3] Zan, Lei. "Causal Discovery from Heterogenous Multivariate Time Series." In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pp. 5499-5502. 2024.

[4] Yuan Meng, Shenglin Zhang,, Yongqian Sun, Ruru Zhang, Zhilong Hu, Yiyin Zhang, Chenyang Jia, Zhaogang Wang, Dan Pei, “Localizing Failure Root Causes in a Microservice through Causality Inference“. IWQoS 2020.

We hope this addresses your concerns and welcome any additional feedback. Thank you!

Dear reviewer yFv3：

Thank you for your invaluable feedback. We would like to address your primary concerns and provide corresponding responses below.

**Response to: It is surprising that many faults happened in 8 days. It seems that these faults are simulated to mimic the real world scenario. **

A: Thank you for your thoughtful feedback. We appreciate the opportunity to clarify the data collection process and the nature of the system faults in our dataset.

1. Duration of Data Collection:

Each system fault in the Product Review dataset is collected over a period of approximately 49 hours. The total of 8 days refers to the cumulative time for all four cases. We have updated the manuscript to clarify this point and avoid any confusion.

2. Isolation of Faults During Collection:

To ensure the accurate capture of distinct malfunction patterns and to prevent potential interference between different system faults, we collected data for one fault at a time. After collecting data for a specific fault, we rebuilt the microservice system before inducing and collecting data for the next fault.

3. Nature of Faults and Simulation:

While we mimic the patterns of real system faults during data collection, we emphasize that this is not the same as generating simulated data. For example:

• In the case of an external storage failure, we mimicked the failure pattern by filling up the external storage disk connected to the Database (DB) pod.
• Importantly, we did not directly simulate system metrics or logs, but instead allowed the microservice system to exhibit natural behaviors resulting from the induced conditions.

4. Sufficiency of the Data Collection Period:

We argue that the two-day data collection window is sufficient to capture comprehensive system behaviors for the following reasons:

• Focus on Malfunction Patterns: Collecting additional data over longer periods, especially for normal patterns, would be unnecessary, as our primary focus is on malfunction patterns caused by faults.
• Duration of Malfunction Patterns: Malfunction patterns lasting several hours to a day are typically sufficient to reveal the malfunction behaviors associated with a system fault. This is consistent with real-world scenarios where identifying faults promptly is critical to avoid significant financial losses (e.g., millions of dollars in e-commerce platforms). We hope this explanation addresses your concerns regarding the data collection process and the nature of the faults. If further clarification is needed, we would be happy to provide additional details.

We hope this explanation addresses your concerns regarding the data collection process and the nature of the faults. If further clarification is needed, we would be happy to provide additional details.

Response to: SWaT and WADI are from existing work. SWat and WADI are already evaluated for RCA in the REASON (Wang et al. KDD2023) paper. Since this is a dataset paper, including existing datasets into the proposed one should not be seen as the contribution.

A: Thank you for your feedback. We would like to clarify our contribution regarding the SWaT and WADI datasets and their relevance to our work.

1. Transforming SWaT and WADI for Root Cause Analysis:

While SWaT and WADI were originally used for anomaly detection tasks, we are the first to transform these datasets into a root cause analysis (RCA) task. This transformation includes preprocessing and adapting the data to create meaningful continuous labels or construct KPIs necessary for RCA. These continuous labels enable more granular evaluations of causality and root cause patterns compared to the discrete labels used in their original anomaly detection context..

2. Sharing the Transformed Data:

The transformed SWaT and WADI datasets have been shared with researchers via private communications and used in subsequent works, including studies such as REASON (Wang et al., KDD 2023). While REASON evaluates these datasets for RCA, it is based on our preprocessing and transformation of the data, making it possible to use SWaT and WADI in this context.

3. Preprocessing Code and Accessibility:

To ensure transparency and reproducibility, we have provided the source code for preprocessing SWaT and WADI datasets. This allows other researchers to validate our approach and further explore these datasets for root cause analysis.

4. Contribution as a Dataset Paper:

While SWaT and WADI are existing datasets, our contribution lies in adapting them for a new task (RCA), enhancing their utility for the research community. This transformation aligns with the goals of a dataset paper, as it extends the applicability of well-known datasets into a new domain.

We hope this explanation clarifies the necessity and significance of our preprocessing approach and our contribution to enabling RCA research using SWaT and WADI. If further details are required, we are happy to provide additional explanations.

Response to: For Product Review datasets, Reason achieves 100% on PR@5 and 95% an MAP@5 with solely metric data. I think this is definitely not far from optimal.

We appreciate the reviewer’s comment and would like to clarify our definition of "optimal performance." We refer to optimal performance as achieving MAP@1 = 1.0 or PR@1 = 1.0. As we mentioned in our earlier response, in real-world applications—particularly in critical domains like e-commerce—investigating the top 5 potential root causes for a system fault can be time-consuming and costly. Such delays can result in significant financial losses, making it important to set a high standard for performance evaluation.

We acknowledge that REASON achieves good performance on the Product Review sub-dataset (e.g., PR@5 = 100% and MAP@5 = 95% using only metric data). However, this is based on one sub-dataset within the IT domain, and REASON’s performance on the more challenging Cloud Computing sub-dataset is notably lower (e.g., PR@1 = 0.167). This demonstrates that the IT operations datasets in LEMMA-RCA are challenging overall, even if one baseline performs well on a specific sub-dataset.

Response to: The difference on injected system faults

We appreciate the reviewer’s follow-up question and acknowledge that describing the Train Ticket faults as "entirely simulated" may have been unclear. Both datasets involve fault injection in controlled environments. However, the faults in our datasets are designed to reflect real-world scenarios observed in production environments, with differences in design, scale, and data richness:

1. Fault Realism:

Our datasets include realistic scenarios such as Silent Pod Degradation, DDoS attacks, and cryptojacking, which emulate operational challenges seen in real IT systems. These faults involve nuanced behaviors, such as subtle latency increases or cascading failures, informed by real deployment experiences.

2. Scale and Granularity:

• The Product Review Platform includes 216 pods and six nodes, collecting metrics at 1-second intervals (~130,000 timestamps per fault).

• The Cloud Computing Platform captures six fault types, with data sourced from AWS CloudWatch Metrics and Logs, offering detailed insights across layers (e.g., API debug logs, MySQL logs).

• In contrast, the Train Ticket dataset involves only 27 nodes, with coarser granularity (1-minute intervals) and ~60 timestamps per fault.

3. Data Diversity and Monitoring:

We collect rich metrics and logs using tools like Prometheus, Elasticsearch, and CloudWatch, enabling comprehensive fault analysis. Faults were monitored for extended periods (e.g., 49 hours per fault in the Product Review Platform), providing high temporal resolution and diverse data types, such as system metrics, API logs, and database logs.

Clarification:

We recognize that the Train Ticket dataset also uses a deployed environment for fault injection. Our intent was to highlight differences in scale, granularity, and fault complexity, which make our datasets particularly challenging for root cause analysis.

Response to: Please write specifically the process of data collection to avoid miss leading

We thank the reviewer for their valuable feedback. In response, we have expanded Appendix B to provide a detailed description of the processes used to induce system faults and mimic real-world patterns during data collection. This includes specific methodologies, data collection tools, and analysis techniques for each fault scenario. These enhancements aim to clarify our approach and ensure that future researchers can replicate and evaluate the quality of our data.

Response to: But the contribution of transforming an existing datasets into a new task is much less than collecting a new dataset. The authors emphasis a lot of multi-domain datasets and OT domain of paper.

We appreciate the reviewer’s feedback regarding the emphasis on multi-domain datasets and the preprocessing efforts for transforming existing datasets. While we agree that collecting a new dataset is a significant contribution, we would like to clarify that the primary contribution of our work lies in the collection of novel datasets from the IT domain, specifically the Product Review and Cloud Computing datasets.

Additionally, we acknowledge that transforming existing datasets like SWaT and WADI into a root cause analysis (RCA) task represents a smaller effort compared to collecting entirely new datasets. However, this transformation is an essential step for addressing the lack of RCA-specific datasets in the operational technology (OT) domain, ensuring broader applicability and relevance.

To better align with the reviewer’s concerns, we have revised the relevant sentences in the paper: e.g.,

Original: "LEMMA-RCA is multi-domain, encompassing real-world applications such as IT operations and water treatment systems, with hundreds of system entities involved."

Revised: "LEMMA-RCA encompasses real-world applications such as IT operations and water treatment systems, with hundreds of system entities involved."

This revision removes the term "multi-domain" to focus more explicitly on the practical applications and the contribution of collecting new datasets from the IT domain while maintaining an accurate representation of our work.

Dear reviewer E7LX：

Thank you for your invaluable feedback. We would like to address your primary concerns and provide corresponding responses below.

Response to: Could the authors consider including the dependency graph? Could the authors benchmark the baselines using the dependency graph instead of the causal graph inferred by the PC? ...

A: We appreciate the reviewer’s insightful suggestion regarding the inclusion of a dependency graph and the comparison with causal graphs inferred by baseline methods.

• Addressing the Dependency Graph in Real-World Systems
  In real-world systems, obtaining a complete ground-truth dependency graph for root cause analysis is challenging due to the complexity and scale of these systems. However, to address this concern, we have included a semi-complete dependency graph in the dataset repository (i.e., in the readme.ppt file) for each system fault, based on domain knowledge.

• Experimental Comparison: Dependency Graph vs. Causal Graph
  To evaluate the difference between the semi-complete dependency graph and the causal graph learned by baseline methods, we conducted experiments on the Product Review sub-dataset (system metrics data only). Following the methodology outlined in [1], we assessed the performance using four commonly used metrics: True Positive Rate (TPR), False Discovery Rate (FDR), Structural Hamming Distance (SHD), and Area Under the ROC Curve (AUROC). | Method | TPR $\uparrow$ | FDR $\downarrow$ | SHD$\downarrow$ | AUROC $\uparrow$ | |--------|------|------|-------|--------| | Dynotear | 0.214 | 0.743 | 0.786 | 0.612 | | PC | 0.112 | 0.892 | 0.861 | 0.563 | | C-LSTM | 0.428 | 0.427 | 0.543 | 0.733 | | GOLEM | 0.126 | 0.847 | 0.823 | 0.571 | | RCD | 0.152 | 0.869 | 0.838 | 0.584 | | ϵ-Diagnosis | 0.084 | 0.905 | 0.874 | 0.554 | | CIRCA | 0.327 | 0.544 | 0.582 | 0.685 | | REASON | 0.634 | 0.217 | 0.347 | 0.846 |

• Evaluation and Results

  • For each system fault, we computed the metrics individually and then averaged the results across four cases.
  • It is important to note that system entities not included in the semi-complete dependency graph were excluded from this comparison to ensure consistency and fairness across methods.
  • To ensure comparability for SHD, which is influenced by the number of nodes in the graph, we normalized SHD by dividing it by the square of the number of nodes for each system fault. Finally, we averaged the normalized SHD across the four system faults on the Product Review sub-dataset.

These results are summarized in the table above, providing a comprehensive comparison between the dependency and causal graphs.

[1] Pamfil, Roxana, Nisara Sriwattanaworachai, Shaan Desai, Philip Pilgerstorfer, Konstantinos Georgatzis, Paul Beaumont, and Bryon Aragam. "Dynotears: Structure learning from time-series data." In International Conference on Artificial Intelligence and Statistics, pp. 1595-1605. Pmlr, 2020

Response to: Insufficient Explanation of Baseline Approaches. The paper does not include explanations of the baseline approaches used, even in the appendix.

A: We appreciate the reviewer’s feedback regarding the explanation of baseline approaches. We would like to clarify that our initial submission includes a description of the baseline methods in Section 4.1 (Lines 373–386). Additionally, in the updated version of the manuscript, we have expanded upon the explanations of the baseline approaches in Appendix F to provide further clarity.

Response to: Limited Explanation of Experimental Results

A: Thank you for your feedback regarding the explanation of experimental results. In the updated version of the manuscript, we have provided a more detailed explanation of the experimental results in Section 4.2, with the new additions highlighted in red for ease of reference. If there are specific aspects of the results that require further clarification, we would be happy to elaborate further.

Thank you for your constructive feedback and for actively engaging with our work during the public discussion phase. We appreciate your thoughtful comments and suggestions.

1. Partial Dependency Graphs:
   Following your suggestions, we have uploaded the semi-complete dependency graphs in CSV format, which you can access here. We will also make the code for the causal discovery experiments available to support reproducibility and further research.

2. Selection of Baselines:
   Regarding the question about the selection of baselines, we have addressed this in our response to Reviewer UCbX. To summarize briefly, most of the methods we employ—such as CORAL, REASON, MULAN, ϵ-Diagnosis, Nezha, CIRCA, RCD, and Baro—are state-of-the-art algorithms specifically designed for root cause analysis in time-series data. This ensures that the comparative analysis is both meaningful and relevant. For PC and Notears, while they were not originally designed for time-series data, they have been widely adopted in recent literature for detecting root causes in similar datasets, as shown in:

• Wang, Lu, Chaoyun Zhang, Ruomeng Ding, Yong Xu, Qihang Chen, Wentao Zou, Qingjun Chen et al. "Root cause analysis for microservice systems via hierarchical reinforcement learning from human feedback." In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 5116-5125. 2023.
• Ikram, Azam, Sarthak Chakraborty, Subrata Mitra, Shiv Saini, Saurabh Bagchi, and Murat Kocaoglu. "Root cause analysis of failures in microservices through causal discovery." Advances in Neural Information Processing Systems 35 (2022): 31158-31170.
• Zan, Lei. "Causal Discovery from Heterogenous Multivariate Time Series." In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pp. 5499-5502. 2024.
• Yuan Meng, Shenglin Zhang,, Yongqian Sun, Ruru Zhang, Zhilong Hu, Yiyin Zhang, Chenyang Jia, Zhaogang Wang, Dan Pei, “Localizing Failure Root Causes in a Microservice through Causality Inference“. IWQoS 2020.

These references validate their utility and relevance for time-series root cause analysis tasks.

3. More Thorough Presentation of Results:
   We encourage the reviewer to refer to Section 4.2 of the updated manuscript, where the red-highlighted text provides a detailed explanation of the experimental results, performance distinctions, and underlying reasoning. Quantitative results are presented in Tables 3 and 4, and we have included additional details in Appendix K for further clarity. If there are specific aspects of the results that require further elaboration, we would be happy to address them.

Thank you once again for your valuable feedback and for recognizing the contributions of our work. We remain committed to addressing your concerns and strengthening the impact of this research.