OCEAN: Online Multi-modal Root Cause Analysis for Microservice Systems

Thank you for your invaluable feedback. We would like to address your primary concerns and provide a response below.

• The notations in the problem statement are not sufficiently clear, e.g., n-1 and d_M in line 158. Additionally, do all system entities share the same "entity metrics" as features? Is n-1 referring to the number of system "entities" or "entity metrics"?

A: Yes, all system entities share the same "entity metrics" as features and $n-1$ refers to the number of system entities. The reason why we use n-1 to denote the number of system entities is that we also include one KPI in our causal structure learning such that the total number of nodes in the adjacency matrix is n rather than n+1.

• Is only one system KPI considered?Is the bold symbol $\bm{y}$ simply a one-dimensional vector? I had assumed multiple KPIs could be monitored.

A: Yes, for simplicity, our method only considers one KPI. $\bm{y}$ is simply a one-dimensional vector. When multiple KPIs are available, we select one KPI that is mostly related to system fault based on domain knowledge.

• I find it challenging to understand the replication of the KPI $d_M$ times to create a tensor version of $\hat{\bm{X}}$. If all entities share the same "metrics," that makes sense; could you clarify this?

A: In the root cause analysis, KPI serves as the “label information” to guide the RCA methods to identify the potential root causes. The reason why we replicate the KPI $d_M$ times is to match the number of system metrics features and then use the same KPI to measure causal relation among different system entities and KPI for different system metrics features or different log features. Notice that When multiple KPIs are available, we select one KPI that is mostly related to system fault based on domain knowledge.

• The 1-D dilated convolution described in Eq. (2-5) is unclear. Is f(t) a scalar or a vector? Could you elaborate on Eq. (2-3) in relation to the tensor input x? What is the rationale for using "two" 1D kernels and activation functions (tanh, sigmoid) in Eq. (3)?

A: f(t) is a scalar. It should be $f\in \mathbb{R}^{K}$ rather than f(t). We have corrected the typo in the updated version. The rationale for using two activation functions (tanh and sigmoid) is similar to the rationale of LSTM. The sigmoid function is used to control the extent to which the cell state should influence the output and the tanh function is applied to the cell state to constrain its range between -1 and 1, ensuring that the outputs remain stable and manageable. The sigmoid part decides "how much" information should be passed, while the tanh part provides a scaled and normalized version of the "what" information to pass.

• It seems that Eq. (8) or Eq. (13) represents the loss for only the i-th batch, yet the authors incorporate them into their final objective (18). What is the precise training procedure for the online framework?

A: In the online setting, every several minutes, a new batch data (e.g., i-th batch) is given. Once the online failure detection is triggered, starting from the i-th batch, we keep finetuning the model with equation 18. Our proposed method is not only fine tuned with the i-th batch but also the next several batches, which aims to ensure the correctness of the final results. We provide the stopping criteria in Section 3.5.

• If the goal is to maximize (14), then L_MI in (18) should have a negative sign.

A: We would like to point out that we add the negative sign in equation 15 rather than equation 18. We will take your suggestion and update both equation 15 and equation 18 to avoid the confusion.

• In Figure 2 (b,c,d), the setting of other hyper-parameters should be revealed.

A: The hyperparameters $\lambda_1$, $\lambda_2$ and $\lambda_3$ were selected from the set [0.01, 0.1, 0.5, 1, 10, 100, 300]. In Figure 2, the vertical red line in each subfigure indicates the specific parameter settings ($\lambda_1$, $\lambda_2$ and $\lambda_3$) used to achieve optimal performance on the AIOps dataset. Similarly, Figures 3 and 4 provide the parameter settings that yield the best results for the other two datasets. To enhance the reproducibility of our experimental results, we will further refine our parameter analysis and highlight the optimal parameter settings throughout the paper.

Thank you for your invaluable feedback. We would like to address your primary concerns and provide a response below.

Main concern 1: The necessity of introducing causal discovery (CD) into root-cause analysis. RCA is a differenct task from CD,as the former requires identifying a subset of variables while the latter requires the identification of orientations. When the prior knowledge is present, e.g., in some cases of microservices, the causal graph is already given. When the prior knowledge is not sufficient, the discovery of causal graph is lack of validation, and the resulting RCA becomes wield. My suggestion is that, based on the (Structural Causal Model) SCM model, designing some statistics from the data rather the first-discover-then-identify approach, as the latter paradigm is somehow incremental and risky.

A: We agree with your opinion that prior knowledge (e.g., physical dependency graph) provides important information to us. However, we would like to point out that the complete physical dependency graph might not be available in many real-world applications. We experiment on two semi-synthetic datasets (i.e., Train Ticket and Online Boutique datasets) and one real-world dataset, i.e., Product Review dataset. Both Train Ticket and Online Boutique datasets do not include the physical dependency graph. In the Product Review dataset, the physical dependency graph is incomplete and it only consists of around 30 nodes/system entities for each case. Nevertheless, the entire dataset consists of more than 200 nodes/system entities, which indicates that the physical dependency graph can only capture a small percentage of the nodes/system entities and this graph could not be directly used for root cause analysis. However, we would like to express our disagreement towards the statement that “When the prior knowledge is not sufficient, the discovery of causal graph is lack of validation, and the resulting RCA becomes wield”. Many existing RCA methods [1,2,3,4] have demonstrated the effectiveness of our proposed method when a physical dependency graph is unavailable.

[1] Dongjie Wang, Zhengzhang Chen, Yanjie Fu, Yanchi Liu, and Haifeng Chen. Incremental causal graph learning for online root cause analysis. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2269–2278, 2023a. [2] Lecheng Zheng, Zhengzhang Chen, Jingrui He, and Haifeng Chen. MULAN: multi-modal causal structure learning and root cause analysis for microservice systems. In Tat-Seng Chua, Chong-Wah Ngo, Ravi Kumar, Hady W. Lauw, and Roy Ka-Wei Lee (eds.), Proceedings of the ACM on Web Conference 2024, WWW 2024, Singapore, May 13-17, 2024, pp. 4107–4116. ACM, 2024 [3] Dongjie Wang, Zhengzhang Chen, Jingchao Ni, Liang Tong, Zheng Wang, Yanjie Fu, and Haifeng Chen. Interdependent causal networks for root cause localization. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2023, Long Beach, CA, USA, August 6-10, 2023, pp. 5051–5060. ACM, 2023c [4] Roxana Pamfil, Nisara Sriwattanaworachai, Shaan Desai, Philip Pilgerstorfer, Konstantinos Georgatzis, Paul Beaumont, and Bryon Aragam. DYNOTEARS: structure learning from time-series data. In Silvia Chiappa and Roberto Calandra (eds.), The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy], volume 108 of Proceedings of Machine Learning Research, pp. 1595–1605. PMLR, 2020

Main Concern 2: As this paper is not the first work to introduce multi-modal information into RCA, I think that this paper should focus on building theoretical understanding on why and how multi-modal information will be beneficial towards RCA. The contribution of this paper, including a causal-structure learning module and a temporal learning framework, seems very incremental such that I do not agree that this paper's novelty reaches the bar of ICLR.

A: We would like to point out that our work is NOT solely targeting multi-modal root cause analysis. Our method is an ONLINE multi-modal root cause method, and the existing methods are OFFLINE multi-modal root cause methods. Each component in our proposed method is designed for the online root cause analysis setting and ignoring this specific setting is unfair to the evaluation of our contribution.

By the way, existing RCA methods on online data also exist. So my critical question stills exists, i.e., what is the unique benefit (either theoretical or empirical results) when introducing multi-modal setting?

Thank you for your invaluable feedback. We would like to address your primary concerns and provide a response below.

Q1. On page 2, line 83, the author introduces a factor attention mechanism to analyze the relationship between different factors and re-evaluate their impact on online causal graph learning. As far as I know, many methods use attention mechanisms to model and analyze the relationship between different variables, as shown in references 1-2. What is the essential innovation of this paper compared with them? [1] Wu X, Ajorlou A, Wu Z, et al. Demystifying over-smoothing in attention-based graph neural networks[C]. Advances in NeurIPS, 2024. [2] Cai J, Zhang M, Yang H, et al. A novel graph-attention based multimodal fusion network for joint classification of hyperspectral image and LiDAR data[J]. Expert Systems with Applications, 2024, 249: 123587.

A: We would like to point out that the attention mechanism used in [1,2] and the attention mechanism in our method are quite different. (1). In [1,2], the attention mechanism aims to measure the importance of the neighbors around the anchor node, while our proposed method aims to capture the interrelationship between two modalities and meanwhile measure the importance of each factor. Basically, the attention map used in [1,2] measures the importance of one node to another node, while the attention map used in our method measures the inter-connection between two factors from two modalities.

Q2. Page 4, line 186 indicates that this paper uses the 2021 MSSA algorithm and claims it is the most advanced online fault detection method. Why is the most advanced online fault diagnosis method from 2021? We searched for several online fault diagnosis algorithms published in recent years as shown in references 1-2. [1] Zeiser A, Özcan B, van Stein B, et al. Evaluation of deep unsupervised anomaly detection methods with a data-centric approach for on-line inspection[J]. Computers in Industry, 2023, 146: 103852. [2] Wang X, Yao Z, Papaefthymiou M. A real-time electrical load forecasting and unsupervised anomaly detection framework[J]. Applied Energy, 2023, 330: 120279.

A: We appreciate your suggestions. We would like to point out that the proposed method in [2] is “a combination of WGAN and encoder CNN, adapted from f-AnoGAN (published in 2019)”, as stated in the abstract of [1]. In addition, [1] is designed to detect anomalies for image data rather than time-series. For [2], although [2] is designed to detect the anomalies for time-series data in the unsupervised setting, it requires the model trained on the time-series data first, then forecast the future values and finally detect the anomaly based on whether the change is within a reasonable range. According to [2], in the experiment, it usually requires the length of the data to be more than 1 year as shown in section 5.1.2 for data preprocessing. However, in our experiment, the largest dataset, (i.e., Product Review Dataset) only consist of 2 day time series data, not to mention two small datasets. This suggests that using [2] to detect anomalies might not be a good choice. Here, we would like to emphasize that the main reason why we select 2021 MSSA algorithm to detection the change point is that this paper has a good performance with solid theoretical performance guarantee.

Q3 & Q6. There are many errors in the organization logic, symbol unification and text expression of the article, so I suggest careful revision.

A: We have corrected all of these typos in our updated version and highlighted the change in red.

Q4. Among the 7 compared algorithms, is it fair to compare them with 4 algorithms that focus on learning causal graphs rather than fault diagnosis algorithms? We list some recent fault diagnosis algorithms as shown in the references 1-3.

A: We would like to point out that the problem setting in our paper is different from the setting in papers [1] and [3].

• Paper [1] assumes that the label information of system faults in the training set are available and system faults in both the training set and the test set are overlapped. With this assumption, [1] aims to learn a classifier trained on the labeled samples from the training set and then predicts the unlabeled samples in the test set. However, in our setting, we do not have such an assumption and no labeled samples/system faults are available to train such a classifier.
• Paper [3] is designed for anomaly detection on the object and it is not designed for root cause identification in a microservice system. These two are totally different settings.
• Paper [2] shares a similar experimental setting with our paper. However, we failed to find the source code either in the GitHub or the author’s homepage.

Thank you for your invaluable feedback. First, we would like to identify a few factuality misunderstood by Reviewer 2.

Q1: Please clearly define online / offline RCA in introduction. RCA is usually triggered by KPI anomalies. Therefore, it is not a function that needs to be conducted continuously, like anomaly detection. I believe many of the existing works have been deployed in the production system. Are they online RCA methods according to the authors' definition? It is not clear to me that the authors stated that most existing methods are designed for offline use. Methods can be trained offline and used online.

A: We would like to point out the main differences of definition between online RCA and offline RCA as follows:

• In the offline setting, only the historical data is available for model training, but the batch data is available and serves as very important indicators for the root cause analysis in the online setting.
• There is no good way to leverage the batch data for the offline methods. In the online setting, our proposed method will first be initialized by training the model with the historical data only and it is only fine-tuned for around 100 iterations for each batch data. In the offline setting, the model needs to be trained from scratch when new data is provided, which can be time-consuming. In addition, we also examine the performance of the model fine-tuned on newly available batch data. The experimental results show that fine-tuning these multi-modal RCA models yields even worse performance than training the model from scratch. This is the major reason why an online RCA method is necessary compared with the offline RCA methods.

Training from scratch (Two Modalities)

Fine-tuning the model (Two Modalities)

We highlight the difference between online setting and offline setting in section 3.1 in red in the updated version.

Q5: It seems that the authors converted log to metric data. The problem is only defined on metric data. What is the difference between the old metrics and new metrics converted from log? For me, at least the method is only for single modal data. It is not clear why existing single model methods cannot be applied in such a setting.

A: We would like to point out that you misunderstand that. We DO NOT convert log data to metric data. Instead, we only convert log into time-series data described in Line 158-160, which has the similar format as metric data. We also provide the details of log feature extraction in Appendix C. Therefore, the log time-series data is different from metric data.

Q9. In Table 2, the proposed method can be used for both metric only and log only settings. The authors are suggested to give both results.

A: We want to point out that you misunderstand our proposed method. First, log data are different from metric data and we do not convert log data to metric data. Second, our proposed representation learning with multi-factor attention aims to capture the interaction between metric and log data. Our proposed method CANNOT only experiment on a single data type.

Thank you for your invaluable feedback. We would like to address your primary concerns and provide a response below.

W1. Limited Innovation and Contribution: While the paper achieves good results, it lacks methodological innovation specifically for online root cause analysis.....

A: We would like to differentiate the difference between our proposed method and these papers.

• Difference between our method and [1] and [2]: Indeed, papers [1] and [2] can capture the long-term temporal dependencies. Our contribution in Section 3.2 is to capture both temporal dependency and the causal relationship by minimizing the forecasting errors in equation 8. This is different from [1] and [2]. Notice that [2] could not capture the causal relation between two system entities (e.g., system entity X -> system entity Y), which is our major goal in Section 3.2.

• Difference between our method and Mulan [3]: Though both MULAN and our method uses contrastive learning loss to maximize the similarity between two modalities, our method also aims to capture the interrelationship among two modalities in section 3.3. This section aims to measure how different factors within two modalities correlate to each other and how these factors contribute to causal structure learning, which is ignored by Mulan. In line 276-282, we explicitly introduce the way to capture the relationship among different factors from both modalities via equation 9. We contribute the superior performance of our method over Mulan to the design of representation learning with multi-factor attention in section 3.3.

• How do method 3 address modality reliability: We would like to point out that the third change (C3) Learning Multi-modal Causal Structures is addressed by the combination of section 3.2 and 3.3. In Section 3.2, we introduce the similarity matrix $**C**$ to measure how different factors between two modalities contribute to causal structure learning via equation 9. Specifically, the smaller value in the similarity matrix $**C**$ indicates that this factor contributes less to the causal structure learning and a large value indicates more contribution this factor made towards causal structure learning. Based on this design, we assess the reliability of each modality based on the importance of factors within each modality via the importance measurement in equation 16. Typically, a larger value of $s_M$ implies that the metric data is more important than the log data. To better assess the reliability of each model by measuring the weight for each modality, we conduct a case study below. In the table, we observe that our proposed method assigns larger weights to metric data across four cases, which is consistent with the observation in the experimental results in Table 2 that all baseline methods achieve better performance with only metric data than those with the log data.

W2. Motivation Not Clear: The primary novelty of this paper appears to focus on "online" multi-modal root cause analysis (RCA) for microservice systems.

A: Here is our special design for the online multi-modal RCA. In the preparation (offline) phase, we will first train the model with only the historical data by minimizing equation 18. Notice that in the preparation phase, the batch data is not available and thus we remove all loss terms involving the batch data. In the online phase, batch data is available and we only need to fine-tune the model for 100 iterations by minimizing equation 18.

W3. Minor Issues: The font in Equation (13) is somewhat inconsistent.

A: We have updated the font in equation 13.