Flow-of-Action: SOP Enhanced LLM-Based Multi-Agent System for Root Cause Analysis

While the research direction is promising, there are several weakness. First, in Section 2.3.1, the authors described “To enhance the precision of RCA, we have devised hierarchical SOPs. Our objective is for the RCA process to progress from a macro to micro level, from a general to specific perspective, mirroring real-world scenarios more closely. For instance, we first address network issues before delving into network partition problems"

How does the SOP knowledge change when a new SOP is introduced? Does the knowledge base update every time?

Second, in Section 2.4, how do the authors confirm that the action set is comprehensive?

Third, failures and fault localization in real-world micro service-based architecture is well-studied problem in software engineering research [1-3]. The experiment design in Section 3.1.1, specifically artificially fault injection in microservice-based architecture is not well justified.

Fourth, how the authors derived the rule-set of MainAgent on page 15 is not clear to me. As this ruleset is crucial in proposed model's performance evaluation, a systemic and well-defined approach is expected. An example of deriving a ruleset based on historical software artifacts can be found in [4].

References.

1. Meng, Y., Zhang, S., Sun, Y., Zhang, R., Hu, Z., Zhang, Y., Jia, C., Wang, Z. and Pei, D., 2020, June. Localizing failure root causes in a microservice through causality inference. In 2020 IEEE/ACM 28th International Symposium on Quality of Service (IWQoS) (pp. 1-10). IEEE.
2. Zhang, S., Xia, S., Fan, W., Shi, B., Xiong, X., Zhong, Z., Ma, M., Sun, Y. and Pei, D., 2024. Failure Diagnosis in Microservice Systems: A Comprehensive Survey and Analysis. arXiv preprint arXiv:2407.01710.
3. Jagadeesan, L.J. and Mendiratta, V.B., 2020, October. When failure is (not) an option: Reliability models for microservices architectures. In 2020 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW) (pp. 19-24). IEEE.
4. Rahman, A., Parnin, C. and Williams, L., 2019, May. The seven sins: Security smells in infrastructure as code scripts. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) (pp. 164-175). IEEE.

• The comparison with other existing methods for RCA is largely missing. The paper merely focuses on improving upong the ReAct framework but didn’t discuss why ReAct is preferred in the first place among all other RCD methods. It only talks about a single RCD method that uses deep learning.
• This paper extends a framework ReAct from a paper that has not been published. It becomes even more difficult to understand the novelty of this extension. Another cited paper is recently published, but there is no discussion on how this paper is compared to the cited one.
• Some parts of the methodology are not described in detail. The paper often discusses many processes only at a high level, which makes it difficult to judge the correctness of the approach-related various claims that often appear subtly in the paragraph.
• The paper advocates that the proposed method is efficient, but it is limited to the context of LLM-based RCA methods. In terms of efficiency, a comparison with other non-LLM-based methods is missing.

Although I appreciate the multi-agent and SOP based solution framework and performance gain in simulated environment, I have a few concerns:

1. The central contribution of the work lies in creating structured SOPs. This is not a challenging task when operating on a simulated environment where faults are artificially injected, and the nature of the faults are known. However, in real production environments, where thousands of inter-dependent microservices are continuously running, identifying the nature of the fault and localization itself is a challenging task. I am not fully sure how these structured SOPs can be prepared in a real-world setting. Is there a way to automate the SOP creation process given the nature and dependency of the services? If SOPs need to be prepared by the service owners, then it is a strong assumption and may not be a practical solution.
2. While multi-agent framework is proposed as a key contribution, several details regarding this is missing. What multi-agent framework is used in the experiment and how these agents are communicating among each other needs to be clarified in detail. Does the execution of agents follow a sequential static flow, or they run asynchronously and somehow manage their communication flow. Please clarify.
3. Experiments are conducted in a simulation environment where faults are injected synthetically. Therefore, the actual performance of the proposed framework in a real-world setting is still questionable as the real situation is much unorganized and chaotic.
4. Lastly, several details on experiments are missing in the paper which raises concerns about reproducibility. Are you planning to opensource the code and simulated data?