SOP-Agent: Empower General Purpose AI Agent with Domain-Specific SOPs

• The primary limitation of SOP-Agent is that, while the paper critiques existing domain-specific agents for relying on hardcoded, human-designed workflows, SOP-Agent itself is subject to the same limitation. SOP-Agent still requires human expertise to design the decision graph, which can become large and challenging in complex domains.
• The explanation of SOP-Agent is insufficient. For example, the interaction between the decision graph and the LLM agent lacks clarity, including how SOP-Agent determines which node to execute, how it evaluates condition satisfaction after each action, what input is provided to the LLM agent at each step, and how the model handles the generation of actions not included in the decision graph.
• The fixed decision graph approach appears similar to behavior tree methods, such as MOSAIC [1]. To highlight the novelty of SOP-Agent, it would be helpful to explain how it differs from these existing methods.
• The use of a predefined decision graph limits adaptability, making it difficult for SOP-Agent to handle unexpected situations.
• There is a concern that the Data Cleaning experiment could not be appropriate for evaluating LLM agent performance.
• The performance gap between SOP-Agent and ReAct is minimal in the ALFWorld and HotpotQA experiments, and in the code generation task, SOP-Agent performs worse than some other algorithms. Justification for SOP-Agent’s advantages over these methods is needed.
• Although the motivation behind the proposed Grounded Customer Service Benchmark is sound, the benchmark’s difficulty appears relatively low. With a maximum depth of 4.52 and an average of 1.94 child nodes per node, as evidenced by SOP-Agent’s high performance (99.8%).
• (Minor) The citation format should be improved.
• (Minor) Using mathematical notation for performance metrics would enhance readability.

[1] Wang, Huaxiaoyue, et al. "MOSAIC: A Modular System for Assistive and Interactive Cooking." arXiv preprint arXiv:2402.18796 (2024).

1. The SOP-agent employs decision graphs to represent Standard Operating Procedures (SOPs). However, it appears that the creation of these decision graphs necessitates substantial input from domain experts. Furthermore, it may be more appropriate to refer to these as decision trees rather than decision graphs.
2. The primary contribution of this paper lies in the integration of SOPs into agent systems through the use of decision graphs. However, as noted in related works, agent methodologies based on SOPs have already been explored.
3. The methodology section of the paper is overly simplistic. It is essential to provide more detailed information regarding the SOP-agent framework. For instance, in the specified decision graph, is the reasoning of the LLM constrained in any way? Can the approach be guaranteed to maintain adaptability to varying environments, as well as scalability and generalization capabilities?
4. Mirror: the formatting of tables should be standardized throughout the document.

Lack of solution novelty. The authors explain the method in very short with many missing details (see questions), and then go on and on about the benchmarks and baselines. Probably this work was suitable for the Dataset and Benchmark track.

1. Novelty: While SOP guidance for AI agents is an innovative approach, the contribution could be seen as incremental, primarily advancing on the design of domain-specific SOPs rather than fundamentally new techniques. Experimental Limitations: The paper could benefit from broader comparisons with additional, more varied baseline models. Additionally, it lacks a detailed analysis of failure cases in the Grounded Customer Service Benchmark, which might provide insights into areas needing further improvement.

2. Robustness: The framework’s reliance on manually crafted SOPs may reduce scalability, as these SOPs might need frequent updates for new domains or changing environments, impacting the agent’s adaptability.