Thanks for the reviewer’s appreciation of our finite state machine design as well as the thorough discussion and experiments.

Re Reference Material:

Thank you for your valuable feedback. To the best of our knowledge, our work is the first to introduce finite state machines for the automatic design of multi-agent systems. We appreciate the suggestion regarding formal verification techniques and RL-based approaches. In our revisions, we will expand the discussion to include relevant formal verification methods for multi-agent systems and provide a more comprehensive analysis of RL-based self-play approaches in the context of task orchestration.

Re Wakeness 1:

Firstly, our framework is orthogonal to the foundation model capabilities. A better foundation model leads to better performance. However, when using GPT3.5 as the designer, our method also shows good performance on the Machine Learning Task. Moreover, we admit this is a reasonable concern. But when using LLM as a designer of a Multi-Agent System and even the LLM Agent domain. Almost every existing work can not avoid the influence caused by the ability of the foundation model. Our framework shows good performance on many kinds of tasks empirically.

Re Weakness 2:

The condition verifier serves as a guider who utilizes the knowledge of the foundation designer model (by following state transition conditions) and helps the agent learn from feedback or evaluation results. So the thing is the same as Weakness 1: when the foundational model goes powerful, our FSM becomes more reliable in uncertain domains.

Re Weakness 3

As for the ‘reuse’, the benchmark on ML bench and Software already show the reuse ability of the designed FSM. Because the FSM use in these benchmarks is designed for general use in the task domain (eg. Software Development). When deployed, the FSM will handle different cases in the domain. Thus, the experiment is showing the ‘reuse’ of the FSM.

Thanks for the thoughtful feedback. We are encouraged that the reviewer agrees the Finite State Machine(FSM) is an inspiring method to the Multi-Agent System field and appreciates our theoretical analysis.

Re Weakness:

Our framework is independent of the foundation model’s capabilities. Our method remains effective even with weaker models. For instance, when using GPT-3.5 as the designer, our approach still achieves strong results on Machine Learning benchmarks. Moreover, while concerns about the reliability of using an LLM as a designer in a Multi-Agent System, to the best of our knowledge, almost no existing work provides a provable guarantee. And Our framework shows good performance on many kinds of tasks empirically.

Re Claim 1:

Simply modifying the prompt is insufficient to enhance the performance of the LLM debate structure on text-based tasks. This is because our finite-state machine (FSM) framework incorporates structural features such as Null-Transition and State Traceback, which cannot be replicated through prompting alone.
To empirically validate this, we conducted an experiment comparing a purely prompt-modified LLM debate with the traditional one. Specifically, we rewrote the LLM debate prompt, instructed the model to refine its responses, and evaluated performance on GPQA (Diamond). The modified prompt achieved a score of 0.56, only marginally better than the traditional one (0.54). This result demonstrates that mere prompting cannot match the performance of our FSM-based approach—it is the FSM structure itself that drives the improvement.
Theoretically, Null-Transition serves as a refinement mechanism, allowing agents to improve responses based on feedback. In contrast, Decentralized Debate follows a rigid sequence where agents present opinions without feedback opportunities, preventing output refinement. This limitation cannot be overcome through simple prompt modifications. However, introducing a condition verifier to refine responses and select speakers naturally transforms the structure into a finite-state machine.

Re Claim 2:

Our MetaAgent Method represents the full realization of a Finite State Machine (FSM), as it incorporates a specialized condition verifier that enables Null-Transition and flexible state transitions, including state traceback.
Section 3.3 demonstrates that existing Multi-Agent Systems can be viewed as limited FSMs; the "Coordinate with Orchestrator" approach, which has a shared verifier, suffers from centralized decision-making that becomes computationally burdensome as states increase. In contrast, FSM's decentralized architecture—with independent condition verifiers at each state—significantly improves scalability and adaptability.

Re Method1：

We evaluate the MetaAgent Method on ML and Software Development tasks using two predefined tools: Python Code Interpreter and Search Engine. Our experiments demonstrate that the designer LLM effectively selects and assigns tools to different agents. We also test FSM with an expanded tool pool; the designer maintains wise selections and consistent performance.

Re Method2:

It is interesting to include the ‘splitting’ action in our method. However, practically, we find the designer LLM always tends to split the given task into trivial sub-tasks, causing a high failure rate because the chain of execution is too long. To fix this drawback, we design the merge action to help the designer LLM rethink and revise the FSM. The ‘splitting’ method is not practical because the initial version of the finite state machine itself tends to be very trivial.

Re Experiment 1

We selected SPP since it is an auto-design multi-agent method. Based on your suggestion, we have added new experiments for the software development task using AutoGen, Task Weaver, and Open Interpreter. This table shows the result:

From the above table, we can observe that our method also outperforms existing baselines by a large margin. Note that we do not include Data Interpreter since it is a frame specifically designed for Data Science. For other baselines in the text-based tasks, including direct prompt, CoT, CoT-SC, and Self-Refine. They are merely prompt-based, single LLM methods instead of the multi-gent framework, and they do not support tool-using functionality. They perform poorly and are not a fair comparison with our method since we do not include them in the real-world coding task.

Re Comment 2:

In appendix F

Re Comment 4:

From DataInterpreter Paper. NPS = $\frac{1}{1 + s}$ (if s is smaller, the better ) or $s$ (if s is bigger, the better ).

Re Question:

Yes. It is included in the ‘design’ stage.’

Thank the reviewer for appreciating the finite state machine as a unified framework of a Multi-Agent System.

Re Reference:

CAMEL is a simple two-agent chat system resembling a Decentralized Debate structure (Session 3.3).

AgentVerse employs two cooperation structures: Horizontal : a Linear System where decisions are summarized and Vertical:a Debate System where a solver and reviewer iteratively refine decisions until reaching consensus. These structures are also discussed in Session 3.3.

The paper "Embodied LLM Agents Learn to Cooperate in Organized Teams" introduces a Communication-Action model, where agents communicate first and act later. However, this model has drawbacks: agents receive no immediate feedback during the Action Phase, delaying behavior refinement, and the leader agent can only issue instructions during the Communication Phase.

In contrast, the FSM structure enables immediate feedback after each action via condition verifiers. Agents refine actions on the spot, determine state transitions, and communicate continuously, ensuring seamless interaction throughout the process.

Re Question 1:

The proposed optimization method relies on a self-iteration procedure, where the system refines itself over time. While incorporating external data could enhance this process by allowing the designer agent to learn from additional information, our observations indicate that most failure cases in the initial version of the FSM are caused by overly trivial agent and state assignments. Therefore, the most effective approach is to prompt the designer agent to self-refine the FSM by merging trivial agents and states, eliminating the need for external data.

Our FSM structure is designed to handle general open-ended, real-world tasks, where user inputs can vary widely. This variability makes it difficult to collect or synthesize high-quality test cases. As a result, relying on the self-iteration optimization method becomes not only effective but also the most practical approach for improving the FSM's performance in these complex, unpredictable scenarios.

Re Question2:

As presented in the optimization method (Session 3.4.3):” This iteration continues until no further states can be merged and the state set stabilizes.” The ‘iteration’ in the ablation study means the optimization method. We will refine our description in the revision.

Re Evaluation:

We have implemented a Virtual Home environment to the MetaAgent Framework and tested it with some housekeeping tasks. The following is a highlighted running log.

Agent Design: LivingRoomAgent, RestRoomAgent, KitchenAgent, and BedroomAgent.

State Design(show state instructions):
Take books from the living room, Clean rubbish in the living room, …, Take books from the bedroom

State Transition Example:
{ "from_state": "1",
"to_state": "2",
"condition": "If books are taken from the living room"
},

When deployed, this multi-agent system can plan and submit actions to the environment in the specific format. (example task: Clean the bedroom)

The Bedroom Agent first plan:
Prioritized Cleaning Plan for the Bedroom:

1. Organize Clothes:

• Fold and organize the clothespiles (15 clothespiles).
• Hang the clothesshirts (3 clothesshirts) and clothespantss (5 clothespantss) in the closets.

2. Clean Surfaces:

• Wipe down the desk, nightstands, coffeetable, and bookshelf.
• Ensure the desk and nightstands are properly closed.
  …

And then it act:

<ACTION> { "action": "Grab", "object": "clothespile", "object_id": 150 } </ACTION> …

Several test cases show that the FSM-based multi-agent system can also understand the Virtual Home environment and interact with it properly.

Re Weakness 1:

When writing the paper, Open source models do not have enough intelligence level to achieve this complicated task of designing a Multi-Agent System. To invest the influence of the foundation model quality, we also apply GPT-3.5 as a weaker model in the ablation study, whose performance is lower than GPT-4o but is still comparable with other baselines.
Recently, we also tried some powerful open-sourced models as designers (like deepseek-v3). It also worked well on our test cases.

Re Weakness 2:

Firstly, our framework is orthogonal to the foundation model capabilities. A better foundation model leads to better performance. However, when using GPT3.5 as the designer (GPT-4o as executor), our method also shows good performance on the Machine Learning Task. (In ablation study, Table 5) Moreover, while concerns about the reliability of using an LLM as a designer in a Multi-Agent System (or even within the LLM Agent domain) are valid, to the best of our knowledge, almost no existing work in the area can avoid the influence caused by a weaker foundation model.

Thanks for the reviewer’s effort and the appreciation in the discussion in Session 3.3. We believe the finite state machine has the potential to be a unified structure of the Multi-Agent System.

Re Claims1:

Our optimization method is inherently self-iterative, meaning it does not rely on external training data. In the ablation study, the description highlights the motivation behind this design choice. Initially, we observed that the first version underperformed on test cases, which led us to develop this self-iteration approach as an optimization strategy. We will refine the description in the revision to improve clarity.

Re Question 1:

This question explores why a Multi-Agent System, where agents are assigned different roles, can outperform a single LLM agent in certain tasks. Intuitively, assigning specific roles to agents activates their specialized knowledge related to those roles. Similarly, when all condition verification tasks are assigned to a single LLM, two main challenges arise. First, as the number of agents increases, a single condition verifier struggles to understand each agent’s situation. Second, dedicated condition verifiers, assigned to specific agents, can better capture and process the state-specific information, leading to more accurate and efficient verification.

Re Question 2:

GPQA evaluates the accuracy of multiple-choice questions, while Writing assigns a score based on keywords. Each keyword has a corresponding list that includes its various forms, and as long as the generated text contains at least one of these variants, it is considered valid for scoring. When designing the objective evaluation criteria for software development tasks, we also select several bad cases for subjective evaluation. Through this process, we gradually update the objective evaluation criteria to make it more reasonable.

Re Question3:

No. It means the optimization method described in the method section. We will refine the description in the revision.

Re Experiment Metric also Question4:

For a batch of tasks (10+), the cost of the MetaAgent architecture is lower than that of other frameworks. The primary reason is that the FSM is more general: it only requires a one-time design for the task domain, whereas other frameworks necessitate case-by-case design(eg, AutoAgents,SPP). As a result, MetaAgent incurs lower costs when handling a batch of tasks.