AutoAgents: A Framework for Automatic Agent Generation

Thank you for your motivating review and concrete suggestions. Detailed responses to your questions are listed as follows.

1. The paper is perceived as having low readability and insufficient reproducibility. The reviewer kindly requests a more granular description of the methodology that enables readers to implement the procedures step by step. For instance, while Table 1 is highly beneficial for positioning this research within the LLM-based Agent framework, in comparison to AgentVerse and SSP, it distinctly highlights the significance of Self-Refinement agents and Collaborative Refinement Action. Nevertheless, the two points mentioned above are not clearly articulated in Section 3.2, "EXECUTION STAGE." They are mentioned in the text and Figure 2, but their presentation as steps is absent, making it challenging for readers to comprehend and evaluate reproducibility.
2. Could you provide a more detailed, step-by-step description of the Self-Refinement agents and Collaborative Refinement Action in Section 3.2, "EXECUTION STAGE," to enhance readability and reproducibility?

In order to streamline the implementation process of self-refinement and collaborative refinement, we have introduced the concept of a 'custom agent'. The intricate design of its prompt has been meticulously detailed in Appendix D.5. For enhanced clarity and understanding, we have itemized the core design principles of this custom agent below:

1. Understanding Previous Results: Begin by analyzing the results of previous agents to grasp the context and progress of the task.

2. Task Analysis and Breakdown: Understand, analyze, and deconstruct the given task. Use available tools to assist in task completion.

3. Current Step Identification and Execution:

• 3.1 Examine completed steps and their outcomes to identify the current step that needs to be completed.
• 3.2 In the absence of completed steps, analyze the task, design a plan for the necessary steps, and accomplish the first one.
• 3.3 If steps have been completed, understand them to determine the next step to be completed.

4. Tool Selection and Action Execution:

• 4.1 Choose the appropriate tool from the given list ({tool}) to complete the current step.
• 4.2 Follow specific format guidelines when using tools like 'Write File'.
• 4.3 Once all steps are completed, use the 'Final Output' action to summarize each step's outputs, ensuring the final output is detailed, comprehensive, and solves the task.

5. Maintaining Format Consistency: Ensure that the final output adheres to the given format example, prioritizing helpfulness, relevance, accuracy, and detail.

Building upon the custom agent's framework, a critical aspect of the refinement process is the configuration of 'completed steps' in step 3.1. The specific procedural steps for self-refinement and collaborative refinement are outlined as follows:

Self-refinement Action: During its first execution, each custom agent omits the outlined step 3.1 and proceeds to complete the remaining steps. If the task's criteria are not met or the step limit has not been reached, the outcomes from this execution are incorporated as 'completed steps' in the prompt for the next iteration of the task. In subsequent executions, the custom agent will execute all steps, continuing this process until the task is deemed successfully completed.

Collaborative Refinement Action: This mirrors the self-refinement action, yet involves active collaboration between multiple agents. For instance, when agents A and B collaborate on a task, A initially bypasses step 3.1 in its first execution. Upon completion, B incorporates A's results as 'completed steps' and then executes all steps. In later cycles, A and B alternate their roles in the task, perpetuating this collaborative cycle until a joint conclusion is reached.

As the rebuttal phase draws to a close, we want to thank the reviewers for their valuable feedback. We believe we have successfully addressed your concerns in our responses and welcome any further questions you might have.

Given the improvements and clarifications made based on your feedback, we kindly ask the reviewers to reconsider their evaluations to account for these updates. We greatly appreciate your consideration.

Thank you for your comments and suggestions. We think there might be a misunderstanding of our contributions. We will clarify these concerns with detailed responses in the following.

Method

Limited novelty: according to Table 1, the main difference between the proposed framework and other existing methods like Social Simulacra, Epidemic Modeling, SSP, and AgentVerse is that this work uses self-refinement and collaborative refinement. This difference is more of a prompting technique and has already been used in many existing works like [1, 2, 3]

Here, we further elucidate the main contributions of AutoAgents, a novel framework that addresses two salient issues: multi-agent generation and multi-agent cooperation. With respect to multi-agent generation, our method surpasses four existing methods in terms of the completeness and adaptability of agent creation, by employing Agent Generalization by Multi-Agent Discussion and Prompt Generalization. Regarding multi-agent cooperation, our framework exhibits the efficacy of Self-refinement Action and Cooperative Refinement Action. As illustrated in the table below, the distinctive features of AutoAgents encompass Agent Generalization by Multi-Agent Discussion, Prompt Generalization, Self-refinement Action, and Cooperative Refinement Action. AutoAgents is the first holistic framework that integrates multi-agent generation and collaboration. Detailed accounts of Agent Generalization by Multi-Agent Discussion and Prompt Generalization are given in below.

• Agent Generalization by Multi-Agent Discussion: Existing agent generation frameworks rely more on a single agent to generate the list of agents, but this often limits their adaptability to different tasks. For AutoAgents， the drafting stage is very important for the auto-generation of agents, as the list of agents to be generated is determined through cooperative discussions between the Planner agent and the two Observers. Additionally, in Appendix A, we further supplement with ablation experiments concerning the Agent Observer and Plan Observer, with the following table providing a detailed comparison. It is evident that in the absence of cooperative discussions among Observers, there is a significant decline in the overall performance of AutoAgents. This also validates the importance of generating reasonable agents for task adaptability.

Note: Please refer to the Appendix A of the revision for the detailed setup of the experiment.

• Prompt Generalization: The prompt employed by AutoAgents exhibits a more universal nature, signifying its capacity to acclimate to diverse tasks without necessitating bespoke customization. Both AgentVerse[1] and SSP[2] have implemented task-specific enhancements for varied task evaluations. In contrast, our methodology leverages a singular, unified prompt format to accommodate an array of tasks. The commendable efficacy in open-ended question answer and trivia creative writing tasks further corroborates the wide-ranging applicability and versatility of prompt design within the AutoAgents framework.

Note: In the revision, we have improved Table 1 and provided additional comparisons in Appendix A.

[1] https://github.com/OpenBMB/AgentVerse/tree/minecraft/agentverse/tasks

[2] https://github.com/MikeWangWZHL/Solo-Performance-Prompting/tree/main/prompts

As the rebuttal phase draws to a close, we want to thank the reviewers for their valuable feedback. We believe we have successfully addressed your concerns in our responses and welcome any further questions you might have.

Given the improvements and clarifications made based on your feedback, we kindly ask the reviewers to reconsider their evaluations to account for these updates. We greatly appreciate your consideration.

Thank you for your motivating review and concrete suggestions. Detailed responses to your questions are listed as follows.

Q1: How the proposed AutoAgents framework expands the scope of collaborative applications and reduces the consumption of resources should be elaborated.

Existing agent technologies are overly dependent on pre-defined prompts. This reliance poses a substantial user barrier and limits their practicality for broader applications. AutoAgents addresses this by automating the creation of new agent roles, which streamlines both the definition of roles and the execution of tasks, thereby enhancing the system's versatility.

AutoAgents significantly extends its adaptability to diverse applications through the automatic generation of multiple intelligent agents, each tailored for specific tasks. This improvement in agent generation quality is achieved through the collaborative efforts of three predefined agents - the Planner, Agent Observer, and Plan Observer. These agents work together to discuss, refine, and finalize the list of agents and their corresponding execution plans. In Appendix A, we delve deeper into this aspect with ablation experiments focusing on the Agent Observer and Plan Observer. Our comparative analysis, as shown in the following table, reveals a marked decrease in AutoAgents' overall performance in the absence of cooperative dialogues among Observers. This outcome not only underscores the efficacy of collaborative refinement but also highlights the critical role of these discussions in enhancing task adaptability and resource optimization in the AutoAgents framework.

Note: Please refer to the Appendix A of the revision for the detailed setup of the experiment.

Q2: The authors do not explain how to determine the number of agents in the section of the framework for automatic agent generation.

AutoAgents fundamentally predefines only a Planner and three types of Observers. Throughout the automatic agent generation phase, neither the number nor the types of agents to be generated are pre-specified. Instead, the decision regarding both the number and types of agents is dynamically and autonomously determined through collaborative discussions among the Planner, Agent Observer, and Plan Observer. This process ensures that the agents generated are optimally aligned with the specific requirements and nuances of each task.

Q3: The section about automatic agent generation is too tedious to introduce too much related works

In Appendix A, we have included an expanded comparison of methodologies for agent generation. Traditional frameworks in agent generation typically hinge on the capability of a single agent to determine the list of agents. This approach, however, often results in limited adaptability across various tasks. In contrast, the AutoAgents framework places significant emphasis on the drafting stage for the auto-generation of agents. Here, the compilation of the agent list is a result of collaborative discussions between the Planner agent and the two Observers, enhancing adaptability and flexibility.

Moreover, the prompts utilized in AutoAgents are designed with a universal approach, reflecting their ability to adapt to a wide range of tasks without the need for task-specific customization. While platforms like AgentVerse[1] and SSP[2] have opted for task-specific modifications, AutoAgents adopts a more holistic, unified prompt strategy. This uniform approach is adept at catering to diverse tasks, which is evident in our system's notable performance in open-ended question answering and trivia creative writing tasks. Such results highlight the extensive applicability and adaptability of our prompt design, making AutoAgents a versatile tool in the realm of agent-based systems.

[1] https://github.com/OpenBMB/AgentVerse/tree/minecraft/agentverse/tasks

[2] https://github.com/MikeWangWZHL/Solo-Performance-Prompting/tree/main/prompts

We appreciate your time in reading our feedback and look forward to further discussion and your suggestions.

As the rebuttal phase draws to a close, we want to thank the reviewers for their valuable feedback. We believe we have successfully addressed your concerns in our responses and welcome any further questions you might have.

Given the improvements and clarifications made based on your feedback, we kindly ask the reviewers to reconsider their evaluations to account for these updates. We greatly appreciate your consideration.

Q4: Regarding knowledge sharing, did you experiment with each agent using different types, or were these predefined onset?

To further elucidate, we delineate the specific agent types that are compatible with the three modes of knowledge sharing, as depicted in Figure 7 of the revision:

• Short-term Memory: This is predominantly utilized by agents that emerge from self-refinement and collaborative-refinement processes. These agents leverage short-term memory to retain historical records throughout the refinement phase. Notably, every agent generated within our framework actively employs short-term memory.

• Long-term Memory: This component archives the execution results of each agent, primarily chronicling the cumulative progress of the execution plan. Access to long-term memory is exclusive to a pre-designated Action Observer. The Action Observer utilizes this memory to evaluate whether additional execution steps are necessary or if there's a need to generate dynamic memory content.

• Dynamic Memory: This memory type is essential for distilling and summarizing pivotal information from long-term memory for diverse agents. Generated by the Action Observer, dynamic memory serves to amalgamate critical aspects of the overall execution plan, thereby facilitating the agents responsible for implementing subsequent tasks. Consequently, the dynamic memory content available to each generated agent may differ, contingent upon their specific roles and tasks.

Due to the fundamental nature of Short-term Memory and Long-term Memory as indispensable components within the system, our investigation was exclusively focused on understanding and analyzing the role of dynamic memory. In the Appendix A of the revision, we have included an ablation study of dynamic memory. The following table shows the results without the dynamic memory. It is observed that the performance of AutoAgents decreases by 1% without the summarization from dynamic memory, which also proves the importance of dynamic memory.

Note: Please refer to the Appendix A of the revision for the detailed setup of the experiment.

Q5: How did you decide to use the number of discussions in the two stages to 3 and 5, respectively (page 7)?

On one hand, it's been observed through experience that a detailed plan and a complete list of agents can generally be obtained after about three rounds of collaborative planning discussions. In cases where a single agent is operating, there's a high probability that the current task can be completed within five rounds. While a greater number of execution rounds typically leads to better outcomes, on the other hand, since we rely on GPT-4, more iterations mean increased costs. This setup is designed to manage and control the costs associated with task execution.

Q6: In multiple places, the agents' discussions may be stopped after some predefined threshold if the agents do not reach a consensus (e.g. during collaborative refinement, page 7). How often did this happen, and if the team did process, what is the quality of the outcome?

The frequency of such occurrences is inherently tied to the complexity of the problem or task at hand. It is acknowledged that a premature termination of the refinement process could potentially compromise the quality of the final output. To address this, once the specified limit is reached, our system prompts with an instruction: 'You should synthesize the responses of previous steps and provide the final feedback.' This directive is designed to guide the agent towards concluding the refinement process, ensuring that a coherent and comprehensive outcome is still achieved despite the early halt in discussions.

We thank you for your constructive comments! We have fixed the grammatical errors in the revision. We also have addressed the other comments below and incorporated the feedback in the revision.

Q1: What motivated the design of the prompt elements for the predefined agents? Did you consider alternatives, or did existing works inspire these?

Existing agent technologies are overly dependent on pre-defined prompts, presenting a significant barrier to users and rendering them impractical for widespread use. By developing a system that automatically creates new agent roles, we can automate both the definition of roles and the execution of tasks, significantly increasing the technology's versatility. AutoAgents, potentially a pioneering framework, is distinctively geared towards generating universal prompts applicable to a wide range of agents. Our design not only ensures the accomplishment of each agent's predefined tasks but also, through careful consideration during the design phase, enhances the adaptability of these prompts to a variety of tasks, thus broadening their general applicability. In the updated appendix, we have incorporated a section at the outset of each prompt detailing the design principles and underlying logic.

Q2: How were the roles, skills, and actions decided for the specific tasks? Were they injected in the prompt, or did the Planner agent generate them?

The Planner agent serves as the architect in our framework, initially crafting a comprehensive array of roles specifically designed for a variety of tasks. This includes not only the creation of prompts, descriptions, and toolsets for each role but also the formulation of an intricate execution plan. This plan distinctly specifies whether self-refinement or collaborative refinement actions are required. Following this initial phase, both the list of roles and the execution plan are subject to further refinement, achieved through collaborative dialogues with Observer agents. It is imperative to note that the choice of specific tools and techniques employed by each agent to navigate the steps in the execution plan is made independently by the agents themselves. A crucial aspect of our methodology is that we do not create unique prompts for each task. Rather, the primary prompts are methodically generated by the Planner agent, ensuring a consistent and streamlined approach across various tasks.

Q3: For the self-refinement process, what was the source of the thoughts, i.e., who decided what thoughts to include and why?

As described in the appendix about the 'custom agent', each agent, upon receiving their assigned task, must first understand and analyze the task. They need to plan the steps required for future action and determine the immediate steps to be taken. The agent then outputs the results of completing the current step. In the next round of feedback, the history of executed steps and their outcomes (short-term memory) are fed back into the prompt under 'completed steps and responses'. This helps the agent to determine the steps needed in the future. The prompt's design rationale is additionally detailed in the appendix of the revision.

As the rebuttal phase draws to a close, we want to thank the reviewers for their valuable feedback. We believe we have successfully addressed your concerns in our responses and welcome any further questions you might have.

Given the improvements and clarifications made based on your feedback, we kindly ask the reviewers to reconsider their evaluations to account for these updates. We greatly appreciate your consideration.