Automated Design of Agentic Systems

Thank you for your review, and for noting our paper is thought-inducing and interesting to a good portion of the community.

We sincerely believe our paper presents a significant and beneficial contribution that would enrich the ML community upon publication. We greatly appreciate your comments on safety concerns and acknowledge that our initial submission did not adequately convey our strong commitment to safety. We believe we have now addressed your concerns, particularly those related to safety considerations. Your current score recommends not publishing this work, but we hope you will keep an open mind to reconsidering in light of our significant improvements to the manuscript, especially describing everything we did to mitigate safety concerns as well as explaining why the work is valuable even with-and even because of- safety issues.

Summary: The authors propose to "form" a new research area

To clarify, our argument is that we are describing a newly forming research area, which includes some prior work in this direction, and then introducing a new entrant in this space that is higher performing and more general.

Weakness 1 & 2: Concerns on safety

Please see the general response.

Weakness 3 Part 1: The practical description of the method is very unclear and far too high-level in the main text.

Thank you for pointing this out. We have revised the relevant sections to provide a more detailed description of the algorithm in Section 3. Additionally, we have included pseudocode in Appendix I to better illustrate the process.

We have added the following text in the method section to make it clearer: “The algorithm proceeds as follows: (1) The archive is (optionally) initialized with baseline agents such as Chain-of-Thought and Self-Refine. (2) Conditioned on the archive, the meta agent designs a new agent by generating a high-level description of the new idea for an agentic system and then implementing it in code. The design then undergoes two self-reflection steps by the meta agent to ensure it is novel. (3) The generated agent is evaluated using validation data from the target domain. If errors occur during evaluation, the meta agent performs a self-reflection step to refine the design, repeating this process up to five times if necessary. (4) Finally, the agent is added to the archive along with its evaluation metrics, and the process continues with the updated archive until the maximum number of iterations is reached.”

Weakness 3 Part 2: the computational cost addition to the inference was not compared to the other, clearly less computationally expensive, approaches such as chain of thought in Table 1

Recent advancements, such as the o1-series model (OpenAI, 2024), highlight the potential of scaling compute during inference for foundation models and agents. Our algorithm is designed to explore effective ways to leverage this scalability rather than focusing on resource-constrained application scenarios. As suggested in our future work, it would be interesting and promising to investigate this further using multi-objective optimization algorithms that balance performance and computational cost.

Weakness 3 Part 3: The method has very limited applicability if it clearly cannot be applied to hundreds of billions of parameters even if you have the supercomputers for that.

We believe this method has strong potential to be applied to foundation models with hundreds of billions of parameters. First, our current approach is already feasible for larger models like GPT-4o; our current limitations (e.g., using GPT-3.5) stem from financial constraints typical of academic labs, rather than limitations of the method itself, since our method demonstrates promising sample efficiency. Second, as suggested in our future work, more efficient evaluation functions could be developed to reduce computational overhead, such as sampling test entries and examining logs in detail instead of evaluating all entries. Third, our model transfer experiments demonstrate that less expensive models can serve as effective proxies for discovering agents applicable to more costly models. Finally, with the consistent trend of exponentially increasing affordability and speed of APIs (e.g., GPT-4o-mini from July 2024 is 133x cheaper than GPT-3.5 from November 2022 while being significantly more powerful), we anticipate these constraints will continue to diminish. For these reasons, we remain confident in the scalability of this method for larger foundation models.

Seems like from the authors' responses that I did not have any gross misunderstanding on the main reasons for my scoring. Therefore I keep the score.

Thank you for your review, and for noting our work is well-motivated and the transferability results are interesting.

We believe we have addressed each of your concerns, including conducting two additional experiments you requested, significantly improving the manuscript as a result. Overall we would have predicted a higher score based just on your comments alone, as it did not feel to us that the level of issues you raised merited such a low score. We sincerely believe our paper presents a significant and beneficial contribution that would enrich the ML community upon publication. Your current score recommends not publishing this work, but we hope you will keep an open mind to reconsidering in light of our revisions, replies, and additional experiments.

Weakness 1: "Given that programming languages are Turing Complete..." is not written with care.

Thank you for pointing this out. We have revised the sentence to ensure it is more precise. “Given that most programming languages are Turing Complete...” (Line 18)

Weakness 2: It should be stated in the main text that the implementation of the meta optimizers is different for different tasks.

Our meta agent implementation is identical across different tasks, with the only variation being task-specific descriptive text included in the prompt. This is explained in Appendix C, but we have added an explanation of it in the main text too (Line 230).

Weakness 3: As an optimization method, it is only compared with handcrafted baselines. More comparison with other optimization methods is expected.

Thank you for your insightful comment! Based on your suggestion, we added experimental results for a state-of-the-art prompt optimization baseline (OPRO, Yang et al., 2024) and added them to Table 1.

The results demonstrate that our proposed Meta Agent Search outperforms OPRO across all domains. This additional comparison further reinforces the discovery that defining agents in code and allowing the learning of all components provides significant advantages.

Weakness 4: I think there are missing citations for self-referential higher-order meta-learning.

Thank you for your feedback. We have added the relevant citation [1] on self-referential higher-order meta-learning as per your suggestion in Section 6 Future Work.

[1] Schmidhuber, Jurgen. "Evolutionary principles in self-referential learning." On learning how to learn: The meta-meta-... hook.) Diploma thesis, Institut f. Informatik, Tech. Univ. Munich 1.2 (1987): 48.

Weakness 5: how the work illustrates the potential of this direction to benefit humanity, which is claimed in the abstract.

Please see the general response.

Weakness 6: For a fair comparison, I think the optimization result starting without any manually optimized initial solution should be presented.

Since one of our claims is that the code space allows for better utilization of existing human efforts (Section 2) rather than starting from scratch, we believe that starting without initial solutions is not necessarily a fairer comparison. However, we agree that it is an interesting investigation to explore the effects of initialization on the archive.

To address this, we conducted experiments running the Meta Agent Search without any initial agent designs. The results show that the discovered agents still outperform hand-crafted baselines across all domains. Interestingly, while good initial solutions generally improve performance, starting from scratch yielded superior results in the math domain. We hypothesize this is because the absence of initial design patterns may result in a broader exploration of different reasoning strategies within limited iterations, which is particularly beneficial in math tasks requiring diverse reasoning approaches. This opens up interesting future work on how initialization impacts search effectiveness across different domains. We have added the results and discussion in Appendix K.

Thanks for your response. I'm satisfied with all of them except weakness number 3. For weakness number 3, your method is compared with OPRO, which is a prompt optimization method but not an agent optimization one. I think a fair compression would be with an agent optimization method with the same initial agent and search space. Given that the motivation of the paper is to effectively find better LLM agents, I believe that showing your method's performance compared to agent optimization methods such as [1, 2, 3] is essential. Therefore, I will keep my score.

[1] Zhuge, M., Wang, W., Kirsch, L., Faccio, F., Khizbullin, D., & Schmidhuber, J. (2024). Language agents as optimizable graphs. arXiv preprint arXiv:2402.16823.

[2] Cheng, C. A., Nie, A., & Swaminathan, A. (2024). Trace is the next autodiff: Generative optimization with rich feedback, execution traces, and llms. arXiv preprint arXiv:2406.16218.

[3] Yuksekgonul, M., Bianchi, F., Boen, J., Liu, S., Huang, Z., Guestrin, C., & Zou, J. (2024). TextGrad: Automatic" Differentiation" via Text. arXiv preprint arXiv:2406.07496.

Dear Reviewer 955X,

Thank you for your feedback and for agreeing that we have addressed all your concerns except for weakness 3 in your review. Is there any chance you might reconsider raising your score to reflect that we have addressed all concerns except one, since that indicates the paper is now better according to your criteria than when you first selected that score?

Regarding the remaining issue, we received your request for additional experiments one day before the end of the discussion period. Had you asked at the beginning of the discussion period, we would have made every effort to incorporate these experiments. Unfortunately, we are unable to conduct new experiments at this stage due to time constraints, so we would like to explain our views regarding the issue you raised.

We understand that you suggest comparing our method with agent optimization methods that use the same initial agent and search space. However, our work is the first to explore the optimization of the entire agentic system in code, which is the least constrained search space. As such, there are currently no prior works that optimize agents in code with the same initial agent and search space as ours. This makes it challenging to perform a direct comparison under identical conditions. Moreover, our main contributions lie not only in the optimization method but also in defining a new search space and problem formulation for agent optimization. We believe that controlling for “the same initial agent and search space” is not thus applicable in our context, or at least that it would risk the reader getting distracted with comparing our innovation (a much broader, general method) with more constrained search in a constrained search space. That would thus risk people missing the point of our paper, which is demonstrating that searching within the most general, powerful search space is possible. The fact that this is true is one of the reasons we think this work is valuable: it opens up a new research frontier, rather than being a new/better way to do something others have tried before.

Regarding the comparison with prompt optimization methods, we chose OPRO as a baseline because prior works primarily focus on prompt optimization for agents. Even in the references you mentioned, [1] includes prompt optimization as an independent step, and [2] and [3] also primarily optimize prompts in agents as one of the applications of their general optimization frameworks.

Given the above arguments, we believe that comparing our method with OPRO is representative and provides valuable insights into how our algorithm in the proposed code search space performs relative to established prompt optimization techniques for agents. We acknowledge that incorporating additional baselines, such as [1-3], could strengthen our results, and we will make an effort to include them in the camera-ready version of the paper. However, due to time constraints, we hope our explanations suffice to address your concerns.

Overall, we believe that the issues you raised do not warrant such a low score, especially given (1) that we have fixed many of the issues you asked us to since you selected that score, (2) we improved the paper substantially based on the other reviewers’ comments as well, and (3) the positive evaluations from the other reviewers (an accept with a score of 8 and a strong accept with a score of 10). We worry that the “anchoring effect” (to your original score) may be at play, and wonder if you might be open to reconsidering your score.

Thank you very much for your time and for at least considering reconsidering your score a final time.

The Authors

Thank you for your comprehensive review and for recognizing the strengths of our work, including the novelty of introducing ADAS as a research area, the robust experimental results, the clarity in presenting the motivation, methodology, and implications, and the potential of automating agent design to significantly accelerate AI development and broaden applicability. We appreciate your feedback and are pleased to note your positive assessment. We address your concerns and suggestions below.

Weakness 1 & Question 1: While the paper mentions safety concerns, a more in-depth analysis of potential risks and mitigation strategies associated with ADAS would be beneficial. How does Meta Agent Search address potential biases or unsafe behaviors that might emerge from automatically generated agents? Are there mechanisms in place to detect and mitigate these issues?

Please see the general response.

Weakness 2: A deeper discussion on why certain baselines were chosen and how they were implemented would strengthen the evaluation.

Thank you for your comment. We have added more discussions about the baselines in Section 4.1 Baselines. The chosen baselines represent agent designs widely adopted in the agent literature. For fair comparisons, we implemented them using the same framework provided for the meta agent. Example code for the baselines and related framework is included in Appendix D. All code, including the baseline implementations, will be open-sourced.

Also, we have incorporated a state-of-the-art prompt optimization method (OPRO, Yang et al., 2024) as an additional baseline (Table 1). The results demonstrate that our proposed Meta Agent Search outperforms OPRO across all domains, further reinforcing our argument that defining agents in code and enabling the learning of all components provides significant advantages.

Question 2: How does the approach scale with more complex tasks or larger domains? Are there computational limitations that need to be considered?

We expect the proposed Meta Agent Search to scale effectively to more complex tasks and larger domains due to its flexibility, which allows for the integration of additional tools and functionalities as needed. While computational limitations may arise in larger domains, these challenges can be addressed through optimizations such as efficient evaluation functions to balance performance and cost. Overall, the use of a flexible code search space makes Meta Agent Search a promising approach for scaling to a wide range of applications. Moreover, as foundation models and computational resources continue to exponentially decrease in cost (e.g., GPT-4o-mini from July 2024 is 133x cheaper than GPT-3.5 from November 2022 while being significantly more powerful), ADAS will become increasingly feasible for application across a broader range of complex domains.

Question 3: Could the authors elaborate on how higher-order ADAS (e.g., improving the meta-agent itself) might be achieved and what challenges are anticipated in that direction?

That is a great question! One potential approach is to introduce a “meta-meta agent” and run a “Meta-meta Agent Search” to improve the meta agent itself. However, this would pose significant computational challenges, as each evaluation of a meta agent design would require a full run of the Meta Agent Search algorithm. An alternative is to enable online adaptation of the meta agent during the search process. While this approach may be more feasible from a cost perspective, it introduces the challenge of assessing the quality of the evolving meta agent. We also found [1,2] provides relevant insights into this intriguing topic.

Thank you for recognizing the contribution of our work. Have we answered all your questions and concerns? Please do not hesitate to bring up any additional questions.

[1] Lu, Chris, Sebastian Towers, and Jakob Foerster. "Arbitrary Order Meta-Learning with Simple Population-Based Evolution." ALIFE 2023: Ghost in the Machine: Proceedings of the 2023 Artificial Life Conference. MIT Press, 2023.

[2] Lange, Robert Tjarko, et al. "Discovering Evolution Strategies via Meta-Black-Box Optimization." The Eleventh International Conference on Learning Representations.

Thank you for your insightful and detailed review. We appreciate your recognition of our work’s strengths, including its strong results, timely and relevant topic, clear presentation, and potential integration with other significant research directions such as designing better evaluators (e.g., LLM-as-a-Judge, Agent-as-a-Judge). Your insights are encouraging and valuable to us.

We believe we have addressed each of your concerns, significantly improving the manuscript as a result. We feel this paper makes an important contribution and the ML community will benefit from it if published. We appreciate your open mind to changing your score, and we hope you can improve the score given our answers to your questions and our improvements.

Weakness 1 & Question 6: There are papers like Voyager out there which incrementally build up sets of tools for solving a specific task. Is this actually fundamentally different from such code-generating systems? Can you compare this with Voyager?

Both our work and Voyager involve code generation, but they differ fundamentally in focus and scope. Voyager generates code as “skills” to control a game API, enabling atomic actions without involving foundation models. In contrast, our approach generates code for agentic systems, which include complex workflows involving multiple calls to foundation models and tools. Voyager is more akin to works that generate tools within agentic systems but focuses primarily on games rather than general agentic tasks. Our framework, by contrast, is designed to search for any possible components in agentic systems (which could include Voyager-style code as tools), supporting a wide range of general tasks. Due to the differing domains and objectives, a direct comparison is impossible (e.g. a code policy for Minecraft could not answer free-form QA), but we will expand on this distinction in the related work section.

Weakness 2 & Question 2: Most things in AI are, in one way or another, Turing Complete, so I wonder if that is really a big deal here … whether the LLMs actually act as Turing Complete systems in practice.

Turing completeness in our work specifically relates to the code search space. Unlike prior attempts to automatically discover LLM agents, which often restrict their search space (e.g., focusing solely on prompt optimization), our method defines agents in code. Because the programming language we use (Python) is Turing complete, our search space theoretically allows the discovery of all possible components of agentic systems, such as advanced tool use and RAG in open-sourced codebases like LangChain, and even beyond. As such, our system has more power versus other attempts to automatically create LLM agents. It is true that there are many systems like artificial life systems or even neural networks that are also Turing Complete, and you are right to raise the issue that having this property is not sufficient, for one needs to be able to effectively search this vast search space to produce useful artifacts. But we do show just that. Because it harnesses the powerful intelligence in Foundation Models, Meta-Agent Search marries a Turing Complete search space with the ability to search it well to discover effective solutions.

Weakness 3 & Question 1: From the main text, it is quite unclear how their proposed system works exactly. Is it just a trivial loop writing a basic program?

Thank you for your comment. To improve clarity, we have added a more thorough description of the algorithm in Section 3 and pseudocode for our Meta Agent Search algorithm in Appendix I to better illustrate how the system works.

We have added the following description: “The algorithm proceeds as follows: (1) The archive is (optionally) initialized with baseline agents such as Chain-of-Thought and Self-Refine. (2) Conditioned on the archive, the meta agent designs a new agent by generating a high-level description of the new idea for an agentic system and then implementing it in code. The design then undergoes two self-reflection steps by the meta agent to ensure it is novel. (3) The generated agent is evaluated using validation data from the target domain. If errors occur during evaluation, the meta agent performs a self-reflection step to refine the design, repeating this process up to five times if necessary. (4) Finally, the agent is added to the archive along with its evaluation metrics, and the process continues with the updated archive until the maximum number of iterations is reached.”

Concerns about the safety and the benefit of this work

Three reviewers (MJM9, 955X, gtAT) expressed concerns about potential risks and safety considerations associated with the automatic generation of agentic systems. Reviewer gtAT worried the approach could be seen as a primitive form of self-replication. Reviewer 955X also inquired about how our work illustrates the potential to benefit humanity.

Implemented Safety Measures

We acknowledge the importance of safety in developing systems like ADAS and apologize for not providing detailed explanations of our safety protocols in the original submission. We had, in fact, implemented several safety measures to mitigate potential risks, consistent with safety standards in the published literature (e.g., SWE-Bench, Yang et al., ICML, 2024). We failed to mention these in the original paper (though we should have), which we will fix in the revision. The safety measures are as follows:

1. Containerized Execution: All generated code is executed within secure containerized environments on remote servers to prevent any unintended side effects or interactions with the host system. This isolation ensures that even if the generated code contains harmful instructions, it cannot affect the underlying system. One could further restrict the sandboxing to proactively block any attempted internet access, but because we have seen zero evidence (see below) that our system is doing anything unsafe with its internet access, and because such access can be important for the science (e.g. to see if it uses web searches in its agentic systems), we currently have left that in place. We will add detailed descriptions of our Containerization practices in the revised paper.
2. Manual Inspection: We conducted thorough manual reviews of our generated agents to get a sense of whether dangerous agents were being generated that could take potentially harmful actions, such as unauthorized file access, network communications, or execution of system commands. We did this both during preliminary experiments and after our final runs. No such behaviors were observed in our experiments, aligning with our expectations given that the foundation models we used are extensively fine-tuned through Reinforcement Learning from Human Feedback (RLHF) to avoid generating malicious content, and because we do not believe there is any incentive for these agents to take malicious actions. We purposefully avoid using foundation models that have not been “aligned” in this way.
3. Clear Warnings and Guidelines: We include warnings in our codebase and documentation that will be open-sourced about the potential risks of executing generated code, advising users to employ appropriate safety measures such as sandboxing and strict code review before deploying any generated agents in real-world settings. We will emphasize these guidelines in the revised paper and code release.

Alignment with Safety Standards in the Literature

Our safety practices align with those in existing literature where code is generated and executed. Works such as Voyager (Wang et al., TMLR 2024), SWE-Bench (Yang et al., ICML 2024), and FunSearch (Romera-Paredes et al., Nature 2024) have similarly generated and executed code in Turing-complete languages, employing safety measures like sandboxed execution and controlled environments. Some systems like AndroidControl (Wei et al., NeurIPS 2024) and WebAgent (Gur et al., ICLR 2024) even permit broader access, such as modifying local files, or executing system commands, which we do not permit, nor is it our aim in our experiments. By adhering to these established standards, we ensure that our approach is consistent with the community-defined best practices.

Safety Contributions of ADAS

Our work offers a pathway to improve the safety of agents. Currently, agent workflows generally fall into two categories:

1. Explicitly Defined Workflows: These agents rely on explicitly coded logic, such as loops, conditionals, and tool calls (as used in our current algorithm).
2. Autonomous Reasoning-Based Workflows: Systems like AutoGPT (Richards, 2023) and O1-series models (OpenAI, 2024) depend on the model's reasoning to decide the next step, action, or tool.