13. Question about BC baselines.

Thank you for your professional question. We would like to clarify the following:

• Zero-shot and few-shot are evaluation approaches for agents, while BC and RL are training approaches. Therefore, zero-shot evaluation is decoupled from the specific training methods.
• In our experiments, we do include three BC baselines: $BC_{base}$, $BC_{large}$, and AgentLM. All of them have been trained on a large number of agent trajectories. Models trained with BC already possess instruction-following capabilities and can distinguish between different tasks based on the system prompt and instruction. Therefore, we evaluate these models, including AgentEvol, in a zero-shot setting.
• Additionally, many works still use BC, such as AutoWebGLM[8] and ToolLLM[9]. However, we do not include them in our baselines for comparison, as these models are task-specific or environment-specific. In our main experiment, our model demonstrates general capabilities across 11 environments.

14. Question regarding typos and minor errors.

Thank you for your careful review. We have corrected all the typos in the manuscript.

• Re: p.5 "golden trajectories". We apologize for the imprecise wording. What we intended to convey is a meaning similar to "golden label". In the BIRD-SQL environment, it includes a crowdsourced text-to-SQL dataset, where the questions and corresponding SQL answers are already provided. An example is as follows. We call this data "golden trajectories (labels)" because these high-quality data don't require any further annotation. We have corrected this term in the revised manuscript. Thank you for your careful review.

  SELECT T3.name 
  FROM student AS T1 INNER JOIN registration AS T2 ON T1.student_id = T2.student_id INNER JOIN course AS T3 ON T2.course_id = T3.course_id 
  WHERE T1.type = 'UG' ORDER BY T2.sat
  DESC LIMIT 1
  

• Re: p.6 "The step of J(q, \pi...)". Our wording may have caused some confusion. In Line 313 and Line 315 on p.6, 'The former step' and 'The latter step' refer to the two parts before and after the minus sign in Eq(5) (i.e., ${\\mathbb{E}_q [\\log p(O=1|\\tau)]}$ and $\text{KL}[q(\tau)||\pi\_{\theta}(\tau)]$ ). We aim to provide a clearer and more intuitive explanation of these two parts, so that readers who are not familiar with RL can better understand some of AgentEvol's theoretical methods. We have followed your suggestion and updated the manuscript accordingly.

References

[1] Dayan P, Hinton G E. Using expectation-maximization for reinforcement learning[J]. Neural Computation, 1997, 9(2): 271-278.

[2] Abdolmaleki A, Springenberg J T, Tassa Y, et al. Maximum a posteriori policy optimisation[J]. ICLR 2018.

[3] Peters, Jan, and Stefan Schaal. "Reinforcement learning by reward-weighted regression for operational space control." Proceedings of the 24th international conference on Machine learning. 2007.

[4] Liu, Xiao, et al. "Agentbench: Evaluating llms as agents." arXiv preprint arXiv:2308.03688 (2023).

[5] Ma, Chang, et al. "AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents." arXiv preprint arXiv:2401.13178 (2024).

[6] Zhang, Jianguo, et al. "AgentOhana: Design Unified Data and Training Pipeline for Effective Agent Learning." arXiv preprint arXiv:2402.15506 (2024).

[7] Zeng, Aohan, et al. "Agenttuning: Enabling generalized agent abilities for llms." arXiv preprint arXiv:2310.12823 (2023).

[8] Lai, Hanyu, et al. "AutoWebGLM: A Large Language Model-based Web Navigating Agent." Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2024.

[9] Qin, Yujia, et al. "Toolllm: Facilitating large language models to master 16000+ real-world apis." arXiv preprint arXiv:2307.16789 (2023).

11. Question about “interaction rounds” mean in Appendix E.1.

Thank you for your question. We apologize for the confusion caused by our description. "Interaction rounds" in Appendix E.1 (now Appendix F.1 in the revised version) refers to the average number of interactions required for an LLM-based agent to complete a task in the environment. This is the same with the metric provided in the last column of Table 2 (Statistics of AgentTraj-L).

In each interaction round, the environment returns the current observation, and the agent takes actions based on this feedback. The number of interaction rounds typically depends on the task goal in different environments and the efficiency of the agent. For example, in the WebShop, the interactions usually include the following rounds:

• Agent executes "Search[something]" on the webpage, and the environment returns search results.
• Agent executes "Click[some button]" to select an item, and the environment returns the product detail page.
• Agent selects the product attributes.
• Agent selects the "Buy" button, and the environment returns the reward.

Additionally, for the BIRD, which is a single-round SQL programming task, the agent only needs to perform one action on the database, so the number of interaction rounds for the BIRD task is set to 1.

This may differ significantly from traditional RL agent tasks. In LLM-based agent tasks, due to the limitation of model's input length, an average task is typically solved in 10~20 rounds of text interactions. We apologize for the confusion this may have caused.

12. Question regarding SOTA models in Table3.

The baselines we compare in Table 3 are still SOTA models. We would like to clarify the following:

• Regarding close-sourced models, we test GPT-4, Claude 3.5, and Deepseek-chat. We expect to test the latest O1 models in the future when sufficient usage and resource are available.
• Regarding open-sourced models, we discuss the relevant works in the LLM-based agent field (see Table 1 for details):
  • AgentBench, AgentBoard, and AgentOhana [4, 5, 6]create excellent benchmarks to evaluate LLMs as agents, but they do not train specific agent models.
  • Pangu-Agent: No publicly available model provided.
  • AgentLM from AgentTuning [7] is an early milestone in the LLM-based agent field. It trains agent models of various sizes for multiple tasks and still shows strong performance. We believe it is appropriate to select AgentLM as SOTA models.
  • Llama-2-chat is not listed as SOTA models. As one of the well-regarded models and our backbone model, we also test it.

We expect to include more models in our work, e.g., Qwen, Gemma, along with more of the latest closed-source models.

7. Question about train sets and test sets.

We apologize for any confusion caused by our writing. The right half of Table 2 on p.4 (revised version) shows the dataset sizes for each environment. We do separate a train set and a test set for each environment. Here, "Traj. Size" represents the size of the small training set (AgentTraj), "Traj-L Size" represents the size of the large training set (AgentTraj-L), and "Eval. Size" represents the size of the test set. If the training set size is 0 for an environment (e.g., WA, AM, and ST), it means that the environment is used solely for testing.

8. Question about general purpose agents.

Thank you for your insightful question. It is very helpful for our future research and extensions. To the best of our knowledge:

• Gato is a very powerful, multi-modal, multi-task, and multi-embodiment generalist agent. Meanwhile, AgentGym is specifically designed for complex, long-term, multi-turn sequential decision-making tasks. We are excited about the possibility of integrating such a strong agent into our framework to evaluate its capabilities.
• OpenVLA is a well-known VLA model designed for real-world robotic tasks. Currently, AgentGym is still a language-based framework. In this context, instructions, actions, and observations are all language-based and textual. Additionally, robotic tasks differ slightly from the focus of our LLM-based agents (see Q1: "Why do we select these environments" in response to Reviewer VNyh). We will include these multi-modal agents in our future work.

9. Question regarding the lack of improvement in certain environments.

Please refer to the first point of the General Response. We also kindly emphasize that AgentEvol is stronger than $BC_{base}$ in 8 environments and stronger than $BC_{large}$ in 7 environments.

10. Question about implementation details of PPO and other methods.

We have provided an analysis and discussion in the second point of the General Response, and included the implementation details of all methods in Appendix E.

5. Question regarding the main results in Table3.

We would like to kindly clarify the following:

• In Table 3 (the main experiment table), we aim to show the testing and training results based on the AgentGym framework. Therefore, we compare with both open-sourced and close-sourced SOTA models. The baselines in Table 3, as well as our AgentEvol method, are all task-agnostic. We believe this comparison is fairer.
• Regarding Fig. 4, due to limited computational resources and implementation difficulties, we are unable to run PPO experiments across all 14 environments. Therefore, we cannot merge these results with the main table. Additionally, to the best of our knowledge, PPO algorithms executed on LLM agent tasks currently are all task-specific. Running PPO across multiple environments is very challenging.
• Regarding Table 5, scaling (model size, data size) is a key focus in the LLM field, so we present it in a separate table for discussion. Following Reviewer ZWi2's suggestion, we add a more detailed analysis and discussion on scalability (see Q1 regarding Reviewer Zwi2).

6. Question regarding M=4 iterations.

Thank you for your thoughtful question. The evolution process requires a substantial runtime. On eight A100-80GB GPUs, a single iteration requires approximately 35 hours. This includes about 3 hours for the learning step, 6 hours for evaluation across all environments, and 26 hours for the exploration step, which involves multiple interactions between the agent and the environment. A summary of exploration times across environments is shown below:

Although increasing the number of iterations $M$ yields performance gains, it gradually converges in later iterations. As described in Section 5.3, we perform an ablation study on the iteration number, and to further validate this, we conduct additional experiments for $M=6$. The performance results are detailed below:

The improvement from $M=4$ to $M=5$ and $M=6$ is minimal, with less than a 1% increase in most environments. Given the substantial runtime required for each additional iteration and the limited performance gains, we select $M=4$ as a trade-off between computational efficiency and performance.

3. Question regarding the RWR algorithm.

Thank you for your insightful question. We would like to kindly clarify the following points:

• The key focus of AgentGym is to build generally-capable agents that can self-evolve across various environments. Therefore, we first considered the popular online RL algorithm, PPO. However, current PPO implementations in the LLM field are mainly suited for single-turn conversation. They perform poorly on multi-turn, long-term agent tasks and pose engineering challenges.
• AgentEvol is derived from the Dayan & Hinton's framework[1, 2], and it is an adaptation and application based on the idea of Dayan & Hinton's framework in the LLM-based agent field, designed to enable agents to achieve self-improvement and interactive training across multiple environments and tasks. RWR [3] is also derived from the Dayan & Hinton's framework, but its practical implementation is different from our method. Specifically, in AgentEvol, we set $r=0$ for failed trajectories, and $r=1$ for successful ones (see p.8 Line 416 in the revised version). This means that although AgentEvol is weighted by rewards, only successful trajectories are included in the loss calculation. Failed trajectories are assigned a weight of 0 in the policy optimization process and are excluded from the loss. For the solidness and thoroughness of our experiments, we evaluate the standard RWR baseline in the General Response, point 2.
• Additionally, we highly acknowledge that AWR and AWAC are more advanced advantage-weighted RL algorithms. However, in our task setting, rewards are at the trajectory level, meaning the agent only receives a reward at the end of a complete interaction trajectory. As such, we cannot calculate stepwise advantages to guide optimization. This is why we did not compare against methods like AWR. (Of course, we could adopt a more fine-grained sampling process, starting from intermediate steps and using Monte Carlo to estimate stepwise values. We aim to provide a more detailed discussion in our future work.)

4. Question about "non-negative reward".

Thank you for pointing out this issue. We apologize for the sudden introduction of "non-negative reward". On p.5 Line 251 (revised version), we mentioned the range of the final reward r, but we admit it wasn’t clear enough and could be easily overlooked. Based on your suggestion, we have updated the manuscript accordingly.

We would also like to clarify that, since we aim to optimize across multiple environments and tasks, we have to standardize rewards from different environments to fit within a $[0,1]$ range for stable training. This might differ from traditional RL/agent practices, leading to some misunderstanding.

Thank you for your detailed reading and valuable feedback. We are also grateful for your recognition of the impact and value of our framework. Your suggestions are really helpful for improving our writing, formatting, and overall structure. We will revise the manuscript based on your comments and do our best to address your concerns.

1. Question about the structure of the paper

Thank you for your professional suggestions. We believe they are crucial for improving our writing and the structure of the paper. We have made the following modifications:

• We add extensive experiments and analyses regarding the AgentEvol method, which are now included in the revised Appendix F.
• We shorten the detailed description of the AgentGym framework, removing a substantial amount of engineering-related content from Section 3 and moving it to Appendix D.
• We merge Section 2 and Section 4 to enhance the overall coherence of our paper.
• Additionally, we would like to kindly point out that the AgentGym framework, database (including AgentTraj, AgentTraj-L, AgentEval), and the AgentEvol algorithm are all key components, specifically designed to facilitate the self-improvement of LLM-based agents across multiple environments. This has been explained in the fourth point of the General Response.

We hope these modifications meet your expectations.

2. Question about the evaluation of AgentEvol.

Due to space limitations, our evaluation of the AgentEvol method wasn’t fully comprehensive. We've added several experiments in the first three points of the General Response to better demonstrate its effectiveness.

2. Re: the suggestion of adjustments to organization of the paper contents.

• To achieve our research goal of exploring the self-improvement of LLM-based agents in diverse environments, we do incorporate several components, including the AgentGym framework, dataset, benchmark suite, and self-improvement methods across diverse environments. These components are crucial building blocks for exploring our research question.
• Following your suggestion, we have moved certain descriptions and engineering details to the Appendix and merged Sections 2 and 4 to better organize the paper, making it more focused on the scientific contribution. We will continue to improve our manuscript following your advice.
• In response to your new suggestion, we will make full use of computational resources and conduct additional experiments to further strengthen the solidness of our work.

References

[1] Zeng, Aohan, et al. "Agenttuning: Enabling generalized agent abilities for llms." arXiv preprint arXiv:2310.12823 (2023).

[2] Lai, Hanyu, et al. "AutoWebGLM: A Large Language Model-based Web Navigating Agent." Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2024.

[3] Chen, Zehui, et al. "Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Models." arXiv preprint arXiv:2403.12881 (2024).

7. Question about Figure 1.

Thank you for your valuable feedback. It has been a great encouragement to us. Indeed, the information provided by this figure is almost identical to the caption. However, we hope that Figure 1 is not just for visual appeal.

• On one hand, we aim to help readers by offering a more intuitive understanding of the Self-evolution process through a comic-like illustration, especially those unfamiliar with LLM-based agent reasoning.Different parts of Figure 1 (such as the transition from classroom to wild environments) can help readers better imagine how LLM-based agents learn and progress in various environments.
• On the other hand, the process shown in Figure 1 has been validated in the subsequent experiments of the paper. We use a similar process to train the agent: 1. Learning basic skills through Behavior Cloning, and 2. Continuous exploration and evolution in unseen environments and tasks.
• Finally, since AgentGym framework is a little bit engineering, we also hope this figure provides some enjoyment for readers while going through the paper. Thank you again for your kind suggestion and understanding :-).

Thank you once again for your valuable comments! If you find our clarifications satisfactory, we kindly ask for your consideration in adjusting your scores accordingly.

References

[1] Wooldridge M, Jennings N R. Intelligent agents: Theory and practice[J]. The knowledge engineering review, 1995, 10(2): 115-152.

[2] Goodwin R. Formalizing properties of agents[J]. Journal of Logic and Computation, 1995, 5(6): 763-781.

[3] Xi Z, Chen W, Guo X, et al. The rise and potential of large language model based agents: A survey[J]. arXiv preprint arXiv:2309.07864, 2023.

[4] Wang L, Ma C, Feng X, et al. A survey on large language model based autonomous agents[J]. Frontiers of Computer Science, 2024, 18(6): 186345.

[5] Liu, Xiao, et al. "Agentbench: Evaluating llms as agents." arXiv preprint arXiv:2308.03688 (2023).

[6] Zeng, Aohan, et al. "Agenttuning: Enabling generalized agent abilities for llms." arXiv preprint arXiv:2310.12823 (2023).

[7] Chen, Zehui, et al. "Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Models." arXiv preprint arXiv:2403.12881 (2024).

[8] Zhang, Jianguo, et al. "AgentOhana: Design Unified Data and Training Pipeline for Effective Agent Learning." arXiv preprint arXiv:2402.15506 (2024).

[9] Ma, Chang, et al. "AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents." arXiv preprint arXiv:2401.13178 (2024).

5. Question about exploring various RL approaches.

We have added experiments and a detailed discussion to address your concern in the second point of the General Response. Additionally, in General response point 1, we analyze why AgentEvol performs better in certain environments (e.g., WebShop) and worse in others (e.g., Weather). We hope this analysis helps clarify your concerns.

6. Questions regarding grid world problems.

Yes, we agree that basic grid world problems are essential for LLM-based agents, as these game environments provide an excellent testbed for verifying an agent's understanding of the environment and decision-making abilities.

• When building the AgentGym framework, we did consider this. We introduced a similar environment, called "Maze". It is a maze problem described in text form. The agent can access information about its current position, goal position, and surrounding walls. We provide an example https://anonymous.4open.science/r/AgentGym-ICLR-rebuttal-1B8D/Maze_example.png to help you better understand this task.
• If you have any more specific suggestions, we would be happy to include these environments in future versions of AgentGym to make our framework more complete.

3. Question regarding the reasoning / planning capacity of LLMs.

We fully agree with your point that the reasoning/planning capacity of LLMs is the foundation of LLM-based agents.

• To the best of our knowledge: GSM-Symbolic is an advanced benchmark created based on GSM8K symbolic templates. LLMs' performance tends to be unstable when changing the format or values of the problem. One hypothesis is that LLMs may have learned fixed patterns from the training data through pattern recognition. PlanBench is a time-tested dataset for evaluating LLM planning abilities. Since GPT-3, many models have made little progress on this benchmark, and O1 has not yet met expectations.
• These works have proposed and tested their hypotheses through extensive experiments. In light of the experimental results, we must admit that many current LLMs may not yet meet the standard in terms of reasoning abilities. However, these works also acknowledge that O1 outperforms other models. Which indicates the potential benefits of expanding the exploration space and enhancing test-time scaling.
• At the same time, building such a foundation is a gradual process. Without a very strong LLM as the base model, the agents constructed on it may not perform well. For example, in our experiments with Llama-2-7B. However, through optimization with the AgentEvol method, the reasoning ability of agents on specific tasks can match or even surpass that of originally stronger models.

The key focus of AgentGym is to enhance reasoning/planning abilities through self-evolution across different environments. The OOD experiments we have added in the third point of the General Response also indicate that improvements in reasoning abilities have a degree of generalization.

4. Question about metrics: optimal solutions or success rate?

We kindly provide the following clarifications:

• In the LLM-based agent community, researchers focus more on task success/reward rate because they want the agent to successfully accomplish specific task [5, 6, 9]. For each environment, we list the corresponding metric in p.5 Table 2. $Reward = 1$ is defined as Success.
• However, we fully agree with your point. For an agent, focusing on optimal sequential decision-making is one of the key metrics. This means the agent can maximize long-term returns. Relying solely on success rate may overlook this aspect.
• In our preliminary experiments, we observe that in simpler environments like Webshop, the agent can achieve high rewards. However, it may not reach the final goal, i.e., the success rate remains low. Our experience is that in LLM-based agent tasks, the optimal solution always achieves the highest reward (e.g. success), but higher rewards do not necessarily indicate an optimal solution. This may differ from traditional RL settings.
• At the same time, using success rate as a metric also helps mitigate reward hacking, preventing the agent from performing irrelevant actions to gain high scores.

Thank you very much for your valuable suggestions and questions. We will improve our manuscript based on your feedback.

1. Question regarding the focus of our paper.

Thank you for your valuable question. We believe it's important to clarify the motivation and focus of our paper. The AgentGym framework and the AgentEvol algorithm are both key components, designed to facilitate the self-evolution of LLM agents across multiple environments. We have explained this in the fourth point of the General Response. We hope this helps clarify your concerns.

2. Question about ReAct and approaches to building LLM-based agents.

Thank you for your insightful question. We are glad to discuss this with you.

• First, we acknowledge that ReAct may not be the best reference method for constructing a general-purpose, powerful LLM-based agent. Based on our understanding, ReAct is a reasoning format that guides and facilitates the interaction between the agent and environments.
• In industrial practice, to the best of our knowledge, some representative and successful LLM-based agents include "Computer Use" proposed by Anthropic, as well as GitHub Copilot and Cursor. We do not have access to the details of how these successful applications were constructed. However, we believe that the reasoning format is just one of the key factors behind the success of these agent applications, rather than the only factor. A powerful LLM-based agent is one that has a decision-making core based on a large language model (LLM), which extends its input (e.g., various types of textual input) and actions (e.g., calling APIs, using tools, performing embodied actions, etc.). Such agents require insightful design across multiple aspects and dimensions [1,2,3,4].
• In academic research, with the rise of O1, we have also noticed excellent works on reasoning, such as Quiet-STaR and AlphaZero-like Tree-Search. However, we would like to clarify that these works are decoupled from our AgentGym framework. AgentGym is a unified framework for training, testing, and sampling. Therefore, whether your agent uses a ReAct approach, Quiet-STaR, or MCTS for reasoning, it can integrate closely with our framework. We help test the final performance of the agent and can also use high-quality reasoning data sampled by the agent for subsequent training.
• Notably, the reason we employed the ReAct format in this work is largely to maintain consistency with other works in the LLM-based agent field (e.g., AgentBench, AgentTuning, AgentFlan, AgentOhana, etc.) [5,6,7,8]. This not only facilitates fair comparison but also enables the community to conduct in-depth research and exploration based on this work.
• Finally, we will incorporate more reasoning/planning paradigms into our future work, enhancing the comprehensiveness of our framework.

2. Question about technical details in Sec 3.

To provide a detailed overview of our framework, we indeed describe many details regarding framework architecture, instruction collection, and trajectory generation in several subsections of Section 3. This undoubtedly affects the focus of our paper and the reader's perception. Based on your suggestion, we have made revisions (See Q1).

3. Question about the varying performance in different environments.

Thank you for your insightful question. We believe your suggestion is very important for our paper. In the first point of the General Response, we provide a more detailed discussion and attempt to analyze the underlying reasons for this phenomenon with experiments.

Thank you once again for your valuable comments! If you find our clarifications satisfactory, we kindly ask for your consideration in adjusting your scores accordingly.

References

[1] Wooldridge M, Jennings N R. Intelligent agents: Theory and practice[J]. The knowledge engineering review, 1995, 10(2): 115-152.

[2] Goodwin R. Formalizing properties of agents[J]. Journal of Logic and Computation, 1995, 5(6): 763-781.

[3] Xi Z, Chen W, Guo X, et al. The rise and potential of large language model based agents: A survey[J]. arXiv preprint arXiv:2309.07864, 2023.

[4] Wang L, Ma C, Feng X, et al. A survey on large language model based autonomous agents[J]. Frontiers of Computer Science, 2024, 18(6): 186345.

[5] Yao S, Zhao J, Yu D, et al. React: Synergizing reasoning and acting in language models. ICLR 2023.

[6] Sumers T R, Yao S, Narasimhan K, et al. Cognitive architectures for language agents. TMLR 2024.

[7] Zhou Y, Zanette A, Pan J, et al. Archer: Training language model agents via hierarchical multi-turn rl. ICML 2024.

Thank you very much for your valuable suggestions and questions. We hope our response addresses your concerns:

1. Question regarding the starting point of AgentGym framework and environments.

Thank you for your valuable feedback. Your kind suggestions regarding the writing of our paper are very important to us. We will incorporate these changes into the new manuscript.

To address your feedback, we have added a discussion on "The choice of environments and tasks in AgentGym" in Section 3 to clarify our starting point. Additionally, we have reduced some of the technical details and moved them to Appendix D.

The starting point of designing the AgentGym framework? The key research question we explore in this paper is: How can we build generally-capable agents that can handle multi-turn decision-making tasks across various interactive environments? We have elaborated on this in the fifth point of the General Response; please refer to the discussion there.

Why do we select these environments? We provide an explanation from two perspectives:

• Firstly, from the definition, an LLM-based agent is an agent with a decision-making core based on a large language model (LLM), extending its input (e.g., different types of textual input) and action (e.g., calling APIs, using tools, performing embodied actions, etc.) [1, 2, 3, 4]. Therefore, the capabilities required for an LLM-based agent include [3, 4, 5]:

  • Input Side: The agent needs to have the ability to receive different types of textual input or observations, such as plain text, HTML, code, etc. Therefore, the environments and tasks we use need to include these different inputs. Thus, we include some textual environments (e.g., ALF, TC), some web environments (e.g., WS, WA), and code environments (e.g., BD).
  • Decision-Making Side: Centered around the LLM, we need to improve/evaluate the agent's basic knowledge, abilities in reasoning, planning, and understanding environmental information. Therefore, the tasks we use need to be challenging, requiring the LLM to complete them through reasoning and planning. For example, in the WS task, when faced with a web page containing many products, an LLM-based agent needs to guide its next actions through reasoning and information extraction. In the Sci task, when facing observations in different rooms, the LLM-based agent needs to read manuals, find raw materials, and synthesize items, which often requires basic common knowledge, strong planning and environmental information filtering and understanding abilities.
  • Action Side: We need to improve/evaluate the agent's ability to output different forms of actions, such as plain text, tool and API calls, code, embodied actions, etc. Therefore, the environment needs to be influenced by different types of actions. For example, WT and MV tool-calling environments provide many APIs for the agent to call; the DB environment requires the output of SQL code; BabyAI and ALFWorld require some embodied actions, while WD requires plain text output.
  • In fact, many environments require diverse capabilities, spanning the input, decision-making, and action sides.

• From another perspective, in the field of NLP, many researchers believe that an LLM-based agent is not for simple chat or single-turn QA, but for complex, long-term, multi-turn sequential decision-making tasks through interaction with the environment. These tasks are also referred to as agent tasks by researchers in the LLM/NLP field [3, 4, 5, 6, 7]. Therefore, in this context, the environments and tasks we need to build in AgentGym must have these characteristics. Specifically, the tasks we include generally require multiple rounds of interaction with the environment to complete, and the context length is usually longer than that of LLM's Q&A tasks or reasoning tasks (because it needs to include instructions, historical information, and current observations). From this perspective, AgentGym is an environment framework specifically designed for LLM-based agents.

Thank you for your kind question and suggestion. Our clarifications are as follows:

1. Re: Clarification on our key research question

Thank you very much for your feedback. It seems that our previous statements may have been somewhat too broad, leading to potential misunderstandings.

• To clarify, we narrow our research question to exploring how to develop LLM-based agents capable of self-evolution across multiple and diverse environments. This is why we include the AgentGym framework, trajectory sets, benchmark suites, and AgentEvol in the paper. We do not intend to tackle the broader challenges of AGI or strong AI, but instead aim to advance the capabilities of intelligent agents within diverse environments.
• Based on your feedback, we will narrow and clarify the scope of our study in the manuscript.

2. Re: "Validity of Environments and Tasks"

"Why were the selected environments and tasks chosen as the basis for evaluation?"

• In our response to "Q1: regarding the starting point of the AgentGym framework and environments," we have clarified why we select these environments. We consider two main aspects: 1. The definition of an LLM-based agent, including the input side, decision-making side, and action side [1,2,3]. 2. The capabilities that the LLM-based agent community is concerned with, e.g. handling complex, long-term, multi-turn sequential decision-making tasks through interaction with the environment.
• In addition, following your suggestion, we have also detailed our thoughts on this question and the related technical features in the revised Section 2.1 of the paper.
• We hope our response addresses your concerns. Please let us know if you have any further questions.

"Do these environments adequately demonstrate the agents’ ability to generalize across various interactive environments? If the chosen tasks are narrow or overly specific, it may undermine the claim of generalizability."

• We have carefully collected a diverse range of environments from the LLM-based agent field, including categories such as web navigating, house-holding, tool-using, programming, text games, and digital games, in order to avoid being "narrow or overly specific". Additionally, compared to related works, we have included a greater number and variety of tasks [4,5,6]. For more detailed information, please refer to Table 1 on p.3 (comparison with other agent frameworks). These environments are sufficient to demonstrate the multi-tasking capabilities of the agents. We expect to include more tasks and environments in the future, as this is a rapidly evolving field, and new challenges continually emerge.

3. Re: "Comparison with Existing Methods"

Thank you for your insightful comment. To clarify the fundamental differentiation of AgentEvol from existing approaches, we highlight the following points:

• AgentEvol enables LLM-based agents to perform exploration and learning across multiple environments.
• Existing LLM-based agent methods typically rely on imitating expert trajectories, such as behavioral cloning. These methods are often task-specific and lack exploration [5, 6, 7]. Additionally, some approaches allow agents to improve through environmental feedback (i.e., self-improvement), but these methods are typically constrained to specific environments, limiting their performance across multiple and diverse environments [8, 9].

4. Re: Question about the data used for an LLM-based agent

"While the concept of self-evolution is highlighted, it appears to focus on collecting more data."

• Self-evolution can be described as self-improvement, which focuses on enhancing the capabilities of LLM-based agents. The goal is to train more capable agents by collecting more and higher-quality data through interaction with various environments. This aligns with the technical approach of many works in the LLM field, such as mathematical reasoning [10, 11, 12], code generation [13, 14, 15], alignment [16, 17], tool using [18].

If the solution depends heavily on data collection, why do current LLMs, despite access to vast datasets, still struggle in these scenarios?

• Although current LLMs have access to large amounts of data, the data is not highly relevant or of high quality for LLM-based agent tasks, which causes them to struggle in these environments. For example, pre-training uses terabyte-scale corpora, such as Common Crawl, C4, Github, Wikipedia, StackExchange, etc. [19, 20, 21], which provides the model with basic language understanding and generation abilities. However, the performance of these base models on agent tasks is moderate, as demonstrated by Llama2-Chat in our main results.

5. Re: Question about overly task-specific

Thank you for your feedback. In the following, we will carefully respond to your question.

• Our goal is to develop LLM-based agents capable of self-evolution across multiple and diverse environments (e.g., web shopping and household), instead of a broader problem of AGI or strong AI. We apologize for the misunderstanding and will clarify this point in the revised version of the paper.
• To achieve this, LLMs provide a solid backbone with prior knowledge and general capabilities to some extent [1, 2, 3]. This foundation enables the agent to perform effectively across multi-task environments.
• Moreover, we have made considerable efforts to collect a diverse set of environments within the framework. Our results demonstrate that AgentEvol shows promising performance across these environments. Furthermore, it shows generalization capabilities on unseen tasks and environments, as detailed in the third point of our General Response. This highlights our approach is not overly task-specific.

6. Re: Recommendation for narrowing the scope

Thank you for your valuable suggestion. We acknowledge that the statements of our key research question may have caused some confusion. To clarify, we have narrowed our focus and now explicitly frame the research question as: "How to develop LLM-based agents capable of self-evolution across diverse environments." We include AgentGym framework, dataset, benchmark suite, and self-improvement methods across diverse environments for the research question.

We will revise the paper to provide a clearer and more focused description of the research question. Thank you again for your helpful feedback.

3. Question regarding overfitting risk.

Thank you for your professional question. In our early experiments, we did observe overfitting. We apologize for not providing a detailed discussion in the paper. We have added the corresponding experimental results below, and we hope this enhances the rigor of our paper and addresses your concerns.

In the AgentEvol framework, overfitting is mainly caused by two potential factors: the continuously iterating model and the growing training dataset.

• Data: We show the ablation results of different data merging strategies in Figure 3 on p.9. The experiment results indicate that accumulating training data generated in each iteration can lead to performance degradation after several iterations. Therefore, we choose to merge the generated data from each iteration with the initial training set. This approach ensures the model's learning of basic skills while also allowing it to learn higher-quality, higher-reward trajectory data over iterations.
• Model: In our experimental setup, we optimize the initial agent at each iteration to mitigate overfitting. We add further experiments and analysis to the third point of clarifications of the General Response. The results are presented https://anonymous.4open.science/r/AgentGym-ICLR-rebuttal-1B8D/Ablation_study_overfit.png.

In summary, we have provided a deeper discussion on overfitting risk from both the data and model perspectives. We hope these analyses clarify your concerns.

4. Question about safety issues during AgentEvol.

We try to address your question from the following perspectives:

• Data Preprocessing: When constructing the AgentTraj dataset, we conduct strict data preprocessing. During the instruction and trajectory collection process, we check all instructions and responses for harmful or sensitive tokens, preventing harmful behavior from the source of the training data.
• Data Postprocessing: In designing the various environments in AgentGym, we add corresponding rules. Harmful responses are assigned lower rewards, and these responses are discarded during the iterations of AgentEvol.
• Agent Environments: Currently, all environments in AgentGym are offline (except for the Tool environment, which requires real-world APIs). Therefore, the agent’s interactions take place in a sandbox, preventing any real-world harm.
• Preference Alignment Strategy: For future work, we aim to extend AgentGym to real-world environments. For example, using OpenTable for restaurant bookings. In such settings, we will not only consider data preprocessing and postprocessing but also plan to introduce preference alignment techniques based on human feedback to reduce harmful behavior (e.g. RLHF, DPO) [1,2].

Thank you once again for your valuable comments! We hope our responses address your concerns and enhance your confidence in our paper.

References

[1] Yang, Zonghan, et al. "Towards Unified Alignment Between Agents, Humans, and Environment." ICML 2024.

[2] Yuan, Tongxin, et al. "R-judge: Benchmarking safety risk awareness for llm agents." arXiv preprint arXiv:2401.10019 (2024).

Thank you very much for recognizing the novelty, contributions, and writing of our work. This has greatly encouraged us. We hope our response addresses your concerns:

1. Question regarding the scalability of AGENTEVOL.

Our analysis is from two perspectives:

• Performance Improvement with Larger Models

  • In Section 4.3, Table 5, we conduct experiments with models of different scales (DeepSeek-Coder-1.3B and Llama-2-13B), and the results are visualized https://anonymous.4open.science/r/AgentGym-ICLR-rebuttal-1B8D/Scalability_of_AgentEvol.png to better illustrate the observed scaling trends. AgentEvol demonstrates significantly improved performance as the model size increases, although we do not perform fine-grained hyper-parameter searching. This indicates that AgentEvol can leverage larger models to achieve further performance gains.

• Analysis of Computational Costs

  • As shown in the right Figure (accessible https://anonymous.4open.science/r/AgentGym-ICLR-rebuttal-1B8D/Scalability_of_AgentEvol.png), we also analyze the inference time for models of varying scales. While the inference time does increase with model size, larger models exhibit higher interaction efficiency, resulting in only a marginal increase in overall inference time. This suggests that AgentEvol can scale to larger models while maintaining an acceptable computational costs.

We appreciate your insightful comments, which have helped us refine our discussion on scalability.

2. Question about comparisons with other exploration-based methods.

We apologize for the brief analysis of other baseline methods. We have added related experiments, training curves, case studies, and in-depth analysis in the second point of the General Response. We hope this meets your satisfaction.

2. Re: Comparison between AgentEvol and other RL / exploration-based methods.

Thank you for your insightful questions. We have added extra RL baselines, Reward Weighted Regression (RWR) algorithm, and provided deeper analysis with case studies and training reward curves. We hope these additional discussions will strengthen the validation of AgentEvol's effectiveness.

• Regarding RWR baseline: Following Reviewer jPpZ’s suggestion, we discuss the differences in implementation details between AgentEvol and RWR [1]. We believe it is valuable to include this classic offline RL algorithm as a baseline. Additional experiments have been added below.

• Regarding DPO baseline: We would like to kindly emphasize that DPO is also a well-known offline alternative to RL in the field of LLMs [2, 3]. The experimental results are provided in p.10, Table 4. We have also added the implementation details of DPO in Appendix E of the revised manuscript.

• Following the suggestions of Reviewer ZWi2 and Reviewer PSRy, we have conducted a detailed comparison of AgentEvol with other RL methods and exploration-based methods. All statistical results are provided below and also included in the revised Appendix F.5.

  • Selection of baselines: Our evaluation is comprehensive and sufficient, including Prompt-based, BC, offline-RL, and online-RL methods. Experimental results demonstrate that AgentEvol achieves superior performance across various tasks when compared with representative algorithms.

  • Training cost: The training costs for $BC_{large}$, RWR, and AgentEvol are set as baselines, as they all optimize the policy in a behavior cloning manner. In contrast, DPO and PPO methods have significantly higher training costs. DPO requires loading both the actor and reference models and computing probability distributions for chosen and rejected responses. PPO, being an online RL method, involves sampling and policy optimization simultaneously, leading to more intensive training times.

    Method	Type	Avg. Training Cost	Accuracy (WS)	Accuracy (ALF)	Accuracy (Baby)
    LLM-Planner [4]	prompt-based	/	18.9	68.5	82.5
    $BC_{large}$	behavior cloning	1 ×	73.5	83.0	74.2
    RWR [1]	offline-RL	1 ×	68.0	76.5	82.1
    DPO [2]	offline-RL	4.3 ×	75.0	86.5	78.3
    PPO [5]	online-RL	15 ×	68.0	83.5	69.8
    AGENTEVOL	offline-RL	1 ×	76.5	88.0	82.7

  • Learning stability: The results are presented https://anonymous.4open.science/r/AgentGym-ICLR-rebuttal-1B8D/Learning_stability.png. For consistency, we set the smallest unit of the x-axis for training time as an epoch. It is clear that algorithms optimized with BC objectives are more stable in performance improvements, leading to faster convergence. While DPO shows significant improvement in the early stages, overfitting occurs quickly as training progresses. PPO, on the other hand, exhibits noticeable instability throughout the training process, with no clear learning trend during the same number of epochs.

  • Case Study: We compare the performance of several RL baselines when executing the same task in the WebShop environment. The results are presented https://anonymous.4open.science/r/AgentGym-ICLR-rebuttal-1B8D/Case_rl.pdf. The RWR and DPO baselines lead to the selection of the first item without considering the price constraint, resulting in task failure. In comparison, the PPO baseline continuously clicks "Next Page" without effectively extracting relevant information from the environment, also failing to find a suitable item. After evolution, the agent demonstrates improved capabilities by accurately parsing product details, conducting effective multi-round interactions, and successfully identifying a long-lasting, lead-free soy candle within the price range.

4. Re: Question regarding the focus of our paper.

To address the reviewers' concerns, we would like to clarify the starting point and motivation of our paper:

• In this paper, the key issue we explore is: How can we build generally-capable agents that can handle multi-turn decision-making tasks across various interactive environments? Our goal is to train LLM-based agents that can explore and evolve themselves across diverse environments.
• To achieve this, we identify three key elements (which we detail in the abstract and introduction). 1. diverse interactive environments for agent exploration, 2. a trajectory set to equip agents with basic capabilities and prior knowledge, and 3. an effective and scalable approach for agent improvement across environments. Specifically:
  • The AgentGym framework implements the first element, while the AgentTraj dataset represents the second element.
  • Based on these elements, we continue to explore the potential of training self-evolving agents and demonstrate the effectiveness of AgentEvol through experiments. This constitutes the third element.
• Based on Reviewer VNyh's feedback, we explain why we select these environments in AgentGym. If you are interested, please refer to Q1 in response to Reviewer VNyh.
• Following the suggestions of Reviewer VNyh and Reviewer jPpZ, we have revised the writing and format of our paper.

References

[1] Peters, Jan, and Stefan Schaal. "Reinforcement learning by reward-weighted regression for operational space control." Proceedings of the 24th international conference on Machine learning. 2007.

[2] Rafailov, Rafael, et al. "Direct preference optimization: Your language model is secretly a reward model." Advances in Neural Information Processing Systems 36 (2024).

[3] Song, Yifan, et al. "Trial and error: Exploration-based trajectory optimization for llm agents." arXiv preprint arXiv:2403.02502 (2024).

[4] Song, Chan Hee, et al. "Llm-planner: Few-shot grounded planning for embodied agents with large language models." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[5] Ouyang, Long, et al. "Training language models to follow instructions with human feedback." Advances in neural information processing systems 35 (2022): 27730-27744.

[6] Zheng, Rui, et al. "Secrets of rlhf in large language models part i: Ppo." arXiv preprint arXiv:2307.04964 (2023).

[7] Trung, Luong, et al. "Reft: Reasoning with reinforced fine-tuning." Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

3. Re: The lack of thorough evaluation of AgentEvol.

Thank you for your valuable suggestions! We have added some additional experiments regarding AgentEvol

1. Adding experiments of evaluation on OOD tasks and environments.

   • To explore performance on tasks or environments not seen during the evolution phase (i.e. OOD tasks), we conduct experiments. In these experiments, the task types and settings in ALF and Baby are unseen by the agent during both the BC and AgentEvol phases, and the entire environments of Tool-Academia(AM) and Tool-Sheet(ST) are entirely new for the agent. The results are presented below, showing that our method demonstrates stronger generalization ability on unseen tasks and environments compared to other methods.

     Method	ALF-OOD	Baby-OOD	AM	ST
     Llama2-Chat-7B	0.0	2.2	0.0	0.0
     AgentLM-7B	57.7	4.4	10.0	14.3
     $BC_{base}$	60.8	6.2	20.0	24.3
     $BC_{large}$	64.9	6.1	20.0	25.2
     AGENTEVOL	67.5	6.2	25.0	26.2

2. Adding experiments about the performance of task-specific BC and task-specific AgentEvol.

   • We have provided a detailed discussion in the first point of the General Response. The results are presented in the first point of the General Response . Here, we mainly highlight the results of the BC and AgentEvol methods running in isolated environments.

3. Adding experiments regarding the choice of base model.

   • Following Reviewer ZWi2's advice, we examine the overfitting risk during our AgentEvol process. Specifically, we conduct experiments comparing two strategies: fine-tuning the initial agent and fine-tuning the agent from the last iteration. As shown in Figure 6 in the Appendix F.2 (also accessible https://anonymous.4open.science/r/AgentGym-ICLR-rebuttal-1B8D/Ablation_study_overfit.png), re-optimizing the initial agent mitigates overfitting and ensures more stable performance across diverse environments compared to continuous fine-tuning. Therefore, we select the initial agent as the base model in each iteration of self-evolution.

[Continuation of 2]

• Additionally, we provide the Mean Training Reward curve for each environment following Reviewer jPpZ's advice. The results are presented https://anonymous.4open.science/r/AgentGym-ICLR-rebuttal-1B8D/Mean_training_reward.png. For WebShop and ALFWorld, we present the results for 10 epochs; for BabyAI, we present the results for 8 epochs. We also observe that PPO encounters instability and fluctuations in training rewards. This could be due to the standard PPO algorithm, which only uses outcome-based rewards and struggles with optimizing sparse, long-term, and multi-turn trajectories, limiting the model’s exploration and learning. In the early stages of training, rewards encounter a noticeable drop when the critic model is not fully optimized. This phenomenon is consistent with the performance of LLMs in the standard RLHF pipeline [5, 6, 7]. Full training details are included in Appendix E of the revised manuscript.