AgentTuning: Enabling Generalized Agent Abilities for LLMs

Thank you for your insightful comments and constructive feedback on our paper. Your questions have provided us with an opportunity to clarify certain aspects of our work, which we hope will enhance the overall quality and understanding of our research.

Weakness 1: While the agent trajectories are very valuable and costly to collect, they are mainly extracted from public tasks/benchmarks by using ReAct with GPT models. The overall process with instruction generation, trajectory collection, and filtering can be useful for other data collection efforts but they are relatively straightforward. For example, ReAct/COT prompting is used with no significant change, reward filtering is also a standard practice in imitation learning. I suggest highlighting main challenges and how significant they are in addition to remedies that you introduced.

We appreciate your insights regarding the perceived straightforward nature of our methodology. However, we would like to emphasize some unique aspects and significant contributions of our work:

• Limitations of prior works: Existing studies on LLMs as agents have thus far largely focused on designing prompts or a framework for completing one particular agent task [1][2], rather than fundamentally enhancing the agent capabilities of the LLMs themselves. In addition, many efforts are dedicated to improving LLMs in specific aspects, involving fine-tuning the LLMs using datasets tailored to specific tasks [1][3]. This overemphasis on specialized capabilities comes at the expense of the LLMs’ general abilities and also compromises their generalizability.
• First General Agent Model to Match GPT-3.5-turbo: As illustrated in Figure 1(b), there is a substantial gap between open-source and API-based models like GPT-3.5 and GPT-4. Our work introduces the first open-source model, AgentLlama-70B, which matches the capabilities of GPT-3.5-turbo in general (unseen) agent tasks while preserving its general abilites. We believe this makes a significant contribution both academically and in the development of downstream applications.
• First attempt to instruct-tune LLMs with agent trajectories across multiple tasks: To the best of our knowledge, we are the first to use agent trajectories from multiple tasks for instruction tuning, achieving performance comparable to GPT-3.5-turbo. Furthermore, our error analysis (Cf Sec 3.3) shows that the lack of alignment is the main issue with the agent capabilities of models, which can be greatly improved with minimal data training. We believe this will provide insights into how to enhance the inherent agent capabilities of models.
• Automated and Scalable Trajectory Collection Methodology: Given that agent trajectories are precious and challenging to collect, we have established an automated and scalable method for collecting agent trajectories and have verified its effectiveness. For the Instruction Generation process, we used two methods, Self-Instruct and Task Derivation, to supplement tasks lacking training instructions. This ensures that we have sufficiently diverse agent trajectories. In the Trajectory Collection process, we interact using GPT-4 and aggressively apply a filtering of $r=1$ to almost all trajectories, ensuring we have high-quality agent trajectories. Both of these steps require no manual intervention and can be easily extended to other agent tasks. By fine-tuning the model on 1,866 trajectories from 6 held-in tasks, there was a significant improvement in performance on 6 held-out (unseen) Agent tasks, demonstrating the effectiveness of our method.

Regarding your concerns about the innovativeness of using the ReACT/CoT prompt method. We wish to emphasize that while previous works [4][5] have indeed innovated in prompting methods to enhance the performance of API-based models, they have not focused on improving the agent capabilities of the models themselves, which is the main focus of our paper. Considering the fact that open-source models are far behind API models in agent capabilities, we have for the first time elevated the general-purpose agent capability of an open-source model to the level of GPT-3.5-turbo, even just by using the common ReACT prompt method. We consider this to be a significant contribution. Of course, we believe that more advanced prompting methods are very worthwhile to research, and we intend to make this our next direction of exploration.

Thank you for the detailed response and additional experiments. It helped clarify many of my concerns. I raised my score accordingly.

I would like to further discuss my point about weakness 2. I am curious about just fine-tuning Llama on all "originally available training data that a task has". For example, WebAgent [1] uses all available trajectories from WebShop to fine-tune a T5 model. It is not a generalist agent and it doesn't utilize CoT rationales but it can use all available training data. I think the problem stems from using GPT to collect interaction trajectories; since GPT can lead to unsuccessful trajectories, without filtering them you would be training on unsuccessful trajectories; which could also be why you get worse result without filtering. An alternative would be using GPT to supplement every trajectory that a task originally has with thoughts using ReAct prompting, instead of collecting trajectories using GPT. My understanding is that you do this only for trajectories generated using task derivation but please clarify if I am mistaken.

One additional thing about training corpus for AgentLlama-7B (w/o CoT). Did you use the same set of demonstrations as in AgentLlama-7B but just removed CoT rationales?

[1] A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis, Gur et al. https://arxiv.org/abs/2307.12856

Overall, my main concern is that the improvements might have come from better instruction following, not agent reasoning.

In our work on AgentTuning, our primary goal was to enhance the performance of LLMs in generalized, unseen agent tasks. Through our experiments, we observed a notable improvement in the instruction-following capabilities, as demonstrated in Figure 3(a). This enhancement in instruction following for the Llama-2-chat models, was significant in reducing simple errors that are often lacking in many open-source models. We believe that this improvement in instruction following, which is essential for the successful completion of agent tasks, stands as a significant contribution of our work.

Regarding agent reasoning, we acknowledge its importance in the interaction of LLMs with real world scenarios. However, we think agent reasoning is hard to precisely evaluated. As observed the held-out tasks, AgentTuning has enhanced the generalized agent performance of LLMs on unseen tasks. We welcome this opportunity for an open discussion to further explore and clarify this concept.

Figure 1 (a), Where does the overall score stand against GPT-4, from which you collected training data?

Our main results in paper show both GPT-4's overall score in both held-in/held-out tasks (Cf Tab. 4). As we choose Llama 2 as our base model, which is a strong open-source model with capabilities comparable to GPT-3.5-turbo in general domains, but there is a huge gap between it and GPT-3.5-turbo in agent tasks. We believe it's possible to surpass GPT-4 using open-sourced LLMs in a specific agent task. Howerver, as our focus is on general agent capabilities, we think that using GPT-3.5-turbo as a baseline for comparison of agent capabilities is reasonable.

Question 1: How does AlfWorld results in table 4 compare to [1], which reported higher score than the highlighted best open-source model?

The ALFWorld evalutaion setting in [1] is different from the evalution setting used in AgentTuning (which follows the setting in an [2]), so the evaluation scores are not directly comparable. Also, the model in [1] used task specific fine-tuning from a pre-trained model and can only solve ALFWorld task. That model probably has no capability to follow general instructions and cannot perform generalized agent tasks. This also makes this model unsuitale for a direct comparason.

References

[1] Micheli, Vincent, and François Fleuret. "Language models are few-shot butlers." arXiv preprint arXiv:2104.07972 (2021).

[2] Liu, Xiao, et al. "Agentbench: Evaluating llms as agents." arXiv preprint arXiv:2308.03688 (2023).

[3] Wang, Guanzhi, et al. "Voyager: An open-ended embodied agent with large language models." arXiv preprint arXiv:2305.16291 (2023).

We would like to express our sincere gratitude for your insightful comments and constructive criticism regarding our paper. Your feedback is invaluable in enhancing the quality and clarity of our research. We understand your concerns and are happy to provide further explanations as follows:

Weakness 1: Figure 1 (b), I don't think the message is fair for this figure, since you trained on AgentBench (although partly), but the other LLMs have not trained on AgentBench. One of the down-sides for open-source LLMs is the ability to generalize to `different' settings from training, but the proposed work has essentially made AgentBench in-distribution by training.

Firstly, it is important to emphasize that AgentBench was not the primary benchmark used to validate the agent capabilities of our model. The purpose of Figure 1 (b) was primarily to illustrate the performance gap between open-source models and API-based models.

Moreover, we have conducted systematic evaluation on 6 held-out agent tasks, which was not included in the training set. This demonstrate the robust generalization performance of our models. The results from these tests, as detailed in our paper (Cf Sec. 3.2), show that our model exhibits strong agent capabilities on held-out agent tasks.

Weakness 2: It seems that GPT models are heavily relied on for generating training data. Do we have some sense of how to go beyond GPT models? Suppose we want to push the boundaries of GPT-4, then GPT-4 data may not work as well.

The use of GPT-4 generated data in our experiments was a strategic choice, primarily for its convenience of data generation and the quality of the trajectories.

We believe that AgentTuning can serve as a foundational method that can be readily combined with other data-centric approaches to foster improvements that are not dependent on the external models like GPT-4. For instance, a key step in AgentTuning is trajectory filtering, which rejects trajectories with poor performace with environment reward. Self-instruct may be combined with trajectory filtering to perform AgentTuning, where a model can be iteratively self-improving, like [1], while preserving general abilities through hybird instruction tuning.

Weakness 3: I have concerns for generalization to other agent tasks that are distinct, but not captured in agent-bench. For example, driving or operating Minecraft.

We deeply appreciate your feedback concerning our agent's generalization ability. Your insights are crucial in ensuring the robustness of our research. In fact, generalization ability is one of our main goals, and we carried out extensive experiments to show that our models hold strong generalization ability.

As described in our evalution setup (Cf Sec. 3.1), we conducted experiments on 6 held-out tasks that are not covered in our training set, with 5 of them not from AgentBench. The tasks vary greatly in their respective domains. Specifically, the 6 tasks cover tasks of digital card games, daily computer tasks, science experiments, web interaction, wiki retrieval and question answering. As shown in Section 3.2 Main Results, our models achieve huge improvement on these held-out tasks despite they have never seen them before, with our 70B model getting +176% improvement in the overall score of held-out tasks, which nearly matches the score of GPT-3.5. This clearly shows that our model can generalize to other unseen agent tasks well.

Playing Minecraft is a rather hard task that even GPT-3.5 performs poorly according to [3]. They carried out experiments and found out that GPT-3.5 and open-source LLMs cannot deal with playing Minecraft well. In their experiments, substituting GPT-4 with GPT-3.5 in code generation leads to a sharp performance degradation, with over 80% drop in number of unique items obtained.

Besides, [3] utilizes a set of complicated prompts to enable GPT-4 to play the game effectively. However, our work focuses on improving the inherent agent ability of models, which will naturally benefit from improvements on prompting methods.

Therefore, in current stage, we did not test our models on these highly difficult tasks. Instead, we leave further enhancing the agent ability of LLMs and finally enabling LLMs to tackle these hard tasks as a future working direction.

Thank you for your valuable feedback and insightful questions regarding our manuscript. We appreciate the opportunity to clarify the methodology used in our study.

Question 1: The authors claim that they use a reward r to filter out low-quality trajectories, but how this reward is calculated/generated is unclear to me. Besides, as the trajectories are generated by GPT3.5/4, I think the trajectories may contain some misleading information. I wonder whether there is a human evaluation of the correctness of the trajectories.

Clarification on Reward Calculation As we adopt 6 held-in agent tasks from AgentBench [1], the detailed reward calculation formula for each task could be found in appendix of AgentBench. Reward represents the completion status of a task, with values ranging between $[0, 1]$, where 1 indicates that the task is fully completed. For a quick clarification, we have summarized 6 held-in tasks as a table below:

Human Evaluation and Dataset Verification We fully understand your concern about the correctness of the trajectories, and we will explain the effectiveness of our data construction method without the need for manual intervention, as well as how we ensure the quality of the data. Since we have used trajectory filtering with $r=1$ for most tasks (except for MindWeb due to its difficulty), we can ensure the correctness of the trajectories in these tasks. Combining the Self-Instruct and Task Derivation methods for Instruction Generation, our data construction process does not require manual participation, making our method scalable and easily extendable to other agent tasks. Table 2 in the paper demonstrates that using trajectory filtering to eliminate incorrect trajectories significantly enhances the model's capabilities in Agent tasks. As for using GPT-4 to generate the Chain of Thought (CoT) reasoning process, its effectiveness has been verified by many works [2][3], and we have also conducted a set of comparative experiments showing that training with the CoT process yields better performance for the model.

Question 2: It leverages a hybrid instruction-tuning strategy, which is a simple multi-task training in essence. Based on my understanding, improving the agent ability while preserving the general language understanding ability is close to a continual learning scenario. So I think it is important to discuss some typical training strategies like weight regularization[1] and parameter allocation[2] used in continual learning. And it would be better if you can further provide some results when such strategies are applied to see whether these strategies can benefit training.

Thank you very much for your valuable suggestion regarding the discussion of typical continual learning strategies with AgentTuning methodology. We certainly agree that discussing and comparing our approach with established continual learning methods like weight regularization and parameter allocation would enrich the paper. However, there are specific challenges in directly applying many continual learning techniques to the fine-tuning of large language models (LLMs).

For example, Elastic Weight Consolidation (EWC) necessitates access to the original supervised fine-tuning (SFT) data from the first task to estimate Fisher information for each parameter. Similarly, Hard Attention to the Task (HAT) requires the original SFT data to learn a task embedding with the original SFT data and modifications to the model architecture, such as adding gates between layers. However, the fine-tuning data for many open-source LLMs, like the Llama series, is not typically available. (Still, it's not totally impossible to combine these methods. For EWC, the fisher information of each parameter could be estimated with extra general task data like shareGPT. For HAT, the task embedding may also be esitmated through extra data also.)

Another challenge the of implementing EWC/HAT entails the techniques used to train language models with huge parameters, like model/pipeline parallelism, distributed optimizers, etc. It requires huge modifications to Megatron training infrastructures (for example, reducing over the square of gradients, and accessing all parameters for EWC loss terms calculation) and would likely necessitate considerable effort in tuning method-specific hyper-parameters.

Understanding the value of your suggestion to include discussions and comparisons with continual learning methods, we propose a different approach. Instead of integrating EWC and HAT methods, we plan to explore LoRA and P-Tuning, both of which are prevalent and parameter-efficient fine-tuning methods in the realm of LLMs and are frequently applied in the continual learning of LLMs. LoRA learns a low rank delta for the parameter matrices, while P-Tuning introduces an additional prefix in each layer. We will conduct experiments with these methods and include a comprehensive and detailed discussion in the Appendix. We are truly grateful for your constructive feedback and are confident that it will significantly enhance the depth and rigor of our study.