FireAct: Toward Language Agent Finetuning

Thank you for your detailed feedback and constructive criticism. We would like to address your concerns as follows:

On the Contribution of FiReAct: While we acknowledge that the concept of data augmentation and multi-task fine-tuning is not novel in itself, the unique contribution of FiReAct lies in its application to language agent fine-tuning. We are among the first to systematically explore how agents can acquire generalizable skills through carefully designed fine-tuning trajectories. Our findings offer preliminary yet promising insights into the potential of language agent fine-tuning, paving the way for future research in this area.

Regarding Experiment Design and Conclusions: We understand your concern about the apparent straightforwardness of some conclusions, such as the comparison between full-model training and LoRA. However, these results are crucial for practitioners in the field, providing concrete, quantitative insights into the performance-efficiency tradeoff in agent fine-tuning. For instance, our findings on the comparative efficiency of LoRA versus full-model training on different hardware setups (detailed in Appendix A.5) are valuable for those implementing these methods in practice.

On the Choice of Datasets: We have expanded our experiments to include tasks beyond QA, as detailed in our General Response. These additional tasks encompass a broader range of real-world scenarios, further validating the effectiveness of our approach.

Concerning the Detailedness of the Appendix: We appreciate your feedback on the balance between the main paper and the appendix. We invite specific suggestions on which experimental results you find less insightful and which fine-tuning details you believe are crucial but missing. Such feedback will be invaluable in refining our paper.

Comparison with Retrieval-Based Methods: Your question regarding the comparison between fine-tuning-based and retrieval-based methods is insightful. It's important to note that the concept of an "agent" in our context goes beyond mere retrieval; it encompasses reasoning and tool usage in a broader sense. Our approach, therefore, offers a more comprehensive improvement of agent capabilities, as evidenced by our inclusion of tasks like Operating System, Database, and Web Browsing from AgentBench mentioned in General Response.

Content of Appendix A.2: Appendix A.2 presents a detailed comparison of results across different tasks and methods, focusing on robustness analysis. We have made efforts to enhance the clarity of this section to avoid any ambiguity.

We appreciate your insightful observations and questions, which allow us to further clarify the contributions of our work.

On the Novelty of FiReAct: We acknowledge the existence of concurrent works such as those by Ho et al. (2022) and Qin et al. (2023), which we have duly cited and discussed in our related work section. FiReAct, however, distinguishes itself by exploring the multi-task, multi-method fine-tuning approach for language agents. Our work delves into new dimensions such as generalization to novel tasks, robustness against noise, and efficiency in agent operation, thereby contributing unique insights to the field.

Experiments on Full Fine-Tuning: Our experimental setup primarily utilized LoRA fine-tuning due to computational constraints. However, we posit that our key findings, particularly regarding the enhanced robustness of fine-tuned agents to tool noise and the efficiency benefits during inference, would likely remain valid in a full fine-tuning scenario.

Robustness and Implicit Method Selection: The observed robustness of fine-tuned agents, especially in comparison to zero-shot counterparts, can be attributed to the inclusion of noisy data in the fine-tuning trajectories. This exposure enables the agents to better handle imperfect tool outputs, a scenario less effectively addressed by few-shot prompting, which often relies on idealized examples (refer to Table 3 for details).

The phenomenon of implicit method selection in multi-method fine-tuned agents is indeed intriguing. Our current hypothesis is that the fine-tuning process, which selectively focuses on successful trajectories, results in a varied distribution of training questions associated with different methods. This could enable the fine-tuned agents to learn associations between question features and the most effective method for addressing them. We believe this presents an exciting avenue for future research to explore in depth.

We are grateful for your detailed feedback and the opportunity to clarify aspects of our work.

Advantages of FiReAct over ReAct: Your query about the benefits of FiReAct compared to ReAct is crucial. FiReAct's fine-tuning process, unlike ReAct's few-shot prompting, involves learning from a broader range of trajectories that encompass diverse task-solving strategies. This diversity is key to FiReAct's enhanced robustness and generalizability. Specifically:

• FiReAct shows improved performance on the test sets of training tasks (refer to Table 2).
• When applied to novel tasks, FiReAct maintains or enhances performance, as evidenced in Table 3 and our General Response. This is a notable departure from traditional NLP fine-tuning, where the focus is often on specific facts or task formats. FiReAct, by teaching LLMs how to reason and utilize tools like Google search, imparts skills that are transferable across various tasks.
• FiReAct demonstrates greater resilience to noise in tool outputs. This is because the fine-tuning process includes trajectories with noise, teaching the model to handle imperfect data. In contrast, few-shot prompting often relies on 'ideal' examples, which may not prepare the model for real-world variability (see Table 3).

On the Alleged Triviality of Agent Fine-Tuning: We appreciate your concern regarding the perceived simplicity of our approach. However, the novelty of FiReAct lies in:

• Addressing the gap in research: Prior work has largely overlooked fine-tuning LLMs specifically for agent roles, with most studies either fine-tuning LLMs for non-agent tasks or using prompting for agent tasks.
• Demonstrating multi-faceted empirical benefits: FiReAct, despite its simplicity, showcases significant benefits in terms of robustness, generalization, efficiency, and cost, which are crucial for practitioners in the field of agent development.
• Exploring new technical depths: FiReAct pioneers the approach of multi-method, multi-task fine-tuning for agents. It demonstrates that even smaller LLMs can effectively acquire generalizable agent skills through supervised fine-tuning, presenting a viable alternative to more complex and less controllable methods like reinforcement learning.

Clarification of Contributions: In response to your suggestion, we have revised our introduction to more clearly articulate our contributions. We now explicitly state how FiReAct advances the field of language agent fine-tuning, detailing the novel methodologies, systematic experiment designs, practical insights, and new research questions that our work introduces.

We sincerely appreciate your insightful feedback and constructive suggestions. Here is our response to the key points raised:

Task and Tool Limitation: We acknowledge your concern regarding the generalizability of FiReAct. In response, we have expanded our experiments to include additional tasks from AgentBench, which extends beyond QA and incorporates a variety of tools. These new experiments are designed to demonstrate FiReAct's adaptability to a broader range of tasks and tools.

Methodology Limitations: We understand the importance of handling multiple prompts, roles, and contexts in agent design. Our current focus on methods like CoT, ReAct, and Reflexion, which align with the ReAct autoregressive format, was a strategic choice due to their compatibility with multi-method fine-tuning. However, we recognize this as a limitation and an exciting avenue for future research, as noted in our revised manuscript.

Multi-Task Constraints: For further details on our expansion beyond QA tasks, please refer to our General Response. Regarding the use of GPT-3.5, it represents the most advanced publicly available LM for fine-tuning at the time of our study. We are actively exploring the integration of more diverse tasks and advanced LMs in our ongoing research.

Application of FiReAct Beyond QA and Advanced Agents: As detailed in our General Response, FiReAct's application extends to various interactive tasks, including those mentioned. Our initial focus on QA was due to its well-defined structure and data availability, making it an ideal starting point for demonstrating FiReAct's capabilities.

Combining Prompting and Fine-Tuning: As discussed in Section 8 of our paper, we advocate for a balanced approach. Fine-tuning is preferable for well-defined, frequently used tasks with sufficient data, while prompting offers flexibility and convenience for less defined or zero-shot scenarios.

Scaling FiReAct to Multi-Task Environments: Our initial experiments do not indicate a performance decrease when scaling FiReAct to more tasks. Future work will investigate the upper limits of task complexity and quantity that FiReAct can effectively manage.

Impact of Newer Models like GPT-4: Our experiments across various models, including Llama-2 and CodeLlama, demonstrate FiReAct's effectiveness irrespective of model size or family. We anticipate that fine-tuning with GPT-4 would enhance performance, generalizability, and robustness, and we look forward to exploring this in future work.