AGILE: A Novel Reinforcement Learning Framework of LLM Agents

We sincerely thank you for your effort and valuable comments in reviewing our paper. We appreciate your recognition of our contributions and your insightful feedback. We have addressed these concerns in the Rebuttal Section within the given time constraints to the best of our ability.

---

Please let us know if our responses address your concerns. We are grateful for your time and effort in helping us enhance our work.

Q1 (Weakness 1 & Question 1): Paper framing

A1: Regarding novelty, we would like to note that we have summarized our core contributions at the end of the Introduction section [line 65], aligning exactly with your suggestions. The Methods section also aligns with your summary. We acknowledge that the "novel framework" statement was somewhat overstated. Hence, we will revise our paper's title to: "AGILE: A Novel Reinforcement Learning Framework for LLM Agents," to emphasize that our main contribution is the end-to-end RL formulation and training of LLM agents. We will also properly update other parts of the paper to clearly convey this point.

Beyond the RL aspects, we want to highlight other novel contributions of our paper. First, training the agent to proactively seek advice requires it to silently estimate its own confidence before making decisions. This capability cannot be achieved through SFT alone, further demonstrating the importance of RL training.

Second, the ProductQA dataset is a novel contribution. It is the first agent task involving very long trajectories, with each containing thousands of correlated QA sessions and millions of tokens in total. The actions the agent takes in one session influence all future sessions. This characteristic is markedly different from other benchmarks where the impact of actions is much shorter-term.

---

Q2 (Weakness 2): Difference between AGILE and other agents

A2: Agents like Voyager are based on prompt engineering, while AGILE is a RL formulated framework that allows end-to-end training. Application of AGILE to other tasks, such as WebShop, is left to future work.

---

Q3 (Weakness 3 & Question 4): Additional results of RL training

A3: First, we provide training curves in the global response PDF file, which indicates that RL training converged after about 500 steps.

Second, following your suggestion, we conducted multiple independent trials of PPO training to study the variation of the result (see Table 2 in the PDF file). On average, RL training improves the total score by 2.6%, with a standard deviation of 0.3%, demonstrating the significance of RL improvements.

Third, we would like to discuss why RL training improved performance by a moderate 2%. We believe the reasons are: (1) The agile-vic13b-sft model is a strong baseline as it imitates the policy of the agile-gpt4-prompt; (2) SeekAdvice decisions made by agile-gpt4-prompt are nearly optimal under the default advice cost (c=0.3). To further study the impact of RL training, we conducted additional experiments in two more settings:

• We re-generated SFT training data for agile-vic13b-sft such that the agent performs SeekAdvice randomly in 25% of cases. This initial policy is simpler but more general. In this setting, we name the SFT model agile-vic13b-sft-random, and the final model trained with RL on top of it agile-vic13b-ppo-random. As shown in the table in global response, RL training brings a 7.1% improvement in this setting. Interestingly, the performance of agile-vic13b-ppo-random is better than that of agile-vic13b-ppo that we reported in the paper. We conjecture that random SeekAdvice is a better initial policy because it enables exploration in all directions.

• In the second experiment, we lowered the advice cost to 0.1. After PPO training, the agile-vic13b-ppo-random agent quickly adapted to the new cost, performing SeekAdvice much more aggressively than the initial agent trained by SFT. In this scenario, RL training brings a 22.3% improvement.

See Table 3 in the PDF file for concrete numbers.

---

Q4 (Weakness 4): RL training for metacognitive skills

A4: RL training helps the model's metacognitive skills. Experiments on ProductQA and MedMCQA datasets show that, after PPO training, the agent seeks advice less often and achieves higher accuracy, indicating improved self-assessment in resolving queries versus seeking assistance.

Q5 (Weakness 5): Related work and our contributions to the topic of uncertainty

A5: We note that AGILE solves a complex self-evaluation problem because the agent must make multiple seeking advice decisions in a long trajectory, each having a long-term influence on future decisions. This differs from existing work in uncertainty. We will add discussion to the revised version.

---

Q6 (Weakness 6): Training details

A6: On ProductQA, SFT takes 3.6 hours and PPO takes 5.5 hours on 8 H800. On MedMCQA, SFT takes 0.9 hours and PPO takes 2 hours. The LLM is trained without using LoRA.

---

Q7 (Weakness 7): Agreement between GPT-4 evaluator and human

A7: We conducted an additional evaluation by randomly selecting 100 triplets (question, reference long answer, model-predicted long answer) from ProductQA and manually labeled the correctness. Our results show a 94% agreement rate between the GPT-4 evaluator and the author.

---

Q8 (Question 2): Analogy to System 1 and System 2 processes

A8: According to the dual-process theory [1], human thinking involves two systems: System 1 (fast and unconscious) and System 2 (slow and conscious). Recent research has explored AI systems incorporating both processes [2]. We propose that AGILE uses LLM for System 1 and integrates external tools for System 2, mirroring human thinking. We will include further discussion in the revised version.

[1] D. Kahneman. 2003. Maps of bounded rationality: Psychology for behavioral economics.

[2] Y. Bengio, Y. Lecun, and G. Hinton. 2021. Deep learning for AI.

---

Q9 (Question 3): Further explanations of MCQA and the Meerkat model

A9: MedMCQA is a multiple-choice QA dataset from medical school entrance exams. Meerkat is a medical LLM trained with high-quality CoT reasoning paths from 18 medical textbooks and diverse instruction-following datasets. We will include these explanations in the revised version.

---

Q10 (Question 5): Citation of RAG paper

A10: We will add it to the paper.

Thank you for your valuable feedback. We would like to address each of your questions as follows.

---

Q1: I also appreciate that you performed additional RL experiments to try and show it helps compared to the strong stf baseline. To make sure I really understand: the original sft policy had 0% [SeekAdvice] rates? And what your new experiment does is start out with higher advice rates and show it leads to more efficient use of human advice (esp when cost of advice falls to 0.1, effectively solving the task by soliciting advice more than half of the time)?

A1: In all training experiments, we consider three types of SeekAdvice rates: 1) the rate in the SFT training data, 2) the rate predicted by the SFT agent after training, and 3) the rate predicted by the RL agent after training.

For both the original and additional experiments, the SeekAdvice rate in the SFT training data (rate 1) is 25%. In the original experiment, GPT-4 makes SeekAdvice decisions. In the additional experiment, these decisions are made randomly but maintain the 25% rate.

In the original experiment, the SFT agent (agile-vic13b-sft) predicts SeekAdvice in 25.6% of cases, closely matching the training data. In contrast, in the additional experiment, the SFT agent (agile-vic13b-sft-random) predicts SeekAdvice in only 1.4% of cases. This discrepancy is due to the LLM's greedy decoding. Although it predicts SeekAdvice with logit scores corresponding to a 25% probability, the greedy decoder typically selects other actions with higher logit scores. We also tried random sampling decoding, and the results are attached to the table below.

In the original experiment, the RL agent (agile-vic13b-ppo) has a SeekAdvice rate of 23.3%, while in the additional experiment, the RL agent (agile-vic13b-ppo-random) has a rate of 30.6%. The new agent achieves a higher overall score, likely because it starts from a simpler and more general initial policy, allowing PPO to find a better final policy. In contrast, the original agent starts with a strong GPT-4 policy and only makes slight adjustments.

When the SeekAdvice cost is reduced to 0.1, PPO training converges to a much more aggressive SeekAdvice rate of 67.1%. This experiment demonstrates that RL training is sensitive to the SeekAdvice cost, always optimizing the policy under specific costs. In contrast, the GPT-4 agent remains insensitive to cost changes, even when the cost is provided as input. The SFT agent is also insensitive since generating optimal SFT data for a given cost is difficult.

---

Q2: I think nothing prevents researchers to build a Voyager agent (with an open weights model) and perform RL on it right?

A2: We agree with you that our main contribution is the end-to-end RL formulation and training of LLM agents.

In addition, our agent can manage very long trajectories. In the ProductQA task, there could be hundreds of question-answering rounds and could generate very long trajectories yield training sequences that span millions of tokens. To address the challenge, two key capabilities are necessary: 1) Some actions can erase or modify the context to prevent the context from infinitely extending and unmanageable for the LLM; 2) session-level RL training. These challenges cannot be resolved simply by applying PPO training on Voyager-like agents. We discuss the solutions to these issues in our work.

Q3: I also appreciate that you emphasize that you build a new task, but from what I can see it doesn't seem very hard since agents (with quite standard architecture, again I think the type of agent you implement is quite common, see also https://arxiv.org/abs/2304.03442) based on 13b models, without RL or search, are able to solve most of it. That's why I was calling for evaluations on other datasets, to be able to compare. (ReAct is not a very strong baseline). I appreciate your new experiments on HotpotQA, and would be interested to know how it compares to the SotA listed in the benchmark homepage (in terms of metrics), and how you implemented advice in the case of this dataset.

A3: Thanks for your comments.

1. Experiment implementation.

   In the HotPotQA task, the agent has the option to either use search tools, seek advice, or directly predict an answer. If the agent chooses to use search tools, it generates a search query to retrieve relevant information, which is then appended to the agent's context. If the agent seeks advice, it obtains a human answer (ground-truth answer in our setting).

2. Comparison to SoTA listed in the benchmark homepage.

   Thank you for your reminder. We will include the SoTA results in our results table. However, we want to highlight two key differences between our AGILE agent and the methods on the leaderboard:

   • Our agent uses an external retrieval tool that returns the most relevant document based on a query. This retriever is the same as the one we used for ProductQA and MedicalQA and is not fine-tuned on the HotpotQA dataset. During training, we focus solely on training the LLM of the agent. In contrast, the top five systems on the leaderboard [1,2,3] (except the second one, which lacks a reference link) all train task-specific retrieval models on the 90K HotpotQA training examples. Training such models significantly boost accuracy, and these works claim it as their main technical contribution. Our paper primarily studies the training of LLMs and treats external tools as black boxes, so using a task-independent retriever aligns better with our objectives.

   • The top systems on the leaderboard [1,2,3] train separate answer extraction models to extract answers from document spans. Our agent directly generates answers from the LLM. While the generative nature may lower exact match (EM) accuracy, we believe it enhances simplicity and generalizability. Besides EM accuracy, we also calculated a GPT-4 accuracy where answers are compared for correctness, recognizing (USA, United States) as correct. As shown in the table below, our system's actual accuracy is much higher than the EM accuracy.

   Regarding baselines, we noted the original ReAct baseline's implementation in [4] was suboptimal. We reproduced their results using GPT-4, leading to improved performance. In the table below, we also provide results of other works that use the same experimental setting as ours (using a black-box retrieval tool and generative answers). Our system's overall performance surpasses all these baselines.

[1] J. Zhang, et al. End-to-End Beam Retrieval for Multi-Hop Question Answering, NAACL 2024.

[2] Z. Yin, et al. Rethinking label smoothing on multi-hop question answering, CCL 2023.

[3] XY. Li, et al. From easy to hard: Two-stage selector and reader for multi-hop question answering, ICASSP 2023.

[4] S. Yao, et al. ReAct: Synergizing reasoning and acting in language models, ICLR 2023.

[5] Z. Gou, et al. CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing, ICLR 2024.

[6] A. Zhao, et al. Expel: Llm agents are experiential learners, AAAI 2024.

[7] S. Qiao, et al. AutoAct: Automatic Agent Learning from Scratch for QA via Self-Planning, ACL 2024.

Dear Reviewer b4yX,

Thank you for taking the time to review our manuscript. We sincerely appreciate your insightful feedback. As the deadline for the discussion phase nears, we would like to kindly remind you of our recent response, in which we diligently addressed the concerns you raised and provided detailed explanations.

Should you have any further questions or require additional clarifications, we would be eager to address them promptly.

Thanks again for your valuable feedback.

We sincerely thank you for your positive feedback and effort in reviewing our paper. Thank you for your constructive questions. In our response, we will quote each question and provide our answers accordingly.

Response to comments

---

Q1 (Weakness 1): It is somewhat like WebGPT (in terms of actions and policy learning strategy), with extensions supporting memory, expert consultation, and reflection.

Answer: Thanks for bringing this to our attention. We would like to clarify the key distinctions between our proposed AGILE framework and WebGPT:

• WebGPT primarily uses a policy learning strategy to train the agent for web operations and does not incorporate memory, expert consultation, or reflection. In contrast, AGILE emphasizes proactively seeking advice from human experts, allowing the agent to achieve high-level accuracy when handling complex and challenging questions. The agent can also enhance its performance over time by reflecting on expert feedback and memorizing it for future use.

• While WebGPT utilizes reinforcement learning, our work addresses the significant challenge of training the policy for invoking various modules and the reasoning, planning, reflection, and seeking advice abilities of the LLM agent in an end-to-end manner. This challenge is particularly pronounced in long trajectory scenarios. To overcome this, we propose a session-level optimization algorithm that facilitates policy learning at the session level, thereby mitigating the difficulties associated with long trajectory optimization.

• As shown in Table 5 in our paper, incorporating memory, seeking advice, and reflection into AGILE results in relative total score improvements of 4.0%, 5.0%, and 1.7%, respectively, on the ProductQA dataset. These results demonstrate the necessity of these modules for LLM-based agents in practical scenarios.

---

Q2 (Weakness 2): The experiments are all related to a single agent. It would be better to show the application for multiple agents and include planning.

Answer: We appreciate your valuable suggestions. Applying AGILE to multi-agent systems is indeed an interesting direction. The AGILE framework can be extended to facilitate interactions with machine agents in various roles, such as students or teachers, and in different formats, such as debates or coordination. For example, a multi-agent version of AGILE could seek advice from a human expert at a high cost or from a more capable machine agent at a lower cost. Furthermore, the planning for multiple agents can be enhanced using end-to-end RL training.

In our paper, we primarily focus on the RL formulation and end-to-end training of LLM agents, along with their seeking advice capabilities, which we believe are fundamental and essential for agent performance. Due to space and time limitations, the application of AGILE to multi-agent systems will be addressed in future work. Thank you for your constructive feedback.

---

Q3 (Question 1): How can this framework be extended to support multiple agents?

Answer: Thanks for your question. Please refer to our response to Q2.

---

Q4 (Question 2): In Table 5, the accuracy of "w/o Memory" and "w/o Tool-Use" is higher than that of agile-vic13b-ppo. Is the metric "Accuracy – Advice Rate" a more straightforward one than Total Score?

Answer: Thank you for this valuable comment. In Table 5, although the accuracy of "w/o Memory" and "w/o Tool-Use" is higher than that of agile-vic13b-ppo, they exhibit a significantly higher Advice Rate compared to agile-vic13b-ppo. This higher Advice Rate results in lower overall performance for "w/o Memory" and "w/o Tool-Use" when assessed using our Total Score metric.

In our paper, the total score is defined as the average reward across all sessions, taking both the advice rate and the accuracy into account. Specifically, the total score is expressed as "Total Score = Accuracy - Seeking Advice Cost * Advice Rate". This formulation encompasses the "Accuracy – Advice Rate" metric as a special case where the Seeking Advice Cost is set to 1.

It's important to note that the Seeking Advice Cost can vary across different real-world applications. For instance:

• In a math problem-solving task, the required human expert might need high professional expertise in mathematics, resulting in a high cost (e.g., a cost of 1).
• In product question-answering scenarios, the human expert could be a normal person who has received short-term customer service training, leading to a lower cost (e.g., a cost of 0.3).

---

Please let us know if our replies address your concerns. Thanks for taking the time to consider this discussion. We appreciate your time and effort in helping us improve our work.

We genuinely appreciate your positive feedback and the time you invested in reviewing our paper. Thank you for your insightful questions. In our response, we will address each question individually, quoting them and providing our answers accordingly.

Response to comments

---

Q1 (Weakness 1): Because the model is only evaluated on two conversational QA datasets (and not on any of the mentioned datasets in Table 8), it's difficult to judge the generality of the proposed method.

A1: Thanks for the helpful suggestion. To address the concern regarding the generality of AGILE, we conducted additional experiments on the HotPotQA dataset, which is one of the tasks listed in Table 8. The main results and ablation study are presented below.

Compared with ReAct implemented by prompting GPT-4, our method (agile-vic13b-ppo) shows a 19.3% relative improvement in accuracy. Furthermore, agile-vic13b-ppo achieves a 10.7% relative improvement in the total score over agile-gpt4-prompt, which is the AGILE agent implemented by prompting GPT-4. The ablation study underscores the indispensability of seeking advice and PPO training in achieving the agent's strong performance.

---

Q2 (Weakness 2 & Question 1): What is the reason behind why there is not comparison with rl finetuning baseline (standard ppo)?

A2: Thanks for your question. In the ProductQA task, there could be hundreds of question-answering rounds. These rounds are correlated. Actions in earlier rounds can write the memory, thus have lasting effects on subsequent rounds. For example, knowledge distilled from seeking advice can help respond in the future. These long trajectories yield training sequences that span millions of tokens and are impractical for standard PPO training. To address this issue, we proposed a session-level training algorithm that takes lasting effect into account, which is detailed in Appendix A of our paper.

---

Please let us know if our replies address your concerns. Thanks for taking the time to consider this discussion. We appreciate your time and effort in helping us improve our work.

We sincerely appreciate your effort and valuable comments in reviewing our paper. Your recognition of our contributions and your insightful feedback are greatly valued. We have addressed your concerns in the Rebuttal Section to the best of our ability within the given time constraints.

---

Please let us know if our responses satisfactorily address your concerns. We are grateful for your time and effort in helping us improve our work.

Q1: Limited Novelty: The reliance on human experts for advice, while useful, is not a novel concept and has been explored in previous works and becomes a common practice. The paper does not sufficiently differentiate its approach to human expert interaction from existing methods.

• Xiao, H. and Wang, P., 2023. Llm a*: Human in the loop large language models enabled a* search for robotics. arXiv preprint arXiv:2312.01797.
• https://python.langchain.com/v0.1/docs/use_cases/tool_use/human_in_the_loop/

A1: Thanks for bringing this to our attention. In AGILE framework, agent can proactively seek advice from human experts when its confidence is low. This is distinct from existing human-in-the-loop methods, which rely on passively receiving human feedback (Xiao and Wang, 2023) or following pre-defined rules (https://python.langchain.com/v0.1/docs/use_cases/tool_use/human_in_the_loop/). Proactively seeking advice is a much more complicated decision-making, since the agent must estimate its own confidence in the current state, predict the potential value of the advice for future sessions, and consider the cost of experts. We will cite the works you mentioned and clearly differentiate our proactive seeking-advice mechanism from existing human expert interaction methods in our revised paper.

---

Q2: Scalability Concerns: The scalability of AGILE with more complex environments is not addressed. It is unclear how well the framework would perform with task complexity.

A2: Thanks for your feedback. We acknowledge the importance of addressing scalability of AGILE. To this end, we have introduced ProductQA, a complex benchmark designed to evaluate the comprehensive capabilities of the agent. ProductQA tests an agent's ability to handle historical information and accumulated knowledge, leverage tools, interact with humans, perform self-evaluation, and conduct reflection. In addition, the training and testing tasks are made disjoint to assess the agent's ability to adapt to new tasks.

Our experimental results show that AGILE agent, based on 13b LLMs and trained with PPO, outperforms GPT-4 agent on ProductQA. This demonstrates the AGILE framework's potential to scale and manage complex tasks effectively. Furthermore, AGILE is a general agent framework, allowing for various extensions, such as integrating additional tools, further enhancing its scalability with increasing task complexity.

---

Q3: Benchmark Selection: Why were ProductQA and MedMCQA specifically chosen as the benchmarks for this study? Are there other benchmarks where AGILE could be tested to validate its generalizability?

A3: While the proposed agent framework is general, in this paper, we evaluate it in complex question answering, a task an LLM agent has the potential of outperforming existing solutions such as the use of an LLM alone. We use ProductQA since it assesses a variety of agent capabilities including knowledge accumulation, tool-use, human interaction, reflection, etc, making it ideal for demonstrating the comprehensive ability of our framework. To validate the generality of AGILE, we select MedMCQA as an additional benchmark. This task requires extensive medical knowledge and the ability to effectively seek expert advice.

In response to your suggestion about exploring more diverse benchmarks, we perform experiments on HotPotQA, featuring natural, multi-hop questions. We train an agile agent using the Vicuna-13b as a base. The experimental results show that the agile agent outperforms GPT-4 agent by 10.8% in relative total score. Ablation studies verify that ppo training improves total score by 1.8%. Detailed results of the additional experiments will be included in the revised version to underscore the generality of the AGILE framework.

---

Q4: Human Expert Advice: Is there a ground truth for when the agent needs to seek advice, and how is this optimized?

A4: Thanks for your questions. Determining when the agent should seek advice relates to the model's confidence and cost of human experts. Since the optimal decision is model-dependent, it is difficult to establish a ground truth and perform supervised fine-tuning. However, our RL framework can effectively address this issue. By defining rewards for both correctly predicting answers and seeking advice, we can optimize the skill as part of the policy model on end-to-end RL training. For example, we assign a reward of +1 for a correct prediction and a penalty of -c for seeking advice, where c represents the human cost. This approach allows the agent to learn an optimal trade-off between relying on its own predictions and seeking external advice. We will explain the method more clearly in the revised paper.

---

Q5: Memory Component: Can you provide more details on how the memory component scales with an increasing number of interactions and how it ensures efficient retrieval of relevant information?

A5: Thank you for your question. The memory is designed as a scalable database where stores agent's trajectories, including question-answering pairs and agent reflections. We employ vector-based retrieval for efficient information access. Specifically, we use the all-MiniLM-L6-v2 model for embedding the text data. When an instance is stored, it is embedded into a vector representation that serves as its key in memory.

For retrieval, we embed the user question using the same model and then perform a cosine similarity search among the stored embeddings. This is a well-studied process and can be performed efficiently even when the memory is large.

Dear Reviewer 5wwv,

We sincerely appreciate the time and effort you have invested in reviewing our manuscript. As the deadline of the discussion phase approaches, we would like to kindly remind you of our rebuttal, which addresses each concern you raised in your feedback.

If you have any further questions or concerns, we would be happy and eager to address them promptly in the discussion period.

Thank you once again for your valuable feedback.