Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization

Dear Reviewer Hrx6,

We are sincerely grateful to the reviewer for the informative feedback, which has helped improve our paper (please see the updated paper and appendix.) Please see our point-to-point response below.

Q1: I think the main weakness is that this kind of prompt tuning may not be necessary for a well trained agent. The agent should be well tuned to understand all kinds of different prompts. So the problem itself may not be very significant.

A1: Thanks for pointing it out! In light of your comments, we’ve added GPT-4, which has impressive decision-making capabilities, as the agents for comparison. Please see the updated Table 4, Fig. 4 and 6. In general, we find that even with GPT-4 as the agent, our Retroformer still shows significant improvements after just 1 episode. This is because even with a well-trained agent like GPT-4, it still needs exploration to cross off some incorrect answers and choose correct trajectories in the next round. If there is anything unclear about the updated results, please kindly let us know!

Q2: Instead of fixing the agents and tuning the prompts, tuning the agents from RLHF feedback may have a more significant effect for the LLM. But this is just my opinion. The authors can discuss whether it's correct or not.

A2: Thanks for the question! In light of the comment, we’ve included one RL baseline, i.e., Soft Actor-Critic [1] that directly finetune the LLM agent online with environment feedback. For a fair comparison with the language agent methods in this paper, which all showed improvements in 5 episodes, we ran the RL method for 5 episodes but did not observe any improvement across all 3 environments. These results show that tuning the agent prompts turn out to be much more efficient than tuning the agent LLMs directly, under our problem setting.

[1] Haarnoja, Tuomas, et al. "Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor." International conference on machine learning. PMLR, 2018.

In my "Weaknesses" section, I realized I accidentally said "Reflexion" where I meant "Retroformer." Sorry for any confusion!

Dear Reviewer SkxB,

We appreciate that you read our paper very carefully and your informative feedback, which has helped improve our paper. Below please see our response to your concerns. In case you find anything unclear in the paper, please kindly let us know.

Q1: It would be useful to see an analysis of how often the plan correctly identifies an improved plan, e.g. have a human hand-label 100 prompts for Reflexion and 100 from the frozen LM and record their success rates.

A1: Thanks for the comment! Given the limited time, we’ve manually labeled 5 reflection responses in the HotPotQA dataset, including the “Teen Titans” examples mentioned in the review. We found that with human labeling of the root cause of failure and next plans, the agent LM can always succeed in the next trial. For the reflection responses from the frozen LM, i.e, the Reflexion method, 1 out of 5 tasks succeeds in the next trial and with our Retrofomer fine tuned by policy gradient, under the same content for the remaining prompt, 3 out of 5 tasks succeed.

For the 100 tasks, one can refer to Figure 4. For example, under GPT-4, at episode 1, Reflexion has 48 successful tasks while Retroformer (r=4) has 51 successful tasks. This means that 8 (48-40) plans of frozen language model identifies an improved plan and 11 (51-40) plans of Retroformer identifies an improved plan, which shows the effectiveness of our approach.

We appreciate your suggestion for hand labeling, and we will label (the rest of) the reflection responses and add them for fine tuning the Retrospective model, as an ablation model for human-in-the-loop expert demonstrations, similar to the Dagger RL method .

[1] Ross, Stéphane, Geoffrey Gordon, and Drew Bagnell. "A reduction of imitation learning and structured prediction to no-regret online learning." Proceedings of the fourteenth international conference on artificial intelligence and statistics. JMLR Workshop and Conference Proceedings, 2011.

Q2: I'm confused how the "f1 score" reward works.

A2: Thanks for spotting this! We’ve updated the Appendix C.3 to include the description of reward functions used in 3 environments. To answer your question, I quote the updated descriptions below.

F1 reward is used in the HotPotQA environment for comparing the matching of a generated answer to a question against the ground truth answer. After removing the stopwords in both answers, we calculate the number of common tokens in two answers. Then Precision is # of common tokens divided by # of generated answer tokens and the Recall is # common tokens divided by # ground truth answer tokens. We can then compute f1 from precision and recall. This reward was defined in the ReAct paper, which is used as the baseline model in this paper.

[2] Yao, Shunyu, et al. "React: Synergizing reasoning and acting in language models." arXiv preprint arXiv:2210.03629 (2022).

Q3-1: I'd like to see the following additional curves added to Fig 4 and 6 (possibly in an appendix), which might make it clearer how the finetuning contributes: Retroformer, rolling out the base policy (before RL finetuning). This would make it easier to see how much of the improvement above Reflexion is due to fine tuning vs other algorithmic details.

A3-1: Thanks for the comment! In light of the suggestions, we’ve added the curve Retroformer (GPT-3, lora r=0) to Fig 4 and 6 to denote the base policy Retroformer agent with the frozen longchat-3b as the reflection model (before RL finetuning). The improvement due to finetuning is now entirely represented by the area, for example, of the curve Retroformer (GPT-3, lora r=4) above Retroformer (GPT-3, lora r=0).

Q3-2: I'd like to see the following additional curves. Retroformer, rolling out the base policy, but with GPT-3 as the reflection model. This would answer the question of whether it's typically a better strategy to finetune a small model or just prompt a more capable cloud model.

A3-2: Thanks for the suggestions! The Reflexion (GPT-3) curve in Figure 4 and 6 is actually the requested “Retroformer, rolling out the base policy, but with GPT-3 as the reflection model”.(By the way, we believe you meant GPT-4 for the reflection model as a more capable cloud model; please let us know if we are wrong.) Therefore, We’ve also tried using GPT-4 as the reflection model, see Reflexion (GPT-4) and found that, for example, Retroformer (GPT-3/4, r=4) still outperformed it by a lot in all environments. This demonstrates that for generating reflective responses in a given environment, fine tuning a small model is more effective than prompting a more capable general-purpose model.

Q4: I'd recommend writing out the exact algorithm you used for PPO finetuning, especially since when PPO is used with language models slight details from the original algo are changed. Also, it's typically an online algorithm, so it would be helpful to detail any changes you made to make it work well in the offline setting.

A4: Thanks for pointing it out! We’ve included an algorithm table for the PPO training in Appendix C.1 - Algorithm 1. We’ve also added a description of the three steps.

The offline PPO algorithm we used for finetuning the Retrospective component in this paper is presented below in \cref{Algo: PPO}. It contains three steps: offline data collection, reward model learning, and policy gradient finetuning. We use the offline ratings data to train a reward model first, and plug in the reward model for PPO finetuning

Q5: Equation 6 seems to be optimizing for each (x, y) pair individually (i.e. not treating a sequence of (x1, y1), (x2, y2) (x3, y3)... as a trajectory. Is this correct? If so, I'd recommend making this clearer in the text.

A5: Thanks for carefully reading our paper! The presentation of this equation is actually correct. $x$ in the equation refers to the instruction prompt, which includes textual descriptions of the agent trajectory, episodic returns, environment feedback at the end of one episode and $y$ denotes the reflective response generated by the Retrospective component taking this prompt as input. Therefore, for fine tuning the Retrospective model (Retroformer’s task), there is only one step in this model’s trajectory. In light of your comment, we’ve updated the presentation to make it clearer!

(note there is only 1 step in the Retrospective model’s trajectory in its episode)

Q8: How much efficiency is gained by using a retrospective model to tune prompts, rather than directly tuning the LLMs?

A8: Thanks for the question. In light of your comments, we added one RL baseline (Soft Actor-Critic) in the updated experiments which directly tunes the LLMs instead of tuning the prompts. For a fair comparison with the language agent methods in this paper, which all showed improvements in 5 episodes, we run SAC for 5 episodes but do not observe improvement across all 3 environments. These results show that tuning the agent prompts are much more efficient than tuning the agent LLMs directly, under our problem setting.

Q9: Are there any details about the hyperparameters of PPO?

A9: Thanks for pointing it out. We have updated in Appendix C.1 the details about the hyperparameters of PPO. We’ve also written out the exact algorithm we used for PPO finetuning in the updated Appendix C.1. Please kindly let us know if anything is not clear.

The offline PPO algorithm we used for finetuning the Retrospective component in this paper is presented below in Algorithm 1. It contains three steps: offline data collection, reward model learning, and policy gradient finetuning. We use the offline ratings data to train a reward model first, and plug in the reward model for PPO finetuning.

(3) Improvements against RL methods

In light of your comments, to show the improvements against traditional RL methods, we added one method, i.e., Soft Actor-Critic [5] as the baseline. We have updated the experiment results in Table 2 and added the implementation details of the RL method in the updated Appendix C.2, which is quoted below. “We include one online RL algorithm, i.e., Soft Actor-Critic [5], or SAC as a baseline model for comparison. Given that the three environments are text-based games, inspired by [1], we do mean-pooling for the embeddings of the generated text outputs, such as ``Search[It Takes a Family]'' as the agent actions. Therefore, the action space is continuous and is of 768 dimensions. We apply LoRA adapters with r=4 on the agent Action model instantiated from longchat-16k, and use SAC to do the online updates, with discount factor gamma=0.99, interpolation factor polyak=0.995, learning rate=0.01, entropy regularization alpha=0.2, and batch size=8.”

For a fair comparison with the language agent methods in this paper, which all showed improvements in 5 episodes, we run SAC for 5 episodes but do not observe significant improvement across all 3 environments in the updated Table 2.

This is mostly due to the fact that the three text-based games have an extremely large state and action space, and thus online exploration and learning within 5 episodes online cannot improve the LLM model with an extensive parameter count. Under this setting, verbal reinforcement with an LLM that is finetuned offline for generating effective reflective feedback (i.e., our Retroformer) is much more effective.

To summarize for Q1, we do believe the improvements brought by Retroformer are not limited, especially if the reviewer considers how traditional RL methods perform under the same problem setting. We do observe significant improvements under HotPotQA and AlfWorld. Webshop requires a significant amount of exploration with more precise search queries but our Retroformer method still performs better than Reflexion and React across all episodes. The results under Webshop indicate that the verbal feedback approach (Reflexion, Retroformer) is not an optimal method for this environment, but our fine-tuning method still proves effective. If there is anything unclear, please kindly let us know!

[5] Haarnoja, Tuomas, et al. "Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor." International conference on machine learning. PMLR, 2018.

Q2: The experiments are not solid enough. It lacks comparisons with RL methods that have been recognized to perform well within the same framework of interaction-feedback-learning.

A2: Thanks for raising the concern! Please refer to the detailed response above for the section Improvements against RL methods. In summary, we’ve added Soft Actor-Critic as the baseline. We run SAC for 5 episodes but do not observe significant improvement across all 3 environments with the updated results in Table 2.

Q3: Additionally, there is no comparison with the currently acknowledged GPT-4 model, which has impressive decision-making capabilities, making it insufficient to demonstrate the contribution of this work.

A3: Thanks for pointing it out! In light of your comment, we have updated the paper draft by adding experiments using GPT-4 (See the updated Table 2, Fig. 4 and 6). In general, we observe consistent improvements at episode 0 where verbal feedback is not applied. This is because of the better decision-making capabilities of GPT-4. Furthermore, we still observe consistent improvements by Retroformer under GPT-4, which again demonstrates the effectiveness of verbal feedback with a well-trained actor (e.g., GPT-4) together with a retrospective component tuned with our policy gradient method. We have updated the draft to include this analysis of GPT-4 results. If there is anything unclear, please kindly let us know!

Dear Reviewer 96C3,

We greatly appreciate your thorough and constructive comments, many of which have helped improve our paper. Both the paper and appendix have been updated extensively to include as much detail as possible. Please see our point-to-point response to your concerns below.

Q1: The improvements brought by Retroformer are limited. There are no significant improvements in HotPotQA and WebShop, only meaningful improvement is observed in AlfWorld.

A1: Thanks for the comments. We’d like to discuss them in the following three aspects: (1) Improvements in HotPotQA, (2) Improvements in WebShop, and (3) Improvements against RL methods.

(1) Improvements in HotPotQA

We believe that there are significant improvements in HotpotQA by using Retroformer. For example, as in Table 2, Retrofomer (GPT-3, lora r=4) was improved by 14% in success rate within just one episode. The improvement is also 6% higher than the Reflexion (GPT-3) method that uses a frozen model without our proposed RLHF finetuning. In light of your comments, we’ve also tested GPT-4 as an agent in the updated experiments and found even GPT-4 gets stuck at 54% success rate. We believe that 54% is the upper bound of the success rate under existing LLMs. It deserves notice that HotPotQA, which is a text-based game, has an extremely large state and action space (you can treat it as a RL problem with a continuous action of 768 dimensions). It should take more than 500 episodes to observe certain improvements in some much simpler text-based environments in [1]. Our improvements (14%) by Retroformer in just one episode are significant.

[1] Yuan, Xingdi, et al. "Counting to explore and generalize in text-based games." arXiv preprint arXiv:1806.11525 (2018).

(2) Improvements in WebShop

Thanks for carefully reading our paper! However, the difficulty of solving this web shop environment is well known in all recent language agent publications [2,3,4] that use this environment. As we stated in our previous submission, we agree that the performance improvement by Retroformer over the frozen baselines is observed but the improvements may be limited. This limitation of improvement was due to the fact that “web browsing requires a significant amount of exploration with more precise search queries”. Nevertheless, our Retroformer method is still performing better than Reflexion and React across episodes, which use a frozen model without our proposed RLHF fine tuning. In light of your comments, we’ve added GPT-4 agents in the updated experiments and found similar results in the updated Figure 6 (b).

We’ve explained these results in the updated draft in Section 4 - Web Browsing. The results probably indicate that the verbal feedback approach (Reflexion, Retroformer) is not an optimal method for this environment, but our fine-tuning method still proves effective.

[2] Yao, Shunyu, et al. "React: Synergizing reasoning and acting in language models." arXiv preprint arXiv:2210.03629 (2022). [3] Shinn, Noah, Beck Labash, and Ashwin Gopinath. "Reflexion: an autonomous agent with dynamic memory and self-reflection." arXiv preprint arXiv:2303.11366 (2023). [4] Liu, Xiao, et al. "Agentbench: Evaluating llms as agents." arXiv preprint arXiv:2308.03688 (2023).

Dear Reviewer 96C3,

Thanks for the prompt response. We are glad that we are aligned that the previous concerns on the comparisons with GPT4 have been resolved.

1. For the sparse feedback problems, let me try to explain this step by step to demonstrate why it is not sparse or repeated. For example, the agent was asked to answer a question in the HotpotQA by using Wikipedia APIs. In the first episode, it failed ($G_0=0$), and then multiple different reflections are generated, in which one reflection helps the agent succeed in the next round ($G_1=1$),; the other one doesn't help so the agent failed again in the next trial ($G_1=0$),. Then the first reflection is good and rated higher ($r=G_1-G_0=1$ and the second reflection is bad and rejected ($r=G_1-G_0=0$). The trajectory in the second episode is even not used for offline training in this example. Can the reviewer explain why the small differences between consecutive episodes or similarity could cause a problem for RLHF fine-tuning?

2. For continuous RL control problem, we are very confident that soft-actor critic is still the most stable SOTA baseline according to our knowledge. That is the reason why we chose it in the beginning during rebuttal. Furthermore, we do not believe any online traditional RL algorithm can learn within 1 episode (~6 steps) in these complex text-world environments (note all state-of-the-art RL algorithms including [2,3] in the paper [1] cited by reviewer have been trained for 1M steps). Could you please provide one specific RL baseline you would like us to try further, besides SAC, given the limited amount of time available?

With best regards, Authors of submission 1478