Policy Improvement using Language Feedback Models

We thank the reviewer for taking the time to review our paper and provide us with valuable insights. We appreciate the acknowledgement of the generalizability and efficiency of our method, as well as improvements to model explainability.

W1: Rejection sampling

We thank the reviewer for drawing connections between our method and rejection sampling (RS). We discuss 2 key differences between the two techniques.

First, LFM removes 50-80% of non-productive steps whereas RS includes entire trajectories (20-100 steps for our envs). That said, we experimented with training on entire rollouts w/ positive reward. This did not significantly improve over the base policy.

Second, LFM filtering can be done during testing, where no reward is present. In this case, reward-based RS cannot be directly applied to perform adaptation during testing. That said, one could train a model to predict rejection, using data from the training environments, then use said model on test environments for adaptation. This is precisely what our proposed method does.

W2: reinforcement learning

We reference prior RL work on ALFWorld and Touchdown (https://arxiv.org/abs/2110.10661), which shows RL underperforms LFM after training for 10 million steps. ALFWorld episodes typically have <30 steps. Training on demos amounts to 30 * 3.5k ~ 100k steps. We train LFM using 1 rollout per training env for another 100k steps. Touchdown episodes are typically <200 steps. Demos and LFM steps are then 1.5 million steps each. For both envs, LFM requires fewer steps than RL and achieve higher task success rate (e.g. 64 vs 23 ALFWorld, 60 vs 15 Touchdown).

W3: Importance of initial demonstrations

We experimented w/ a random base policy, but found collecting trajectories that demonstrate productive behaviour difficult, as random policies are mostly not productive. Consequently, most data given to the LLM for feedback annotation do not exhibit productive behaviour, therefore increasing the cost of LLM usage. We can alleviate this using reward-based RS, as the reviewer suggested in W1. That is, we sample many trajectories, then give those w/ non-trivial reward to the LLM for feedback annotation. We will explore this in future work.

Q1: BC vs Prev SOTA

We use the same demos as prior SOTA, provided by the env designers. However, we use stronger, newer base models. For instance, ALFWorld prior SOTA fine-tuned a GPT2 model. Touchdown prior SOTA trained a CNN/RNN network. In contrast, we fine-tuned a FLAN T5 Large model, which performs well in language grounding tasks (https://arxiv.org/abs/2210.11416). We perform similarly to prior SOTA for ScienceWorld using the same model but without ScienceWorld-specific prompts (e.g. tracking object types, rooms visited).

Q2: ActPred k steps

The significance of this k-step is due to the token limitation. We want diverse trajectories over many envs. However, each trajectory may consist of >100 steps (ScienceWorld and Touchdown are often 50-200 steps). Predicting actions from the start using LLMs is biased towards early-trajectory steps - we will exceed the token limitation before encountering late-trajectory steps. Completely rolling out one env at a time will exceed token limitation before processing many of the envs. To balance between coverage of envs and trajectory depth, we first roll out k steps, then predict action using LLM. k is sampled from the max trajectory length of demos.

Note that this coverage problem is less significant for LFMs. Because LFM data is queried in windows of 20-steps, we can maintain high coverage of both environments and of trajectory depth. We also experimented with asking GPT4 to retroactively relabel actions in 20-step windows, however the labels were significantly worse than labeling actions one step at a time.

We will add explanations of the k-step significance to the manuscript.

Q3: average perplexity vs logprob of sequence

We did explore with log probability, which performed slightly worse than averaged perplexity.

Q4: Scoring large action spaces

We emphasize that training strictly maximizes log probability of tokens of the correct action, so its complexity is independent of the size of the action space. Inference requires scoring large action sets, however is cheap due to lack of gradient-tracking. The only optimization we did was compute scores in chunks, which are concatenated to select the optimal action.

Q5: initial phase trajectories

We clarify that it is 10k windows of 20 steps sampled from trajectories. These windows contain many negative examples, consequently we do not add them to the dataset for policy improvement. The data used for policy improvement are only steps from base policy rollouts that identified by the trained LFM as productive.

Q6: analysis of explanations

The examples in Table 5 are actually from the held-out test set tasks. Unfortunately, we did not perform quantitative analyses. Our qualitative findings are that a 770M model, trained on LLM feedback for a specific domain, is able to provide accurate feedback and explanations for said domain. One limitation was that LFMs are sometimes not consistent in evaluating productiveness of ambiguous actions such as exploration. For instance, if the instruction is to find cups and wash them, then even a good policy will spend time searching the room for cups. To some observers, this search procedure is productive, to others, it is not productive. In our experience, LLMs and LFMs are sometimes not consistent in assigning productiveness to such actions. In future work, we would like to analyze the limits of what types of feedback LLMs (and VLMs, for that matter) are able to provide in multi-modal grounded environments.

Summary

We sincerely thank the reviewer for taking their time to help us improve this work. We hope we have addressed the reviewer’s concerns and questions. If so, would the reviewer please consider increasing their score to show support for our work?

We thank the reviewer for taking the time to review our paper and provide us with valuable insights. We appreciate the acknowledgement of the novelty of our method, as well as clear demonstrations of improvements using our method across three representative benchmarks.

W1: sample-efficiency and generalizability

We define sample-efficiency to mean that the agent requires few human labeled demonstrations in order to achieve high performance. In this work, we assume access to a small set of human labeled demonstrations to train a base policy. With no further human labeled demonstrations, we can synthesize demonstrations using the LFM automatically, with which we can improve policy performance. Consequently, we describe the proposed method as sample-efficient because it improves policy performance with no additional human labeled demonstrations.

We define generalizability to mean that the agent generalizes to new environments not seen during training. Let’s consider the game NetHack. In NetHack, there is one implicit goal for the agent, which is to obtain the highest score it can, given the rules of NetHack. In other words, every NetHack agent is given the implicit instruction “get the highest score you can in NetHack, subject to the rules of the game”. In contrast, all three environments we investigate have different instructions between training and evaluation. These environments require generalization to new scenes (like NetHack) as well as to new instructions (unlike NetHack). For ALFWorld, the agent may be trained to “find and wash glasses” and “put apples in the fridge”, but during test evaluation it may be asked to “find apples, wash them, then put them on the dining table” in new rooms. Similarly, in ScienceWorld the agent is required to follow new compositional instructions it has not seen before during training, in new spaces (e.g. the determine the boiling temperature of a new substance). In Touchdown, the agent is required to navigate between new starting and end points in new neighborhoods. Consequently, we describe the proposed method as one that generalizes because it improves performance on new environments with new scenes and new instructions not seen during training.

We will clarify these two points by precisely defining sample-efficiency and generalization in the manuscript. We thank the reviewer for identifying this point.

W2: desirable behaviour

We want to emphasize that the proposed framework consists of two components. The first component involves learning a language feedback model (LFM). The second component involves using this LFM to improve policy. The D_k the reviewer refers to has to do with the second (improving policy), not the first (learning a feedback model). We will summarize both components here.

We start from a base policy trained from a small set of human demonstrations. In the first component of learning a LFM, we roll out the base policy to collect trajectories. These trajectories are given to an LLM to create a dataset (this is not D_k) to train the LFM. This dataset consists of tuples of (agent behaviour, LLM feedback). We train a smaller LFM to imitate LLM feedback. That is, given the agent behaviour, the LFM critiques in natural language whether this behaviour is productive. We then freeze this LFM.

In the second component, we use the trained LFM to improve the base policy. First, we roll out the base policy to collect trajectories. We then use the LFM to predict whether each step in the trajectory is productive. We take the subset of productive behaviour (eg. context and agent action) and add it to our collection of productive demonstrations. Then we update the policy by training on this collection of demonstrations. We do this in multiple rounds, in each round K, we roll out the policy, identify good behaviour, and add it to the collection of demonstrations. We refer to the collection of demonstrations during round k as D_k.

W3: compare expert LLM to policy improvement with LFM

We do perform two such experiments in our ablation study. Experiment A compares to using GPT4 (0315) on-policy zeroshot. Table 3 shows that our method (and other methods that train a smaller policy model) achieves significant improvements over zero-shot GPT4. Experiment B uses GPT4 to label what the agent should do, as opposed to critique how the agent performed. We then train a policy using demonstrations as well as GPT-4 labeled actions. This method of using LLM as expert policy to label actions (ActPred), underperforms using LLMs to train feedback models (LFM).

Q1: number of rollouts

Dataset statistics are shown in Appendix C Table 6. To train the initial policy, we use 3.5k demos for ALFWorld, 3.6k for ScienceWorld, and 6.5k for Touchdown. These correspond to 1 demo for each unique environment. When rolling out, we perform one rollout per unique environment. Consequently we have an identical number of rollouts as initial demos. That is, we perform 3.5k rollouts for ALFWorld, 3.6k for ScienceWorld, and 6.5k for Touchdown.

Q2: desirable actions vs. transition function

The revised trajectories correctly reflect env dynamics because they are steps that took place during the rollout, according to env dynamics. This observation also explains underperformance of ActPred. In ActPred, an LLM is used to label what actions the agent should take. However, the LLM can hallucinate actions, resulting in demonstrations that do not correctly reflect env dynamics. In contrast, our proposed method only identifies real actions that took place as productive behaviour. Consequently all demos collected using LFMs correctly reflect environment dynamics. This also explains why LFM improvement is larger than ActPred.

Summary

We sincerely thank the reviewer for taking their time to help us improve this work. We hope we have addressed the reviewer’s concerns and questions. If so, would the reviewer please consider increasing their score to show support for our work?

Dear Reviewer,

We wanted to send a friendly reminder that we are awaiting your response. With the deadline for the reviewer-author discussion approaching on August 13, we would greatly appreciate it if you could provide feedback at your earliest convenience. We hope we have addressed your concerns and questions. If so, would you please consider increasing your score to show support for our work?

Sincerely, Authors

Thank you for providing thoughtful responses to my comments. For now, I maintain my initial rating. However, I am open to AC's decision on this paper.

We thank the reviewer for taking the time to review our paper and provide us with valuable insights. We appreciate the acknowledgement of the novelty of our method, as well as clear demonstrations of improvements using our method.

W1: interpretable feedback

To clarify, we have 2 claims. First, policy improvement from LLMs feedback is better than using LLM as policy. There are two experiments that support this. a) we compare to using GPT4 (0315) on-policy zeroshot. Table 3 shows that our method achieves significant improvement. b) we use GPT4 to label what the agent should do, then train a policy using demos as well as GPT-4 labeled actions. This method of using LLM as expert policy to label actions (ActPred), underperforms using LLMs to train feedback models (LFM).

The interpretable feedback experiments show that instead of producing LFMs that only identify good behaviour, we can train descriptive LFM-Ds that also state why a behaviour is good. We show that LFM-Ds perform similarly to LFMs, achieving both interpretability and high levels of policy improvement.

W2: related work

Thank you, we will add this. LFM differs from MOTIF in that the former improves policies via imitation learning while the latter derives a reward model for RL. While MOTIF shows results on a single game NetHack, we show results across three different settings in household (ALFWorld), scientific experiments (ScienceWorld), and real-scene navigation (Touchdown). In the latter two settings, RL is substantially worse than LFMs (https://arxiv.org/abs/2110.10661). In the future, we will investigate RL using language feedback for multimodal grounded settings.

W3: adaptation rounds

Thank you for the comment. We describe one-round adaptation in section 5.2 on line 249. We will describe two-round adaptation in the manuscript. Round-wise adaptation is described on ​​line 148: we use the policy from the previous round to perform rollouts, then filter for good behaviour using the trained LFM, then imitate them to improve the policy.

W4: small things

Thank you, we will make the corresponding corrections to the manuscript.

Limitations

We discuss limitations and broader impacts of this method in Appendix Section A and B, we will move them before references.

Q1: efficient vs. naive feedback learning

Let’s suppose we have 20 steps. The "naive" method queries the LLM each step, resulting in 20 queries. The kth query has steps 1…k-1 in the context and asks the LLM whether the kth action is productive. The "efficient" method batches feedback requests into 1 query, which asks the LLM to list which steps were productive (Figure 2). The efficient method is much cheaper, requiring 20x fewer API calls to the LLM (line 130). Once feedback is collected, the subsequent training would be identical.

Q2: reward function

We do not train a reward function - we only use imitation learning for policy improvement, not reinforcement learning.

Q3: iteration/rounds

After training a LFM, we can improve the policy in rounds. In round 1, we start with the base policy P1 trained on initial demos. We roll out P1, and then identify its good behaviour using our LFM. We then add the good behaviours into the demo set and train the policy P2. We then repeat this in round 2, where we identify good behaviours using P2, then use those to train P3, and so on.

Q4: training vs evaluation goals

Let’s consider NetHack. NetHack has 1 implicit goal for the agent, which is to obtain the highest score it can. In contrast, we consider settings w/ different instructions between train and test, which require generalization to new scenes (like NetHack) and new instructions (unlike NetHack). For ALFWorld, the agent may be trained to “find and wash glasses” and “put apples in the fridge”, but during test it may be asked to “find apples, wash them, then put them on the dining table” in new rooms. Similarly, in ScienceWorld the agent is required to follow new instructions in new spaces (e.g. the determine the boiling temperature of a new substance). In Touchdown, the agent is required to navigate between new starting and end points in new neighborhoods.

Q5: summarize with “Before”

In Table 2, the “Before” is not a summary, it is the observation of the step right before the window starts. In this case step 20. We will clarify this in the manuscript.

Q6: Token limitation for LLM usage

As described in Table 3, we limit LLM interactions to 100k output tokens. We do not limit input tokens as they are much cheaper than output tokens. We collect feedback for as many windows as possible until we run out of 100k output tokens, then we use this feedback to train the LFM. For ActPred, we label actions for as many steps as possible, until we run out of 100k output tokens, then we use this annotated set along with demos to train the policy. We will clarify this in the manuscript. Thank you!

Q7: 20 steps

Empirically, 20 steps fits over 90% of observation windows into the model’s context length (8k for GPT4 0315). We will investigate using new LLMs w/ longer context (128k) to train LFMs.

Q8: Data ratios and sample diversity

This ratio differs between settings (60% not productive for ALFWorld, 70% ScienceWorld, 90% Touchdown). All settings use 10k 20-step windows as feedback data.

We have diversity across tasks, instructions, and envs. LFM data collection is biased by how good the base policy is. If the base policy is bad at task A and good at B, then we tend to identify more “good behaviour” from trajectories from B. In this work, we do not do sophisticated filtering to rebalance on a task level, but we are interested in exploring this in the future.

Q9: new environments

Please see Q4.

Summary

We sincerely thank the reviewer for taking their time to help us improve this work. We hope we have addressed the reviewer’s concerns and questions. If so, would the reviewer please consider increasing their score to show support for our work?

We thank the reviewer for taking the time to review our paper and provide us with valuable insights. We appreciate the acknowledgement of the strengths of our paper, including its clarity, novelty, and demonstration of generality across a range of tasks.

W1: improvement from distilling language feedback from stronger models

Our primary contribution is a novel framework that combines language feedback with policy improvement. While it is true that using a single model for both policy learning and language criticism is possible, our results show that a weaker feedback model results in insignificant policy improvement (we show results using Llama2 70B in the Appendix, which is better than Llama3 8b on most benchmarks: https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md#benchmark).

What our findings suggest are that

1. a very large model (let’s say an LLM) is difficult to use as a policy because it is expensive to train and slow to inference.
2. a small model can be trained to provide a reasonable tractable policy, however it is not capable of providing high quality feedback (without training on such feedback).

Our framework uses existing large LLMs to provide high quality feedback, without further training, to improve small tractable policies for specific environments we care about.

W2: ALFWorld baselines

We also appreciate the feedback on the comprehensiveness of our experiments. Although Table 3 presents a paper from 2021, it is the best-performing imitation learning method without external knowledge on ALFWorld according to the ALFWorld authors. As mentioned in W1, we consider the setting where the LLM is not available during test time. Consequently we did not use reflective techniques nor CoT techniques that tend to show strong benefits with very large models (50B+, Li et al https://arxiv.org/abs/2306.14050). Are there specific works the reviewer would like comparisons to?

W2: reinforcement learning

We reference prior results that use RL on ALFWorld and Touchdown (https://arxiv.org/abs/2110.10661). Crucially, we show that in these challenging environments, RL underperforms LFM (and the baseline model) after training for 10 million steps. In contrast, ALFWorld episodes typically have <30 steps. Training on the demonstration dataset amounts to 30 * 3.5k ~ 100k steps. LFM improvement using one rollout per training environment, as is the case in our experiments, results in another 100k steps. For Touchdown, episodes are typically <200 steps. Demonstration steps and LFM improvement steps are consequently 1.5 million steps each. For both these cases, imitation learning and LFM improvement require substantially fewer steps than RL and achieve substantially higher task success rate (e.g. 64 vs 23 ALFWorld, 60 vs 15 Touchdown).

W2: reflection

We are very interested in adapting LFM to RL and with reflection, as the reviewer suggested. In this preliminary work, we wanted to scope our problem to investigate the most fundamental setting where the large model (the LLM LFM) is not used during policy improvement. Improving policies with LLMs in the loop (e.g. to provide rewards, to provide reflection) results in hundreds of steps per rollout, and is very expensive for environments we consider. However, we would like to investigate this direction in future work, for instance by learning a small language reflection model in parallel with the policy.

Q1: references

We appreciate the reviewer’s suggestions of several relevant works, including [1], [2], and [3]. We agree that these papers share similar motivations and approaches with ours, and we will include them in our related work section. In the current manuscript, we specifically discuss the ReAct, Reflexion, and InnerMonologue line of work in the last paragraph of our related works section.

Q2: GPT-4-0315

Unfortunately due to internal infrastructure policy it was the only GPT-4 model widely available to us at the time of our experimentation. We also experiment with Llama2 70B, the results for which are in our appendix. In ongoing and future work, we are investigating more recent LLMs, including VLMs, as policy critics.

Q3: Chain of Thought

In this work, we make the assumption that the LLM is not available during test time, only training time. Current evidence suggests that CoT is only significantly helpful when the base model is sufficiently large (50B+, Li et al https://arxiv.org/abs/2306.14050). As mentioned in response to W1, a very large model (let’s say an LLM) is difficult to use as a policy because it is expensive to train and slow to inference. Consequently, in this preliminary work, we do not investigate CoT. In future work, we would like to study how to distill large base models into small models such that they are capable of providing high quality CoT. Our result on LFM-D is a first step towards this direction, where we ask the model to provide evidence for its critique, akin to asking a question answering model to provide reasoning steps for its answer.

Q4: LFMD and LFD and baseline comparison

We want to make an important clarification that LFM and LFM-D do not perform rollouts on the test set - they perform rollouts only on the training set for policy improvement. The only method that performs improvement via test-set adaptation is LFM-A, which achieves significant improvement via adaptation compared to LFM. The comparison to LFM-D (D for descriptive) serves only to show that we can increase the interpretability of the feedback model without suffering performance degradation.

Summary

We sincerely thank the reviewer for taking their time to help us improve this work. We hope we have addressed the reviewer’s concerns and questions. If so, would the reviewer please consider increasing their score to show support for our work?

Dear Reviewer,

We wanted to send a friendly reminder that we are awaiting your response. With the deadline for the reviewer-author discussion approaching on August 13, we would greatly appreciate it if you could provide feedback at your earliest convenience. We hope we have addressed your concerns and questions. If so, would you please consider increasing your score to show support for our work?

Sincerely, Authors