True Knowledge Comes from Practice: Aligning Large Language Models with Embodied Environments via Reinforcement Learning

We thank the reviewers for their time and comments. Please see the above official comment to all the reviewers first and then our answers to your questions. We hope our answers will clear up the doubts about our work, and please let us know if there is any other clarification we can provide.

---

Q1: As for now it is hard to consider LoRA to be part of the proposed method: hard to see the benefits of the LoRA contribution to sample efficiency and no catastrophic forgetting without an ablation on with and without LoRA. The contribution of the method seems to be the action normalization and some prompt design (qualitatively assessed by the authors in 4.4). Could you add experiments on these points?

A1: Thanks for your valuable comments. We want to point out that we use LoRA mainly for parameter efficiency instead of sample efficiency. Even using LoRA, TWOSOME still has about 500M parameters to update. In contrast, the PPO network only has about 10K parameters to update, which is 50,000X less than TWOSOME. But TWOSOME still exhibits much better sample efficiency than PPO with the same amount of data, we attribute this to the prior knowledge of LLM, instead of LoRA.

As mentioned in the first point of the above official comment, in the section of Parameter-efficient Fine-tuning Architecture, one of our main baselines, GLAM requires 8 NVIDIA A100 80GB GPUs to train their framework with a much smaller LLM, Flan-T5 780M without using LoRA. They report using 18880 GPU hours on Nvidia A100 80GB in total. Our model is 9X larger than theirs. We estimate that at least 32 NVIDIA A100 80GB GPUs are required to conduct the experiments to train LLaMA-7B without using LoRA, otherwise, there is even no way to load the model and do a complete backpropagation. Training LLaMA-7B without using LoRA is significantly far beyond most research labs’ ability, including us. It also shows the importance of our non-trivial parameter-efficient architecture, which enables the whole framework to run in a single A100 40G GPU. It greatly lowers the threshold for researchers to conduct the following research.

---

Q2: Paper mentions that (Carta et al, 2023) “focus on primitive actions in toy environments without rich semantics“. I am not sure actions are primitive in (Carta et al, 2023) but rather semantically described actions, e.g. “go forward”, “turn left”, “drop and toggle”. What makes them more “primitive” seems to be the environment they chose for experimenting (BabyAI vs Overcooked/VirtualHome).

A2: Yes, you are correct. That is one of the main reasons why we do not think BabyAI is a suitable environment to exhibit the performance of LLM-based agents. We argue that LLM-based agents should be deployed in environments as close to daily life or their learned specific domains as possible, like SayCan with high-level daily skills (household robotics). Then they can make full use of the prior knowledge learned in the training process. If LLMs are not familiar with the observation and actions, not only will they fail to apply their learned prior knowledge but also dramatically slow down the training process for their tremendous parameters. It also emphasizes the importance of prompt design.

As we mentioned in the above official comment, both the observation and action in BabyAI have relatively poor semantics. Here is an example prompt used by GLAM in BabyAI, “You carry a blue ball, You see a wall 4 steps forward, You see a wall 2 steps left, You see a grey key 1 step right and 1 step forward, You see a red box 2 steps forward”. It is rare for LLMs to see a corpus like this in the training process and we all know that even GPT-4 struggles with simple navigation tasks. It might partially explain why GLAM spent as much as 18880 GPU hours on Nvidia A100 80GB to get the results. LLM-based agents should not be applied to solve tasks in these environments under primitive settings. They are exactly the strong areas of RL agents. This is the reason why we insist on deploying our framework in daily environments with rich semantics for observation and actions.

---

We thank the reviewers for their time and comments. Please see the above official comment to all the reviewers first and then our answers to your questions. We hope our answers will clear up the doubts about our work, and please let us know if there is any other clarification we can provide.

---

Q1. How does the word normalization help LLM agents to learn better policy? What is the intuition of the word normalization?

A1: For the intuition of the word normalization, we provided a detailed explanation in Section 4.1 in the main paper. Briefly speaking, word normalization can provide a more accurate and reasonable initialized policy for the LLM agents to interact with environments. So reasonable good actions, no matter whether they are long or short, always have higher probabilities to be sampled at the beginning of the training process, which would greatly accelerate the exploration process.

We observe that the probability of each token is always less than 1. Therefore, longer action prompts tend to have lower token-level probabilities, even though the longer action prompts may be more reasonable in the current situation. So we propose the token normalization to normalize the joint probability by the number of tokens (length).

For example, if we have two actions, the first action has 20 tokens. To to simple, let the probability of each token in this action be as same as 0.8, which means this action is well-written and highly related to the task and the current observation. However, the joint probability of the action is 0.8^20 = 0.011. In contrast, the second action has only two tokens, with a probability of 0.2 for each token, which means this action is much less relative to the task and the observation, and even the relation between these two tokens is quite weak. The joint probability of the second action is 0.2 * 0.2 = 0.04. Though each token in the first action is much higher than the second action, the first action finally has much fewer chances to be chosen, compared to the second action, which is not a desirable result. With the use of the token/word normalization by taking the lengths of the action into consideration, the joint probability of the action is still 0.8 and the joint probability of the second action is still 0.2 The distribution is much more reasonable. And the first action has a higher probability of being sampled.

However, we find the token normalization is slightly excessive. We observe that if a word is divided into several tokens, the first token usually has a relatively low probability, while the probabilities of the rest of the tokens tend to be remarkably high, which are often almost close to 100%. Therefore, it is more reasonable to regard the several tokens made up of one word as an integrated symbol. Then we propose our word normalization method to normalize the joint probabilities of actions by the number of words. Our experiments also demonstrate the same conclusion that TWOSOME without normalization is very unstable and fails to learn the optimal policy. Both TWOSOME with token normalization and word normalization have much better performance. And TWOSOME with word normalization converges much faster than token normalization since token normalization is slightly excessive.

In Appendix F, we also provide a detailed comparison and analysis of the probabilities generated by TWOSOME without normalization and with token/word normalization across all the tasks.

---

Q2. Even though this paper proposes a promising direction towards fine-tuning a LLM agent with a RL algorithm, the main contribution seems rather marginal. It seems to compensate a text action with larger number of words, but I am not sure that this kind of simple division is a proper way to normalize varying length text actions sampled from a LLM policy.

A2: Thanks for your valuable comments. We kindly remind the reviewer that the contribution of the whole framework may be underestimated. There are some previous similar works like SayCan and GLAM, but we managed to polish and optimize every module in the framework, resulting in high parameter-efficiency and better generalization and sample-efficiency. Please refer to the first point of the above official comment, About Technical Contribution, for a clear explanation.

For the word normalization, we have introduced it in the previous question, Q1. We would not claim that our solution is the ultimate solution to the issue, but it turns out to be an easy-to-implement and feasible way to alleviate the issue, which is well acknowledged by the other three reviewers. Our method has shown significant improvement including both performance and stability according to the corresponding experiments and ablation studies.

Not like the traditional natural language generation task where the output can be as long as several hundred tokens. The actions in traditional decision-making tasks are typically composed of a verb-object structure (e.g. pick up the apple), or Verb + Object + Preposition structure (e.g. put the apple on the table). They are usually within 20 tokens. Our experiments have covered the most common cases, which show that even in this range, the length of the prompts still has a significant influence on the joint probability of the actions and introduces much instability to the training process.

---

Q3. I am not sure that TWOSOME can properly learn optimal policy in a complex environment like VirtualHome. When showing the performance of TWOSOME in VirtualHome, the authors masks out unrelated actions to reduce the complexity of the action space.

A3: Sorry for the confusion. We intend to use action masking to prove that compared to traditional RL methods, like PPO, it is more convenient for our method to mask actions and solve the tasks with a large action space. When doing action masking, the network of PPO still needs to calculate the values of the masked actions, but we just do not use their values to form the policy in the final step. However, for our method, we just directly delete the masked actions from the action prompt list in the input, which also saves the computation for the values of these masked actions. Furthermore, it is also worth noting that our framework is also very flexible in extending the action space, by just directly add new actions to the action prompt list as the input, which is impossible for traditional RL methods. They need to train new agents from scratch with the new extended action space.

Moreover, we argue that struggling with large action space has been one of the main challenges for all decision-making methods, which is not a unique weakness of our method. Action masking is a very common trick to deal with large action space in decision-making tasks. The action space of all the environments is also a subset of the action space of reality, where these environments also mask out unrelated actions to reduce the complexity. It is fairly reasonable to mask invalid actions and unrelated items in the action space.

Finally, we want to kindly remind the reviewer that even with the use of action masking, the difficulty of VirtualHome might be underestimated. Because we adopt a sparse reward setting, only when the agent successfully finishes the task will it receive +1 reward, as shown in Figure 5, even PPO with action masking, still fails to solve the task of Entertainment. In contrast, our method, TWOSOME with word normalization manages to learn the optimal policy, which already shows that our method has the ability in a complex environment.

---

Q4. Recently, in RLHF, some enhanced RL algorithms such as Direct Policy Optimization (DPO) and Pair-wise Proximal Policy Optimizaition (P3O) have been proposed. Can these enhance RL algorithms be applied to solving decision making tasks? And then, can we expect some performance improvements of TWOSOME?

A4: To our understanding, compared to RLHF, both DPO and P3O attempt to directly use the preference data to train LLMs, saving the effort of training an accurate and explicit reward model or value function. However, in the traditional decision-making setting, the reward function is already contained in the environment. We can directly use rewards to train our models instead of using preference pairs to infer the reward/value distribution. Therefore, DPO and P3O do not help in our standard RL decision-making settings.

Dear reviewer fSyb:

We just added some new results in Appendix G.2 (on the last page of the paper), to show that even without action masking, our method, TWOSOME with word normalization, can still manage to learn the optimal policy, which shows the superior robustness and performance of our method and demonstrates that our method has the ability to deal with large action space.

We hope these new results can solve the reviewer's doubts. As the deadline for the rebuttal period is approaching, is there any other clarification we can provide?

We thank the reviewers for their time and comments. Please see the above official comment to all the reviewers first and then our answers to your questions. We hope our answers will clear up the doubts about our work, and please let us know if there is any other clarification we can provide.

---

Q1: The idea of querying joint probabilities of each valid action with LLM to form a policy is explored in previous works such as GLAM (Carta et al., 2023). The reviewer found, in the current version, the credit is not clearly given to authors of previous works.

A1: Thanks for pointing this out. To our best knowledge, it is SayCan that first proposed to use joint probabilities of each valid action to form a policy. And GLAM also acknowledges this point. Here is an excerpt from GLAM in Page 5, “Finally, Ahn et al. [2022] proposed to directly use the LLM to compute the (log) probability of each action by computing the conditional probability of each token in action given the prompt p”, where Ahn et al. [2022] refers to SayCan. We also mention this in our 4.1 Section that the way we used to form the policy is very similar to SayCan without the affordance function. We will add GLAM behind SayCan, since it also uses the same way to form the policy.

---

Q2: The reviewer has some concerns on the baselines used in the experiments. There are many existing works that leverage LLM to form RL policies [1, 2, 3]. Comparing approaches that also leverage knowledge in LLM could make the experimental section more convincing.

[1] Pre-Trained Language Models for Interactive Decision-Making, Li et al., NeurIPS 2022
[2] Do As I Can, Not As I Say: Grounding Language in Robotic Affordances, Ahn, CoRL 2022.
[3] Grounding large language models in interactive environments with online reinforcement learning, Cartra et al, 2023

A2: Thanks for your pointing out. Please refer to the third point of the above official comment, About Baselines Selection, for a clear explanation. Actually, we do compare our method with SayCan[2] and GLAM[3]. Because we need to make some modifications and apply them to our tasks and domains, and we found that they also served as our ablation studies, for consistency, we call SayCan “TWOSOME without finetuning” and GLAM “TWOSOME without action prompt normalization” or directly “TWOSOME without normalization”.

As for [1], it is actually a purely supervised-learning-based method. Instead of explicitly calculating the probabilities of each action to form standard RL policies, they directly generate actions using LLMs. The loss they show in Equation 1, is exactly the same as the autoregressive loss. They just formulate the loss into a policy-like format. There is no difference between autoregressive supervised learning and imitation learning if actions are the only output of LLMs. And the authors also do not claim they are an RL-based method. In their paper, they also rely on pre-collected high-quality expert datasets. We do not think that they directly compete with our method. We can first apply their methods to do offline training if provided with a high-quality dataset, and then use our method to do online training by interacting with the environments.

---

Q3: The technical contribution of the paper is somewhat limited. Specifically, the idea of forming valid policy is proposed by previous works and LoRA for fine-tuning is a standard method. The review found the main new method is the proposed normalization approach. That to be said, the reviewer still considers the combination of existing methods might be valuable to our embodied AI community.

A3: We appreciate that the reviewer acknowledges our contribution to the embodied AI community. Please refer to the first point of the above official comment, About Technical Contribution, for a clear explanation.

---

Minor Q4: In Eq (2), using ak in the summation is confusing.

A4: We greatly thank the reviewer for catching the typo. We have corrected it in the latest version.

We thank the reviewers for their time and comments. Please see the above official comment to all the reviewers first and then our answers to your questions. We hope our answers will clear up the doubts about our work, and please let us know if there is any other clarification we can provide.

---

Q1: The paper does not provide a theoretical analysis of the proposed framework.

A1: Thanks for pointing it out. We indeed do not provide much theoretical analysis of the proposed framework because the whole framework is still basically under the traditional online actor-critic (PPO) framework, where the convergence of PPO should also apply. Though LLMs are usually transformer-based architecture, the last layer of LLMs is still an MLP layer to map the values generated by Transformer blocks to the probabilities of each token in the vocabulary. We use the values to form the policies, which have no essential difference from the traditional RL agents, which usually have some CNN/MLP layers and use the output of the last MLP layer to form the policies. Finally, LoRA only updates part of the parameters, which may indeed prevent RL agents from learning the optimal policies. Empirically, LoRA shows superior performance not only in our work but also in tremendous CV/NLP works. We agree that it is non-trivial to provide a theoretical analysis of LoRA, especially in RL settings, but it is beyond the scope of this work.

---

Q2: The paper does not evaluate the performance of TWOSOME on a wider range of tasks and environments.

A2: Thanks for your valuable comments. We fully understand the reviewers’ wishes to see more results in different scenarios. Please refer to the second point of the above official comment, About Evaluations on More Tasks and Environments , for a clear explanation.

---

Q3: The paper does not discuss the potential limitations of the proposed framework.

A3: Thanks for pointing it out. Due to the limitation of pages, we do not provide a detailed analysis of the potential limitations. However, we do mention our limitations in the final Discussion and Conclusion Section, ‘However, our method suffers a major limitation that training a PPO agent from scratch is far faster and cheaper than finetuning an LLM. TWOSOME needs to feed all the valid actions to the LLMs for every action sampling, resulting in multiple times the amount of computation and a small batch size.’ Another limitation is that because we use an LLM as the backbone of our framework, we need to convert the observation into text and feed it into LLM, which inevitably causes information loss. The future work will be replacing the LLM with a VLM to directly use the image as input. We will put them in a separate section called Limitations and provide a more detailed discussion in the camera-ready version as there is more space provided.

---

Q4: The paper does not introduce anything novel, rather just combines existing components together to generalise to new tasks (apart from word normalisation which is fairly trivial).

A4: Thanks for your valuable comments. We kindly remind the reviewer that the contribution of the whole framework may be underestimated. There are some previous similar works like SayCan and GLAM, but we managed to polish and optimize every module in the framework, resulting in high parameter-efficiency and better generalization and sample-efficiency. Please refer to the first point of the above official comment, About Technical Contribution, for a clear explanation.

---

Q5: The paper does not compare to other LLM fine tuning baselines which generalise to new unseen tasks.

A5: Thanks for your valuable comments. To our best knowledge, only GLAM uses RL to train LLMs in embodied environments, which also serves as one of our main baselines. For the other supervised-learning-based methods, we do not think that they directly compete with our method. We can first apply their methods to do offline training if provided with a high-quality dataset, and then use our method to do online training by interacting with the environments. Please refer to the third point of the above official comment, About Baselines Selection, for a clear explanation.

2. About Evaluations on More Tasks and Environments
Our methods do not rely on any assumptions or features of the environments and tasks, which are compatible with any standard RL decision-making settings. We have tried our best to select the most representative environments and tasks to show the generalization of our method, a classical sequential decision-making environment, Overcooked, and a simulated household environment, VirtualHome with more actions and items available.

One of our baselines, GLAM, only tested their methods in a simple grid world environment, BabyAI with easy tasks, such as going somewhere or picking up something. Our environments are much more complex and practical. And our well-designed tasks are a combination of several of their tasks. For example, in our VirtureHome task, the agent needs to find chips and milk, get close to them, grab them, carry them to the living room, put them on the coffee table, turn on the TV, get close to the sofa, and sit on the sofa. We fully understand the reviewers’ wishes to see more results in different scenarios. There are also some previous impactful works[1][2], only evaluated their methods in Overcooked[1] and VirtualHome[2]. We believe our current environments and tasks are enough to show the robustness and generalization of our methods.

Besides, we want to kindly remind the reviewers that the computational consumption of our experiments might be underestimated. We have three fine-tuning-based methods to compare, TWOSOME without normalization and with token/word normalization. Each method needs to be evaluated on four different tasks in the two environments. And according to RL standard, the performance of each method should be reported on the average of at least three seeds to show robustness. Thus, to obtain the results shown in Figure 5, we have to finetune at least 3X4X3=36 LLaMA-7B models without taking hyperparameter tuning into account. It is difficult for RL-based methods to get evaluated in dozens of tasks like supervised-learning-based methods since they only need to train the model once with a pre-prepared high-quality expert dataset, while we need to collect and learn the noisy low-quality data from scratch.

---

3. About Baselines Selection

We find that some reviewers may get confused about the baselines we use and miss the discussions in the extended related work in Appendix A. Due to the limitation of pages in the main paper, we had to put more discussions in the Appendix. We will move them to the main paper in the camera-ready version since more space will be provided.

As introduced in Section 5.1, we compare the performance among five methods, “PPO (adapted from CleanRL (Huang et al., 2022a)), TWOSOME without finetuning (similar to SayCan (Brohan et al., 2023) with affordance value set to 1), TWOSOME without action prompt normalization (similar to GLAM (Carta et al., 2023) under decoder-only architecture and high-level actions) and TWOSOME with token normalization or word normalization.” So we do adopt SayCan and GLAM as our baselines. Because we need to make some modifications and apply them to our tasks and domains, and we found that they can also serve as our ablation studies, for consistency, we call SayCan “TWOSOME without finetuning” and GLAM “TWOSOME without action prompt normalization” or directly “TWOSOME without normalization”.

As for the other supervised-learning-based methods, like [2] and [3]. They usually require pre-collected high-quality expert datasets. It is also the key difference between reinforcement learning and supervised learning. In our assumption, the agent does not have any specific prior knowledge of the environment and it needs to interact with the environment by itself, which is usually a more generalized and realistic way. Moreover, our method is not contradictory to the supervised learning methods like [2] and [3]. A high-quality dataset can always improve performance. We can first train the LLM with the provided dataset by supervised learning for offline training and then use our method for online training. Because LLMs are originally trained by supervised learning. It is easy to shift between supervised learning and reinforcement learning. It is another advantage of our method, compared to traditional RL methods and supervised-learning-based methods. So these supervised-learning-based methods do not directly compete with our method.

[1] Strouse, D. J., et al. "Collaborating with humans without human data." Advances in Neural Information Processing Systems 34 (2021): 14502-14515.
[2] Xiang, Jiannan, et al. "Language Models Meet World Models: Embodied Experiences Enhance Language Models." Advances in Neural Information Processing Systems 36 (2023).
[3] Li, Shuang, et al. "Pre-trained language models for interactive decision-making." Advances in Neural Information Processing Systems 35 (2022): 31199-31212.

To all reviewers:

We have added two new ablation studies in Appendix G.

1. In Appendix G.1 (on the last but one page of the paper), the new experiments show that our novel parameter-efficient architecture can not only dramatically save the GPU memory but also improve stability and accelerate convergence.
2. In Appendix G.2 (on the last page of the paper), the new experiments show that even without using the action masking in both VirtualHome tasks, our best method, TWOSOME with word normalization can still manage to learn the optimal policy, which shows the superior robustness and performance of our method and demonstrate that our method has the ability to deal with large action space.