Large Language Models as Generalizable Policies for Embodied Tasks

We thank the reviewer for the comments and suggestions. We address the reviewer’s points below.

1. The proposed method uses an MLP to map LLM output to discrete actions. I am not sure if the policy can fully leverage the world knowledge from the LLM.

Modifying LLaRP to operate directly in the language token space rather than through the action decoder MLP is an interesting extension. We note extending LLaRP to implement this is non-trivial. Firstly, a single action in Language Rearrangement consists of at most 4 tokens (for example, “nav left counter”). This LLaRP variant requires predicting the correct sequence of 4 tokens from a vast number of tokens which decode to an invalid action. Secondly, it slows the RL process, because now the policy needs to auto-regressively predict 4 tokens per action prediction step rather than the single prediction needed in LLaRP. Despite these challenges, we agree this is an exciting direction for a follow up work and we updated the conclusion to reflect this (changes in red).

We also note that we empirically demonstrate the policy is able to use the world knowledge from the LLM since LLaRP generalizes better than LLaRP-Scratch which does not have the pre-trained LLM weights.

2. Why not use a VLM backbone but instead use a language-only backbone?

It is possible to change the network weight initialization in LLaRP with VLM weights instead of using VC-1 and LLaMA as in the paper. However, we started from LLaMA and VC-1 since we found the zero-shot performance of LLaMA was higher than the Flamingo VLM (12% vs. 6% success rate).

Dear Reviewer 92Kq, we would be grateful if you can comment on whether our response addressed your concerns or if issues remain.

We thank the reviewer for the comments and suggestions. We address the reviewer’s points below.

1. Incorrect claims and missing related work: the paper states "To our knowledge, there are no prior work which demonstrate that the linguistic knowledge in LLMs can be used in online RL problems to improve generalization in embodied AI settings". There are many examples in the literature that actually have demonstrated that.

We corrected this claim to “To our knowledge, there is no prior work which demonstrates that LLMs can be used as vision-language policies in online RL problems to improve generalization” (changes in paper in red). This updated claim is accurate and the mentioned works do not demonstrate this. We apologize for this: this was an error that played out in the final editing of our first submitted draft (please refer to the general response). We compare to the works mentioned by the reviewer in detail below:

• SayCan, CodeAsPolicy and ProgPrompt are zero-shot applications of LLMs that require describing the environment in text [1-3]. LLaRP adapts the LLM for visual decision-making with online RL. By online RL, we mean learning from interactions with the environment.
• PercieverActor, EmbodiedGPT and PaLM-E [4-6] learn from static expert collected datasets with supervised learning. LLaRP doesn’t need expert data and instead learns with RL.
• ELLM uses LLMs as a reward to guide non-LLM policies from text-based state descriptions [7]. We use the LLM directly as the decision-making policy from visual observations.
• HELM uses LLMs (CLIP) as a “memory mechanism”, which we see as orthogonal from our contribution of using LLMs directly for policy decision-making [8]. Unlike LLaRP, HELM does not modify the LLM, use it for better generalization, or use it directly for decision-making.

2. Lack of task details.

We expanded Supp. A.4 to describe the simulation in detail. We also added a description of the key simulation details in a new paragraph in the main paper (Sec 4.1). Due to limited space, we described how the agent executes the skill and skill definitions in Supp. A.3 and the PDDL definition in Supp. A.1.

We never claim the defined PDDL is a contribution, it is only used in the implementation of the Language Rearrangement task (which is a contribution).

3. Lack of comparison to existing benchmarks.

Thank you for the suggestion, we include a table highlighting the differences between Language Rearrangement and prior benchmarks below without citations and in full in Table 4, supplementary A.7.

Language Rearrangement has more instructions than prior benchmarks. It has almost 18x more instructions than ALFRED (151k vs 8k) and 8x more instruction templates than CALVIN (282 vs. 35). “# instructions” refers to the number of linguistically distinct instruction types, meaning “pick the apple” and “pick the pear” are counted as two separate instructions. “# Instruction Templates” refers to distinct instructions regardless of specific entities, so the same picking example is only counted as a single instruction. Also unique about Language Rearrangement is that it has dense rewards for language specified tasks. All other tasks with language instructions only define sparse success-based rewards. Since Language Rearrangement is implemented with Habitat 2.0, it also inherits the same fast simulation speeds.

4. Limited comparison to existing work.

We thank the reviewer for suggestions around additional baselines, but believe they are not suited for our problem setting.

• CodeAsPolicies [2] relies on perception modules which Language Rearrangement doesn't have. This is also similar to our ZS-ChatGPT baseline which replans based on environment feedback.
• PercieverActor [4] requires offline data for imitation learning and depth observations for constructing point clouds. Language Rearrangement assumes access to neither.
• While Semantic HELM [8] can be combined with LLaRP and baselines to increase learning efficiency and interpretability it does not address how to generalize to unseen instructions.

Our experiments compare LLaRP to three RL and three zero-shot baselines. We compare three different LLM/VLMs (LLaMA, ChatGPT, and Flamingo).

Citations

[1] Ahn, Michael, et al. "Do as i can, not as i say: Grounding language in robotic affordances." arXiv preprint arXiv:2204.01691 (2022).

[2] Liang, Jacky, et al. "Code as policies: Language model programs for embodied control." 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023.

[3] Singh, Ishika, et al. "Progprompt: Generating situated robot task plans using large language models." 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023.

[4] Shridhar, Mohit, Lucas Manuelli, and Dieter Fox. "Perceiver-actor: A multi-task transformer for robotic manipulation." Conference on Robot Learning. PMLR, 2023.

[5] Mu, Yao, et al. "Embodiedgpt: Vision-language pre-training via embodied chain of thought." arXiv preprint arXiv:2305.15021(2023).

[6] Driess, Danny, et al. "Palm-e: An embodied multimodal language model." arXiv preprint arXiv:2303.03378 (2023).

[7] Du, Yuqing, et al. "Guiding pretraining in reinforcement learning with large language models." arXiv preprint arXiv:2302.06692 (2023).

[8] Paischer, Fabian, et al. "Semantic HELM: A Human-Readable Memory for Reinforcement Learning." Thirty-seventh Conference on Neural Information Processing Systems. 2023.

Dear Reviewer DZLf, we would be grateful if you can comment on whether our response addressed your concerns or if issues remain.

Dear authors,

thank you for your response.

• compare to existing benchmarks: Table4 is indeed very insightful, but does not answer the question why LLaRP is not additionally evaluated on existing benchmarks such as BabyAI to show how they compare to existing methods that have already shown results on BabyAI. This adds more evidence for generality of the proposed approach, which currently is lacking.

• compare to existing methods: while I appreciate the authors detailed the differences between the proposed method and the discussed existing work, I would still be interested to see how other more advanced existing methods compare to LLaRP, even though they may differ in some aspects.

minor:

• claims: the updated claim reflects the contributions more accurately. Though, some previous work could still qualify for the given definition of "LLMs used as vision-language policies in online RL problems to improve generalization".

• lack of details: the additional details clarify some of the missing information, thank you for extending the paper. I recommend to also extent the clarifications with the differences between the current benchmark and existing ones on executing low-level policies or kinetically updating with skill post conditions, in which the latter introduces less challenges.

• I recommend adding the discussions about the related work and existing benchmarks to the paper.

We thank the reviewer for the comments and suggestions. We address the reviewer’s points below.

1. Technical novelty is not high. Using VLMs to control a task is not new.

We disagree and believe our work is novel in training VLMs with reinforcement learning (RL) to control a task. While other works have trained VLMs from LLMs in offline settings, no other works have trained VLMs with online RL (i.e., without expert demonstrations). RT-2 and PaLM-E are offline: they are given 130,000 human teleoperated demonstrations collected over almost 2 years as training data [2,3]. LLaRP only has access to interaction data collected from its current policy (it uses PPO, which is on-policy), and we demonstrate that, by adapting LLMs in this way, LLaRP achieves generalizable decision-making.

The architectural differences from online LLaRP and prior offline VLM decision-making models are rooted in the relative differences between online vs offline learning. RT-2 and PaLM-E use expert demonstrations, so their training is similar to supervised learning: they take a single image as input along with the task specification, and fine-tune the LLM to generate the text-based action instructions from the expert demonstrations. LLaRP does not have access to this expert data, and so trains using an online RL algorithm. Exploration is a critical capability for online learning, so in order to help in this regard, LLaRP takes as input all observations from the current episode (up to 32). Since LLaRP does not have access to expert actions (e.g., as text or otherwise), we add an action decoder module and train it with RL using the reward signal from the environment.

Finally, beyond technical novelty, a useful contribution of our paper is the empirical analysis of the generalization capabilities of using an LLM as a VLM policy trained with online RL, as we explain in the next response.

2. While the one of the main claim or implication of the paper might be linguistic knowledge in LLMs can be used in online RL, but the statement itself is already validated in prior studies

This was a writing error on our part since our main claim is about adapting LLMs for vision language policies with online RL. We mistakenly omitted that the novelty is specific to vision-based policies with online RL in the related work section. The reviewer is correct that [1] shows LLMs for online RL in textual environments. We updated the related work writing to address this and the connection to [1] (changes in red). Thank you for bringing this to our attention.

We apologize for the confusion, but the corrections are minimal and do not change our main contribution of adapting LLMs for VLM policies with online RL. [1] also uses LLMs in online RL, but focuses on environments with text-inputs and text-outputs, thus this method does not apply to the visual decision-making in our setting. The LLaRP architecture is also different from [1] since we train an observation encoder and action decoder module, which are additional components to operate from visual observations and general action spaces.

The empirical analysis in Sec. 5 also contains novel insights about LLMs for embodied AI, which itself is a contribution. We benchmark a wider variety of baselines than [1]. We include zero-shot baselines with powerful LLMs (ZS-ChatGPT, ZS-Flamingo, ZS-LLaMA), while [1] only compares to Flan-T5. We train policies for over 100x more experience (200M vs 1.5M steps). We show generalization to 1,000 new instructions across 10 distinct splits. [1] only shows generalization to new objects, compositions, synonyms, and translation (French vs. English). Unlike [1] we also show the efficiency of fine tuning to new downstream tasks (Fig 4b).

3. The paper lacks in depth analysis of the failure case.

Thank you for the suggestion. We included failure analysis in supplementary section D to support the success qualitative examples in figure 6 and section D.

Citations

[1] Carta, Thomas, et al. "Grounding large language models in interactive environments with online reinforcement learning." arXiv preprint arXiv:2302.02662 (2023).

[2] Brohan, Anthony, et al. "Rt-2: Vision-language-action models transfer web knowledge to robotic control." arXiv preprint arXiv:2307.15818 (2023).

[3] Driess, Danny, et al. "Palm-e: An embodied multimodal language model." arXiv preprint arXiv:2303.03378 (2023).

Dear Reviewer vC8Q, we would be grateful if you can comment on whether our response addressed your concerns or if issues remain.

We thank the reviewer for the comments and suggestions. We address the reviewer’s points below.

1. What distinguishes this work from [1,2,3,4], which also appear to emphasize the evaluation of LLMs' generalization abilities in embodied settings?

What distinguishes our work from [1,2,3,4] is that our main contribution is to demonstrate that an LLM can be adapted with online RL to serve as a generalizable vision language (VLM) policy for embodied AI tasks. We apologize for the confusion here. We updated the writing in sections 1 and 2 to better reflect this (changes in red) and refer to our general response for more details.

We appreciate the opportunity to talk about this main contribution w.r.t. the prior works mentioned. [1,3] primarily focus on finetuning a text only LLM with supervised learning in text-based observation and action spaces. LLaRP adapts an LLM with online RL for visual, instruction tasks directly in the environment action space. [2] also uses RL, but does so in text-based environments. We adapt LLMs to visual tasks and environment action spaces. [4] along with related works [6,7] are "zero-shot policies for interactive decision-making tasks, without task specific training, in settings where the states and action spaces are both text-based" (Sec. 2).

2. More qualitative examples for understanding limitations of employing pretrained LLMs on embodied tasks.

Thank you for the suggestion. We included failure analysis in supplementary section D to support the success qualitative examples in figure 6 and section D.

3. Valuable to gain insights into the performance of simpler baselines.

The LLaRP-Scratch baseline is a simple baseline that does not rely on LLMs (Sec. 5.1). LLaRP-Scratch shows the necessity of LLMs due to its lesser generalization performance (17% vs 42% for LLaRP in Table 2) and efficiency (it takes 500M vs. 50M steps to converge, Fig. 4a).

In this rebuttal, we added a comparison to using EmbCLIP [8] in LLaRP. As with [8] we used the pre-trained CLIP-ViT-L as the visual encoder for the policy. This performed slightly worse than LLaRP, achieving 40% total success rate, vs. 42% success rate for LLaRP. VC-1 [9] also reported better performance than EmbCLIP for other Habitat tasks. We also note that [5] is a method solely for improving training wall clock time, while it could help scale all our experiments, by itself it will not improve generalization performance.

4. How are the current baselines chosen, why not compare with other VLMs?

We chose Flan for the LSTM-Flan baseline because Flan is a powerful encoder model that can embed instructions into fixed length vectors, which the LSTM then takes as input. Using a decoder-based model instead, such as Flamingo, is better suited for text generation, not encoding instructions. We show this with the LSTM-LLaMA model performing significantly worse than the LSTM-Flan model (Table 2).

We used IDEFICS since, at the time of submission, it was the state-of-the-art VLM model.

Citations

[1] Li, Shuang, et al. "Pre-trained language models for interactive decision-making." Advances in Neural Information Processing Systems 35 (2022): 31199-31212.

[2] Carta, Thomas, et al. "Grounding large language models in interactive environments with online reinforcement learning." arXiv preprint arXiv:2302.02662 (2023).

[3] Xiang, Jiannan, et al. "Language Models Meet World Models: Embodied Experiences Enhance Language Models." arXiv preprint arXiv:2305.10626 (2023).

[4] Huang, Wenlong, et al. "Language models as zero-shot planners: Extracting actionable knowledge for embodied agents." International Conference on Machine Learning. PMLR, 2022.

[5] Wijmans, Erik, Irfan Essa, and Dhruv Batra. "VER: Scaling On-Policy RL Leads to the Emergence of Navigation in Embodied Rearrangement." Advances in Neural Information Processing Systems 35 (2022): 7727-7740.

[6] Huang, Wenlong, et al. "Grounded decoding: Guiding text generation with grounded models for robot control." arXiv preprint arXiv:2303.00855 (2023).

[7] Liang, Jacky, et al. "Code as policies: Language model programs for embodied control." 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023.

[8] Khandelwal, Apoorv, et al. "Simple but effective: Clip embeddings for embodied ai." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[9] Majumdar, Arjun, et al. "Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?." arXiv preprint arXiv:2303.18240 (2023).

Dear Reviewer Daer, we would be grateful if you can comment on whether our response addressed your concerns or if issues remain.

Hi authors, I found that the work from Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, Hang Li, Tao Kong: Vision-Language Foundation Models as Effective Robot Imitators. ICLR 2024

looks similar to yours, do you consider both works are tackling the same research problem?