Compositional Instruction Following with Language Models and Reinforcement Learning

Thank you for your time and feedback on our work. We've addressed the comments in the order listed.

1. We utilize a learned semantic parser that uses in-context learning and feedback from the environment. To the best of our knowledge, both our semantic parsing method and the scale and diversity of simultaneously learned language-RL tasks is unique in the literature. It is not a priori certain that semantic parsing should work in the RL setting on this quantity of tasks. Our method learns the parser from environment feedback, a much weaker form of supervision than is typically assumed. Further, we use our method to solve language grounding tasks involving negation, which are under-explored (Pillai and Matuszek 2018).

We evaluate our agent on a much larger set of simultaneous tasks than previous works using WVF, which instead evaluated on individual tasks with hand-specified compositions of value functions. These works assumed ground-truth Boolean expressions, which is a very strong assumption. Our method relaxes this assumption and instead learns these Boolean expressions from environment feedback which is non-trivial.

Our work does build on pre-existing methods. But many related works contain applications of existing methods, including SayCan (Ahn et al. 2022) which combines skill learning from demonstration with few-shot LLM planning, PaLM-E (Driess et al. 2023) which combines pre-existing vision and language representations to solve vision-language tasks. But we also go beyond the existing literature and demonstrate the utility of a novel semantic parsing method for RL settings and the advantage of compositionality at both the RL and language levels for solving a large set of language-RL tasks.

2. The original BabyAI domain is intentionally structured such that a perfect parser for language instructions can be defined, and this makes it the right domain for developing and evaluating our method. Because we can define a parser, we are able to benchmark our method against an Oracle and validate our method by showing that our method converges to the Oracle’s performance. Many recent works published at top conferences have utilized BabyAI for evaluation for similar reasons including Carta et al. (2022) EAGER: Asking and Answering Questions for Automatic Reward Shaping in Language-guided RL and Li et al. (2022), Pre-Trained Language Models for Interactive Decision-Making. In future work we plan on generalizing our method to other domains.

3. We appreciate your feedback on our baseline. We are updating the baseline to utilize pretrained language representations: the SOTA all-mpnet-base-v2 sentence embedding model available in the SentenceTransformers library (Reimers and Gurevych 2019). Initial results show that this agent still requires more steps than our agent and does not generalize well to held-out tasks. We plan to share updated baseline results by the end of the discussion period. The similar performance of the new baseline is expected, as neural networks struggle with compositional generalization and to generalize systematically in this domain requires both compositional RL representations and language representations. While the new baseline agent may generalize better between language commands, it must still learn each task value function without the aid of compositional structure. To solve this many compositional tasks in a generalizable way requires compositional generalization at both the language and policy level, which our method enables.

Carta et al. (2023) does not appear to be an appropriate baseline as they only evaluate in a custom text-world environment with a significantly simpler observation space. Our method utilizes value function approximation from images and text, providing greater applicability and we evaluate in a more challenging observation space.

4. We demonstrate that our method attains a higher success rate in fewer timesteps than the non-compositional baseline and also offers a simple means of interpretability which black-box methods like joint text-image value function learning do not. We utilize this interpretability to analyze the failure mode of the ablation using the less capable GPT-3.5 Agent. In terms of performance, our method significantly reduces (by more than 50%) the error rate over the baseline for an equal number of environment steps at convergence. Our method also generalizes better to held-out tasks as shown in Figure 4.

5. We appreciate your feedback and will improve the explanation of the WVF learning for a camera-ready submission and include additional details in the appendix.

Thank you again for your feedback and questions. We have updated Figures 3 and 4 with the results of using pretrained language representations in the baseline denoted "LM-Baseline". These new results do not show significant learning differences when using only pretrained language representations. As expected the agent cannot generalize compositionally without access to representations that allow for such generalization for both language and RL.

In Figure 4, we have also run the baseline for more steps (21 million).

Thank you for your time and feedback on our work. We've addressed the comments in the order listed.

1. We appreciate your suggestions. We plan on including those suggested ablations in a camera-ready submission: a fine-tuned LLM baseline and a RL-based semantic parser that doesn’t utilize pre-trained LLM representations. In our experience we have found GPT-4 to be significantly better than other models for semantic parsing. We note that to our knowledge, our method is the first to learn semantic parsing using in-context learning with feedback from policy rollouts. This is a much weaker form of supervision than is typically assumed in semantic parsing. We believe this is an important contribution to existing work that maps language to structured representations for language instruction following tasks.

2. We believe BabyAI is the most appropriate domain for our work. It provides a scalable RL learning environment for a suite of compositional tasks. BabyAI is also easily extensible, allowing us to augment the domain with the largest set of simultaneously learned language-RL tasks (to the best of our knowledge). Other recent related works published at top conferences also select BabyAI for evaluation for similar reasons including (Carta et al. 2022 EAGER: Asking and Answering Questions for Automatic Reward Shaping in Language-guided RL), (Li et al. 2022, Pre-Trained Language Models for Interactive Decision-Making). The newer Carta et al. 2023 also use BabyAI, but they convert BabyAI to a simpler text-world domain (Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning).

3. We appreciate your comment and plan on investigating learning of WVFs without a curriculum in future work.

Questions:

1. Our approach builds on similar in-context semantic parsing work like Toolformer, but goes beyond these methods as it is the first to use in-context learning from environment feedback. We have not been able to devise an effective method where the agent utilizes negative examples for in-context learning. We can however utilize negative samples in the fine-tuning setting through unlikelihood training (Welleck et al. 2019).

References:

Carta, Thomas, et al. "Eager: Asking and answering questions for automatic reward shaping in language-guided rl." Advances in Neural Information Processing Systems 35 (2022): 12478-12490.

Li, Shuang, et al. "Pre-trained language models for interactive decision-making." Advances in Neural Information Processing Systems 35 (2022): 31199-31212.

Carta, Thomas, et al. "Grounding large language models in interactive environments with online reinforcement learning." arXiv preprint arXiv:2302.02662 (2023).

Welleck, Sean, et al. "Neural text generation with unlikelihood training." arXiv preprint arXiv:1908.04319 (2019).

Thank you for your time and feedback on our work. We've addressed the comments in the order listed.

Need for user-defined primitive tasks: Current robotics deployments require pre-defined primitive tasks and for human design of the low-level controllers. For example, SOTA language instruction following methods like SayCan (Ahn et al. 2022) rely on extensive human-design of both high level tasks and low-level control through manual demonstrations. This work is among the first to relax these requirements (the low-level controllers are learned) while providing the ability to compose these controllers, and is the largest set of such tasks learned simultaneously with language and RL.

Unfair comparison to baseline: We are updating the baseline to utilize pretrained language embeddings from the SOTA all-mpnet-base-v2 sentence embedding model from SentenceTransformers library (Reimers and Gurevych 2019) . Initial results show that this agent still requires more steps than our agent and does not generalize well to held-out tasks. We plan to share updated baseline results by the end of the discussion period. The similar performance of the new baseline is expected, as neural networks struggle with compositional generalization and to generalize systematically in this domain requires both compositional RL representations and language representations. While the new baseline agent may generalize better between language commands, it must still learn each task value function without the aid of compositional structure. To solve this many compositional tasks in a generalizable way requires compositional generalization at both the language and policy level, which our method enables.

Only Boolean combinations supported: Prior work in semantic parsing with LLM: to our knowledge, our method is the first to learn semantic parsing using in-context learning with feedback from environment feedback. This is a much weaker form of supervision than is typically assumed in semantic parsing and represents an important contribution to existing language instruction following literature that maps natural language to formal representations.

Even for cases where the mapping from task primitives to symbols is known, one challenge is that language is ambiguous and there is a many-to-one mapping of language instructions to individual tasks, and language symbols to task primitives. Indeed knowing only the semantics of the symbols is insufficient in our domain for learning to form valid Boolean expressions as our tasks involve combinations of primitive task negation, intersection, and conjunction. Lastly, Boolean combinations are able to express many interesting tasks. Because we support Boolean task specifications, our method also naturally generalizes to task negation. Negation is understudied in the grounded language literature (Pillai and Matuszek 2018) and to the best of our knowledge our paper also solves the largest and most diverse set of language-negation RL tasks. We leave temporal compositions of tasks to future work.

Questions:

1. Your question is well taken, and we are updating the baseline to utilize pretrained language embeddings as mentioned above.
2. We plan on adding ablations where the agent has access to the underlying semantics of the WVFs in a camera ready submission. Previous exploration has shown that this improves the sample efficiency of the agent.
3. We believe that temporal compositions are useful, but fall outside of the scope of this work. Instead we evaluate a task-space based on Boolean expressions which can express many useful types of tasks, for example “serve breakfast with coffee and no milk.”

References:

Ahn, Michael, et al. "Do as I Can, Not as I Say: Grounding Language in Robotic Affordances." CoRL (2022).

Pillai, N., & Matuszek, C. (2018). Unsupervised Selection of Negative Examples for Grounded Language Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1). https://doi.org/10.1609/aaai.v32i1.12108

Thank you again for your feedback and questions. We have updated Figures 3 and 4 with the results of using pretrained language representations in the baseline denoted "LM-Baseline". These new results do not show significant learning differences when using only pretrained language representations. As expected the agent cannot generalize compositionally without access to representations that allow for such generalization for both language and RL.

In Figure 4, we have also run the baseline for more steps (21 million).

Thank you for your time and feedback on our work. We've addressed the comments in the order listed.

Sample Efficiency: Your comment on the results of Figure 3 are well taken. We are claiming a sample efficiency gain and benchmark our model against the best performing non-compositional baseline DQN agent we could train. This baseline agent does continue to increase in performance, though at a decreasing rate of improvement. The x-axis scale change could also make this hard to see. Given that we are working in the function-approximation setting and learning value functions from text and images, the agent is not guaranteed to converge to the optimal action-value function. If we were evaluating in a tabular setting where convergence is guaranteed, it would be appropriate to run our baseline until convergence. Our setting is more challenging than previous similar compositional domains. To the best of our knowledge it contains the most simultaneously learned language-RL tasks. Since we have no guarantee of convergence, we instead run the baseline agent for many more steps than it takes for our method to converge to the Oracle agent’s performance. We thereby show that our method requires fewer steps than the baseline to reach the Oracle agent’s performance.

Baseline: We are updating the baseline to utilize pretrained language embeddings from the SOTA all-mpnet-base-v2 sentence embedding model from SentenceTransformers library (Reimers and Gurevych 2019) . Initial results show that this agent still requires more steps than our agent and does not generalize well to held-out tasks. We plan to share updated baseline results by the end of the discussion period. The similar performance of the new baseline is expected, as neural networks struggle with compositional generalization and to generalize systematically in this domain requires both compositional RL representations and language representations. While the new baseline agent may generalize better between language commands, it must still learn each task value function without the aid of compositional structure. To solve this many compositional tasks in a generalizable way requires compositional generalization at both the language and policy level, which our method enables.

Formalities: We will run the baseline for more steps for Figure 4 as suggested, to provide a more fair comparison of generalization performance at each timestep. We plan to update these results by the end of the discussion period.

Questions:

1. Thank you for your suggestion. We arrived at that hyperparameter empirically but have not completed a full multi-trial evaluation of ablated values. We can include these in a camera-ready submission.
2. We believe this helps demonstrate the applicability of our method to situations where the semantics of the value functions are either not easily expressed in language or are otherwise challenging to enumerate.
3. During training each parse in the beam is evaluated for 100 episodes, until a high success rate parse is found, or all parses in the beam have been exhausted. The section will be updated in a camera-ready submission to make this clearer.

References: Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL https://arxiv. org/abs/1908.10084.