Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks

We thank the reviewer for providing a detailed review and for appreciating the strength of our experimental results and quality of writing.

“It is a paper about robotics. However, experiments are based on simulations only.”

In this work, we focus on studying the question: how can we train policies to solve long-horizon robotics tasks? Our main contributions are an algorithm for guiding RL agents for learning low-level control using LLMs via motion planning and a series of insights for practically training such policies. To study this question effectively, we perform extensive empirical evaluations on 25 tasks across 4 evaluation domains, validating the strength of our method on established benchmark tasks for comparison. We leave extensions to the real world for future work and briefly discuss two possible directions for doing so: 1) sim2real transfer by training local policies in simulation and chaining them using motion planning and LLMs at test time 2) directly running PSL in the real world as it is far more efficient to train than E2E methods.

“It is about long horizon robotics tasks. However, the largest number of stages is 5 in the experiments.”

We emphasize that even solving tasks with up to 5 stages is quite difficult for end-to-end based methods; they make no progress at all in most cases beyond 1-2 stages. Furthermore, we only evaluate tasks with up to 5 stages, as a majority of the benchmarks for robotic control have only up to that many stages, not because our method cannot be applied beyond 5 stages. As shown in the experiments section, prior methods such as E2E [1], RAPS [2], MoPA-RL [3], TAMP [4] and SayCan [5] do not reliably solve the benchmark tasks with up to 5 stages. In contrast, for our method, whether we have 1, 5 or even 10 stages (we include these new results at the end of this response), we can still solve the task because our modular, hierarchical method effectively decomposes the task and simplifies the learning problem significantly.

“"Large Language Models (LLMs) are highly capable of performing planning for long-horizon robotics tasks" This is very arguable. There are evidences that it is not the case. And if it is so, there is no need to write the paper.”

We have toned down this point in the updated version of the paper by instead stating that LLMs have been shown to be capable of high-level planning. Our changes are shown here in italics.

While LLMs perform poorly on general purpose planning as noted in the work cited by the reviewer [6], in our work the LLM is not required to perform general purpose or fine-grained planning. Instead, in PSL, the LLM is only outputting a very coarse high-level plan - where to go and how to leave the region - which is simple and does not require significantly complex reasoning ability. For such tasks, we find that the semantic, internet-scale knowledge in LLMs is sufficient to produce high-quality plans. Empirically, on the tasks we consider, the LLM achieves 100% planning performance.

We additionally note that our method is not necessarily tied to using an LLM as a task planner. We can also use classical task planners such as STRIPS [7] and use an LLM to simply translate the natural language prompt into a format for task planning as done in LLM + P [8]. In this way, we inherit the guarantees and benefits of classical task planners while guiding the RL agent to efficiently solve the task from a natural language task description.

“"Language models can leverage internet scale knowledge to break down long-horizon tasks (Ahn et al., 2022; Huang et al., 2022a) into achievable sub-goals" How valid is such claim? Why is it so? What if the tasks are not available or not frequent in Internet texts?”

We have re-written this statement to be appropriately qualified: “Prior work (...) has shown that when appropriately prompted, language models are capable of leveraging internet scale knowledge…” There is a large body of recent work [5, 9, 10, 11, 12] in this area that empirically illustrates such capabilities for long-horizon robotics tasks. We have updated the main paper with this change. Our changes are shown here in italics.

However, we emphasize the existence (or lack thereof) of general purpose planning capabilities of LLMs is orthogonal to the claims of our paper. Our focus, with respect to the Plan Module, is on a simple, coarse planning interface for the LLM with demonstrably high performance across a wide range of robotics tasks.

We thank the reviewer for their detailed review and for recognizing the clear motivation for PSL, novelty of our method, clarity of writing and strength of our experimental results.

“I feel like the author could illustrate the details in the main paper and also a better explanation of how stage termination and training details are implemented.”

We have updated the discussion in Section 3.3 of the main paper to include additional details regarding the stage termination and LLM planning implementation details. We have also included further details in Section 3.5 of the main paper. Additionally, we emphasize we will release the code for PSL upon acceptance - enabling the community to replicate our results. The code will unambiguously specify the requested implementation details.

“Although Citing 'Inner Monologue' and 'Bootstrap Your Own Skills(BOSS)', they are not used for comparison or experiments, as these methods share many similarities.”

For our experiments, the high-level planning success rate is 100% - the bottleneck is performing effective low-level control. To that end, prompting techniques such as Inner Monologue [1] would not affect the performance. Inner Monologue would achieve the same results as SayCan [2]. Furthermore, Inner Monologue could be readily incorporated into PSL to improve planning performance when necessary; we leave this extension to future work.

With regards to BOSS [3], as we note in the paper, this is concurrent work with our own. It was released on Arxiv on October 16, 2023 - after the ICLR submission deadline. That notwithstanding, there are several reasons why comparisons to BOSS is currently infeasible: 1) The code for BOSS is not released and re-implementing the method is non-trivial as it requires training policies online using IQL in the loop with an LLM. Once the code is released, we will attempt to perform a fair comparison if possible. 2) Their method operates with a different assumption set than ours: existence of a pre-trained skill library while we evaluate training from scratch to learn unseen low-level skills 3) BOSS specifically uses a language labeled demonstration dataset to pre-train skills - no such dataset exists on most of the tasks we evaluate. Furthermore, the environment code for BOSS is not publicly available either - they use a “modified version of the ALFRED [4] simulator” which is not released to our knowledge. Finally, we would like to note that the contribution of BOSS is orthogonal to our own, our method focuses on learning to efficiently solve a single task while BOSS aims to expand a pre-existing repertoire of skills. In principle PSL can be combined with BOSS to efficiently learn and incorporate a new skill into an existing library, particularly when starting from any empty skill set.

“Trade-off: Planning without a library of pre-defined skills is mentioned as a strength in the paper, but this comes at the cost of relearning the whole process compared to other methods.”

In our experiments, we show that learning policies from scratch can outperform methods that use pre-trained/defined skills such as SayCan [2] and RAPS [5] by over 2x in terms of raw success rate. Online learning enables the agent to adapt its low-level control to the task it is solving while avoiding cascading failures. However, we acknowledge that pre-trained skill libraries come with many practical benefits and in principle PSL can also take advantage of and fine-tune pre-defined skills as we discussed above. We leave this extension to future work. Ultimately, we agree with the reviewer that it is desirable to combine online learning with pre-defined skills, however, in this work our aim was to study and improve the learning process in isolation from pre-defined skills.

We thank the reviewer for recognizing the novelty of our modular framework and our extensive experiments as well as for appreciating the strengths of our results on a wide range of tasks and domains.

“The long-horizon task seems to be divided into only 'grasp' and 'place' (from the paper and appendix), it is unclear if there are more sub-tasks / skills that the LLM divided. From the webpage, I find other tasks besides the pick-and-place series so wonder how to implement these.”

To clarify, PSL is not limited to only using ‘grasp’ and ‘place’ termination conditions. In general, it can take advantage of any stage termination condition when performing LLM planning. The RL agent can then learn the corresponding local control policies. We simply require the following: a function that takes in the current state or observation(s) of the environment and evaluates a binary success criteria as well as a natural language descriptor of the condition for prompting the LLM (e.g. ‘grasp’ or ‘place’). Then the LLM can subdivide the task based on these conditions. We have updated the paper to make this point clear.

Furthermore, in our experimental results, we do not only use ‘grasp’ and ‘place’ conditions. We also use conditions such as ‘push’ (OS-Push), ‘open’ (RS-Door, K-Microwave, K-Slide) or ‘turn’ (K-Burner). These conditions can be readily estimated using vision: they are all dependent on pose estimates. We have updated the paper to include detailed descriptions of how to estimate pushing, opening and turning. Finally, we refer the reviewer to our reply to Reviewer KErm, in which we experimentally show that our method is not dependent on the user specifying exactly the termination conditions that are necessary for the task. We find that providing a superset of termination conditions will also work; the LLM will only output sub-tasks that are necessary for solving the given task.

“Does it need to redesign the reward function of the RL process?”

As we note in Section 4.3 of the main paper, we do not modify the reward function of the environment for any task. Instead, we use the Plan and Sequence modules to move the RL agent to relevant regions of space and specify conditions for exiting those regions (stage termination conditions). The RL agent then learns local interaction based on the overall task reward.

“Moreover, how does the sparse and dense reward influence the learning process?”

Our experiments include results on dense (Robosuite [1], Metaworld [2], Obstructed Suite [3]) and sparse (Kitchen [4]) reward tasks. We find that PSL performs well in both settings. One reason why PSL is capable of effectively solving sparse reward tasks is that it addresses one major component of the exploration problem in sparse settings: finding the object with which it needs to interact. By initializing the RL agent close to the region of interest, we greatly increase the likelihood that random exploration leads to coincidental successes which can be used to bootstrap learning.

[1] Y. Zhu, J. Wong, A. Mandlekar, and R. Martin-Martin. robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009.12293, 2020.

[2] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, S. Levine. "Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning." Conference on Robot Learning, pages 1094–1100, PMLR, 2020.

[3] J. Yamada, Y. Lee, G. Salhotra, K. Pertsch, M. Pflueger, G. S. Sukhatme, J. J. Lim, P. Englert. "Motion Planner Augmented Reinforcement Learning for Obstructed Environments." Conference on Robot Learning, 2020.

[4] J. Fu, A. Kumar, O. Nachum, G. Tucker, S. Levine. "D4RL: Datasets for Deep Data-Driven Reinforcement Learning." arXiv preprint arXiv:2004.07219, 2020.

Dear Reviewer,

We would like to follow up on our rebuttal as there is only one day remaining of the discussion period. If there are any outstanding concerns that you would like us to address, please let us know. Thank you and we look forward to your response.