RePLan: Robotic Replanning with Perception and Language Models

Thank you very much for your feedback! To address your concerns:

1. Missing prompts. We apologize for not originally including the prompts used. In our updated draft, we have included all prompts used in Appendix B and indicate how they are used in Algorithm 1 and throughout the text.

2. Limited evaluations. We agree with the reviewer that our experiment setting was limited. We have increased the number of experiments on two fronts. The first is that we added two new environments and three new tasks. The first added environment contains a cabinet that is initially locked, and a lever. The robot has to open the cabinet to deposit an item in there, but it cannot. There is also nothing physically blocking it, and so the robot has to reason that it must somehow interact with the lever in order to unlock the cabinet. The second environment is a scene with a red crate and two cubes, one red and one yellow. In the first task, the robot must place the cube with the same colour as the crate on the crate. This requires conditional reasoning with the Perceiver. The second tasks also requires the robot to place the red cube on the crate, but this time there is a physical blocker: the yellow cube is already on the crate, and there is only room for one. Thus, the robot must first remove the yellow cube in order to accomplish the task. We have also increased the number of runs of each experiment from 3 to 10. We also include a breakdown of the number of actions the robot does in order to do the tasks. In total, the robot performs 401 MPC actions and 101 VLM actions. This is more than previous work (L2R, 170 MPC actions). To address the reviewer’s concerns about experimental errors, we include a more thorough breakdown of model components and an error analysis (see Appendix C2 and C3).

3. Prior work. Thank you to the reviewer for pointing out these prior works. Indeed, we include many of them in our updated draft to clarify our contributions. Specifically, Inner Monologue (IM) does not actively use feedback from VLMs to replan during a task; it uses feedback from humans or object recognition models to know what objects are present or absent (this would be most similar to row 3 in our Table 1). IM uses a success detector in order to determine whether it could resample the low-level policy for a pick-and-place action, but it would not be able to handle obstacles and does not reason on why the action failed. It has been shown in works since binary success detection is not enough to perform complex tasks [1]. Voyager also shows this by receiving feedback from a chat bot in the Minecraft API on why code execution errors were generated. They (in addition to other works) also show that Verifiers are important for long-horizon task completion. We agree that the use of Verifiers and environment feedback is extremely important. We have added these references to our Related Works section. Towards a Unified Agent relies on previously collected observations to finetune a CLIP reward model. Our method does not require any model training. We believe our method is novel because we combine several key insights in order to execute complicated, long-horizon tasks without the need for model re-training or human guidance. First, we use perceiver models to diagnose problems and gather information about object states in order to replan if the robot encounters issues. We create hierarchical plans for both high-level reasoning and low-level reward generation, allowing for adaptive robot behaviour that has only been previously shown on single-step tasks. Finally, we use verifiers at both high-level and low-level reasoning to enable long-horizon tasks of up to 17 steps.

4. Replanning iterations. Our replanning framework is set up recursively, so that the agent can replan when performing at any subtask. However, we set a threshold on recursion depth (r=2) and the number of replanning stages allowed (p=2) to lower runtime and API call costs.

5. Failure cases. We include a more in-depth error analysis of all components in our system in this draft. Specifically, we showcase Perceiver errors in Appendix C3. The Perceiver is often able to successfully identify what the reason for plan failure is, but sometimes the interaction between the Perceiver and Planner results in the wrong summary reason. We also include an ablation study in C2 that reports the object recognition and object reasoning performance of two state-of-the-art open-source VLMs and GPT-4V.

6. Citations formatting. We apologize for the citation issue and have fixed it in the current draft, we thank the reviewer for pointing it out to us.

[1] Lei Wang et al, “A Survey on Language Language Model based Autonomous Agents”. (2023)

Task 4: VLM is asked to diagnose why the robot cannot open the microwave.

Input to VLM:

1. A robot is in a simulation environment where it can interact with any object like in the real world. The robot would like to open the microwave but it cannot. Is there something in this scene preventing that, other than the robot? Assume the robot can interact with anything. These are the names of the objects in our scene: blue kettle, microwave, cabinet
2. In a simulation, a robot wants to open the microwave but can't. Is anything else, besides the robot, blocking it? Check the objects in the scene: blue kettle, microwave, cabinet.
3. Robot in a simulation wants to open the microwave, can't. Something else stopping it? Objects in scene: blue kettle, microwave, cabinet.
4. A robot can engage with any item. It wants to open the microwave but can't. Is an object in this scene, apart from the robot, hindering it? Objects present: blue kettle, microwave, cabinet
5. I would like to open the microwave but I cannot. Is there something in this scene preventing that, other than the robot? These are the objects in the scene: blue kettle, microwave, cabinet
6. I would like to open the microwave but I am unable to. Is there something in this scene preventing me from doing that? Ignore the robot. These are the names of the objects: blue kettle, microwave, cabinet

Outputs from VLM:

1. There is a blue kettle in the scene, which is preventing the robot from opening the microwave.
2. In the scene, there is a blue_kettle on the microwave and a cabinet above the microwave. The robot is not blocking the microwave, so the only thing preventing the robot from opening the microwave is the blue_kettle on the microwave.
3. The robot in the scene is unable to open the cabinet because there is a blue_kettle placed on the surface in front of the cabinet. This object is blocking the robot's access to the cabinet, preventing it from opening it.
4. There is no existence of a blue kettle in the image description.
5. There is a blue kettle on the countertop that could be preventing you from opening the microwave.
6. There is a robot standing on the counter in front of the microwave. It is blocking access to the microwave.

LLM Summary:

The most probable reason why the robot cannot open the microwave is that there is a blue kettle placed on the microwave, blocking the robot's access to it.

In Appendix C3, we also show some error cases of where the interaction betewen the Perceiver and High-Level Planner fails. As an example, in Task 4, the robot tried to open the microwave door but failed due to a kettle obstructing the path. The Perceiver gave five diagnoses, of which three claimed that the kettle was blocking the way, one claimed the cabinet door was blocking the way, and one did not give any conclusive diagnosis. The summary LLM concluded that it was the cabinet door that blocked the action. The robot went on to interact with the cabinet and never removed the kettle. However, in 9 runs out of 10, the High-LeveL Planner was able to correctly summarize the reason from the VLM outputs.

PERCEIVER-PLANNER MOTIVATION

As an additional experiment, we also report the ability of GPT-4V to solve Tasks 3 and 6 out-of-the-box by prompting it to create a plan for opening the cabinet and placing the cube with the same colour as the crate on the crate (Appendix D). In both cases, the generated plans did not provide a working solution. In the first case, plan did not identify that there was a block across the handles of the cabinet the robot needed to first remove, and in the second case, the plan did not specify the name of the cube to use. This shows combining the VLM with specific LLM prompting is essential in order to solve these tasks.

We again thank you for your feedback, please let us know if there are additional analyses you wish to see.

[1] Kirillov A et al, Segment Anything. (2023)

We are glad that the reviewer finds our idea clear and well-motivated. We also appreciate the reviewer’s insightful feedback. Below we address the reviewer’s concerns:

1. Improved literature review. We agree with the reviewer’s observation that the literature review could be improved. We have updated our introduction section and related works section to provide a more thorough and structured literature review, and have re-written sections to better put our work in the context of previous papers and ideas.

2. Lack of thorough experiments and comparison with prior non-LLM works. We appreciate the reviewer for their suggestion to improve our evaluations. We have worked on both of the axes by including three new tasks in two new scenes and increasing the number of runs from three to ten. One of the new tasks involves reasoning about an interaction with a lever-locking mechanism, and the other new tasks emphasize perceiving and reasoning about colors and executing conditional plans. We believe the inclusion of these new tasks can improve the variety of our evaluation. We also improve the analysis of existing tasks in the Results section and Appendix C by analyzing the number of actions required by our tasks (401 MPC actions + 101 VLM actions in total), as well as analyzing common errors. Finally, we include two additional baselines for a few tasks: PDDL (Appendix E) and GPT-4V (Appendix D). We find that out-of-the-box, GPT-4V is not able to solve our tasks. We include a PDDL domain and problem definition for Tasks 1-3. We show that they are not trivial to write and rely on human-provided ground truth information (for example, that the door is blocked). We also consider using PDDLStream to solve the problem. In this case, some of the human-provided ground-truth information is certified by Stream (by using a motion planner it verifies the feasibility of action execution at the planning time); however, it’s inefficient to solve. In terms of hierarchical RL, our work assumes text-based goals with no access to pretrained skills. To the best of our knowledge, there are no prior RL works that succeed under these assumptions. The closest one we find is [1], which is only evaluated in single-motion tasks, while the majority of our tasks are multi-step. In addition, our algorithm can be evaluated on the fly while RL methods typically require extensive training, which we believe can be an unfair comparison.

3. Simulator ground truth for MPC. We believe using simulator ground truth is a necessary compromise to make, and is currently what state-of-the-art works use (Language to Rewards [2], Inner Monologue [3]). Using a ground truth simulator does not jeopardize transferring the system to the real world. For example, [2] and [3] utilize a vision model to map the real-world objects to the simulator and then execute the simulator-produced actions in real world. Another possible way could be collect demonstrations in the simulator and distill an image-based policy from them, which can be directly applied to the real world.

4. The introduction reads more like related works. We apologize for the possible confusion. We have updated our introduction to provide a better motivation for our work. We have also fixed the citation formatting issue. We thank the reviewer for pointing this out.

5. Missing prompts. We have included the detailed prompts in Appendix B in our updated manuscript, as well as how they are used in Algorithm 1 and throughout the text. We hope it can provide enough information for the reviewer.

• [1] Wenhao Yu et al, “Language to Rewards for Robotic Skill Synthesis”. (2023)
• [2] Tianbao Xie et al, “Text2Reward: Automated Dense Reward Function Generation for Reinforcement Learning”. (2023)
• [3] Wenlong Huang et al, “Inner Monologue: Embodied Reasoning through Planning with Language Models”. (2023)

We appreciate the reviewer’s detailed comments and suggestions. We address the reviewer’s specific concerns below:

1. Missing information about VLM Perceiver. We apologize for not providing sufficient details about the VLM Perceiver. We have updated our draft to include all prompts used by the Planners, Perceiver, and Verifier in Appendix B and annotate each line in Algorithm 1 with corresponding prompts. In developing our framework, our experiences completely align with your comments that vanilla, off-the-shelf VLMs are not sufficient to solve these tasks. As a motivating example, we naively ask GPT-4V (which has demonstrated amazing vision-language capabilities) to create a plan for solving Tasks 3 and 6 (the reviewer can find these in Appendix D). In both cases, GPT-4V was unable to identify the correct plan. For Task 3, it was unable to identify that there was block in between the cabinet handles and in Task 4 it could not name the correct cube right off the bat. However, one key insight we find in this work is that we can use an LLM (which has better reasoning skills) to determine what questions it should ask the VLM in a constrained, VQA-style setting in order to update its understanding of the environment. For example, if the Planner asks the VLM: “Robot in a simulation wants to open the microwave door, can’t. Something else stopping it?” The VLM is able to provide the caption that there is a kettle in front of the door. To prevent overfitting to one question, the LLM asks the VLM multiple iterations of the question (see Figure B9 in Appendix B) and then based on its own knowledge of the world state, it provides a summary answer based on feedback from the VLM of what the most likely explanation. The Planner can also query the Perceiver to update its knowledge on object states, for example: “What is the colour of the crate?” or “Do you see a green apple?” (see Figures B14-B16 in Appendix B). We find that combining LLMs to effectively prompt VLMs and using Verifiers to make sure their outputs are consistent are essential for VLMs to provide useful feedback. We also provide an ablation study in Appendix C2 on the performance of two open-source models and GPT-4V on object recognition and reasoning.

2. Limited evaluation complexity. We apologize for the unclear writing and have fixed our description of the number of actions the robot does to accomplish each goal. When we consider an action as any time the robot needs to do something to satisfy an MPC reward or decides to call the VLM to obtain information, most tasks requires on average between 7 and 11 subtasks without human intervention (with some runs using up to 17 depending on how the robot decides to replan). We show a Table in Appendix C (Figure C1) breaking down the number of MPC actions vs VLM actions for each task. This is comparable to the number of actions required by the longest-horizon tasks in SayCan, but those tasks do not have obstacles preventing the robot from following the high-level instructions and do not require replanning based on environment feedback. Furthermore, SayCan utilizes primitive skills to maintain a reliable subtask execution, which we do not assume to have. Not using primitive skills also poses significant challenges to the reliability of the method. Few evaluation trials. We have increased the number of trials for each task to 10. We have also added two new environments and three more tasks.

3. Clarification of core claim. We apologize for not making the core contributions clear. Our core contribution with this work is as follows: we develop an agent to handle long-term, multi-stage tasks by combining four key aspects: using perceiver models for high-level replanning, creating hierarchical plans with language models, verifying outputs from these language models, and adaptive robot behavior through reward generation. We have updated the introduction to better reflect this. Thank you for pointing out a missing reference to Inner Monologue (IM). While both our works show that robots can perform more complex and longer-horizon tasks by receiving feedback from different sources while executing a task, we are different in the following ways: (a)Their framework uses a VLM at the beginning of the scene for object recognition, but while the scene is progressing, they obtain feedback from humans in some experiments, or an object recognition model to update knowledge about what objects are present/absent in the scene. They do not rely on VLMs for replanning or providing reasons when the success detector fails. This is most similar to row 3 in our Table 1. Our tasks are not solvable from the presence/absence of objects alone, and our framework is about to solve them without human intervention. (b) When the success detector fails, the robot retries or selects another low-level policy to redo the task (not necessarily to determine reasons why the task failed based on the scene state).

Thank you for your feedback, and we are glad that you find our work interesting. To address your concerns:

1. More environments and tasks. We have included two new environments in this iteration with three new tasks. A description can be found in Section 4.1 and the results in Table 1. The environments are a cabinet scene with a lever that unlocks the cabinet (the cabinet is locked when the scene starts). The robot must reason that there are no physical obstacles in the scene and must figure out that pulling the lever unlocks the door in order to accomplish the task. The other environment contains a red crate with two blocks, one red and one yellow. The first task in this environment is to place the cube with the same colour as the crate on the crate, and so the robot must use the Perceiver to determine the correct cube (otherwise it could just get 50% by guessing). The final task is in the same environment; the robot must put the red cube on the crate but there is a yellow cube already on the crate, preventing placement of the red block. The robot must first remove the yellow cube. In addition to integrating these new tasks, we also increased the number of times we ran each of the 7 tasks (from 3 to 10) to further investigate the robustness of our method.

2. Testing planner module. For each important motion in our tasks, we first do multiple MPC runs with ground-truth reward functions to test the stability. We pick four of the important motions and include the test results in Appendix C.1 in our updated draft.

3. VLM Perceiver Weakness. We thank the reviewer for their suggestion. We have updated our prompting mechanism to alleviate the problems incurred by the VLM perceiver (the reviewer can refer to Appendix B in our updated draft for the detailed prompts, as well as Algorithm 1). As a result, we believe the VLM perceiver is no longer a dominant issue. In Table 1, we demonstrate the impact of only providing a text description of the objects in the scene at start-up (RePLaNn [no Perceiver]), which significantly drops the performance. We include experiments in Appendix C2 demonstrating the object recognition and object reasoning skills of two open-source VLMs and GPT-4V. We find that combining open-source models with segmentation models can also boost object recognition capabilities (Section 4.2 and Appendix C2). We include an additional example of why LLM-constrained VLM prompting is important for our method to work in Appendix D, where we ask GPT-4V to solve Tasks 3 and 6 out-of-the-box, and find that it is unable to. Finally, we include a more in-depth error analysis of the failure cases in Appendix C3 and find that there are still some small issues with the Perceiver-Planner interaction, which could be improved with better prompting. For example, in Task 6, the Planner asks the Perceiver “Which cube has the same colour?” which is very vague, and the Perceiver ends up reply nonsensically and the robot picks the wrong cube.