Learning Planning Abstractions from Language

R4.1: Related approaches for deriving state abstraction.

Thank you for the suggestion! We agree that these methods can potentially help with state abstraction. However, we want to emphasize the key differences between these related approaches and our method, and therefore highlight our novelties.

The vision-language representation LIV is trained on the objective such that (1) neighbouring frames in a video are close by in the embedding space, (2) their distances to the language goal smoothly decrease. These properties are useful for language-conditioned manipulation; however, we note that the representation is only trained on and evaluated on short-horizon interactions (e.g., open microwave, put hat on the bottle, and pineapple in the black pot). These interactions are analogous to a single abstract action in our definition. In comparison, our approach learns a state and action abstraction that is amenable to generating plans that involve multiple high-level actions.

The key idea of Peng et al. is to identify task-relevant features of a visual scene conditioned on the input language. The abstraction process is done by extracting a list of objects along with their properties (e.g., silver pan, blue pot, and red square) and then using an LLM to determine which objects are relevant. Compared to this method, we do not assume that groundings of the object concepts are known, and show that our model can learn to reason about these concepts from language-annotated demonstrations. More importantly, the key idea of our approach is that the state abstraction should be latent. We illustrate that our model is able to predict that placing an object into the sink is not feasible if the sink is full. We do not explicitly symbolize "full" because it depends on the size of the object and the available space in the sink. Discretizing a raw state into explicit symbols, a process named textualization by Peng et al., prevents the method from performing fine-grained geometric reasoning.

Finally, we want to highlight the challenges of applying CLIP and PointCLIP to our domain. CLIP mainly aligns images with textual descriptions of the visual contents. This is different from our abstract state representation which is tightly coupled with actions. In addition, existing research has found that CLIP behaves as a bag of words. This suggests that CLIP and its extension PointCLIP would not be able to infer the sink is currently full such that "placing a pan into the sink" is not feasible.

A promising approach is to finetune these vision-language representations. We are currently running an experiment where we replace the point-cloud-based encoder with the LIV image encoder. Because our original data is collected with point clouds, we are currently recollecting the multi-view RGB image data to train the model. We will provide an update soon.

R4.2: Details for the model.

A: Thanks for the suggestion. We have included a new section (Appendix A) in the supplementary to document details of our models and training details. In addition, we have included new sections in the supplementary to document details of the baselines (Appendix B) and environment settings (Appendix C). We have also included more visualizations of the environment and the generated plans by our algorithm in Appendix D. We will also release the code.

Regarding your particular question on inputs:

• In BabyAI, the input to all methods is a 2D grid encoded with one-hot per-grid features (colors, object categories, agent's facing directions, etc.). This follows the original BabyAI paper [D1].
• In Kitchen-World, the input to all methods is segmented 3D point clouds. In particular, the scene is composed of a set of objects; there is a colored point cloud for each object.

[D1] Chevalier-Boisvert et al. BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning. ICLR 2019.

R4.3: Accuracy for feasibility function.

A: Thank you for the great suggestion! We include a new evaluation on the accuracy of the feasibility function in the Kitchen-World environment in Appendix D (which is also copied down below). We break down the evaluation across different generalization environments and different prediction horizons. In short, our model is capable of generating highly accurate feasibility predictions.

R3.1: Difference between key-door and two-corridor.

A: Thank you for bringing this up. We have added a detailed discussion of the different task settings in Appendix C. In short, two-corridor is one specific kind of key-door environment and two-corridor is designed to be a harder task than randomly sampled key-door environments on average.

In particular, in key-door, the connectivities between nearby rooms are randomly sampled. Therefore, it is likely that the required number of high-level actions is small or there is only one possible high-level action that is feasible at a state. In two-corridor, we specifically designed an environment where there are two corridors so that the planning length is long (maximum 7 high-level actions if the goal is to reach the room at the very end), and that at each given state, there will be at least two feasible actions. Therefore, in order to achieve the goal within a limited number of steps, the agent must unroll its learned transition model to select which corridor to enter.

R2.1: Implementation and experiment details.

A: Thank you for the suggestion. We have included new sections in the supplementary to document details of our models (Appendix A), baselines (Appendix B), and environment settings (Appendix C). We have also included more visualizations of the environment and the generated plans by our algorithm in Appendix D. We will also release the code.

R2.2: Scalability as number of concepts increases.

A: Thank you for bringing up this point. We agree that scalability would be an issue for both learning and planning when the number of actions and objects increases. However, we should point out that this is exactly the reason we learn a compositional space of candidate actions. In particular, we use LLMs to decompose instructions into steps, and steps into verbs and their object arguments. Therefore, our method can generalize to unseen compositions of concepts (see Table 1 and Table 2, the "novel concept composition" columns).

Planning with a compositional space of possible actions would be another challenge, for all methods. In our paper, we have used a simple tree search algorithm with action feasibility pruning to generate plans. For more complex scenarios, more sophisticated algorithms, such as using symbolic planning tools [B1] or learning high-level policies to guide the search as in AlphaGo [B2] would be possible. We have tried a similar version with learned high-level policies; however, in practice, we find this policy barely generalizes to longer plans and hurts the performance.

[B1] Helmert, Malte. "The fast downward planning system." JAIR (2006): 191-246. [B2] Silver, David, et al. "Mastering the game of Go with deep neural networks and tree search." nature 529.7587 (2016): 484-489.

R2.3: Definition of task and language goal.

A: Thanks for the suggestion. We have updated our problem formulation section to include definitions of tasks. Your understanding is correct that in this paper, each task $t$ in the environment is associated with a natural language instruction (the language goal) $L_t$.

R2.4: Details for state abstraction function.

A: Thank you for pointing these out. We have made these points more clear in the revision. Here are the direct answers to your questions:

• Point clouds: As stated in Sec 4.2 State abstraction function, we assume access to segmented object point clouds. The number of objects in each scene is $N$ and is known.
• End-to-End training: The state abstraction function $\phi$, the abstract transition model $\mathcal{T}'$, and the feasibility prediction model $f_{a'}$ are trained together end-to-end.
• Pooling operations: We use max pooling and the per-object latent feature has a dimension of 352. We provide in Appendix A a list of hyperparameters for the model.

R2.5: Details for metric.

A: Thank you for bringing this up. We have added evaluation details in Appendix C. In summary, for BabyAI, models need to reach a goal within 200 low-level steps; otherwise, it's considered as a failure. For Kitchen-Worlds, models need to reach a goal within 5 steps; otherwise, it's considered as a failure. For all experiments, we evaluate on 100 task instances and report the average success rate.

R2.6: Abstract transition model clarification.

A: Your understanding is correct! We meant to say that the abstract transition model takes in the current abstract state $s'\_t$ and abstract action $a'$, and predicts the next abstract state $s'\_{t+1}$. We have updated the text to clarify this point.

R2.7: Notation clarification.

A: Thank you for the suggestion. We have chosen to use $s_t$ and $s'_t$ to represent the raw state and the abstract state at step $t$, respectively. Since we always use the subscript $t$ there are no ambiguities. However, we do agree that this will potentially create confusion for readers. We plan to use different symbols $x$ and $s$ to represent raw states and abstract states in the final version.

R2.8: Loss clarification.

A: Yes, we use L_2 norm. We have updated the equation to highlight this.

R1.4: Details for training (Section 4.2).

A: We present training details in Sec 4.2. We have updated the text to further clarify the training procedure. Here we summarize the main points:

• Data: Our model is trained on sequences of paired states and abstract actions, where each is in the form of $s\_0, a'\_0, s\_1, \cdots, a'\_{\ell-1}, s\_\ell$. At each iteration, we first randomly sample a suffix from a demonstration trajectory to obtain $s\_k, a'\_k, s\_{k+1}, \cdots, a'\_{\ell-1}, s\_\ell$. Next, we use $\phi$ to encode $s\_{k}$ as $s'\_{k} = \phi(s\_{k})$, and recurrently apply the Transformer-based abstract transition model and feasibility prediction model: $s'\_{i+1} = \mathcal{T}'(s'\_{i}, a'\_i)$ and $\overline{*feas*}\_i = f(s'\_i, a'\_i)$, for all $i=k,\cdots,\ell-1$.
• Objective: The training loss is defined as $\mathcal{L} = \sum\_{i=k+1}^{\ell} \| s'\_i - \phi(s\_i) \|\_2 + \sum_{i=k}^{\ell-1} \text{BCE}(\overline{*feas*}\_i, *feas*\_i)$, where $\text{BCE}$ is the binary cross-entropy loss, and $*feas*\_i$ is the groundtruth feasibility label. $*feas*\_i$ is true for all $i < \ell$, and is true for $i=\ell$ if the trajectory execution is successful (i.e., the last action is successful) or false otherwise.
• Model: A single model is learned for all abstract actions because the abstract feasibility prediction model $f(s', a')$ and the abstract transition model $\mathcal{T}'(s', a')$ are both conditioned on the given abstract actions. The Point Cloud Transformer is trained as part of the state abstraction function $\phi$ with the loss described above.
• Data: For BabyAI, we train all models on 100,000 trajectories. For Kitchen-Worlds, we train all models on around 10,000 trajectories. We provide details in Appendix C.

R1.5: Details for planning (Section 4.3).

A: We always choose the sequence of abstract actions with the highest feasibility scores. This design ensures that the agent will always pick an action at any state. To deal with situations where the action selected by the agent is not helpful for reaching the goal, we perform a close-loop replan after the agent finishes executing each action, therefore allowing the agent to recover.

In this paper, we assume that the set of actions derived from language annotations of the training trajectories covers the space of actions required to complete the testing tasks. As discussed in Sec 4.1, in contrast to representing each action term as a textual sentence, the decomposed action and object concepts allow us to enumerate all possible actions an agent can take easily. In the experiments, we empirically validate whether our method can generalize to new combinations of action and object concepts in by evaluating our model on "Novel Concept Combinations".

We note that it's non-trivial to generalize a manipulation policy to completely new actions (e.g., from pick-and-place to opening a microwave door). As discussed in a recent paper representative of the state of generalizable manipulation [A2], there are four levels of generalization: (1) different initial object placements, (2) different background and distractor objects, (3) new object-skill combinations, (4) new environment layouts. In this paper, our method shows strong generalization in all four levels.

[A2] Bharadhwaj et al. Roboagent: Generalization and efficiency in robot manipulation via semantic augmentations and action chunking. arXiv:2309.01918, 2023.

R1.1: Parsing language with LLMs.

A: Thanks for the suggestion. Following your suggestion, we would like to clarify how we are "learning from language" and "how we use language models."

Our goal in the paper is to recover from language a symbolic abstract action space, composed of object-level concepts (such as colors and shapes) and action concepts (e.g., pick up). These object and action concepts can be recombined in a compositional manner to form the entire action space. In particular, we learn the grounding of these concepts: we learn recognition models for objects and policies for actions from paired demonstrations and texts, then use them in a planning algorithm to achieve a goal specified in language. In short, our method learns object and action concepts from language, and uses these concepts in language for planning.

In order to decompose natural language instructions into individual concepts, we leverage large language models to parse sentences. We agree with the reviewer that the parsing can be done by other methods as well; we choose to use LLM to perform parsing due to its simplicity and accuracy. We have added example prompts we use for GPT-4 in Appendix A.1.

R1.2: Incomplete language instructions.

A: Thank you for bringing up this point. In our paper, the LLM is only used to generate the high-level goal (e.g., reaching a target object); the generation of high-level action sequences is performed by our planning algorithm, which takes environmental states as its input (for example, if reaching the target requires opening particular doors, the planner will handle such situations.)

For scenarios where a user did not fully specify their goals, additional clarifications would be needed. This could be implemented by, for example, generating questions to the user. We did not consider such scenarios, therefore, we added a discussion of this limitation in Appendix E.

R1.3: Details for abstract plans and actions.

A: Thanks for bringing this up. We briefly discuss the abstract plans in the Setup subsections for BabyAI (Sec 5.1) and Kitchen-Worlds (Sec 5.2) due to the page limit. We have now added more details in Appendix C. We summarize the details below.

For the BabyAI experiment, the most challenging case requires 5 abstract actions to solve. This is a nontrivial task because the models are only trained on task instances that require at most 3 abstract actions to complete. In addition, there are strong dependencies between the actions (e.g., the agent must first find the key to unlock an intermediary door before gaining access to the next key). Our empirical results show that baselines obtain significantly worse performance than our model.

For the Kitchen-Worlds experiment, the most challenging case requires 4 abstract actions to solve (please see the Setup subsection in Sec 5.2). Similar to the BabyAI experiment, the abstract actions also have strong dependencies (e.g., the agent needs to first pick the object from the sink, place the object in empty space, then pick up the target object, and finally place the target object in the sink). In comparison to CALVIN [A1], which is a widely-used language-conditioned benchmark for long-horizon manipulation in 3D environments, our task is more challenging because (1) we significantly vary the environment layouts and (2) at test time, our goal instruction only contains a single goal action, and the agent needs to plan out additional subgoals before the target action; in contrast, CALVIN provides step-by-step instructions for solving tasks with 1-5 steps.

[A1] Mees et al. Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. RAL, 2022.