Video Language Planning

Thank you for your detailed comments. We respond to each concern below.

Text LLM for Model Dynamics. To effectively use a LLM to model dynamics and plan on the precise manipulation settings we consider, the LLM must be able to accurately simulate changes in the visual state subject to each possible natural language action. This requires the first a text-based serialization of the objects in an image as well as resultant simulation based off this serialization. In appendix A.1 (Figure XV), we illustrate that existing LLMs, i.e. GPT-3.5, are unable to accurately simulate the fine dynamics changes of such a serialization.

Complex Tasks. In our paper, we illustrate the ability to construct complex video plans across three separate platforms. in Language Table, we illustrate how our approach can synthesize complex plans to precisely manipulate blocks to form a line. On a 7 DoF mobile manipulator, we illustrate how our approach can synthesize a complex plan to place fruits into a cabinet by first opening the cabinet, putting fruits in it, and finally closing it. Finally, on a 14 DoF dexterous manipulator, we illustrate how our approach can synthesize multi-camera video plans of dexterous manipulation involving stacking utensils, cups, and bowls on top of each other. We believe this already represents several axes of difficulty in synthesizing video plans.

Baseline Comparisons. Since LLMs are unable to accurately simulate the detailed video dynamics for object manipulation, we cannot apply RAP to our setting. In terms of GATO and VIMA, we already compare VLP against a multimodal transformer, RT-2 which directly predict actions (which both GATO and VIMA can be seen as instances of) and find that our approach substantially outperforms this baseline. We have added references to each of these papers in the baseline section and clarified our comparisons against them.

Comparison with HiP. HiP leverages consensus between a text-to-video and LLM to synthesize a possible language instruction given the current visual observation of the scene. In VLP, we directly learn a VLM to predict text actions to execute given an image observation, which serves as an upper bound on the consensus operation in HiP. Since HiP does not do planning, its performance is upper bounded by the VLP (No Value Function) base which VLP substantially outperforms. We have clarified this in the main paper.

Inference Speed. We have added slow inference as a limitation of our approach. We believe that inference speed can be substantially improved by either distilling the diffusion model, or by using alternative generative decoding methods such as that in Phenalki.

Code and Models. We will release an open-source codebase for reproducing the results in VLP by the camera-ready version of the paper. In terms of models – they are trained using a company's internal codebase, so we are unable to directly open-source the models that are trained in the paper. However, we will aim to release pretrained weights for models trained using the released open-source codebase.

Thank you for your response, please see our additional clarifications below.

Duration of Videos. 30 minutes corresponds to the long horizon video plans shown on the website at https://vlp-iclr2024.github.io/ and corresponds to approximately 15-second videos (for example depicting a robot arranging a set of blocks into a line or moving blocks to a corner of the board).

Comparisons to Text-Based Dynamics Simulation. We agree it would be better to compare with a text-based dynamics approach with access to training examples of serialized dynamics predictions. However, our existing training dataset only consists of sequences of images, as opposed to a text serialization, and there is no easy way to convert an image directly into a text representation to train a dynamics model. The most straightforward manner would be directly to tokenize images into multimodal tokens and predict dynamics in this manner and use this for planning (this is actually the first approach we experimented with -- see our response in generating multimodal tokens). If the reviewer believes this baseline is important, we are happy to add such a comparison in the final version of the paper.

Complexity of Videos. The video models in our approach are based on those from [1], and as demonstrated in [1] on the Bridge dataset, the video models are able to accurately synthesize dynamics in cluttered kitchen counters as long as there is not significant partial observability (i.e. limited camera movement). However, we found that our approach does not work well when there is substantial camera movement (i.e. when someone is walking down a hallway into another room) as it can not preserve the world state across video frames.

Generating Multimodal Tokens. The first approach we explored in our project was actually to directly use the existing PALM-E model to generate image goals using multimodal tokens. Unfortunately, we found that this approach was unable to accurately synthesize image goals in high resolution, perhaps due to the challenge of predicting 256 accurate tokens. Furthermore, this approach could not synthesize video due to context limits as decoding a single video required 16*256=4096 tokens (16x16 tokens per image multiplied by 16 frames).

[1] Learning Universal Policies through Text-Guided Video Generation

Thank you for your detailed comments and positive evaluation. We respond to each question below.

Multimodal Value Function. We would like to clarify that in our experiments with PALM-E, our demonstrations to accomplish long-horizon tasks were already multimodal. For instance, demonstrations for the task of making a line were made by 10 different teleop contractors with very different strategies to arrange a line. To deal with this multimodality, our PALM-E model formulates the task of predicting the number of steps left to finish a task as a language prediction problem. We notice that when sampling from the model, it would thus generate multiple different predictions on the number of steps left to finish a task – corresponding to the different ways to solve this task. To get a single deterministic answer, we directly sampled to 0 temperature decoding of the number of steps left to finish a task, which corresponds to the most likely strategy to finish the task. We have added additional details in Appendix A.4.

Long Horizon Tasks. We found that VLP had no difficulties with longer horizon tasks – we artificially set the task length to 1500 so that the evaluation of an episode did not take too long (as a single environment evaluation with 1500 steps already take 1 hour).

Incomplete Videos Lacking Language Labels. We would like to clarify that the videos used to train VLP in both Language Table, as well as the 14-DoF dexterous manipulation setting, already lack full language labels. In both settings, videos of the full long-horizon task are collected by teleop workers. However, there is no way then to directly segment videos into corresponding language labels, as there is no clear language label for each consecutive temporal segment of the video. Instead, we then asked a separate group of workers to go through video and select and label any sub-snippets of the video that had coherent language labels. These sub-snippets would often overlap in time in the video and many parts of the video would not have consistent language instructions. We have clarified this in the main paper as well as in Appendix A.3.

Overestimation of Value Function. Yes, we indeed sometimes found that out-of-distribution goals would lead to overestimation of value function . We have added a reference to the CQL paper and methods in offline RL as ways to avoid the exploration of out-of-distribution states.

Typos. Thank you, we have fixed the typos you mentioned.

Thank you for your detailed comments. We address each of the above issues below.

Long Horizon Tasks. In the Language Table domains considered by this work, we believe that there is still some dependency between subgoal successes/failures. In the pair blocks by colors task, a move on one colored block significantly impacts changes the behavior necessary to push another block to pair it. This is further amplified in the make line task, where one incorrect move in one block significantly changes the downstream means necessary in all other blocks. The difficulty of both this pair by color task and the make line task is analyzed in the MAGICAL robust imitation learning benchmark on synthetic 2D shapes [1]. Even in this simplified setting, this benchmark found that existing imitation learning techniques (without planning) get essentially 0 performance on both tasks, indicating the necessity of reasoning between subgoal successes/failures.

In addition, outside of the Language Table domain, we also evaluate our approach on two other long-horizon tasks. We first illustrate how our approach to solve the task of placing all the fruit on a cabinet on a 7 DoF mobile manipulator, which requires reasoning about both when to open and close cabinets and what objects to put in a cabinet. We further show how we can synthesize long-horizon video plans for 14-DoF dexterous manipulator to place utensils, cups and bowls together on top of each other. This task again requires a model to be able to reason about which bowls to place on top of each other, as will as cups and utensils.

Limitations. We found several limitations with our method. First, as an image is not a full state representation of the world, our approach cannot deal with occluded objects. Second, our video prediction model does not perfectly preserve the shapes of objects, and they often warp their shape over the video rollout. Finally, the underlying generation speed of video is very slow (as we run a diffusion process to generate each video snippet) preventing the approach from be used for realtime control. We have added additional limitations in the limitations section of the paper in Section 5.

[1] Toyer et al. The MAGICAL Benchmark for Robust Imitation

Thank you for your detailed comments and positive evaluation. We respond to each question below.

Runtime. We reported the runtime of our approach in the original submission in Appendix A.2. Our approach takes approximately 30 minutes to synthesize a full long-horizon video and approximately 1 hour to execute a long-horizon task such as making a line of blocks. To make this more visible, we have explicitly added this sampling speed to the limitations sections of the paper. We believe that the runtime can be substantially improved by either distilling the diffusion process or using alternative decoding procedures such as that done in Phenalki.

Searching with Text. While it would be desirable to search first for a text plan and then through video prediction, this is difficult because the precise text action to execute next in the text plans depends heavily on the exact continuous state the previous text plan ends on. As an example, consider the text instruction “push the green circle to the left”. If the final goal is to group blocks by color, if the continuous execution of the text instruction pushes it to the left of the green square the next text instruction becomes “push the green circle slightly to the right”, while if its still to the right of the green square the, the text instruction becomes “push the green circle slightly to the left”. Thus to effectively plan, it is important to plan with both text and videos.

Heuristic Function. The VLM is trained to predict the number of steps left to execute a task through language prediction on a tokenized representation of the steps left. This allows the VLM to sample a multimodal set of possible number of steps left until task completion. In practice, during search, we sample at 0 temperature from the VLM, corresponding to the mostly likely probability mode of the number of steps left to finish execution of the task. We have clarified this in Appendix A.4

Replanning. Yes, replanning can be made dynamic by making replanning occur based off the pixel MSE distance between the current observation and the goal image. We have added a discussion of this in the method section of the paper.

Video Model Clarification. The videos are approximately at 15FPS, so 16 frames of video corresponds to one second of video. The long video rollouts shown in the submission cite correspond to between 10-15 video rollouts chained together.