Video2Policy: Scaling up Manipulation Tasks in Simulation through Internet Videos

Dear Reviewer eE6D:

Thank you for your detailed comments and advice! We hope the following addresses your concerns.

For Weaknesses:

The generated tasks remain limited to simple tabletop scenarios; incorporating more complex or long-horizon tasks would enhance the framework's versatility.

We focus on tabletop scenarios as this is the most common evaluation scenario in related work [1, 2, 3, 4]. Also note we have evaluated hard tasks such as throwing, insertion, and sliding. Future work can also extend the method to reconstruct bigger scenes for mobile manipulation.

[1] Ma et al., Eureka: Human-Level Reward Design via Coding Large Language Models

[2] Wang et al., Gensim: Generating robotic simulation tasks via large language models

[3] Octo Model Team, Octo: An Open-Source Generalist Robot Policy

[4] Kim et al., OpenVLA: An Open-Source Vision-Language-Action Model

This work primarily constructs a pipeline by integrating previous works as components, but it doesn't address technical challenges through unique design innovations.

We focus on evaluating a framework to generate data for training generalist policies. Our approach is novel and is more scalable due to the use of internet videos jointly with LLM reward design. We also would like to highlight the technical contributions required for the performance we report in section 4.2. Track prompting, in contrast to prior work, grounds the model in the real video and improves success function generation. Rejection sampling further reduces hallucinations in reward and success function generation. Please refer to App.A.3 to see these contributions ablated.

No real-world experiments were conducted to validate the approach.

We intend to perform sim2real experiments to further evaluate the performance of the method and we will post an update once we are able to do this.

For Questions:

In the object reconstruction phase, generating accurate reconstructions from masked images can be challenging due to limited pixel data for each object. With low-resolution images, tools like InstantMesh often struggle to produce high-quality results. Has this limitation been encountered in this work, and if so, what solutions have been implemented to address it?

Yes, we find that the mesh results are better when the resolution is higher. However, for our setting, it can generate good and reasonable mesh by resizing the width of the images to 512 by interpolation.

It is an interesting question about how to generate good results by limited pixel data. We think a natural way is to use some high-resolution recovering method. Another method can be some mesh generation model under image and text input, which can use some text priors for reasonable geometry generation. However, we haven't found any model that can work well on internet images.

Given the focus on data collection, comparing only success rates may not provide a fair assessment. Factors like the time-intensive iterative reward design process in this framework should be considered. A more meaningful comparison might be the time required to generate a successful trajectory.

Thank you for your suggestions. We have drawn the curve with the iteration v.s. success rate in Fig. 10 (Line 864-880). We compare the performance of Video2Policy, Eureka, and the RobotGen. The results show that our method significantly outperforms the other baselines across multiple iterations.

We believe it is unfair to compare the time required to generate one successful trajectory, as the successful trajectories produced by a policy with a 1% success rate are too limited and lack diversity, making them inadequate as high-quality data. Instead, we plot the curve of the success rates v.s. 1 / training time, which demonstrates the performance and efficiency in Fig. 10. Our method demonstrates superior Pareto optimality, effectively balancing multiple objectives to achieve optimal trade-offs compared to other approaches.

Thank you again for your detailed review. We have revised our paper in OpenReview. We highlight the important changes and essential details that reviewers have mentioned in blue color.

Why do you need 6D pose tracking? And how does it relate to policy training? You are not imitating the estimated trajectory from the video right?

We attempt to leverage some explicit visual information for VLMs to generate codes, which is a part of the prompts. We inform the GPT-4o that it needs to focus more on the relative position, especially on the initial and final states. This introduces richer information for designing success and reward functions. We do not imitate the estimated trajectory from the videos.

To evaluate the importance of pose tracking, we further conduct an ablation study that removes the 6D position information as follows:

As shown above, after removing the tracking information, the performance drops for the tasks with multiple objects. We find that more stringent success conditions can be added sometimes under tracking info.

Take the task 'Throw Garlic into Bowl' as an example, sometimes it will add an extra condition:

"""

def compute_success(self, states):

#garlic should be inside the bowl's boundary in Z-axis

z_condition = (lower_z_of_garlic < upper_z_of_bowl) & (upper_z_of_garlic > lower_z_of_bowl)

#garlic should be inside the bowl's boundary in XY-plane

xy_condition = xy_distance < torch.min(states["bowl_size"][:, :2] / 2 - states["garlic_size"][:, :2] / 2)

success = z_condition & xy_condition

distance_condition = states["dist_garlic_to_bowl"] <= 0.0396

success = success & distance_condition

"""

Here, 0.0396 is the distance by calculating the delta of 3d pos (we provide this in the prompt). Moreover, without tracking information, our method can also work better than Eureka since we have applied CoT for designing each code part.

Thank you again for your detailed review. We have revised our paper and highlighted the important changes and essential details that reviewers have mentioned in blue color. Please let us know whether this addresses your comments and whether you still have any remaining concerns.

Moreover, the efficiency of the training pipeline is unclear, since it involves multiple rounds of the reward reflection. Is there any metric on the efficiency of this part?

Thank you for your suggestions. We have drawn the curve with the iteration v.s. success rate in Fig. 10 (Line 864-880). We compare the performance of Video2Policy, Eureka, and the RobotGen. The results show that our method significantly outperforms the other baselines across multiple iterations. Our method demonstrates superior Pareto optimality, effectively balancing multiple objectives to achieve optimal trade-offs compared to other approaches.

Again, given that some baselines do not utilize the reflection, how do you ensure fairness?

We provide the Eureka baseline that utilizes reward reflection. We note the other baselines - 'code-as-policy' and 'RoboGen' - do not utilize reflection because they have no success metric for updating the reward functions. We think code generation for policy learning is a novel topic in recent years, and the chosen baselines cover most directions to solve the problem.

Furthermore, the reconstruction does not include any scene information, how do you guarantee, it can be generalized to more complex scenes? Do you have some studies on the effect?

We focus on tabletop tasks as these are the most common evaluation scenarios, and are more efficient to iterate with reinforcement learning. Our main contribution is evaluating the algorithmic framework of converting videos to data for training generalist policies on the example task of tabletop manipulation. Future work can extend the method to reconstruct bigger scenes for mobile manipulation.

Again, since there is no limitation of applying the proposed to more videos, why not include videos from a much larger dataset to verify the benefits of the pipeline?

Reconstructing scenes is fast, but training RL policies requires approximately 1 hour per iteration. To perform experiments in this paper, we only used 8 GPUs, which is why we focus on 60 videos to illustrate the scalability of our method. Future work can evaluate further scaling.

To further demonstrate scalability, we performed additional experiments up to 100 videos. The results can be found in the revised paper at line 464-478. Overall, increasing the number of videos continues to improve the performance of the BC-V2P model, ultimately achieving an average success rate of 75% on 10 novel videos, up from 64% on 60 videos. Different from current real2sim methods which manually constructed a few scenes as [1], our process can leverage large amounts of scenes and learn policies from them more autonomously.

[1] Torne et al., Reconciling reality through simulation: A real-to-sim-to-real approach for robust manipulation.

Dear Reviewer MeiF:

Thank you for your detailed comments and advice! We hope the following addresses your concerns.

For the Weaknesses and Questions

The study of the robustness of the proposed pipeline is limited. For example, what is the performance of the DINO grounding, and how does the detection or segmentation affect the trained policy, in which sense? Also, how does the reconstruction quality affect the generalization?

Thank you for this suggestion. To quantify the robustness of our method, we evaluate vision models used in our pipeline individually by manually annotating their reconstruction success rates. For instance, when assessing the grounding accuracy of DINO, we sample 20 videos. Similarly, for evaluating the segmentation accuracy of SAM-2, we sample 20 successful bounding boxes generated by DINO. Following this approach, we systematically test the robustness of each module in the pipeline. If the reconstruction is broken or identifies the wrong object, it is classified as a failure case. The results are as follows. We can find that the segmentation and mesh reconstruction parts are more robust than the others.

It is interesting to investigate how the reconstruction quality affects the generalization. Since we have no access to the ground-truth objects from the videos, it is impossible to make ablation studies to compare the results. However, we conduct experiments to investigate the effects of the number of reconstruction videos. We find increasing the number of videos will improve the generalization results, which proves that increasing the noisy reconstruction scenes can help with generalization.

Why is the reconstruction performed with only one image? Why not use more images for a better reconstruction?

In this work, we focus on RGB internet videos with a single view. Reconstruction from videos is an important direction for future work that will help address more complex datasets and task such as mobile manipulation. However, existing tools are geard towards multi-view reconstruction and we did not find them helpful for the something-something dataset, which has little camera movement.

The proposed use of an off-the-shelf component UniDepth for size estimation, since there is much noise, is there any study showing how the noise affects the final performance?

Thank you for this suggestion. We conduct experiments to evaluate the noise effects of the depth estimation component. For one thing, we use the Depth-aware Realsense Camera to get the ground-truth depth of the self-recorded video 'Sliding' (We can only access the GT depth for this video, as the other videos are recorded outside of the lab) and evaluate the d1 metric (higher is better) for the video.

$\text{d1} = \frac{\text{Number of pixels where } \frac{|d_{\text{pred}} - d_{\text{gt}}|}{d_{\text{gt}}} > \text{0.1}}{\text{Total number of pixels}}$

The results are as follows. We see that the depth estimation of the object region is accurate enough for our purposes.

The size error under the depth prediction is as follows. The error of the size prediction is small enough to use objects for task generation.

Moreover, we resize the objects to the GT size and apply the previously trained model to study how the noise affects the final performance. We keep the same state inputs and the results are as follows. The performance drops slightly, but the model can still solve the task at a high success rate.

I am confused with the design logic of the 6 components in the task code generation module. Any discussion on the why and why such a design is efficient?

This design follows the basic generation pipeline from the previous works that focus on code generation for simulation tasks (ma23’, wang23’). It is helpful for LLM to infer codes by prompting in a well-organized way. We clarify this in the revised paper.

Dear reviewer MeiF,

Please check out our latest response and general response (with the sim-to-real results), and let us know if there is anything we can further address.

More elaborations about your concerns "I am not convinced by the answer to the question "Furthermore, the reconstruction does not include any scene information, how do you guarantee, it can be generalized to more complex scenes? Do you have some studies on the effect?" Showing the capability to train a generalist policy can not be restricted to a simple desktop environment and also by ignoring scene information.

Our current sim-to-real solution is to segment the object of interest by using SAM-2, which can give us a segmented mask of the object, and we can also obtain segmented depth with a depth camera. Only using object-based information is the most common approach (e.g. in [1, 2, 3]). This is typically because of two reasons 1. The RGB rendering in simulation has a large domain gap with the real world, so it's hard to deploy the policy trained with RGB directly; 2. Depth sensing is noisy and only accurate in a certain range, so people will remove the depth information from unrelated objects (e.g. tables, background objects that are far away) and only focus on the target objects.

To clarify, for table-top manipulation we are studying, is not necessary to have scene-level reconstruction. Furthermore, we don't have very good tools for scene reconstruction from in-the-wild videos. The majority of the scene reconstruction work is based on scan, which has more assumptions that are infeasible for in-the-wild videos.

However, we also agree that scene information is important for other more complex tasks (e.g. mobile manipulation), which we leave for future work. We want to emphasize that our current goal is not to create a 1:1 digital twin in the simulation environments but to leverage the diversity of video to ground the objects and strategies of manipulation and obtain diverse data and more generalizable policies.

[1] Qi et al., General In-Hand Object Rotation with Vision and Touch

[2] Lin et al., Twisting Lids Off with Two Hands

[3] Chen et al, A System for General In-Hand Object Re-Orientation

Dear Reviewer EeTh:

Thank you for your detailed comments and advice! We have performed initial experiments to address some of the concerns. We will further perform additional experiments and will post them as soon as we have them.

Why use only 60 videos?

…

How does the performance scale with the number of videos?

Reconstructing scenes is fast, but training RL policies requires approximately 1 hour per iteration. To perform experiments in this paper, we only used 8 GPUs, which is the reason why we focused on 60 videos to illustrate the scalability of our method. To further demonstrate the scalability of our method, we performed additional experiments up to 100 videos. The results can be found in the revised paper on lines 464-478. Overall, increasing the number of videos continues to improve the performance of the BC-V2P model, ultimately achieving an average success rate of 75% on 10 novel videos, up from 64% on 60 videos. Different from current real2sim methods which manually constructed a few scenes as [1], our process can leverage large amounts of scenes and learn policies from them more autonomously.

[1] Torne et al, Reconciling reality through simulation: A real-to-sim-to-real approach for robust manipulation.

There's no real-world evaluation

…

The proposed method doesn't handle material and physical properties

We intend to perform real-world evaluations to address this concern. For material and physical properties, we can adopt standard domain randomization techniques, which are also manually picked or designed in other previous work [2, 3, 4, 5]. We would also like to note that our main contribution is a new algorithmic framework, that can perform automatic environment construction and policy training, which we primarily focus on simulation experiments for the current version.

[2] OpenAI, Solving rubik’s cube with a robot hand.

[3] Padir et al, Sim2real2sim: Bridging the gap between simulation and real-world in flexible object manipulation.

[4] Peng et al, Sim-to-real transfer of robotic control with dynamics randomization.

[5] Handa et al., Dextreme: Transfer of agile in-hand manipulation from simulation to reality.

The technical contribution and the novelty of the paper is unclear.

We propose a framework to automatically create tasks and train policies with better generalizability. We believe this is more scalable and autonomous than existing digital twin approaches which require manual efforts to create scenes, and can obtain better policies than other generative simulation methods. Our approach is also more practical as we ground our tasks in real videos. It has the potential to leverage large-scale videos to train a more generalizable policy.

The environment is assumed to be a flat tabletop surface, which is a big assumption. It's unclear how the method can be applied to complicated real-world scenarios.

We focus on tabletop tasks as they are the most common evaluation scenarios, and more efficient to iterate with reinforcement learning. Our main contribution is evaluating the algorithmic framework of converting videos to data for training generalist policies on the example task of tabletop manipulation. Future work can extend the method to reconstruct bigger scenes for mobile manipulation.

calling the learned policy a "generalist policy" is a massive overclaim. … the paper essentially proposes a 3D/4D reconstruction framework, the title video2policy seems quite inappropriate. Besides, the policy ... cannot be generalized back to the scene in the original video.

Thank you for pointing this out. We have clarified that we are not comparing to robotic generalist policies (e.g. RT-X, Octo) and removed this word from our paper to address the confusion. We call our method "Video2policy" as we can generate tasks grounded by videos, and train a policy to obtain similar behaviors automatically through the help of 3D reconstruction tools, LLM, and RL. We also welcome suggestions on different names for the method.

We also want to emphasize that we aim to obtain a more generalizable policy from diverse videos, but not perform standard real2sim2real. The latter is not feasible as we don't have access to the original physical scenes. However, we showed that with the help of diverse videos, our method can generate useful trajectory data to train a policy that generalizes beyond training scenes. We leave its real-world applications as future work.

Calling a video a task is misleading.

We have changed the wording ‘task’ to ‘task instance’ in the paper. In our framework, a ‘task instance’ includes a different scene, such as different objects, a different task, such as ‘uncover’, and a different reward function. We welcome suggestions on another term to use.

Thank you again for your detailed review. We have revised our paper in OpenReview. We highlight the important changes and essential details that reviewers have mentioned in blue color.

Dear reviewer EeTh,

We kindly remind you that the final stage of discussion is ending soon, and so please kindly let us know if our response has addressed your concerns.

Here is a summary of the responses and revisions:

• We conduct sim2real experiments on objects with novel shapes or types in the real world for the task lifting to further study the efficacy of our pipeline.
• We further add more videos from 60 to 100 to investigate the scalability of our methods.
• We clarify the reasons and necessities of solving manipulation tasks on the tabletop setup.
• We made modifications for the misunderstanding words mentioned in your review.
• We revised our paper and updated it on the website.

Thanks again for your time and reviews, we will be happy to answer if there are additional issues or questions.

How precise does the tracking of the objects have to be? ... So what success/failure rate did you notice during the experiments if the 6D position for example is way off?

For tracking, we attempt to leverage some explicit visual information for VLMs to generate codes, which is a part of the prompts. We inform the GPT-4o that it needs to focus more on the relative position, especially on the initial and final states. This introduces richer information for designing success and reward functions.

We agree that it is important to investigate the tolerance of 6D position errors, so we conduct an ablation study that removes the 6D position information as follows:

As shown above, after removing the tracking information, the performance drops for the tasks with multiple objects. We find that more stringent success conditions can be added sometimes under tracking info.

Take the task 'Throw Garlic into Bowl' as an example, sometimes it will add an extra condition:

"""

def compute_success(self, states):

#garlic should be inside the bowl's boundary in Z-axis

z_condition = (lower_z_of_garlic < upper_z_of_bowl) & (upper_z_of_garlic > lower_z_of_bowl)

#garlic should be inside the bowl's boundary in XY-plane

xy_condition = xy_distance < torch.min(states["bowl_size"][:, :2] / 2 - states["garlic_size"][:, :2] / 2)

success = z_condition & xy_condition

distance_condition = states["dist_garlic_to_bowl"] <= 0.0396

success = success & distance_condition

"""

Here, 0.0396 is the distance by calculating the delta of 3d pos (we provide this in the prompt). Moreover, without tracking information, our method can also work better than Eureka since we have applied CoT for designing each code part.

How do you evaluate the validity of a generated task code? How often does hallucination occur when generating child success/reset functions using Chain-of-thought?...

Thank you for your suggestion. To evaluate the validity of the generated task code, we have some instructions.

For example, for correctness, we inform the GPT-4o to read and analyze the success part; for reasonability, we inform the GPT-4o to avoid picking the code that assumes some scalar value or states.

To quantify the frequency of the hallucination, we run Video2Policy (16 samples, 1 iteration) for the 3 tasks (Lifting, Uncover, Throw). Here we remove the while-loop generation so that not all samples are runnable. (Previously, if one sample fails, we regenerate again until all 8 samples are runnable.)

Here the hallucination samples include non-runnable samples and zero-score samples. We can find that after picking by GPT-4o, the hallucination problem alleviates in a degree.

‘Task code’ is a relatively unknown term. You define this in parts, but just making this term bolded and clearly stating what this means at the end of paragraph 2 of the introduction (somewhere around line #050) would make the paper much clearer.

Thank you so much for this suggestion. We have added more explanation for this and revised the paper.

Figure 3 shows some task codes, but please add what the task description is and the goal of these tasks are to the figure.

Thank you for your suggestion. We have added the descriptions in the Figure and revised the paper.

Other Minor Comments

L70-82 gives a lot of related work that seems misplaced/breaks the flow of the contribution description. Preferably condense into a few lines and move the rest to the related works section

Thank you for your suggestion. We have revised the paper about this part.

Line 268/269 are unclear. What is the ‘parent’ reset function vs the ‘children’ reset functions? Please clarify this.

The parent reset function is the manual default reset function that resets all the objects randomly without considering some conditions between the objects' initial positions (such as A should be on top of B). The children reset function is written by LLM in the task code. We have revised this part to make it more clear.

Thank you again for your detailed review. We have revised our paper and highlighted the changes and details that reviewers have mentioned in blue color.

Dear Reviewer CR8X:

Thank you for your detailed comments and advice! We hope the following addresses your concerns.

For the Weaknesses:

It's important to include some examples of the task codes that are generated … Also, the types of reward functions/success functions that are generated and how they evolve over the several iterations should be described.

Thank you for your suggestions. We have added examples of the generated codes in Lines 756-788: (1) JSON file content, Lines 759-760; (2) reset function, Lines 765-766; (3) success function, Lines 775-781; (4) additional observation, Lines 767-772; (5) additional observation space Lines 773-775; (6) reward function Lines 822-849; and the process of reward evolutions in Lines 791-850.

How can we verify the iterative reward function is improving the policy overtime and a necessary part of the pipeline? A graph with the iteration vs success rate for the tasks is good to include for this. X-axis would be the iteration number, and the y-axis would be the success rate.

Thank you for your suggestions. We have drawn the curve with the iteration v.s. success rate in Fig. 10 (Line 864-880). We compare the performance of Video2Policy, Eureka, and the RobotGen. The results show that our method significantly outperforms the other baselines across multiple iterations. Our method demonstrates superior Pareto optimality, effectively balancing multiple objectives to achieve optimal trade-offs compared to other approaches.

However, there is not much subsequent discussion about when/what GPT-4o generates these in the experiments that are performed. Furthermore, how does GPT-4o know what additional observations to generate (there are no iterative updates for generating this part right)? Only the reward and success functions are generated with CoT so there is some learning about the environment that can be seen?

We add more explanations and examples in Line 756-788. We give instructions and prompts on how to add observations for new tasks as follows:

"""

For example, if the task is considering the target position or orientation, the distance between the target and the current should be added. Or if the task is about some dynamic motions, the relative velocity or direction can be added.

"""

Also, we provide the CoT examples of analyzing why/what to add.

So when generating the task code, the GPT-4o can generate some interesting states to learn the task.

And yes, there are no iterative updates for generating this part. Instead, we will generate 8 examples of the task code at the beginning and choose one of them for the reward iteration stages.

For the Questions:

Is it possible that the reset functions that are generated by V2P end up allowing the success state to be more easily reachable than the hard coded reset states that other methods such as CoP, Eureka, or RoboGen have? ... Maybe one method of comparison is to check qualitatively they all reset to very different states that are equally ‘hard’ or ‘easy’ to achieve success form there.

We conduct ablation studies by choosing the same 'reset' function for the baselines. For the code-as-policy, we generate the policy code and execute it in the Issac gym. Thus, the 'reset' function is the same as the Video2Policy since they share the task code. We choose 3 tasks, one of a single object and two of multiple objects. The results are as follows:

As we can see, for tasks with a single object, the results are the same because the generated 'reset' function calls the base reset function as follows:

'''

class XXX(Task):

def reset_objects_states(self, env_ids):

​ super().reset_objects_states(env_ids)

'''

For tasks with multiple objects, the results have limited changes. The reason is that when generating the reset function, the LLM introduces certain constants. These constants may vary, for example:

It can be 'book_state[env_ids, 2] = pen_state[env_ids, 2] + 0.1' or 'book_state[env_ids, 2] = pen_state[env_ids, 2] + 0.15'.

However, the variance has limited effects on the final results. It indicates that there is no reset cheating.