Learning to Act from Actionless Videos through Dense Correspondences

Q3 Explanation of why overfitting did not happen.

The video generation performance is seemingly quite good even when very few training trajectories are provided (just 165 videos for Meta-World). Can you comment about how overfitting can be avoided or provide some intuition?

A: In general, we find that predicting trajectories with diffusion has less tendency to overfit than single forward models, as the diffusion model must learn the task of denoising multi-modal samples similar to trajectories of the training distribution (which gives much more supervision than directly learning a single state/action mapping). This can also be validated by comparing a simple BC baseline and Diffusion Policy. Here, we show an extended experiment below: a Diffusion Policy (Chi, Cheng, et al. "Diffusion policy: Visuomotor policy learning via action diffusion") with a similar network architecture outperforms the BC baseline.

However, the task performance of AVDC still surpasses the Diffusion Policy by a large margin. We posit that video prediction is more data-efficient way to learn behavior than behavior cloning-based methods because it only needs to learn to predict object motions (how things should move in a video). By contrast, BC-based methods, including Diffusion Policies, require "joint" learning of object motion and how to control robots.

We thank the reviewer for the thorough and constructive comments. Please find the response to your questions below.

Q1 Generalization to more complex tasks.

The most apparent weakness of this work is that not all tasks that a robot might want to solve can be solved by a trajectory of target object poses. For example, it is unclear how to plan a task that would require the robot to use another object as a tool. It is also unclear how deformable objects could be handled. There also may be tasks such as pressing a button on a microwave, which do not involve the microwave moving, but require a particular amount of force to be applied, for which this may not be applicable.

A: We thank the reviewer for bringing up these points. We will revise the paper to discuss them.

We would like to clarify that tasks requiring the use of tools can be indeed achieved with our proposed method. In particular, our method can achieve this by predicting the movements of a tool and rendering actions that induce such movements. For example, in Meta-World, the hammer task requires using a hammer as a tool.

The current implementation of our method cannot complete tasks that require interacting with deformable objects. One possible extension would be to consider tracking key points (e.g., corners of a piece of cloth) and use the key points to recover robot motion. However, this requires additional knowledge, such as key points or other representations of deformable objects.

Another limitation of learning-from-passive-video frameworks is the missing of force information. Therefore, for forceful manipulation tasks, additional training or models for forceful actions would be needed (but being able to predict that a button should be pushed down would still be useful).

Q2 Related work by Bharadhwaj et al.

The work "Zero-Shot Robot Manipulation from Passive Human Videos" by Bharadhwaj et al. presents very similar (although not identical) ideas and is not discussed in the prior work or cited. Could the authors discuss and ideally perform a comparison to some of the ideas in that work?

A: We thank the reviewer for suggesting this paper. It is definitely sharing the same spirit and some ideas with our work. We will discuss this paper in the revision in detail. While this paper and our work tackle the problem of learning from passive videos, we notice three differences between our method and Bharadhwaj et al.

First, our video prediction model is conditional on a textual description of the task. By contrast, Bharadhwaj et al. used an unconditional generation or goal-image conditioning. Second, we predict an object-centric trajectory given the initial frame; by contrast, Bharadhwaj et al. learned to predict hand poses. While hand poses contain more details of manipulator-object interactions, they may have cross-embodiment transfer issues. Finally, Bharadhwaj et al. directly predict pose trajectories, while ours predicts videos first. In our early experiments, we have found that predicting video frames gives better results than predicting object motion (via prediction optical flows). We posit that this is because video prediction is a dense prediction problem, and convolutional neural networks have been designed for this task; by contrast, optical flows are more sparse for most of the scenes because there are usually only a few objects moving in the scene.

Unfortunately, comparing with Bharadhwaj et al. is not possible because the authors have not yet released their code/models at this moment.

We thank the reviewer for the thorough and constructive comments. Please find the response to your questions below.

Q1 Grasp point => push point in H.1.2.

In the appendix H.1.2, the authors say "We calculate such direction by extrapolating the line between the grasp point and the first subgoal more than 10cm away from the grasp" Should it be push point?

A: Yes. We thank the reviewer for catching this and will fix it in the revision.

Q2 Re-initialization in the push mode in H.1.3.

In Sec. H.1.3, "For the push mode, we re-initialize the gripper as described above "is not clear. What does the re-initialization refer to?

A: When replanning is triggered, we re-initialize the gripper by (1) synthesizing a video plan, (2) sampling the grasp and calculating subgoals, and (3) calculating the direction for placing the gripper by extrapolating the line between the grasp point and the first subgoal. Once the re-initialization is done, we can start the robot execution. We will revise the paper to make this re-initialization procedure clear.

Q3 Why not applying adaptive frame sampling to iTHOR?

When learning the diffusion model for the IThor environment, why not apply the adaptable frame sampling technique in this case?

A: We thank the reviewer for raising this question. We did not use adaptive sampling because iTHOR uses discrete actions such as moving and rotating for a constant distance or angle. We will clarify this in the paper.

We thank the reviewer for the thorough and constructive comments. Please find the response to your questions below.

Q1 Related work.

The literature review is missing a number of relevant works.

• V-PTR: similar high-level motivation of using video-based, prediction-focused pre-training and then action-based finetuning. This should have likely served as a baseline for the proposed method.
  • [A] Bhateja, Chethan, et al. "Robotic Offline RL from Internet Videos via Value-Function Pre-Training." arXiv preprint arXiv:2309.13041 (2023).
• Diffusion policy: diffusion policy has shown very good results in terms of multi-task, low-data regime performance.
  • [B] Chi, Cheng, et al. "Diffusion policy: Visuomotor policy learning via action diffusion."
  • [C] Ha, Huy, Pete Florence, and Shuran Song. "Scaling up and distilling down: Language-guided robot skill acquisition."

In particular, my biggest concern with the paper is the lack of comparison to a strong BC baseline. As the AVDC method uses diffusion to predict images, it seems natural to baseline against a diffusion policy (e.g., [B, C]). R3M is a fairly old representation at this point (e.g., Voltron [D] would be a better representation) and the simple MLP-based BC policy would strongly underperform diffusion policy. This is confirmed by, for example, the very poor baseline performance in Tables 1 and 2. In Sec. 4.3, it is mentioned that since 'R3M is pretrained on robot manipulation tasks ... it might not be suitable for visual navigation tasks'. Something like [E] could be a better baseline here.

• [D] Karamcheti, Siddharth, et al. "Language-driven representation learning for robotics." arXiv preprint arXiv:2302.12766 (2023).
• [E] Shah, Dhruv, et al. "ViNT: A Foundation Model for Visual Navigation." arXiv preprint arXiv:2306.14846 (2023).

A: We appreciate the reviewer pointing out these relevant works. We will discuss them in the revised paper.

Video pre-training for robots (V-PTR): V-PTR learns from videos (the Ego4D dataset) without action labels by extracting a value function during Pre-Training Phase 1. Then, V-PTR learns a Q-value function during Pre-Training Phase 2, which requires actions. Finally, V-PTR is fine-tuned on downstream tasks, which requires learning by interacting with environments. In contrast, our proposed method requires no action labels or learning with environment interactions. With the differences in problem formulation, we believe V-PTR and our method are not comparable. Furthermore, the V-PTR paper was posted on arXiv on Sep 22, six days before the ICLR 2024 deadline (Sep 28), which makes it difficult for us to include a discussion or comparison to V-PTR by the time we submitted our work.

Diffusion Policy: We thank the reviewer for suggesting this method. We have implemented and conducted experiments with the Diffusion Policy on Meta-World tasks. Following the paper, the image observation is encoded with the adapted ResNet-18 with group norm and spatial softmax pooling. For the diffusion backbone model, we experimented with the 1D convolutional FiLM U-net architecture proposed in the paper. The backbone model is adapted to take in task embeddings, i.e., the CLIP-Text task embeddings are concatenated with the observation embeddings. The hyperparameters $T_o$ (the number of past frames used as a condition), $T_p$ (the number of future actions to predict), and $T_a$ (the number of actions to execute before replanning, 0<$T_a$ <=$T_p$), were set to align with the configurations used in most of the paper's experiments. Specifically, we set $(T_o, T_p, T_a) = (2, 16, 8)$. The results are presented in the following table.

The results show that Diffusion Policy outperforms BC, and our method AVDC outperforms Diffusion Policy in 10/11 tasks and the overall performance by a large margin. Note that both BC and Diffusion Policy have access to action labels that are not accessible to our proposed method; therefore, the comparison is unfair. We will include the Diffusion Policy in our revised paper.

Q3 Baseline that uses pseudo action labels.

AVDC relies on object/robot masks - a simple baseline would be to use those as a proxy for the actions. One could get pseudo action labels from the videos and train a policy.

A: We thank the reviewer for the suggestion. However, this is not directly applicable to our setting because computing robot actions requires depth, and we do not assume depth information during training, and none of the Bridge and Visual Pusher datasets have depth data.

Still, to provide an idea of the performance of using object masks as a proxy for actions as suggested by the reviewer. We have conducted experiments with the following setting: We trained a model that takes in a segmented object and directly predicts optical flow within the segmentation (without diffusion). Then, we used the procedure as AVDC to calculate actions. The results are presented as follows.

The results show that our proposed AVDC outperforms the method that predicts the object masks as a proxy for the actions. We will revise the paper to include this experiment.

Q4 Not applicable for tasks that require using tools.

AVDC assumes that all objects are going to be directly manipulated by the robot directly but this is not the when one object as a tool.

A: We would like to clarify that tasks requiring using tools can be indeed achieved with our proposed method. In particular, our method can achieve this by predicting the movements of a tool and rendering actions that induce such movements. For example, in Meta-World, the hammer task requires using a hammer as a tool.

Q5 Action-free baseline for navigation.

Navigation approaches have many action free baselines which should be explored as well

A: We thank the reviewer for the suggestion. However, we are unsure what the "action-free" navigation baselines mean. Can the reviewer kindly provide specific papers or instruct on conducting additional comparisons?

Q6 Real-world experiments.

It would be good to see real world experiments

A: We have shown real-world experiments with a Franka Emika Panda arm in Section 4.5; a further discussion can be found in Appendix H.3. Please also kindly see our videos in the supplementary website.

Q7 Video prediction quality.

It would be good to have more of an analysis on the quality of the video prediction model. I suspect it has a

A: We thank the reviewer for the suggestion. To measure the quality of synthesized videos, we provide qualitative results in Figure 5, Figure 7, Figure 8, Figure 10, and Appendix G.4. Also, we quantify the quality of synthesized videos by reporting the MSE between the last frames of synthesized videos and ground truth videos, presented in Appendix E and on the supplementary website. We believe this sufficiently justifies the quality of the synthesized videos.

Since the reviewer asked for additional analyses, we further quantitatively compared the synthesized videos to the ground truth videos regarding PSNR, SSIM, MSE, and LPIPS. Specifically, we synthesized videos with our trained Meta-World video model given unseen first frames, and compared every synthesized video frame to the corresponding ground truth video frame. We report average PSNR, SSIM, MSE, and LPIPS (AlexNet) scores over 15 videos for each view, totaling 15*3*11=495 videos being evaluated. Also, we report the scores comparing the last frames of synthesized videos and ground truth videos. The following table summarizes the result.

The results show that our proposed video diffusion model can reliably synthesize videos of task execution.

We thank the reviewer for the thorough and constructive comments. Please find the response to your questions below.

Q1 Setting limitations: robot video and 3D.

I think one of the main limitations is the setting: AVDC needs videos of robots performing the task. I believe this is a contrived setting as it is very likely that if video + 3D information is available, then this was a robot demonstration, and one can just collect action data. To me, it is unclear how this approach will scale beyond robot data.

A: We would like to clarify the following two points:

• Our work does not need any 3D information during training, and therefore, our proposed method learns from only videos with RGB frames. The 3D information is only required during inference for rendering robot actions.
• Our work can learn from videos of humans or robots that are different from the hardware setup performing actions. Specifically, to show the ability of our proposed method to transfer across different embodiments, we have shown successful results with visual pusher experiments in Section 4.4, where the robot learns from human video (whose actions are not available). Additionally, in our real-world Franka Emica Panda arm experiment presented in Section 4.5, the robot learns from a large offline dataset from other robots, whose actions are not directly usable, and a small number of human data without action labels. AVDC only needs videos of an arbitrary agent performing tasks, such as a human or a different robot. This ability to learn from diverse data sources underscores its potential for real-world applications.

Q2 BC Baseline.

I am concerned by the reported results for the BC baseline. Due to action data being available, as well as the robot data being in-domain for the task a simple BC or kNN baseline should work very well. There are many cases where the results are < 5% success. This should be addressed. I would be willing to increase my score if this weakness is addressed.

A: Thank you for the suggestion. To validate our BC implementation, we included an additional experiment of training BC with more data. The results are reported below.

The experiment results above show that plain BC needs 20 videos per view (in total 660 videos with action labels, compared to AVDC trained with 165 videos without action labels) to perform similarly to our method AVDC (43.1% overall). That said, BC needs around four times more videos and action labels than our method, which highlights the efficiency of our proposed method. More importantly, BC still cannot learn most tasks that require grasping (e.g., pick and place, use tools) even with 50 demonstrations per view (BC-1650).

We would like to emphasize that we included the performance of BC to calibrate the difficulty of the tasks, and BC has access to action labels that are not accessible to our proposed method; therefore, the comparison is not fair.

Dear Reviewer QCWJ

Thank you for reviewing our submission and your valuable feedback. We hope our clarifications on problem settings and new results (BC with more data, pseudo-action labels, and qualitative and quantitative results for video generation) can address your concerns. We are happy to discuss with you and answer any further questions. As the deadline for discussion is approaching, we very much look forward to your feedback.

Thanks, Authors