VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation

Q2: Controversy of the use of web data and OXE as pre-training.

A2: We used web data and OXE for pre-training, emphasizing their role in situations with insufficient fine-tuning samples. Table 2 in our paper shows experiments conducted on RLBench with 10 demos, meaning each task has only 10 demonstrations for training. Regarding the controversy over using web data and OXE data, we have the following rebuttals. The reason why web data contributes very little is that it is non-optimal data. Open-Sora uses internet data, which has a significant gap with robotic manipulation scenarios. Another reason is that we only loaded the first 12 layers to make the model training more efficient. The OXE dataset or other large dataset indeed plays an important role in the generalization of the model. To clarify this point better, we have converted Table 2b into the table below. Additionally, we conducted experiments using other datasets for video prediction pre-training. Specifically, we replaced the first-stage pre-training dataset with Ego4D, while keeping other settings consistent with those in Table 2b. From the table below, we can see that our model's baseline performance on RLBench-10 is 51.2. When using robotic trajectory data (OXE), the performance improves to 57.1. When the pre-training dataset is non-robotic but still contains dynamics (Ego4D), the performance slightly decreases but still shows an improvement over the initial performance, reaching 55.5.

Q3: Comparison of our results with GR-1.

A3: For a fair comparison with GR-1, we also conducted experiments with our model on the CALVIN benchmark, using the design choices of diffusion, OXE++, and action only. The results, as shown in following table, indicate that our model outperforms GR-1 on the same benchmark by 0.61.

Next, we discuss these three design choices.

1. diffusion vs autoregressive

In Table 1 of our submission, we have already included a comparison between diffusion and autoregressive methods. Additionally, we evaluated the diffusion version of GR-1, as implemented in [5], on CALVIN. The following two tables show the results of two different designs of our method on RLBench-100 (copied from Table 1 of the submission) and the results of two different designs of GR-1 on CALVIN. The results indicate that, regardless of the method, diffusion performs better. This is likely because fine-grained diffusion design capture the intrinsic dynamics of videos better than coarse-grained autoregressive design.

2. Ego4D vs OXE

Ego4D is a hand dataset containing approximately 3,500 hours of first-person perspective videos of human-object interactions. OXE++ is an expert-annotated robotic dataset with about 1.4M trajectory data points. Intuitively, in terms of performance, training on OXE is better than training on Ego4D. We conducted an ablation analysis of the pre-training data in the first stage for both benchmark, and the results are shown below. There is no doubt that using OXE for video prediction pre-training is better than using EGO4D. This is because OXE is expert-annotated robotic data with higher quality. Of course, Ego4D also has its advantages: it is easy to collect, allowing for data scale expansion and the resulting scale effects.

Dear Reviewer,

Thanks for your constructive comments. Below, we would like to address all the weaknesses and questions in details.

Q1: ... by changing the number of cameras used ... the number of seeds used ... [ACT3D, RVT] achieved big leaps in 2023 and the current SOTA on RLBench is [3DDA].

A1: We address this issue in the following four aspects.

1. We used an overhead camera and a wrist camera because the pre-training dataset OXE typically contains only these two cameras. To align the pre-training dataset with the fine-tuning dataset, we used a unified input. Since we adopted the STDiT structure of Open-Sora, it can easily be extended to multi-view input by simply concatenating tokenized image patches as input.

2. We thoroughly examined PerAct[1], Act3D[2], RVT[3], 3DDA[4]. The original papers of PerAct and Act3D used 1 seed. RVT used 5 random seeds during inference because it did not directly output actions but used a sampling-based motion planner which introduced randomness. 3DDA used 3 random seeds but has not been formally published yet. To align with methods like RVT, we also reported the variance of 5 seeds in the evaluation on RLBench.

3. Directly comparing our method with RVT is not entirely fair. Our proposed method has the following key differences from RVT and other state-of-the-art approaches: a. RVT utilizes four RGB-D cameras, including a front camera, left shoulder camera, right shoulder camera, and wrist camera. In contrast, our method only uses an overhead camera and a wrist camera, and it does not incorporate depth information. b. RVT predicts the position and rotation angles of key frames, then uses a sampled motion planner to move the robotic arm to the corresponding position and rotate to the corresponding angle. Our method, on the other hand, outputs a 6-dimensional vector end-to-end, which is directly executed by the end-effector controller. An absolutely fair comparison may not be entirely feasible, but we can adopt some relatively fair comparisons. Specifically, we have standardized the first difference mentioned above by also using multiview input in our proposed method. Fortunately, due to the flexibility of our model, we only need to replace our overhead camera input with multiview cameras with the same resolution 128 x 128 (including front camera, left shoulder camera, right shoulder camera and wrist camera ) as the input of STDiT in the 2nd stage.

4. We compared our method with PerAct, RVT, and Act3D on RLBench-100 benchmark. We are not comparing with 3DDA for now, as it has not yet been published in a conference or journal temporarily. The results are as follows.

3. joint frame-action prediction vs action only

To investigate which is better, we conducted the following experiments: (a) our model on RLBench using joint frame-action prediction and action only, and (b) GR-1 on CALVIN using joint frame-action prediction and action only. The results indicate that different architectures may require different designs. Our model performs better when only predicting actions because we use cross-attention to integrate information into the action query and employ an additional diffusion head for output. Action prediction and frame prediction follow two separate branches, and frame prediction might disrupt the gradients back-propagated for action prediction. GR-1 performs better with joint prediction, possibly because it uses a single branch where frame prediction can better learn the representation of the video.

Q4: A comparison of the research motivations between our method and GR-1.

A4: We can restate the differences of motivation from GR-1 in the following aspects: a. Diffusion-style methods, compared to GPT-style methods like GR-1, can predict longer future frames. This is because diffusion can simultaneously denoise multiple frames to generate future frames, while GR-1 can only generate one future frame image in a next-token generation manner based on historical frames. Generating future frames for more time steps has been proven beneficial, as can be seen in Table 7 of our paper. We copy the table below.

b. GR-1 models inputs and outputs at the token level, which lacks perception of the image space and can lead to information loss. Our model, on the other hand, models inputs and outputs at the image level and has a spatial layer for fine-grained perception of the image.

Q5: The reasoning capability of diffusion model.

A5: We are not the first to utilize the reasoning capability of diffusion models. [9] propose an approach that leverages the expressiveness of latent diffusion to model in-support trajectory sequences as compressed latent skills. [10] use a latent diffusion model with a variational autoencoder that can reconstruct the neural network weights, to learn the distribution of a set of pretrained weights. [11] introduces diffusion models to search for topological logic. Additionally, OpenAI's official website [12] also describes Sora's capability to understand the physical world, e.g., “The model understands not only what the user has asked for in the prompt, but also how those things exist in the physical world.”

Q6: Discussion on the relation of our paper to "Learning Universal Policies via Text-Guided Video Generation"[7]

A6: [7] proposes explicitly generating future frames and using the variance between future frames and current observation images to generate action sequences. In contrast, we implicitly represent future frames without an additional denoising process, directly generating action sequences from the features. The benefit of this approach is that it eliminates the very time-consuming iterative denoising process, making it effective for real-time robotic manipulation.

References：

[1] Perceiver-actor: A multi-task transformer for robotic manipulation.

[2] Act3d: Infinite resolution action detection transformer for robotic manipulation.

[3] Rvt: Robotic view transformer for 3d object manipulation.

[4] 3D Diffuser Actor: Policy Diffusion with 3D Scene Representations.

[5] Unleashing large-scale video generative pre-training for visual robot manipulation.

[6] Zero-shot robotic manipulation with pretrained image-editing diffusion models.

[7] Learning Universal Policies via Text-Guided Video Generation.

[8] https://github.com/EDiRobotics/GR1-Training.

[9] Reasoning with Latent Diffusion in Offline Reinforcement Learning, ICLR 2024.

[10] Diffusion-based Neural Network Weights Generation, ICML 2024.

[11] Aligning Optimization Trajectories with Diffusion Models for Constrained Design Generation, NeurIPS 2023.

[12] https://openai.com/index/sora/

I thank the authors for their clarifications and I endorse their effort on running several baselines, ablations and benchmarks. I feel the paper is much more complete now.

My main concerns were regarding the evaluation and the motivation. On the evaluation aspect, I asked for a more direct comparison on RLBench and more ablations on the relationship with GR-1, preferably on CALVIN, which is a benchmark the authors hadn't used in the first version of the paper. The rebuttal did the following:

• RLBench: direct comparison with RVT and previous SOTA. The results are obtained using 2D images only as input and no depth, please correct me if I'm wrong. This could be one of the best 2D models deployed on RLBench, which is dominated by 3D approaches. This suggests that the technical contributions of the paper are strong. However, I strongly recommend that 3DDA is included in the comparison, even if the proposed approach underperforms it. The paper will be very insightful if it shows where 2D policies have reached and where 3D policies stand. One suggestion would be to split the table into 2D and 3D. Another suggestion/question is whether the proposed method can incorporate RGB-D. Maybe the first stage can predict the future depth frames as well. It would be interesting to test that.

• CALVIN: the newly added benchmark allows for the method to shine. Since GR-1's strong results, I expected that future prediction is important for this benchmark. An even broader question is whether video prediction is the best representation for forecasting in robotics, but I do recognize its strengths and it's impossible to answer this in one paper.

• The ablations now are much clearer.

Overall, the experimental updates significantly upgrade the paper's quality. Please consider including 3DDA in the comparison, as a reader working on the field I would find the clarity and candor very insightful.

Regarding the motivation, I'm not completely satisfied, but I do recognize the rebuttal's effort in providing more context and discussion. I think the introduction could be punchier on the specific contributions of the paper that made this idea work, rather than promoting the idea in general, because it's not novel on its own. I feel that this paper has more technical contributions that theoretical and it's fair to be portrayed as such. I'm also not a fan of connecting the two-stage approach to dual process theory, it feels a bit forced.

I disagree with 5fDE calling the contributions of this paper "engineering", but I believe they convey what I wrote above, that this work has solid technical contributions, rather than theoretical. I do agree with 5fDE that NeurIPS papers should have deep insights and inspire future research. I believe that this papers provides some insights, even if the scope is limited within the field of imitation learning for tabletop manipulation. Since i) NeurIPS is not mainly a robotics conference and ii) the paper stands more on the technical/experimental side rather than theoretical, I feel that these contributions are good enough for acceptance, but I strongly disagree with yiak proposing this paper for award and I honestly advise the AC not to recommend this paper for an award.

Therefore, I'm increasing my score to borderline accept. However, I would appreciate complete candor in the final version which should include both CALVIN and RLBench (together with the other sets used), as well the results of all related approaches, even if they outperform the proposed approach - especially since this happens on RLBench, which is 3D-dominated, as I mentioned earlier. Thank to the authors for the discussion.

Dear Reviewer,

Thanks for your constructive comments. Below, we would like to address all the weaknesses and questions in details.

Q1: Ablate the choices of adaptation, such as an output projector.

A1: In our approach, we've adopted layer-wise adapters because current literature supports that in-model parameter adjustment methods are the most parameter-efficient ways [2, 3] to fine-tune models. While it is also feasible to transfer a pre-trained model using non-intrusive techniques—such as prefixing tokens to the input or appending a projector to the output— these methods tend to require a larger volume of fine-tuning data and parameters to match the performance of in-model adapters. Furthermore, these non-intrusive methods primarily alter the model's behavior by adjusting the input/output levels, rather than directly influencing the model's core competency in aggregating and inferring knowledge across long-range cross-tokens. The latter capability is considered a fundamental enhancement of the Transformer architecture[6] over traditional RNN and LSTM models. To see the benefit of our layerwise cross-attention adapter, we also include a new set of experiments in Table (2c), where we have removed the layer-wise adapter and instead appended a learnable token to the observation tokens (similar to the cls token in ViT). To further validate the effectiveness of our layer-wise adapter, in the second stage, we pooled the output tokens and then passed them through an MLP to project them into a 2x1152 tensor. This tensor was then fed into the action head. The results are shown in the table below. Our approach outperforms pooling MLP projector and concatenating tokens by 8.0% and 4.0% on RLBench-10 benchmark, respectively.

Furthermore, since the Open-Sora model we've adopted wasn't pre-trained with explicit latent planning as described in [1], we don't presume that its hidden representations can be utilized in the same manner. We've observed that the diffusion planning in the latent space, as presented in [1], is computationally expensive and still fundamentally relies on the model's hidden representations. Given the Transformer architecture's demonstrated strong reasoning capabilities, we respectively don't agree that there's a compelling need to engage in explicit planning based on the hidden representations, as done in [1].

Q2: Comparison with some representation learning baselines (e.g., R3M) and explain what useful information is extracted in the representation and why cross-attention with these features is sufficient.

A2: Thank you for your constructive feedback. We've taken your suggestions on board and have now included comparative experiments with R3M in the following table. Before delving into the results, we'd like to distinguish between our approach and the latent representation learning, e.g., R3M. As previously mentioned in our last response, we've incorporated a cross-attention adapter to directly modulate the Transformer model's attention mechanism. This adjustment aims to enhance the effectiveness of the modified cross-attention by training the Open-Sora layers and aggregating more valuable information via the fine-tuned adapter, thereby reducing loss. In contrast to our method, R3M and similar approaches apply direct supervision to the latent representation using optimization techniques such as contrastive learning. They impose manual regularization on the learned representations, which may give the appearance of greater 'interpretability'. We employ the reproduction of R3M from GR-1 for comparison on the CALVIN benchmark, which uses R3M to encode observation images and leverages a GPT-style transformer to output actions. The comparison results between ours and R3M are shown as follows.

Regarding the pursuit of interpretability, we believe its value lies in the potential inductive bias it provides for purely data-driven methods, potentially enhancing performance. Ultimately, however, performance is the key metric that determines the usefulness of such 'interpretability.' In this regard, performance is the end goal, not interpretability itself. As a playful aside, the most powerful and enigmatically uninterpretable machine we know is, arguably, the human brain :).

As for the discussion on the sufficiency of the cross-attention adapter we've adopted for distilling knowledge acquired in Stage 1, a few points must be clarified. First, using adapters to adjust how models behave is recognized in the literature [2, 3, 4, 5], as one of the most efficient ways. Notably, in robotics, studies like [2, 5] are using attention-based techniques to refine knowledge from transformer layers. We stand by our choice, confident that it is sufficient to achieve significant improvements over strong baselines, as evidenced by our experimental results. We've also tested out other non-cross-attention methods, such as directly using a projector, which appears to be less effective than ours. Our claims are based on our experiments and we're not claiming to have the ultimate method for squeezing every last bit of knowledge out of the Open-Sora model. And honestly, we're also skeptical that there will be a method, including those mentioned by the reviewer, that can claim their methods are already sufficient for their objectives.

In conclusion, we believe that the interpretability of the representation and the absolute sufficiency of the cross-attention adapter should not be barriers to the approval of our article. Instead, these aspects should stimulate further research and discussion within the field.

Q3: How to select the Open-Sora layers.

A3: Sorry for the confusion. Open-Sora originally had 28 layers, but we only selected the first 12 layers, totaling 315M parameters. Our model has two deployment modes: one that only trains the layer-wise adapter and diffusion head, which has 80M parameters, and another that trains everything, totaling 395M parameters. This is a point that was not clearly explained in the text. We will improve it in the next revised version.

References:

[1] Large-Scale Actionless Video Pre-Training via Discrete Diffusion for Efficient Policy Learning, 2024.

[2] Vision-Language Foundation Models as Effective Robot Imitators, 2023.

[3] Flamingo: a Visual Language Model for Few-Shot Learning, 2022.

[4] Unsupervised Cross-Modal Alignment for Multi-Person 3D Pose Estimation, 2020.

[5] Octo: An Open-Source Generalist Robot Policy, 2023.

[6] Attention is all you need, 2017.

Firstly, I would like to remind you that you should post your rebuttal using the 'rebuttal' button instead of the 'official comment'. I have to reply to your response by commenting after my initial review to ensure the visibility of other reviewers.

Overall, after reading your response and other reviews, I still have the following concerns:

• While I appreciate your technical contribution and the proposed two-stage pipeline, which performs well in RLBench and other benchmarks (including the experimental results added during the rebuttal period), I have some questions about the performance in RLBench. The success rates of the tasks 'screw bulb', 'place wine', 'sort shape', and 'push buttons' shown in the global PDF differ significantly from those in your original paper. What caused these discrepancies? Did you revise your method?
• This paper's main contribution lies in engineering, such as the incorporation of various modules like open-sora and diffusion policy. Both the modules used in this paper and the two-stage training design are not new. [1] and [2] use a similar two-stage training. For a paper to be accepted at NeurIPS, there should be some novel methods or insights to inspire future research. This is the main reason why I do not give a high rating.

[1] Large-Scale Actionless Video Pre-Training via Discrete Diffusion for Efficient Policy Learning, Arxiv 2024.

[2] Unleashing large-scale video generative pre-training for visual robot manipulation, ICLR 2024

Thank you for your response and further raised questions. The responses to the comments are as follows. We have made rebuttals visible to all reviewers.

1. The experiments in Table 1 of the attachment expand our method to multi-view cameras to make a fair comparison with RVT and Act3D. Note that, our method and RVT differ in many aspects, including the model architecture, inputs, and outputs, please refer to the #A1 response to the Reviewer yiak. Although an apple-to-apple comparison is difficult, our method can be easily extended to handle multi-view sensory input, To achieve this, we only need to replace our overhead camera input with multiview cameras with the same resolution 128 x 128 (including front camera, left shoulder camera, right shoulder camera, and wrist camera ) as the input of STDiT in the 2nd stage. We found that adding additional multi-view camera inputs significantly improves performance on tasks involving small objects often occluded by the robotic arm, such as buttons, blocks, and bulbs, or tasks requiring placing objects in specific locations, such as racks. This could explain why our method shows noticeable improvements in these tasks, such as 'screw bulb', 'place wine', 'sort shape', and 'push buttons'.

2. The reviewer has noted that the two-stage training approach is a common practice for developing representations in action prediction tasks. However, we would like to clarify that we view the two-stage training as a training framework rather than a motivation to conclude either others or our method. In this regard, we respectfully disagree with the reviewer's oversimplification of our motivations and then equating it with the two-stage training pipeline.

• We must point out that our motivation is fundamentally different from GR-1 [1]: As stated in the GR-1 paper, its goal is to verify that video datasets are beneficial to robot action learning, so their motivation guided their research work; whereas we believe that learning in System 2 is beneficial for modulate the precise execution of System 1, and combining the dual-process theory, we propose that slow thinking (denoising diffusion) in Stage-1, which is carefully design to enable the fast execution (one-pass inference) in Stage-2 of our method, thus designing the model. It is not difficult to find that our training goals and the model choices for each phase are closely linked to the motivation.
• As for VPDD [2], its main contribution lies in the first stage, where a mask-and-replace diffusion strategy is proposed to predict actionless videos, while the second stage still predicts both actions and videos simultaneously, which can often slow down real-time inference during deployment. Our core motivation, however, is focused on slow thinking and fast execution, which perfectly aligns with the intuitive thinking of robotic cognition and behavior. Additionally, our method leverages the Open-Sora STDiT architecture and utilizes a layer-wise adapter to integrate the intrinsic dynamic knowledge from videos. This gives our model strong scalability and extensibility, allowing it to be effectively trained on large-scale robotic trajectory datasets like OXE or human demonstrations like Ego4D, and easily extended to multi-view inputs. VPDD has not been published in an official conference or journal, but its method has some valuable insights. We will discuss the differences between our approach, GR-1, and VPDD in the revised version.

References:

[1] Unleashing large-scale video generative pre-training for visual robot manipulation, ICLR 2024

[2] Large-Scale Actionless Video Pre-Training via Discrete Diffusion for Efficient Policy Learning, Arxiv 2024.

Dear Reviewer,

Thanks for your constructive comments. Below, we would like to address all the weaknesses and questions in details.

Q1: In-depth analyses for the effectiveness of the adopted two-stage strategy.

A1: We greatly appreciate the thoughtful feedback you've provided. In our initial manuscript, we carried out an array of experiments designed to underscore the efficacy of our two-stage strategy. Upon review, we've come to understand that the organization of our experimental findings did not fully convey the comprehensive benefits of our approach. Furthermore, it has come to our attention that there are supplementary experiments that could offer more explicit validation of our methods' effectiveness. In the following content, we are committed to incorporating the existing and newly added findings to highlight that the strengths of our adopted two-stage strategy.

a. Is two-stage training better than single-stage in the final result? Based on the reviewer's suggestion, we also conducted experiments in an offline evaluation setting. The experimental setups for 1-stage and 2-stage are as follows: for 1-stage, action prediction is trained on OXE for 300K steps. For the 2-stage, video prediction is trained on OXE for 100K steps in the first stage, and action prediction is trained on OXE for 200K steps in the second stage. Finally, both are evaluated offline on the Bridge dataset. The results are shown as follows. Additional experiments showed similar performance to the original experiments conducted on RLBench-10 (the leftmost column.), proving that 2-stage training is more effective than 1-stage training.

b. Can pretrained with non-robotic data, e.g. Ego4D, at the first stage still possess a similar advantage against the single-stage alternative? We replaced the training dataset in the first stage with EGO4D (a first-person video dataset of human-object interactions), which also showed performance improvement compared to 1-stage training. The results are shown in the table below. This demonstrates that in 2-stage training, what is needed is to learn the intrinsic representation of dynamics from the video prediction training in the first stage. The training dataset in the first stage is dynamic but non-robotic, and it also benefits the performance of action prediction.

c. How does our method compare to other two-stage methods, such as GR-1? GR-1 also employs a two-stage training approach and achieves superior performance on CALVIN. In the first stage, it uses Ego4D for video generation pre-training. In the second stage, it generates actions and the next frame using the next token generation method. For a fair comparison, we replaced our first-stage dataset with Ego4D and compared our model with GR-1 on the CALVIN benchmark. Experiments show that our method outperforms some two-stage methods, which are shown as follows.

Q2: Comparison with RVT.

A2: Thanks for your suggestion. Directly comparing our method with RVT is not entirely fair. Our proposed method has the following key differences from RVT and other state-of-the-art approaches: 1. RVT utilizes four RGB-D cameras, including a front camera, left shoulder camera, right shoulder camera, and wrist camera. In contrast, our method only uses an overhead camera and a wrist camera, and it does not incorporate depth information. 2. RVT predicts the position and rotation angles of keyframes, then uses a sampled motion planner to move the robotic arm to the corresponding position and rotate to the corresponding angle. Our method, on the other hand, outputs a 6-dimensional vector end-to-end, which is directly executed by the end-effector controller. An absolutely fair comparison may not be entirely feasible, but we can adopt some relatively fair comparisons. Specifically, we have standardized the first difference mentioned above by also using multiview input in our proposed method. Fortunately, due to the flexibility of our model, we only need to replace our overhead camera input with multiview cameras with the same resolution 128 x 128 (including front camera, left shoulder camera, right shoulder camera and wrist camera ) as the input of STDiT in the 2nd stage. The results are shown as follows.

Q3: The meaning of "intrinsic dynamics" and "inverse dynamics".

A3: Intrinsic dynamics encompass the world dynamics that a model implicitly learns from video data, which includes both forward and inverse dynamics. The forward dynamics, our primary focus in the first stage, describe how observations evolve from one moment to the next, essentially a temporal sequence prediction task. In our specific application, this refers to the diffusion-based process of video generation. Inverse dynamics, on the other hand, pertain to the inference of an action given consecutive observations (e.g. images) before and after the action is executed. In our context, this involves action prediction process with the aid of the video generator at the 2nd stage. Since we are validating that the video generation model can effectively infer actions, it indicates that the video model can provide a good inductive bias for inverse dynamics.

To be more concise, our approach to efficiently infer action does not involve explicitly predicting frames. Instead, we predict actions based on the hidden representations within the video diffusion generator. To this end, our method relies on intrinsic inverse dynamics, utilizing the model's latent knowledge to forecast actions, rather than relying on explicit inverse dynamics that predict actions from generated images.

Q4: Discuss more about the limitations.

A4: Thank you for your suggestions. We will expand the limitations as follows:

1. Our current model operates solely on 2D vision and does not possess 3D perception capabilities. This limitation affects its performance on tasks that demand accurate 3D spatial understanding. Our future work will prioritize the integration of 3D perception into the model.
2. Our model's capacity to comprehend complex human instructions is limited. It currently lacks the sophistication of a T5 language encoder. We are considering state-of-the-art Large Language Models (LLMs), such as LLama-3.1, as a promising avenue for future exploration to enhance these capabilities.
3. The model's perception is not fine-grained. It processes the entire image as a single input for action prediction. We believe that incorporating fine-grained auxiliary inputs, such as object-bounding boxes or masks, could significantly enhance the model's performance. We are exploring ways to integrate such detailed inputs to refine the model's perceptual abilities.

Dear Reviewer,

I hope this message finds you well. I wanted to follow up regarding the response I sent earlier addressing your insightful comments on our paper, particularly the discussion around the "intrinsic" and "implicit" dynamics.

We have carefully considered your feedback and have started implementing the suggested changes, including the revision of terminology to ensure clarity and consistency throughout the manuscript. We believe these revisions will significantly improve the quality and precision of our work.

If you have any further comments or require additional clarification, please don't hesitate to let us know. We are committed to making the necessary adjustments and greatly appreciate your time and expertise.

Thank you again for your valuable feedback, and we look forward to your thoughts on the revised manuscript.

Dear Reviewer,

Thanks for your constructive comments. Below, we would like to address all the weaknesses and questions in details.

Q1: Similar with diffusion-diffuser.

A1: For trajectory diffusion for planning and inverse dynamics for action prediction, this should be the most intuitive research path for methods that infer actions from videos. Decision-diffuser uses classifier-free guidance with low-temperature sampling to generate a sequence of future states. It then uses inverse dynamics to extract and execute the action that leads to the immediate future state. We emphasize that we have the following distinct contributions. a. we exploit "intrinsic" dynamics from the diffusion model, i.e., we do not explicitly infer the next state from the previous state; instead, it is done implicitly. Decision-diffuser is explicit, which is not suitable for real-time robotic manipulation tasks. b. We emphasize our design specifically for utilizing large amounts of robotic video data, such as the two-stage training method and the layer-wise adapter.

Q2: Discussion about how to extend our method to sub-optimal datasets.

A2: To verify the effectiveness of our method on sub-optimal trajectories, we replaced the first stage training dataset with Ego4D[1]. Ego4D is the world's largest egocentric (first person) video dataset, with 3,600 hrs (and counting) of densely narrated video. Ego4D has a smaller domain gap compared to internet data for robotic manipulation tasks and is easier to collect than high-quality expert trajectory datasets. With the above features, Ego4D can easily serve as a non-optimal dataset. The results are shown in the flowing table. When the domain gap of the pre-training dataset is close to that of the robotic manipulation task, the performance of our second-stage training also improves.

Q3: Whether fine-tune video tokenizer or not.

A3: In our implementation, we have chosen not to fine-tune the video tokenizer in order to minimize training costs. The question of whether to fine-tune the tokenizer of a pre-trained model has been a topic of interest in the field of parameter-efficient fine-tuning. These researches[3, 5, 7] have indicated that employing in-model adapters is often a more parameter-efficient approach and is less likely to compromise the original capabilities of the pre-trained model. While adjusting the tokenizer can be beneficial when dealing with extensive datasets, it may also lead to overfitting and could easily alter the function of the pre-trained model in cases that the finetuning data are too different from the pretraining data. Consequently, many methods that leverage pre-trained models, such as MAE[2] in GR-1[3], CLIP[4] in Octo[5], and DinoV2[6] in OpenVLA[7], opt not to modify the pre-trained tokenizer.

Q4: PerAct's results are lower than those of the original paper.

A4: We checked the results reported in our paper and compared them with those in the PerAct paper and found that they are consistent. Additionally, we found that the PerAct results reported in RVT are higher than those in the original PerAct paper, which may have caused some confusion. The RVT paper states that they reevaluated the PerAct model weights in their experimental environment, which resulted in higher performance. The following table shows the results from the original PerAct paper and the reproduced results by RVT.

Q5: Compared with RVT.

A5: We have identified several challenges in making a direct comparison with our approach. Firstly, RVT leverages four RGB-D cameras, significantly enhancing environmental perception capabilities. Our method, however, adopts a more cost-effective and widely applicable setup, typically utilizing just an overhead camera, sometimes complemented by a wrist camera. Secondly, RVT focuses solely on predicting target positions and rotation angles, delegating motion planning to a separate system, whereas our method offers end-to-end control. Both approaches have their merits, with RVT highlighting the value of advanced sensing and our method emphasizing embodied control. As shown in the following table, without the need for a time-consuming rendering process and explicit motion planning, our method is much more efficient during inference compared to RVT. Despite the differences, we see no inherent conflict between our method and RVT. In fact, our approach is adaptable to various sensor inputs, which has the potential to enhance performance. To illustrate, we have standardized the first issue by incorporating multiview inputs into our method. Specifically, we have replaced the overhead camera input with a set of multiview cameras—each with a resolution of 128x128, including front, left shoulder, right shoulder, and wrist cameras—as used by STDiT in the second stage. The results are shown as follows.

References:

[1] Ego4d: Around the world in 3,000 hours of egocentric video.

[2] Masked Autoencoders Are Scalable Vision Learners.

[3] Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation.

[4] Learning Transferable Visual Models From Natural Language Supervision.

[5] Octo: An Open-Source Generalist Robot Policy.

[6] DINOv2: Learning Robust Visual Features without Supervision.

[7] OpenVLA: An Open-Source Vision-Language-Action Model.

Thank you for your additional questions and for the opportunity to further clarify our work.

1. To clarify, at Stage 2 of our method, we do not directly predict future frames based on past frames when predicting future actions. Specifically, through the first stage of training, our model acquires the ability to predict future frames. In the second stage, we adapt the model to learn to directly predict the actions given the historical observation and task instruction, without the need for iteratively decoding frames explicitly. This allows the model to effectively acquire fine-grained real-world dynamics via the multi-step video denoised diffusion at the first stage, and efficiently leverage such learned dynamics to predict actions by adapting the model to function as a one-pass inverse dynamics model. In other words, the image observations are encoded into latent representations which may include information about future frames due to the pretraining objective. However, explicit future predictions, either latent or pixel, are not explicitly generated. An action is computed based on these inputs.

2. a) We did not directly use Ego4D for robotic action prediction. Instead, we employed them in the first stage of video prediction pre-training. This allowed our model to acquire the intrinsic dynamic knowledge embedded in the videos. Similar to ours, methods such as [R3M, GR-1] have also utilized Ego4D to acquire its intrinsic dynamics. Therefore, using Ego4D for learning prior knowledge of robot control in common and rationale in the field. b) We used the forecasting Hands and Object (FHO) annotations from the Ego4D dataset. FHO captures how the camera-wearer changes the state of an object by using or manipulating it. For a single video, it might include two narration processes: narration_pass_1 and narration_pass_2. We segmented the video data based on these two narration passes and filtered out clips longer than 300 seconds and shorter than 30 seconds. In the end, we obtained approximately 283k and 340k clips, totaling around 623k clips. OXE dataset contains about 800k robot trajectories. Regarding the web data, we only loaded the Open-Sora weights and did not perform any training on the web data itself. Due to our iterative training technique, with a batch size of 384 and 100k training iterations in the first stage, we believe the model has been adequately trained on both the OXE and Ego4D datasets. We will expose all of the data processing details and training details in our revision in case of interest.

3. Since our method is based on the Open-Sora STDiT architecture, instead of changing the architecture, we treat multiview images as tokens and input them into the transformer. Following [RVT], tokens belonging to the same image are processed via the first four transformer layers for each image. Finally, the processed image tokens are jointly processed using the last eight transformer layers. This attention masking approach can automatically maintain multi-view consistency which is also employed by RVT. In our original paper, our model takes in a single-view image of size 256×256 with a patch size of 16, resulting in a total of 256 tokens for the image. To extend the model to handle four images, we resize each image to have the size 128×128, also resulting in 256 tokens in total for images. Thus, this change does not slow down the training or inference time.

We hope this provides clarity on the concerns you raised. All of the mentioned details will be provided in our revised manuscript to ensure that our work is transparent and robust, and we appreciate your continued engagement and feedback.

References:

[R3M] R3M: A Universal Visual Representation for Robot Manipulation.

[GR-1] Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation.

[RVT] Robotic view transformer for 3d object manipulation.

Dear Reviewer,

I hope this message finds you well. I wanted to kindly follow up on the detailed response we provided to your recent comments. We greatly appreciate the time and effort you have already invested in reviewing our paper, and your insights have been instrumental in refining our work.

We believe that our latest response has addressed your concerns comprehensively. With the recent acknowledgment from Reviewer yiak, Yu5P and 5fDE, we are confident that our paper’s contributions are both technically robust and distinct within the field. While we understand that you may still have reservations, we hope you agree that our revised submission meets the high standards expected by NeurIPS.

In light of these revisions and the feedback from other reviewers, we believe our submission merits a higher score than the original version, and we would be grateful if this could be reflected in your final assessment.

If there are any additional questions or concerns, we are more than willing to address them promptly. We are committed to ensuring that every aspect of our work is fully understood and vetted.

Please let us know if there’s anything more we can do to assist in this process. Your continued feedback is invaluable, and we are eager to provide any further clarification you may need.

Thank you once again for your support and consideration.

Dear Reviewer,

Thanks for your constructive comments. Below, we would like to address all the weaknesses and questions in details.

Q1: Compare with more related SOTA methods.

A1: Thank you for your insightful suggestions. In response to the reviewers' feedback, we have expanded our comparison with more baseline methods and evaluated in SIMPLER environment in the revised manuscript, demonstrating that our method continues to excel under fair conditions. A particularly pertinent comparison is with GR-1, which employs next-token prediction autoregression pretraining on Ego4D data, achieving state-of-the-art results on the CALVIN benchmark. CALVIN is a challenging benchmark focusing on learning language-conditioned policy for long-horizon robot manipulation. We perform experiments on the ABC->D split of data. The comparison results on CALVIN are shown as follows. The average length in the last column, computed by averaging the number of completed tasks in a row of 5 in all the evaluated sequences, shows the long-horizon capability in a comprehensive way. Since our method can utilize the diffusion model's perception of the intrinsic dynamics of videos and is pre-trained on a large amount of expert-annotated trajectory, our model significantly improves the average length of successful tasks by 0.61.

And, we also conducted validation in the SIMPLER environment. SIMPLER is a simulation environment that includes two common setups: Google Robot and WidowX. We evaluated our model on 25 episodes per task, and the results are shown as follows.

When it comes to RVT, we've identified several challenges in making a direct comparison with our approach: Firstly, RVT leverages four RGB-D cameras, significantly enhancing environmental perception capabilities. Our method, however, adopts a more cost-effective and widely applicable setup, typically utilizing just an overhead camera and a wrist camera. Secondly, RVT focuses solely on predicting target positions and rotation angles, delegating motion planning to a separate system, whereas our method offers end-to-end control. Both approaches have their merits, with RVT highlighting the value of advanced sensing and our method emphasizing embodied control. As shown in the following table, without the need for a time-consuming rendering process and explicit motion planning, our method is much more efficient during inference than RVT. Despite the differences, we see no inherent conflict between our method and RVT. In fact, our approach is adaptable to various sensor inputs, which has the potential to enhance performance. To illustrate, we've standardized the first issue by incorporating multiview inputs into our method. Specifically, we've replaced the overhead camera input with a set of multiview cameras--each with a resolution of 128x128, including front, left shoulder, right shoulder, and wrist cameras--as used by STDiT in the second stage.

It's important to mention that, given our current implementation and the video DiT backbone we've adopted, which were not originally optimized for diverse camera inputs, we anticipate further performance improvements as these methodologies evolve. Our method, as a pioneering effort, holds significant value for the field.

Q2: More Generated Video Results.

A2: To avoid violating the double-blind policy, we have provided more complex and diverse frame predictions in the attached PDF. An anonymous link to showcase our results will be provided after the rebuttal process. In particular, to effectively convey the dynamics of predicted frames, the new results are created with a sampling frequency reduced to one-third of what was originally presented in the manuscript.

Q3: Scalability.

A3： As the reviewers have noted, we have indeed observed a slight improvement with the pretrained Open-Sora model. We suspect that this could be attributed to two main factors: First, there may be a domain gap between the internet data utilized by Open-Sora and the robotic data we are working with. Second, to expedite the training process, we opted to load only the 12 layers from the pretrained Open-Sora, rather than the complete set of parameters. To validate these hypotheses, we conducted an experiment where we replaced our initial training data with Ego4D and fully finetune the backbone, mirroring the approach taken by GR-1. This dataset is rich in first-person perspective videos depicting hand-object interactions. We applied the same preprocessing techniques as documented in [GR-1] and [R3M]. The outcomes of our method are as follows.

When the domain gap between the pre-training dataset and the expert-annotated robotic trajectory dataset narrows, the performance will also become closer, even if it is non-robot domain data, which supports our hypothesis. In conclusion, our method can have the similar scalability to non-robot domain data to existing methods.

Q4: Complete tasks.

A4: We apologize for the oversight regarding the results of several RLBench tasks that demand high-precision control, which were missing from Tab. 1 in the original manuscript. In our experiments, we noted that the baseline methods did not perform as strongly on these tasks as they did on others. Therefore, in our previous submission, we chose to focus on the comparisons that are more competitive to save space. After viewing the reviewers' feedback, we recognize that including the results in Tab. 1 would be less confusing for readers. Accordingly, we have revised our manuscript and incorporated the missing results as suggested.

Q5: Accuracy metric.

A5: Following Octo, we use continuous action space. XYZ accuracy denotes did we predict the XYZ delta within 0.5 radians and 50% norm when moving. Euler angle accuracy denotes did we predict the rotations delta within 0.5 radians when moving. These two metrics can be found in Octo's codebase.

Q6: Computational efficiency.

A6: During inference, our method consumes 18 GB on a single NVIDIA 4090 GPU. We used the action chunk method with a chunk size of 5, achieving a speed of 18.3 fps.

References:

[GR-1]Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation.

[R3M] R3M: A Universal Visual Representation for Robot Manipulation.

My response to the rebuttal was previously written here, but I have moved my (updated) response as a reply to the individual rebuttal now that individual rebuttals are visible to all reviewers.