Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation

$\textcolor{brown}{Weakness 1}$ Missing limitations.

We added limitations in Section 6: "The limitations mainly lie in two aspects. Firstly, we only evaluate 6 downstream tasks. A broader spectrum of high-precision and contact-rich tasks remain to be explored. Secondly, evaluating across different robots is also necessary to test Seer’s cross-embodiments capability."

$\textcolor{brown}{Weakness2}$ $\textcolor{brown}{and}$ $\textcolor{brown}{Question 1:}$ Comparison with existing sub-goal works.

As shown in table below, our approach outperforms existing sub-goal methods in the CALVIN ABC-D benchmark. The reason is that their sub-goal prediction module and action execution module are separate, and cannot be optimized end-to-end like Seer.

$\textcolor{brown}{Weakness 3:}$ Network architectures.

We have attached the detailed network architecture and descriptions in Supp. Section A.2. Seer consists of the following modules:

• image encoder: a MAE pre-trained ViT-Base
• perceiver resampler: a module to compress the image tokens
• robot state encoder: MLPs, encoding robot states
• language encoder: a CLIP ViT-B/32 text encoder
• transformer backbone: 24 layers of GPT-2 transformer blocks.
• action decoder: MLPs that decode the latent feature into 7-DOF action.
• image decoder: a ViT-based transformer followed by a linear layer.

The parameter configuration can be seen in the table below:

$\textcolor{brown}{Weakness 4:}$ Comparison in network sizes and MACs.

We calculate trainable parameters and MACs for baseline methods and ours, and compare performances across CALVIN ABC-D, LIBERO-LONG, and 4 original real-world tasks. Our method has a larger network but fewer MACs and better performance than GR-1. Our method has the same parameters and MACs as MVP and MPI but surpasses them in simulations and real-world evaluations. Against OpenVLA, our method uses fewer parameters and MACs and performs better in LIBERO-LONG and real tests, confirming its superiority.

• CALVIN ABC-D | Method | Trainable parameters | MACs | Avg. Len. | :------: | :------: | :------: | :------: | GR-1 | 46.0 M | 358.0 G | 3.06 | Ours (DROID) | 65.0 M | 287.5 G | 3.98

• LIBERO-LONG | Method | Trainable parameters | MACs | SR
  | :------: | :------: | :------: | :------: | MVP | 65.0 M | 287.5 G | 68.2 | MPI | 65.0 M | 287.5 G | 77.3 | OpenVLA (LoRA, r=32) | 97.6 M | 2.13T | 54.0 | Ours (DROID) | 65.0 M | 287.5 G | 87.7

• REAL | Method | Trainable parameters | MACs | SR / Score | :------: | :------: | :------: | :------: | MVP | 65.0 M | 287.5 G | 55.0 / 29.8 | MPI | 65.0 M | 287.5 G | 48.4 / 29.3 | OpenVLA (Full FT) | 7.54B | 2.13 T | 16.7 / 11.0 | Ours (DROID) | 65.0 M | 287.5 G | 78.4 / 39.5

$\textcolor{brown}{Question 2:}$ In Table 3, How does it work without both loss functions?

The first line omits two loss functions. It uses vanilla behavior cloning, and directly predicts actions from past and present observations.

$\textcolor{brown}{Question 3:}$ Details about GPU resources.

We used eight RTX 4090 GPUs for pretraining (40 hours for CALVIN ABC-D, 30 hours for LIBERO-LONG) and finetuning (24 and 6 hours, respectively) in simulations. In real-world experiments, the pre-training process costs 32 A100 and 4 days, and fine-tuning process costs 8 A100 and 1 day.

$\textcolor{brown}{Question 4:}$ Details about sub-goals selection.

The image from the third subsequent frame of each observation serves as the subgoal. For example, at the current time step t, the image at time step t+3 will be regarded as the subgoal.

$\textcolor{brown}{Question 5:}$ Effectiveness in high-precision and contact-rich tasks?

We added two precision, contact-rich tasks: "Push Button," requiring a robot to press a 2x2cm button and and exceed 3/4 of the scale, and "Insertion," to place a 2x9cm camera into a 2.8x9.5cm groove without collision. As seen in the table below, DROID pre-training brings obvious improvements, showing Seer's effectiveness in high-precision and contact-rich tasks. More details are in Supp. Section A.6.2.

$\textcolor{brown}{Weakness 1:}$ Relevant works.

We added discussions to Section 2 in the paper:

Another line of work focuses on visual expectations guiding actions, termed the Predictive Inverse Dynamics Model (PIDM) [1, 2, 3, 4]. Firstly, a video generation model predicts future visual sub-goals and is pre-trained on general visual datasets. Then, an inverse dynamics (goal-conditioned) low-level policy is trained on downstream robot data to predict intermediary actions.

[1]: Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulation. Arxiv 2024.

[2]: This&That: Language-Gesture Controlled Video Generation for Robot Planning. Arxiv 2024.

[3]: VideoAgent: Self-Improving Video Generation. Arxiv 2024.

[4]: IGOR: Image-GOal Representations Atomic Control Units for Foundation Models in Embodied AI. Arxiv 2024.

$\textcolor{brown}{Weakness 2:}$ Statistical significance in experiments with low number of trails.

Here we use the t-test to verify the statistical significance in LIBERO-LONG and real-world experiments with low numbers of experimental trials (20 and 15 respectively). We hypothesize that the SOTA baseline, Ours (Scratch) and Ours (DROID)'s success rates satisfy independent normal distribution $N(\mu_1, \sigma_1^2)$, $N(\mu_2, \sigma_2^2)$, $N(\mu_3, \sigma_3^2)$. For all tests, the null hypothesis is rejected when the p value is smaller than the significance level $\alpha=0.01$.

For SOTA baseline and Ours (DROID), we use the Levene test to verify whether $\sigma_1^2 = \sigma_3^2$. Then we use the t-test. Results:

In both benchmarks, the p-values are smaller than the significance level $\alpha=0.01$. Therefore, we reject the null hypothesis, and there exists statistical significance. The sample mean of Ours (DROID) is 87.7 and 71.1, higher than the counterparts (77.3 and 48.9). Therefore, we can regard $\mu_1 < \mu_3$ empirically. This proves that our method outperforms state-of-the-art baselines.

For Ours (Scratch) and Ours (DROID), we use the Levene test to verify whether $\sigma_2^2 = \sigma_3^2$. Then we use the t-test. Results:

In both benchmarks, the p-values are smaller than the significance level $\alpha=0.01$. Therefore, we reject the null hypothesis, and there exists statistical significance. The sample mean of Ours (DROID) is 87.7 and 71.1, higher than the counterparts (78.7 and 53.3) in LIBERO-LONG and real-world experiments. Therefore, we can regard $\mu_2 < \mu_3$ empirically. This proves the effectiveness of pre-training.

$\textcolor{brown}{Weakness 3:}$ Does GR-1 optimize vision and action prediction synergistically.

Seer is pre-trained and fine-tuned on robot datasets, enabling optimization vision and action prediction during both phases. In comparison, GR-1 model is pre-trained on the Ego-4d dataset and only supervised via vision loss during pre-training.

Furthermore, Seer optimizes vision and action loss in a synergistic way, where action predictions are dependent on vision predictions, optimizing in an end-to-end manner. In comparison, vision prediction in GR-1 only serves as an auxiliary loss and does not serve as a direct condition for action execution.

$\textcolor{brown}{Weakness 4:}$ Missing limitations.

We have added limitations to Section 6 in the paper:

"The limitations mainly lie in two aspects. Firstly, we only evaluate 6 downstream tasks. A broader spectrum of high-precision and contact-rich tasks remain to be explored. Secondly, evaluating across different robots is also necessary to test Seer’s cross-embodiments capability."

$\textcolor{brown}{Weakness 5:}$ Typo.

The typo has been corrected.

$\textcolor{brown}{Question 1:}$ Prediction frequency and definition of intermediate actions.

The predicted frequency of "action" is three times that of "image". The term "intermediate action" refers to the action required to transition from the current time step to the future time step. Concretely, at each time step t, the image at t + 3 will be forecasted, and actions at t, t+1, t+2 will be predicted.

$\textcolor{brown}{Question 2:}$ Numbers of observation history and predicted frames.

The history length for CALVIN is 10, and that of LIBERO and real world experiments is 7. 3 future frames are predicted for all three benchmarks.

$\textcolor{brown}{Question 3:}$ Action chunking and temporal ensembling.

During Inference, we implement with a action chunk size of 3, and we apply the temporal ensembling across 3 predictions.

$\textcolor{brown}{Weakness 1}$ $\textcolor{brown}{and}$ $\textcolor{brown}{Question 1:}$ Generalization across embodiments.

Here we add the real-world experiment, pre-trained on OXE and fine-tuned on self-collected data. We refer the subset mix-up recipe in Octo [1], remove all the subset that includes franka robots, filter subsets with odd action labels, and save the rest subsets as OXE pre-training dataset. We also add 2 challenging tasks, Press Button and Insertion, that require high precision and rich contact. The results are listed as follows:

As can be seen in the table, pre-training on the OXE dataset only provides marginal improvements in most tasks. Moreover, in some high-precision tasks, the OXE pre-trained version even brings negative effects. We attribute marginal improvements in general manipulation tasks to the diversity of objects, tasks, scenes, and language instructions in OXE. For the slight decrease in some high-precision tasks, we suspect two reasons:

• For high-precision tasks, images from the eye-on-hand camera (wrist view) are quite important, since this camera captures a lot of local and detailed information for policy. However, in most OXE subsets, only images from third-person view with diverse camera poses are provided, with most wrist view images missing.
• Admittedly, the gap in cross-embodiments and cross-action-controller types exists in our settings. High-precision tasks require a concentrated and precise action distribution. However, pre-training on OXE may offer a distant action prior due to the aforementioned physics gap.

[1]: Octo: An open-source generalist robot policy. Arxiv 2024.

$\textcolor{brown}{Weakness 2}$ $\textcolor{brown}{and}$ $\textcolor{brown}{Question 2:}$ Risk of overfitting DROID dataset and discussion about task generalization.

The DROID dataset is diverse, containing 76k episodes, 564 scenes, 52 buildings and 86 tasks or verbs. Therefore, it is difficult for a model to overfit a specific task. We believe it is our method's ability of distilling action and vision priors from DROID that brings obvious improvements in downstream tasks. Besides, in our six (4 original + 2 newly added) downstream tasks, only limited objects (e.g. bowls and cups) and tasks (e.g. simple pick and place) may be seen in DROID, while most objects (e.g. small delicate items, dustpan, brush, insertion-related objects) and challenging tasks (e.g. precisely stack, insert, press, sweep) are rare in DROID. In this way, our method could remain a satisfying result, demonstrating great objects and tasks generalization.

$\textcolor{brown}{Question 3:}$ OpenVLA's version on LIBERO.

We utilize the OpenVLA-7B checkpoint, fine-tuned via LoRA (r=32) on LIBERO-LONG, and adhere to the official instructions for launching LIBERO evaluations. The checkpoint is available at https://huggingface.co/openvla/openvla-7b-finetuned-libero-10.

$\textcolor{brown}{Question 4:}$ Typo.

Thanks for your sincere advice. The typo has been corrected.

$\textcolor{brown}{Weakness 1:}$ Discuss works using inverse dynamics models in the past.

Thanks for your sincere advice. We have discussed in Section 2 as follows:

Some studies [1, 2] integrate current and goal information to extract effective features or serve as an auxiliary objective. Subsequently, a standard behavior cloning approach is applied during downstream task implementations.

$\textcolor{brown}{Weakness 2:}$ Demonstrate the scalability in the model capacity axis.

We perform validation across four model sizes using the CALVIN ABC-D benchmark, with each model trained from scratch. The results for the Avg.Len. metric are presented in the table below. It is observed that performances improve with the increase in model size.

$\textcolor{brown}{Weakness 3:}$ Whether taking account of history?

The original model has already taken historical inputs into account. We apply varying history lengths across different benchmarks: for CALVIN, the history length is 10, and for LIBERO and real-world experiments, it is 7. The details are outlined in Table 5 in the Appendix.

$\textcolor{brown}{Question 1:}$ Pre-train with inverse dynamics only.

Here we add an additional ablation experiment in Table 3(b). During pre-training, only the inverse dynamics loss function is used, where actions are inferred from history and ground truth future information. We remain the original fine-tuning objectives. As can be seen below, being pre-trained via only inverse dynamics brings improvements, which is also validated in [1]. However, being pre-trained via both inverse dynamics and forward dynamics yields better improvements, demonstrating our method's superiority.

[1] Inverse dynamics pretraining learns good representations for multitask imitation. Neurips 2024.

$\textcolor{brown}{Question 2:}$ How to fine-tune the model?

We finetune all parameters except those of the CLIP text encoder and MAE-pretrained VIT-Base vision encoder. We do not apply LoRA , since finetuning all trainable parameters (65M) is affordable and effective.