GEVRM: Goal-Expressive Video Generation Model For Robust Visual Manipulation

Q1: This work employs a diffusion policy for action prediction and interacts with the environment in a close-loop manner, which may help learn multi-modal action distribution but can be time-consuming and often not desirable in real-time deployment.

A1: To effectively alleviate the problem of low computational efficiency of diffusion models, we propose a goal-guided diffusion policy with multi-step action prediction (4 steps) and allow open-loop control. This setting enables the model to effectively improve computational efficiency while having little effect on the task success rate. For more analysis and experiments, see General Response 3.

Q2: The effectiveness of techniques like random masking compared to simply masking out the last few frames for decision-making scenarios, should be supported by ablations.

A2: We conducted comparative experiments on the random mask and the mask-only last few frames method, as shown in the following table. The results show that the random mask mechanism significantly improves the overall algorithm.

Q3: It would be great if the authors could thoroughly discuss the limitations of the proposed method.

A3: Robust language-based visuomotor control is an important but underexplored field. Our work is just a small step in this direction, and there are still many limitations. In the control tasks of long-term external disturbances, the success rate of the VLA model we proposed still has a lot of room for improvement. How to further deepen the VLA model's understanding of the laws of the physical world and infer stable and reliable (explicit or implicit) behavioral goals is a promising research direction. Moreover, how to integrate these physical knowledge and behavioral goals into the real-time control of the robot in a more efficient and low-cost solution to achieve robust and safe autonomous decision-making is also an important research topic.

Moreover, in the GEVRM model we proposed now, the robot behavior planner and the goal guidance policy are trained separately, which means that it is difficult for the behavior planner model to know the function of the goal guidance policy. Intuitively, by letting the behavior planner know the function of the goal guidance policy, the robustness of the model can be further improved. Therefore, solving this limitation is also an important future work.

Q4: I wonder if this contrastive learning process will learn to focus on only trivial consistent features in the background, which are likely to be similar across all timesteps, and ignore some differences (e.g. the position change of objects or the robot arm) between the current and goal states that are essential to help the model make correct decisions.

A4: Previous work [1] has demonstrated that contrastive learning can be used to effectively obtain representations of the current and future states of a robot trajectory, so that the representation of the future state is closer to that of random states (derived from other trajectories).

Moreover, in our prototype contrastive learning, we use the Sinkhorn-Knoppal algorithm, which is a fast iterative solution to the entropy-regularized optimal transport problem that considers the Wasserstein distance between high-dimensional data distributions. Compared with the commonly used KL divergence and JS divergence, the Wasserstein distance takes into account the structural information within the probability distribution of the data and can well characterize the distance between distributions even when the "overlap area" is small.

Therefore, some differences between the current state and the target state, such as the position change of an object or a robotic arm, can also be well described by the Wasserstein distance. The visual analysis in Figure 6 of the manuscript shows that after using this method, the state representations between different tasks have better cluster centers and classification boundaries, which can effectively distinguish them.

[1] Eysenbach, B., Zhang, T., Levine, S., & Salakhutdinov, R. R. (2022). Contrastive learning as goal-conditioned reinforcement learning. Advances in Neural Information Processing Systems, 35, 35603-35620.

I appreciate the author's efforts in addressing my concerns and providing additional results. I believe the discussion of inference efficiency and task generalization evaluation will further improve the thoroughness of this work. It would be great if the authors could incorporate these into the final version. I will maintain my positive score.

Q1: The connection to IMC. The proposed approach seems to use the current robot state and the desired output to generate the control signal. I think the proposed structure is just a goal conditioned policy, and I don’t see the strong connection to the IMC.

A1: In classical control theory, the IMC framework is a widely recognized control strategy that uses the internal model of the system to predict future behavior and adjust the control action accordingly, making it highly resistant to interference. The insight of our work is how to effectively implement the idea of IMC principles in the modern learning-based VLA framework to achieve robust control. Our work mainly considers how to flexibly and organically leverage powerful deep learning components to realize the learning-based IMC control principle, rather than using it in a rigid manner.

Therefore, we mainly draw on this broad control idea to design specific learning-based components to resist disturbances during deployment. The robot behavior planner proposed in our work is implemented by a diffusion video generator. By performing prototype contrastive learning on the goal image state and the current image state, the state alignment is achieved and the robust action prediction of the control policy is guided. For more discussion, see General Response 1.

Q2: Image encoder. The experiment is missing ablation studies on the encoder. Would be interested in how an off-the shelf encoder like DinoV2 performs.

A2: The state encoder we use is of the ResNet type, which has been widely used in previous related work (DP, SuSIE, etc.). Recent work (OpenVLA, etc.) uses off-the-shelf DinoV2 to obtain image features. Our core idea is to enhance the state encoder with prototype contrastive learning, rather than focusing on which encoder to use. Moreover, due to current GPU resources and rebuttal time constraints, a comparative experiment between the two encoders under our model is currently underway. We will update the reply as soon as the latest comparison results are available.

Dear reviewer 9iWM:

For your main concerns, such as connection with traditional IMC, comparison of image encoders, etc., we have explored the relationship with traditional IMC in depth and conducted relevant comparative experiments. We are sure that we have addressed all your questions and concerns.

The interactive discussion period is about to end, but we have not received further feedback. We hope that we can have more discussions and exchanges with you in time and convince you to consider raising the score from negative to positive. Please feel free to ask for any additional information or clarification that may be needed.

Thank you for taking the time to give insightful feedback in your busy schedule.

Dear Reviewer 9iWM:

The deadline for the rebuttal is approaching. We are eager to have you spare a little time from your busy schedule to provide valuable feedback. We are sure that we have addressed all your questions and concerns. We sincerely hope to get your further feedback and convince you to consider raising the rating from negative to positive.

Q1: The effectiveness of GEVRM on real-world robots was not validated.

A1: To verify the effectiveness of the proposed GEVRM in real-world robot manipulation tasks, we collected more than 400 expert trajectories for picking and placing cups, bowls, and tiger plush toys on a real robotic arm UR5. We trained GEVRM based on these real data and tested it in real scenarios. The results show that GEVRM can be effectively deployed to pick and place tasks of common objects in real scenarios. The specific real-world experiments of GEVRM have been supplemented in the appendix section of the manuscript “Real-World Tasks.”:

“Protocol. To examine the effectiveness of the proposed GEVRM on real-world robotic manipulation tasks, we propose a real-machine deployment protocol. We evaluate GEVRM on a robotic arm UR5 for the pick-and-place tasks of a cup, a bowl, and a tiger plush toy. Specifically, we use a camera to capture third-person images as the observation space (image width 640, height 480), and relative poses and binarized gripper states as the action space (7 dimensions). The total number of collected real-world teleoperation expert trajectories is over 400, with trajectory lengths ranging from 20 to 120 steps and a control frequency of 5Hz.

Experiments. We train and evaluate GEVRM under real-world protocols. The VAE and DiT in the behavior planner are trained for 30,000 and 12,000 iterations, respectively, while the goal-guided policy is trained for 100,000 iterations. Other hyperparameters remain the same as in the experiments in CALVIN (and Bridge). Fig. 7 shows the policy execution process of our proposed GEVRM on three types of real-world tasks, indicating that our method can be effectively deployed on real machines. In terms of task success rate (SR), we evaluated each type of task 10 times. The experimental results show that compared with the grasping and placing of cups (or bowls) with regular shapes (success rate of about 0.8), the grasping of tiger plush toys with soft materials and irregular shapes is more challenging (success rate of about 0.6). Further improving GEVRM's perception of real-world scenes and task execution accuracy is an important future work.”

Q2: Could the authors provide further details regarding the reproduction of GR-1? Furthermore, would it be possible for the authors to incorporate additional powerful methods for comparison by adjusting their input modalities to RGB only? Additionally, I would like to mention that Susie's results on Calvin ABC-D should be included in Table 2.

A2: GR-1 reproduction details: For the sake of fair experimental comparison, we only use third-person images, without first-person and proprioceptive information as model input. At the same time, when predicting future image frames, only third-person images are considered, that is, future images from the first person are not predicted. Other pre-trained models and hyperparameter configurations are consistent with the original GR-1 paper.

More baseline comparisons: The main problem studied in our work is how to make VLA produce robust actions in a perturbed environment, so the core lies in goal state planning and action execution in a perturbed environment. For the goal state planning task (Table 1 of the manuscript), we added SuSIE. The SuSIE results have been added to Table 2 of the revised manuscript. For the perturbed CALVIN ABC-D environment generalization task (Table 3 of the manuscript), we added RoboFlamingo and GR-1. Experiments show that our method achieves better performance, see General Response 2.

Q3: The authors should give more credits to the paper "Closed-Loop Visuomotor Control with Generative Expectation for Robotic Manipulation" … however, it is not mentioned in the introduction or related works sections.

A3: We added a description in the “Internal Model Control (IMC) framework” section of the Related Work section of the manuscript, summarizing the main research content of this work: “… More recently, inspired by classical closed-loop control systems, a closed-loop visuomotor control framework has been proposed that incorporates feedback mechanisms to improve adaptive robot control.”

Q4: The paper lacks a comprehensive analysis of the computational requirements. How about the inference efficiency in Calvin simulation environment?

A4: Our work proposes corresponding strategies from three aspects to effectively alleviate the problem of low computational efficiency of diffusion models:

(1) Use Rectified Flow instead of DDPM to train the behavior planner;

(2) During inference, call the behavior planner at a fixed step interval (20 steps);

(3) Goal-guided diffusion multi-step action prediction (4 steps) allows open-loop control.

All these settings enable the model to effectively improve the computational efficiency without having much effect on the task success rate. For detailed analysis and experiments, see General Response 3.

Dear reviewer siH9:

For your main concerns, such as the lack of real machine experiments and high computational overhead, we have supplemented real machine experiments, and efficiency comparison experiments, and improved the manuscript. We are confident that all your questions and concerns have been addressed.

The interactive discussion period is about to end, but we have not received further feedback. We hope that we can have more discussions and exchanges with you in time and convince you to consider raising the score from negative to positive. Please feel free to ask for any additional information or clarification that may be needed.

Thank you for taking the time to provide insightful feedback.

Q1: It would be good to see GEVRM compared with a consistent set of baselines for video generation and policy rollouts in a few more environments. It would be convincing to see GEVRM compared with HiP, UniPi, GR-1, and pure behavioral cloning, each with data augmentation.

A1: We added the SuSIE algorithm to the goal generation task; added the SuSIE algorithm to the CALVIN ABC-D generalization task; added RoboFlamingo (pure imitation learning) and GR-1 algorithms using data augmentation in the perturbed CALVIN ABC-D generalization task. Considering the extremely poor performance of HiP in the CALVIN ABC-D generalization task (the average success length is very close to 0), its implementation in the perturbed environment is not necessary. In these three types of tasks, our method GEVRM shows superior performance. For more experimental details, see General Response 2.

Q2: Could you clarify the key contribution of GEVRM? How is GEVRM different from prior work in its architecture, and what design choices contributes to the improved performance?

A2: The core contribution of our work is to propose the necessary components in the VLA framework that can effectively implement the IMC principle (mainly including goal state generation and goal alignment), thereby improving the robustness of decision actions to resist external interference during deployment.

Comparison with the structure of previous works and design choices: The network structure used in previous works: AVDC (Clip+U-Net+DDIM), HiP (GPT3.5-turbo+PVDM+ViT-B), UniPi (T5-XXL+U-Net), GR-1 (Clip+ViT+GPT-style Transformer+MSE), Roboflamingo (ViT+Flamingo+LSTM+MSE) . Different from these works, when we instantiate IMC in VLA, we consider the structure of T5-XXL+DiT+2DVAE & 3D Causal VAE + Rectified Flow. The main designs include using 2D & 3D Causal VAE to achieve efficient compression encoding of robot image sequences, random masking mechanism for effective understanding of object dynamics and time series correlation, and prototype contrastive learning for state alignment (SA) to simulate responses. The quantitative ablation comparison results of each component are shown in the table below. See General Response 1 for more discussion.

Q3: Baseline (video generation): is AVDC/GR-1/GEVRM trained with data augmentation?

A3: The experimental results in Table 1 of the manuscript are mainly used to verify the ability of various methods (AVDC/GR-1/GEVRM) to generate robot image goal states, so data augmentation methods are not used. Moreover, for experiments on policy execution, these models use data augmentation methods.

Q4: How does setting the state alignment loss λ=0 affect task success rate? Quantitative results in addition to the t-SNE plot would be convincing.

A4: The ablation experiment of state alignment (SA) loss λ=0 (Ours w/o SA) has been shown in Figure 5 of the manuscript. For a clearer comparison, the relevant numerical results are summarized in the following table. After using SA, the metric Avg. Length increases from 2.56 to 2.83, bringing a 10.5% performance improvement. This proves the effectiveness of the SA component in the proposed algorithm GEVRM.

Dear reviewer TJEW:

For your main concerns, such as inconsistent evaluation and unclear main contributions, we have improved the relevant experiments and contribution descriptions. We are confident that all your questions and concerns have been addressed.

The interactive discussion period is about to end, but we have not received further feedback. We hope that we can have more discussions and exchanges with you in time and convince you to consider raising the score from negative to positive. Please feel free to ask for any additional information or clarification that may be needed.

Thank you for taking the time to give insightful feedback in your busy schedule.

Perturbed CALVIN ABC-D generalization task (Only static camera is used)

3. Inference efficiency of the proposed method.

To effectively alleviate the problem of low computational efficiency of diffusion models, our work proposes corresponding strategies from three aspects:

(1) During the training phase of the behavior planner, we utilize the currently advanced Rectified Flow instead of DDPM to train the video generation model. Rectified Flow promotes the learning of the mapping from noise to the real image distribution by solving ordinary differential equations along the straight path between samples. This method has been proven to be a more effective training paradigm that can significantly reduce the video sampling steps, thereby improving the model training speed and reducing its inference time.

(2) During the test phase, we call the behavior planner only after a fixed interval of the lower-level goal-guided diffusion policy execution step (fixed to 20 for all experiments) instead of every step.

(3) When training the goal-guided diffusion policy, we predict multiple steps (fixed to 4 for all experiments) instead of a single-step action. This allows us to perform open-loop control during the test phase to increase computational efficiency.

The comparative analysis of the computational efficiency and task success rate of the behavior planner is shown in the following table (Noise Interference task). Due to the good properties of the adopted Rectified Flow, when the video sampling steps are reduced, the model inference time is greatly reduced, while the success rate is not significantly reduced.