VIP: Vision Instructed Pre-training for Robotic Manipulation

We believe the Reviewer has significant misunderstandings of this work. In the following, we address the concerns one by one using more precise explanations and sufficient experiments. The paper will be revised accordingly to avoid these misunderstandings.

Q1: Insufficient support for vision instruction

We believe we do not mention RoboFlamingo in the paper. What we claim is in the current manipulation data status (very expensive to collect), vision instruction is more efficient for disclosing task information.

We agree models like OpenVLA and PI0 can pick up the shown green box given text instruction We do not claim that text instruction cannot specify target. What we highlight is the efficiency difference between text and vision instructions. Robot manipulation data is expensive, and models like OpenVLA have used up most public data resources (Open X-Embodiment dataset covers most popular manipulation datasets). However, we find that OpenVLA overfits to specific text representation. For example, when we use the instruction ''grasp the cup", it works well. However, when we use ''pick up the cup'', the policy fails. This means even though we combine almost all public manipulation datasets, the data is still insufficient to cover common text instructions. By contrast, the shown generaliazation performance of our method is much better.

To further support our claim, we compare the performances of the models pre-trained by OpenVLA, PI0, and our VIP. As the zero-shot manipulation results of OpenVLA and PI0 are limited, we tune them with the same data as our policy for fair comparison. Specifically, We use the pre-trained feature extractors to extract feature, and employ three Transformer decoders to decode the extracted feature as actions to execute. Each pre-trained policy is test in two settings, freeze or not freeze the feature extractor during tuning. Text instructions are provided to VLA-based policies. The success rates are as follows:

According to the results, although PI0 and OpenVLA use much more pre-train data than VIP, VIP achieves better performance. This result supports our claim that vision instruction is more efficient than text instruction. The ablation study in Table 2 of the paper also supports this claim.

In summary, our method achieves superior or at least similar performance as OpenVLA and PI0 based on much less data, which favors addressing the expensive data collection cost in robot manipulation.

Q2: Vision instruction is more comprehensible

In this work, "more comprehensible" means the policy can learn to follow instructions based on less training data. As reminded by the Reviewer, we will replace "comprehensible" as "easy to follow".

We understand the concern of the Reviewer is VLA may can understand text instruction easily as it has been pre-trained using numerous image-text pairs. Actually, we have tried many VLAs during developing this work. The observation is large-scale image-text pre-train favors understanding text instruction in robot manipulation, but the required manipulation data volume required by text instruction is still significantly larger than vision instruction to learn to complete a task. This is due to the gaps between common image-text data and manipulation data. Image-text-action data is precious and far from being sufficient to train the policy to follow general text instruction following. The experiment results in the Reply to Q1 supports this observation.

Q3: How the cropped region is obtained

In test, we give a text instruction, and the text instruction is converted as vision instruction (cropped region) based on a 2D object detector.

We do not claim we should not use text instruction. It is more convenient for human to give instructions based on text than taking a photo. What we claim is vision instruction is more efficient to serve as the direct input to the policy. Therefore, we convert the text instruction from human as vision instruction and input the vision instruction to the policy. The experiment results in Table 2 of the paper indicates this design leads to much better performance than inputting text instruction to the policy.

Q4: Comparison with more pre-training methods

As suggested by the Reviewer, we conduct more comparison with the recommended pre-train methods. Besides the results in the Reply to Q1, find more comparison results in the anonymous link: Exp Result.

We have addressed the concerns of the Reviewer one by one in the following. The paper will be revised accordingly.

Q1: Experiments with more distraction.

As suggested by the Reviewer, we have added experiments with stronger distraction in both simulated and real robot environments. For simulation, we select a set of challenging tasks in RLbench, and these tasks require challenging actions like insert, open, and screw. We compare our method with previous SOTAs using the same training data and testing protocol. The results are reported as follows:

We can see that our method outperforms all previous methods.

In addition, we devise two new real robot tasks to verify the performance of our method. In both tasks, there are multiple targets randomly placed on a table. In the spoon deliver task, the robot hand needs to pick and deliver randomly placed spoons to a tray. This task is more challenging because spoons are smaller and the robot must learns to grasp the necks of the spoons in diverse poses. In the wire deliver task, the manipulation target is non-rigid and presents non-regular shapes. The policy must delivers all seen objects to achieve one successful trial. The success rates of different methods on these two new tasks are as follows:

We can find that VIRT outperforms other methods more significantly on these more complex tasks. Moreover, we present anonymous video links of our method on these two tasks here: Spoon Deliver and Wire Deliver.

Q2: Method for Moving Object Manipulation.

Yes, our method supports moving object manipulation due to its fast response speed and precise manipulation operation. We design a new task that the robot needs to pick the moving spoons on a moving belt and delivers them to a box. This task is challenging because (1) The spoons are small and in diverse poses. The robot hand must grasps the necks of spoons for successful picking up. (2) The spoons are moving, which means the policy must learns to predict the positions of spoons at the grasp moment rather than the current moment.

The success rate of VIRT arrives 0.82, while all the other test methods fail. This result attributes to two advantages of our VIRT: (1) VIRT shows very fast response speed. (2) The vision instruction helps VIRT focus on the moving spoons and better predict the future positions of these spoons.

We present an anonymous video link of our method on this new task here: Move Manipulation. To the best of our knowledge, this is the first end-to-end imitation policy that can perform moving object manipulation.

Q3: More discussion with relevant works.

We thank the Reviewer for this reminder. As suggested by the Reviewer, we will add all the references mentioned by the Reviewer and discuss the relevance of our work with them in the paper. The papers mentioned by the Reviewer are mostly about using video data cheaper than robot manipulation data (like human ego-centric videos and web videos) to conduct pre-train. The advantage of these works is the data is easy to obtain, but their effectiveness is limited due to embodiment gaps. Differently, what we study is another setting: how to use robot manipulation data to conduct pre-train, where the data is more expensive to collect but provides more direct generalization. Notably, these two settings of pre-train do not conflict with each other. We can first pre-train a policy using cheap web data and then fine-tune the policy based on the robot manipulation data based on our proposed method.

Q4: Parameter number.

As suggested by the Reviewer, we will report the parameter numbers of these methods in the paper. For the convenience of review, we also list them as follows:

According to the results, we can find that VIRT is the most lightweight model. This is mainly because VIRT adopts a very concise architecture consisting of only Transformer encoders and decoders.

We have addressed the concerns of the Reviewer one by one in the following. The paper will be revised accordingly.

Q1: Visualization details of the teaser figure.

In the teaser figure, we visualize the attention map between the action query $q_T$ at the last timestamp $T$ and image feature tokens $F$ in the last decoder. We use $q_T$ as query and $F$ as key to compute the attention map $A$. Then, we take the mean of $A$ across all channels and resize the obtained feature to the same shape of the input image. Finally, we visualize the feature and get the teaser figure. For fair comparison, the text and vision instructed policies adopt the same network except the input instructions are different. The policy parameter number is 38.6M.

Q2: The success of previous text-instructed methods.

• The mentioned pre-train methods like R3M are based on natural videos (like human hand operation videos). Although these models provide promising initialization weight for robot applications, their effectiveness is limited due to embodiment and task gaps. Differently, what we study is another setting: how to use robot manipulation data to conduct pre-train, where the data is more expensive to collect but provides more direct generalization.

• There are text-instructed VLAs based on robot manipulation data, like OpenVLA. However, we find these methods overfit to specific text instructions. For example, OpenVLA behaves well given ''grasp the cup'' but fails given ``pick up the cup''. This is because manipulation data is expensive to collect and not diverse enough to cover common text instructions, although the works like OpenVLA have used up most public data resources.

• We have added comparison experiments with previous pre-train methods. For fair comparison, we integrate their pre-trained backbone into VIRT in the same way as our VIP pre-trained backbone. If the pre-train method relies on text instruction, we provide the policy with text instruction during fine-tune and test. We report comparison results using all the simulated and real-robot tasks as follows:

According to the results, we can find that although the compared methods generally use more pre-train data than ours, our proposed method achieves better performance.

Q3: Comparison with other pre-train methods.

As suggested by the Reviewer, we have conducted more comparison between our pre-train method and other pre-train methods. Due to character limit, the results are reported in the above Table in the Reply to Q2. The results suggest that our pre-train method achieves the best performance on all tasks, which further reveals the superiority of our method.

Q4: Comparison on public benchmarks.

As suggested by the Reviewer, we compare our method with previous SOTAs using the ManiSkill-v3 and RLBench benchmarks. The training data and testing protocols are the same between VIRT and the compared methods. Refer to the reply to Q2 of Reviewer uDoz for more experiment details.

The results on ManiSkill-v3 are as follows:

For RLBench, we use the training data of PerACT and follow its testing protocol. The results on RLBench are as follows:

The results on the ManiSkill and RLBench benchmarks show VIRT achieves the best results, which further confirm the superiority of our method.

Q5: How image and point flow are integrated.

As shown in the pipeline of the paper, the images and points are encoded as tokens separately. Then, we directly concatenate them as the feature input to Transformer decoders. We will release our code publicly to help other researchers reproduce our work.

Q6: Image random mask in pre-train.

The image random mask improves the robustness of the pre-train policy but cannot directly enable the policy to learn useful representation. Therefore, the future action prediction task is necessary in pre-train.

We have addressed the concerns of the Reviewer one by one in the following. The paper will be revised accordingly.

Q1: Ablation study in real-world experiments.

In the paper, we conduct ablation study mainly in simulation as the environment is easy to control, which benefits to ensuring fair comparison. Our experience shows that the experimental results between our built simulation environment and real robot are consistent.

As suggested by the Reviewer, we also conduct ablation study using the real robot and report the success rates on all the three our designed real-robot tasks. These three tasks evaluate the performance of robot manipulation policies from different perspectives.

The real-robot experiment results corresponding to Table 2 of the paper are as follows. We can see that the improvments of our proposed designs are more significant in the real-robot experiments.

The real-robot experiment results corresponding to Table 3 of the paper are as follows:

According to the reported results, all the designs in our proposed method are also effective on the real robot.

Q2: Experiment on public simulator benchmarks.

In the paper, we conduct experiments based on our own built simulation environment rather than previous public benchmarks because these public benchmarks usually generate demonstration data by rule-based motion control. Their generated data has a distribution gap with the real robot data collected based on teleoperation. Differently, our used training data in our own built simulation tasks is also obtained from teleoperation, so the experimental results are more convinicing.

As suggested by the Reviewer, we compare our method with previous SOTAs using the ManiSkill-v3 and RLBench benchmarks. We select them because they provide unified training data generation protocol, which ensures fair comparison. The training data and testing protocols are the same between VIRT and the compared methods.

The tasks in ManiSkill-v3 cover navigation, humanoid robot, manipulation, etc. We select the manipulation tasks in ManiSkill-v3 to conduct experiments. As mentioned before, the training data is generated based on classic motion control. The results on ManiSkill-v3 are as follows:

For RLBench, although there are totally 100 tasks, many tasks are easy and previous methods have achieved very high success rates. Due to the reply character limit, we select six challenging and representative tasks that the success rate of the best method is still below 90%. We compare our method with the other methods that have reported their results in RLbench. We use the training data of PerACT and follow its testing protocol (the other compared methods also adopt this protocol). The results on RLBench are as follows:

The results on the ManiSkill-v3 and RLBench benchmarks show VIRT achieves the best results, which further confirm the superiority of our method.