Zero-Shot Robotic Manipulation with Pre-Trained Image-Editing Diffusion Models

Thank you for your feedback and a positive assessment of our work. We address your concerns below and have updated the paper to clarify each of the questions.

Please let us know if your questions are resolved, and if so, we would be grateful if you are willing to upgrade your score. We are happy to discuss further if any questions are remaining.

RT-2-X performs much better on scene C, so why does it perform better only on scene C?

After the submission deadline, we made some improvements to the subgoal diffusion model as we discuss in Appendix A.4. We re-evaluated in all scenes and found that our method now outperforms RT-2-X, even in Scene C. You can find the updated results at this link, as well as in Table 2 in the updated PDF.

Regarding why RT-2-X performs badly in Scene B, this is because each scene is challenging in different aspects. Semantically, Scene C is the most challenging since the robot needs to carefully ground the language command to identify the unseen objects and requires a high generalization capability. However, each object is in a shape that is relatively easy to grasp. In contrast, Scene B is challenging in the sense of manipulating the plastic bell pepper --- this object requires particular precision to grasp due to being light, slippery, and almost as wide as the gripper. We have updated the PDF to clarify this point.

We found that while RT-2-X is good at semantic generalization for unseen objects, it lacks precision in picking up challenging objects like the bell pepper. In contrast, as we discussed in Section 5.4, the subgoal generated by SuSIE improves the precision of low-level control for manipulating these challenging objects.

There is not much clear analysis or intuition why the fine-tuned image-editing model performs better than the video prediction model.

Modeling an entire video puts a very high burden on the generative model, requiring the frames to obey strict physical consistency. Unfortunately, we find that current video models often produce temporally inconsistent frames (hallucinations), which only confuse the low-level controller, inhibiting it from completing the task. After the submission deadline, we evaluated the video model baseline (UniPi) on the real robot but we found that it performed poorly due to the hallucinations. We show the updated table at this link. Qualitative examples of UniPi rollouts can be found at this link. While it’s definitely possible that better video models in the future can reach better performance, we found no open-source video model that could get good results despite considerable effort (no code for the original UniPi work was ever released, though we tried to follow a reimplementation used by one of its authors for another paper).

The results from the simulated environment(CALVIN) don’t show much significant performance difference.

Perhaps there is some misunderstanding here. The simulation results of SuSIE in the CALVIN benchmark (0.75, 0.46, 0.19, 0.11, 0.07) improve over the next best baseline AugLC, which scores (0.69, 0.43, 0.22, 0.09, 0.05), by a fair margin.

Furthermore, after the submission deadline, we realized that we used sub-optimal values of classifier-free guidance parameters for the SuSIE’s evaluations on CALVIN. After simply tuning these values, it led to further performance gains posted in the latest draft of the paper (0.87, 0.69, 0.49, 0.38, 0.26). Kindly refer to these numbers as the measured performance of SuSIE. The updated table can be seen at this link.

It would be better if the authors could provide additional experiments with the baselines which utilize low-dimensional image embeddings as subgoals.

We are willing to add this baseline, but we are not sure which method this is referring to. Do you have any specific methods or papers in mind that you would suggest we include?

I wonder if how much the change in k value makes a difference in performance.

While a full ablation of this hyperparameter would be ideal, retraining the diffusion model and re-evaluating the policy many times is prohibitively expensive. As such, we chose $k$ by inspecting each dataset and choosing a value that will be about one-third of the average length of the trajectories in that dataset. For example, in the case of BridgeData, since most of the trajectories are around 20-40 steps, we set $k$ to 10. In the case of the CALVIN dataset, as most of the demos are 65 steps, we set $k$ to 20. Though simple, we found this choice to generally lead to good results. We have added this clarification in Appendix A.6 in the updated PDF.

Dear Reviewer choL,

Thank you for your feedback! We were wondering if you have gotten a chance to go over our responses. We would appreciate it if you could have a look and let us know if your questions are resolved. We are happy to discuss further.

Thanks!

Hi, thanks for your response and the positive feedback! We are glad to hear that most of the concerns have been addressed!

a more specific explanation of the changes that led to SUSIE's better performance in real-world experiments

Thanks for your question! We have updated the paper to discuss the improvements we made to SuSIE since the submission in detail in Appendix A.4. Concretely, the main source of improvement is an updated subgoal diffusion model. Since the deadline, we trained a version of our subgoal diffusion model by fine-tuning on a combination of the Bridge dataset and a human video dataset, Something-Something. Note that human video datasets can be incorporated seamlessly into our approach, as the diffusion model does not make any specific use of robot embodiment or actions and only requires a tuples of frames and future frames, annotated with a natural language command. Upon visual inspection, we found that the diffusion model trained on a combination of Bridge data + human video data produced subgoals with higher quality than the one trained on robot data only (see this figure). Upon evaluating the full SuSIE approach with this model, we found that we were able to outperform RT-2-X in Scene C, while preserving or improving performance of SuSIE across the other two Scenes. This is despite the fact that RT-2-X utilizes a 50 times bigger model than SuSIE (55B vs 1B), and is trained on orders of magnitude more data compared to Bridge + Something-Something, which also includes 1 billion pairs of open-domain vision-language data. Due to this reason, we believe that this updated comparison is more fair than the comparison between SuSIE and RT-2-X in the original submission since RT-2-X co-trains on open-domain vision-language data alongside robot data, whereas the SuSIE model in the submission was not fine-tuned on any open domain data (but the new model utilizes some open-domain data, though in a substantially lesser quantity).

Overall, we believe that this result highlights a strength of our approach, demonstrating that the subgoal diffusion model can leverage knowledge from not only robot data but also from open-domain video datasets, that are unrelated to the robot. Due to an uncaught wording error, our previous response to your comment seemed to imply that the improvements came from a better low-level policy, but this was incorrect (we apologize that this wording error was not caught earlier) -- the improvements came from a better diffusion model, and we have edited our response above to correct this wording error.

We hope this resolves your remaining concern! Thank you!

InstructPix2Pix is not capable of performing viewpoint changes, how would this challenge be reflected in a real-life or research scenario where there are often multiple or moving cameras present?

While solving tasks involving changes in viewpoint is beyond our current focus, we believe this challenge can be addressed by training on more video data (e.g., Ego4D or Open X-Embodiment) that includes multiple or moving cameras. Another potential solution is to condition InstructPix2Pix on both historical and current observations. However, this is outside the scope of our work. We’ve added a discussion of this limitation of our evaluation setup to Section 6.

The gripper may be obscured in the camera view due to other objects. How would the scene play out, would the robot be able to recover or get stuck?

In our robot setup, the gripper will never be completely obscured by obstacles. As discussed in the previous question, while this is beyond our focus, a potential solution in such partially observable tasks is to condition InstructPix2Pix on both historical and current observations, or add multiple cameras to the setup.

What happens when it encounters, say, a similar current state but a different subgoal (that was not in the dataset) during test time?

We believe that our evaluation in Scene B already follows a similar setup to answer your question – this scene has a seen tabletop, a seen object (bell pepper), and a combination of a seen container (orange pot) and an unseen container (ceramic bowl). This means that while the initial state is similar to the state in the dataset, for the task “put the bell pepper in the ceramic bowl,” our method must produce a correct out-of-distribution subgoal. Our experiments demonstrate that our diffusion model can not only generate reasonable subgoals, but the low-level policy can also effectively handle these out-of-distribution subgoals.

Does your robot dataset consist of only completely successful trajectories?

Yes, our datasets consist of successful trajectories. It will be an interesting direction to utilize negative trajectories and failure cases, which we leave to future work.

There is a minor error in the table references. In Section 5.3 and A2.2.2, you mentioned "Table 5.3," which refers to "Table 1."

Thanks for the catch! We have fixed the reference in our updated PDF.

Some parts could be reformulated/reworded better in Section 4

Thank you for pointing this out. We have now reformulated the notation in Section 4.1 in the updated PDF.

The information regarding the datasets used can be provided more comprehensively and at an earlier stage in the paper.

We wanted to avoid excessively repeating information that is covered in a prior BridgeData V2 paper (Walke et. al., 2023), but have now added additional information about datasets in Appendix A.7. We are happy to move these details to the main paper for the final version, where there would be an extra page available.

Thank you for your feedback and a positive assessment of our work. We address your concerns below and have updated the paper to clarify each of the questions.

Please let us know if your questions are resolved, and if so, we would be grateful if you are willing to upgrade your score. We are happy to discuss further if any questions are remaining.

In essence, how do you select/optimize for the feasible/relevant subgoals?

Our model generates feasible subgoals because it is trained on feasible subgoals: the training data consists of predicting an intermediate frame in a real trajectory that corresponds to the desired task. We do not do anything special to pick out “important” subgoals, so it is quite possible the model will produce “less important transitionary scenes” – that is a very valid point. However, we empirically observe that the model learns to generate useful subgoals from this training alone and the generated subgoals are sufficient to improve the performance of the downstream low-level controller, as evidenced by the good overall performance of our method in the real-world evaluations. We chose $k$, the horizon for the subgoals, by inspecting each dataset and choosing a value that will be about one-third of the average length of the trajectories in that dataset. For example, in the case of BridgeData, since most of the trajectories are around 20-40 steps, we set $k$ to 10. In the case of the CALVIN dataset, as most of the demos are 65 steps, we set $k$ to 20. Though simple, we found this choice to generally lead to good results. We do agree that a smarter way of adaptively choosing subgoals could lead to even better results. We have added this clarification in the updated paper.

Table 1 uses some success rates from another paper (Ge et al., 2023). However, the approach proposed in this cited paper achieves higher success rates

We believe that the reviewer is referring to the non-zero-shot evaluation from Ge et al. 2023, which assumes access to interaction data from environment D (the target environment). All of the other methods we report, including ours, are performing zero-shot transfer, using the training environments (A, B, C) and without ever training on any trials from environment D. We believe it is reasonable to include only the zero-shot evaluations, and we’ve clarified this in the paper.

That said, our updated CALVIN numbers in the revised draft (0.87, 0.69, 0.49, 0.38, 0.26) also substantially outperform this omitted result from Ge et al., 2023 (0.72, 0.47, 0.30, 0.13, 0.11). After the submission deadline, we realized that we used sub-optimal values of classifier-free guidance parameters for the SuSIE evaluations on CALVIN. Simply tuning this value led to the currently reported result. The updated table can be seen at this link.

It was mentioned that Scene C should be a more difficult task than Scene B, however the results in 5.3 show that both MOO, RT-2-X and SuSIE perform significantly better on scene C than on B. Can you explain what the reason might be?

We apologize for the confusion. Semantically, Scene C is the most challenging since the robot needs to carefully ground the language command to identify the unseen objects and requires a high generalization capability. However, each object is in a shape that is relatively easy to grasp. In contrast, Scene B is challenging in the sense of manipulating the plastic bell pepper — this object requires particular precision to grasp due to being light, slippery, and almost as wide as the gripper. As such, because of these differences, it is unclear if absolute performances of different methods should be compared across these scenes. We have updated Section 5.1 in the new PDF to clarify this point.

Would the robot in this pipeline be able to pick up such challenging objects, such as a heavy and slippery object, even after a few tries?

This is a good question. As discussed in the previous question, the bell pepper in Scene B is a challenging object to pick up – being light and slippery. We found that the subgoal generated by SuSIE improves the precision of low-level control for manipulating the challenging objects, as we discussed in Section 5.4. We also show qualitative videos on the website under “Enhanced Precision -> Comparison Videos.”

Thank you for your feedback and a positive assessment of our work. We added new experimental results to compare SuSIE to the model without pretraining, and also included clarifications about the physical plausibility and low-level precision.

Please let us know if your questions are resolved, and if so, we would be grateful if you are willing to upgrade your score. We are happy to discuss further if any questions are remaining.

W1.1: Is the pre-training improving the downstream policy performance?

This is a good question! We have conducted an additional real robot experiment evaluating the from-scratch model in Scene C. As shown in the table, the from-scratch model is completely unable to perform the task, which demonstrates that the InstructPix2Pix initialization is essential for robust generalization. The videos of the from-scratch model can be found at this link, including the generated subgoals in the top row. More additional qualitative examples can be found at this link showing that in unseen environments, the model without InstructPix2Pix initialization (the column “from scratch”) produces noticeably worse images.

W1.2: Are the predicted subgoals really physically plausible? Authors should conduct experiments to show that having physically plausible sub-goals improves the performance of the low-level policy.

We apologize for the confusion. While we do not explicitly evaluate the plausibility of a given subgoal, we found that the predicted intermediate subgoals tend to provide more fine-grained feedback on gripper orientations and pose at the intermediate steps of a trajectory, which allows the policy to be more precise. The previous comparison with the from-scratch model empirically supports this claim – while the from-scratch model generates implausible subgoals and is useless for performing the task, the subgoals generated by SuSIE are sufficiently plausible for effective low-level control.

In addition, to further investigate if having plausible sub-goals improves the performance, we conducted a comprehensive comparison with Oracle GCBC (GCBC with the ground-truth final goal). SuSIE constantly achieves higher success rates even compared to executing privileged ground-truth final goals as shown in this table. We also show qualitative videos on our website under “Enhanced Precision -> Comparison Videos.”

We are happy to further revise the wording in the paper if that'd be helpful to improve clarity.

W2: [Major] How are findings from Figure 4 generalizable? W3: [Major] RT-2-X outperforms the method on certain tasks.

After the submission deadline, we made some improvements to the subgoal diffusion model as we discuss in Appendix A.4. We re-evaluated in all scenes and found that our method now outperforms RT-2-X, even in Scene C. You can find the updated results at this link, as well as in Table 2 in the updated PDF.

In addition, to further support our results presented in Figure 4, we conducted a comprehensive comparison with Oracle GCBC as we showed in the previous question. These results demonstrate that the findings from Figure 4 are generalizable.

W4: [Minor] The policy action space is not described in the main paper.

The action space is seven-dimensional, consisting of a 6D relative Cartesian motion of the end-effector, and a discrete dimension to control the gripper action. We have added the description in Section 4.3 in the updated PDF.

W5: [Minor] Additional details should be added directly in the main paper following this sentence.

Due to the page limit, we describe the details of the goal-reaching policy in Appendix A.1.1. We will include the description in the main paper for our final version (which will have an extra page for).

Dear Reviewer UFZY,

Thank you for your feedback! We were wondering if you have gotten a chance to go over our responses. We would appreciate it if you could have a look and let us know if your questions are resolved. We are happy to discuss further.

Thanks!

Thank you for your detailed feedback! To address the issues raised in your review, we added new experiments to show the physical plausibility and precisions of subgoals and also added clarification regarding error detection and recovery, difficulty of tasks, and goal sampling interval K.

Please let us know if your questions are resolved, and if so, we would be grateful if you are willing to upgrade your score. We are happy to discuss further if any questions are remaining.

There is no discussion on how to handle the situation if the generated subgoal is incorrect, nor are there any mechanisms for error detection and recover

Although there is no explicit error detection, we find that the replanning mechanism at test-time execution (Algorithm 1) makes SuSIE surprisingly adept at recovering from failures – as our algorithm regenerates a new subgoal every K steps, even if an incorrect subgoal is accidentally generated, it can still successfully achieve the task as long as it generates the next subgoal correctly. You can find some qualitative examples of failure recovery at this link, as well as more on our website under “Enhanced Precision -> Comparison Videos.” We have added this discussion to the paper.

Most of the tasks presented in the paper appear to be easily solvable. Real-world evaluation is conducted in just three scenes.

We evaluated a total of 9 tasks across three different scenes, and we believe that the difficulty and number of tasks in our evaluation are sufficient to answer the 4 research questions we pose at the beginning of Section 5. It is worth noting that we have adopted a common evaluation scope, as seen in other recent works on data-driven robot learning, that were also published at top ML and robotics conferences such as ICLR, ICML, CoRL, and RSS [1,2,3,4,5,6,7].

[1] studies only simulation tasks and did not include any real-world evaluations. [2] only evaluated 1 single tabletop scene for 4 tasks on the real robot. [3] also studies only 1 single scene for 9 tasks, and all tasks are picking and placing. [4] considers 3 scenes, which consist of 2 seen scenes in the training dataset. [5] studies 3 scenes, but all scenes look similar and have less variability than ours. [6] and [7] study a similar number of scenes as ours, consisting largely of picking and placing on a tabletop.

[1] Yu et al., ​​Using Both Demonstrations and Language Instructions to Efficiently Learn Robotic Tasks, ICLR 2023.

[2] Ma et al., VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training, ICLR 2023.

[3] Ma et al., LIV: Language-Image Representations and Rewards for Robotic Control, ICML 2023.

[4] Kumar et al., Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trials, RSS 2023.

[5] Myers et al., Goal Representations for Instruction Following: A Semi-Supervised Language Interface to Control, CoRL 2023.

[6] Walke et al., BridgeData V2: A Dataset for Robot Learning at Scale, CoRL 2023.

[7] Stone et al., Open-World Object Manipulation using Pre-Trained Vision-Language Models, CoRL 2023.

How do you choose the goal sampling interval K to ensure the subgoals are plausible enough?

While a full ablation of this hyperparameter would be ideal, retraining the diffusion model and re-evaluating the policy many times is prohibitively expensive. As such, we chose $k$ by inspecting each dataset and choosing a value that will be about one-third of the average length of the trajectories in that dataset. For example, in the case of BridgeData, since most of the trajectories are around 20-40 steps, we set $k$ to 10. In the case of the CALVIN dataset, as most of the demos are 65 steps, we set $k$ to 20. Though simple, we found this choice to generally lead to good results. We do agree that a smarter way of adaptively choosing subgoals could lead to even better results.

In 5.2 you mention "an understanding of dynamics to predict how to move and rotate the gripper to grasp it." How can you determine the plausibility of the prediction in terms of dynamics, given that the generating subgoals and learning low-level control policies are separate?

We apologize for the confusion. We have revised this sentence in the updated PDF. While we do not explicitly evaluate the plausibility in terms of the dynamics of a given subgoal, we found that the predicted intermediate subgoals tend to provide more fine-grained feedback on gripper orientations and pose at the intermediate steps of a trajectory, which allows the policy to be more precise, and this is what we meant by this sentence.

In addition, we found that pre-training is crucial for our model to generate fine-grained subgoals for the low-level policy to handle. We conducted an additional experiment to compare SuSIE to the model without InstructPix2Pix initialization (trained from scratch on robot data). As shown in the table, the from-scratch model generates implausible subgoals and is completely unable to perform the task. This demonstrates the fact that the subgoals generated by SuSIE are sufficiently plausible for the low-level control. The qualitative videos of the from-scratch model can be found at this link, including the generated subgoals in the top row.

In short, while explicit detection of subgoal plausibility will likely improve the performance further, our experimental results show that our current system, although simple, is already capable of effectively performing the tasks. We are happy to further revise this wording if that'd be helpful to improve clarity.

It seems that an extremely accurate pose of the end effectors is needed. I am somewhat concerned that the precision of the generated subgoal frame may not be sufficient to achieve it.

As mentioned above, whether or not the subgoals are precise enough is an empirical question that we answer with our experiments. Our comparison with Oracle GCBC (Section 5.4) demonstrates that SuSIE consistently achieves higher success rates even compared to executing privileged ground-truth final goals. In addition, we further compared SuSIE with Oracle GCBC in all scenes and the results can be found at this link. Qualitative examples are provided in Figure 4 as well as on our website under “Enhanced Precision -> Comparison Videos.” These results demonstrate that the subgoals generated by SuSIE are sufficiently accurate to improve the precision of low-level control. Generally, we do not believe that it is necessary to have particularly accurate poses for the end effector, nor did we observe this to be a significant factor – perhaps this stems from some misunderstanding of the method? Could you clarify why you believe that the end effector poses need to be highly precise? If this is due to confusing writing on our part we would like to correct it!

Dear Reviewer rGNF,

Thank you for your feedback! We were wondering if you have gotten a chance to go over our responses. We would appreciate it if you could have a look and let us know if your questions are resolved. We are happy to discuss further.

Thanks!