Make a Donut: Language-Guided Hierarchical EMD-Space Planning for Zero-shot Deformable Object Manipulation

W1 - Technical insights are unclear for the deformable and the general robotics manipulation community.

We appreciate your critical evaluation of the technical insights presented in our paper. The tasks of creating dough shapes such as donuts and baguettes are indeed not pre-existing benchmarks within deformable object manipulation; thus, they offer a novel challenge that extends beyond the capabilities of established frameworks like PASTA.

The key distinction of our approach lies in its zero-shot learning ability, which enables it to adapt to novel tasks without task-specific fine-tuning or training. PASTA, while effective within its demonstrated scope, requires a dataset to train on in order to sample feasible intermediate states during planning and is thus inherently limited in its ability to generalize to new shapes, e.g., donut. All their modules like VAE, cost predictor, etc., are tailored to their collected training data. This data-driven dependency hinders its application to the more complex tasks our framework successfully tackles. We also want to point out that PASTA needs to specify the desired number of stages to execute before the planning, and we already helped them out by specifying 2 stages for making a donut, 2 stages for making a baguette and 3 for making two pancakes, but their results are still worse than ours. You can find some visualizations of PASTA in our updated A.5 (sorry to put that in the appendix due to space limit).

We have now included a detailed analysis in the results section that elucidates the limitations of PASTA and similar baseline approaches. This includes a discussion on their reliance on extensive datasets and the inability to adapt to tasks involving complex manipulations such as squeezing, which our method can handle adeptly. Please see the Results paragraph of section 4.1 in our updated manuscript. We have augmented our manuscript with analyses pinpointing the failure introduced by PASTA’s inherent plan learning algorithm. This breakdown helps clarify the innovative aspects of our LLM-based planning technique.

Our LLM hierarchical planning framework's strength lies in its generative capability, which allows it to conceptualize intermediate subgoals that are not only geometrically feasible but also physically plausible. This is a significant leap over the sampled-based methods, which may not provide a feasible path or require extensive data for complex shapes. The Large Language Model (LLM) plays a crucial role in our framework by charting a high-level planning path, which serves as a guide for the subsequent execution by the low-level Earth Mover's Distance (EMD) space planning algorithm. This hierarchical structure is pivotal; the LLM alone cannot translate its generated plans into the raw actions required for robotic manipulation. Conversely, without the strategic direction provided by the LLM, the EMD space planning lacks a coherent objective, struggling to discern what end states are physically plausible for the robot to achieve. This interdependence is crucial for successful task completion and is evidenced by the comparative results in Figure 5(a), Section 4.3, where the EMD space planning alone is shown to be inadequate without the LLM's high-level input.

To ensure the message is clear and convincing, we have enriched the paper with the requested discussions, visualizations, and comparative analyses. This additional content will reinforce our argument that our LLM-based planning framework is indeed a significant advancement over previous methods like PASTA, particularly in its zero-shot generalization capabilities and task adaptability.

W2 - Unconvincing real robot experiments.

We are grateful for your insights on the real robot experiment demonstrations. We acknowledge the instances where the robot does not perfectly execute the splitting of the dough, as highlighted at the 01:51 timestamp.

The human operator's presence was solely to address the stickiness of the dough, which adhered to the rubber cushion, a non-interactive element of the environment. This was a measure taken to ensure the continuity of the experiment and in no way contributed to the task's execution, as the operator did not interact with the dough or the robot arm during the manipulation process.

We have since made strides to improve our setup, including the modifications to secure the cushion in place. These improvements will be reflected in additional experiments and visualizations, which we are committed to providing in a few days and will update them in the response.

I deeply thank the authors for their efforts in making updates and edits. They are appreciated.

W1 & W2 - Technical insights & Real-world results.

I disagree that PASTA's or other comparative approaches' results are worse, on both approach and result levels.

A. Generality and task adaptability.

On the approach level, the incorporation of LLM indeed makes everything zero-shot. I agree that there is a trend of research in our community heading in this direction and I recognize their values. However, I wouldn't agree that learning-based approaches such as PASTA are worse or limited due to the need for task-specific training, regardless of the amount of training data needed.

In this work, I think the proposed approach was able to circumvent policy training mostly because the control tasks in the study offer a good gradient optimization landscape for trajectory optimization. However, it is well-known in classical control that gradient-based trajectory optimization is limiting in many ways, especially in contact-rich tasks where contact-making and breaking are not differentiable. In deformable object manipulation, the optimization landscape is especially challenging for gradient-based approaches. In these scenarios, learning-based methods such as PASTA and Diffusion Policy [1] are good solutions in my opinion.

Considering the challenges the approach might face in these scenarios, it is hard for me to appreciate the technical insights of the paper from a zero-shot and task-adaptability standpoint which the paper advertises strongly.

B. Interpretability and quality of the results

On the results, as mentioned in the initial review, I think the technical insights are also unclear from an interpretability standpoint. In perception, how does the approach handle partial observation that was addressed in PASTA and DiffSkill via point cloud encoder and image encoder? For example, how is perception performed in real-world experiments? How are the agent states of the real world aligned with analytical simulation in the new updated video?

In terms of the quality of the real-world results, PASTA's real-world results are more interpretable (in my opinion) and show overlayed subgoals in its videos, and how these subgoals are achieved.

[1] Chi et al., Diffusion Policy: Visuomotor Policy Learning via Action Diffusion, RSS 2023.

W3 & W4

I thank the authors for their time and effort in making these updates and edits.

Dear reviewer,

We are very grateful for your feedback and we would like to give more clarifications in complement to our last response.

A. Generality and task adaptability.

We acknowledge the significance of your point regarding the landscape of gradient optimization. It's crucial to note that even in scenarios involving contact-making and breaking, the trajectories remain differentiable, as demonstrated by frameworks such as PlasticineLab[1] and DiffTaichi[2], which effectively handle contact-rich tasks using differentiable physics. Antonova et al. [3] have highlighted the actual challenges of gradient-based trajectory optimization, particularly its susceptibility to rugged loss landscapes and local optima. In response, our method utilizes LLM to decompose complex tasks into shorter, more manageable substages. This decomposition simplifies the optimization landscape, making the trajectory optimization process smoother and more reliable, as ablated in Section 4.3. As you said, our method benefits from a favorable optimization landscape and should be taken as our advantage.

[1] Huang, Zhiao, Yuanming Hu, Tao Du, Siyuan Zhou, Hao Su, Joshua B. Tenenbaum, and Chuang Gan. "Plasticinelab: A soft-body manipulation benchmark with differentiable physics." arXiv preprint arXiv:2104.03311 (2021).

[2] Hu, Yuanming, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan Ragan-Kelley, and Frédo Durand. "Difftaichi: Differentiable programming for physical simulation." arXiv preprint arXiv:1910.00935 (2019).

[3] Antonova, Rika, Jingyun Yang, Krishna Murthy Jatavallabhula, and Jeannette Bohg. "Rethinking optimization with differentiable simulation from a global perspective." In Conference on Robot Learning, pp. 276-286. PMLR, 2023.

Furthermore, we argue that while learning-based methods like PASTA are valuable, they inherently struggle with tasks that fall outside their training data, limiting their applicability to new, complex tasks such as doughnut making. The sim-to-real gap further exacerbates this limitation, as their learned policies often fail to transfer effectively to the real world. This is also evidenced in both visualizations and quantitative comparisons in our paper.

B. Interpretability and quality of the results

In our efforts to enhance interpretability, we have updated our supplementary materials to include videos and visualizations with overlaid subgoals. These improvements aim to provide a clearer and more intuitive understanding of how our planning framework guides the robotic manipulations towards the desired outcomes. We invite you to review these enhancements in our updated video and the visual representations found in Appendix A.9. As we explained in our previous response, PASTA uses heuristic policies (they are detailed in Section 3.1 of their appendix, one example: The roll policy first moves the roller down to make contact with the dough. Then, based on the goal component’s length, the policy calculates the distance it needs to move the roller back and forth when making contact with the dough.). While this can produce visually appealing results, it does not reflect the adaptive, generalized problem-solving our method aims to achieve. Our results may not look as perfect as theirs since our actions are directly derived from our optimizations rather than heuristics.

We hope that these updates address your concerns comprehensively. Your feedback has been invaluable in this process, and we look forward to any additional comments you may have.

I thank the authors for their replies and engagement in the discussion.

A. More thoughts on generality and task adaptability.

The ability to break down the optimization landscape into favorable pieces could be a plus when we only consider the specific tasks presented in the study. However, since the claims of the paper focus on zero-shot and task adaptability, it is important to recognize that the low-level differentiable physics optimizer could become a significant limitation.

B. More thoughts on results in both simulation and the real world.

Simulation. Many subgoals generated by LLM seem infeasible or not well-executed. For example, on the bottom-most row of Figure 9, the generated subgoals on the left seem to differ significantly from the results. Additionally, the donut shape seems incomplete for the second row from the top in Figure 9. I have not found a well-presented donut shape in the results, a shape championed in the title and throughout the paper.

Real world. Thank you for clarifying the details of your perception pipeline, such as the use of SAM. I am unsure why the manuscript did not proactively include these details in the last update. I would like to encourage the authors to include these details for how real-world experiments are conducted for reproducibility, since the paper aims to speak to the robotics community.

The real-world results need a lot of care to convince the robotics community. I appreciate the authors recognizing their shortcomings in real-world presentation. Starting at 01:17 in the video, the real-world shapes are heavily misaligned with the simulation / generated subgoal on the right. That is also not to mention that in the real world, a gripper is grasping on the rolling pin, which differs from the rolling pin setting the authors employ in the simulated environment. I also observed that many contacts with the dough come from the gripper instead of the rolling pin.

In any case, I deeply appreciate and recognize the authors' time and efforts in producing such real-world results during the rebuttal.

The solution proposed in this work is heavily tailored towards the deformable object manipulation setting. It also requires the point-cloud representation and a differentiable physics simulator. This limits the range of applications of the method.

We acknowledge your observation regarding the specificity of our framework towards deformable object manipulation. This focus is intentional, as deformable objects present a unique set of challenges distinct from rigid body manipulation. The paper aims to advance the field within this defined scope, contributing a versatile planning framework tailored to the complexities of deformable materials. The utilization of point-cloud representations and a differentiable physics simulator aligns with contemporary methodologies in the field[1][2][3], ensuring that our approach is both relevant and comparable to existing research.

[1]Li, S., Huang, Z., Chen, T., Du, T., Su, H., Tenenbaum, J. B., & Gan, C. (2023). DexDeform: Dexterous Deformable Object Manipulation with Human Demonstrations and Differentiable Physics. arXiv preprint arXiv:2304.03223.

[2]Li, S., Huang, Z., Du, T., Su, H., Tenenbaum, J. B., & Gan, C. (2022). Contact points discovery for soft-body manipulations with differentiable physics. arXiv preprint arXiv:2205.02835.

[3] Xingyu Lin, Carl Qi, Yunchu Zhang, Zhiao Huang, Katerina Fragkiadaki, Yunzhu Li, Chuang Gan, and David Held. Planning with spatial-temporal abstraction from point clouds for deformable object manipulation. In Conference on Robot Learning (CoRL), 2022c.

The LLM's plan is open-loop. In this case, the next stage assumes the first stage is feasible. When the planner fails to complete a stage, the next stage starts at a bad initial state.

The planning framework we propose is inherently flexible and does not rely on the success of previous stages, as each subgoal generated by the LLM is independent. The online EMD space planning algorithm is robust to variations in initial states and is capable of directing the manipulation towards the desired subgoal at each stage though not perfect. We also provide visualizations on intermediate subgoals and the next reachable point cloud at each timestep to reach the subgoal, in the updated A.5, A.6. We acknowledge the complexity of implementing a closed-loop system for LLM that involves real-time perception and reconstruction. While we recognize this as an important direction for future research, our current work is focused on the planning aspect within the constraints of available technology.

The input to the LLM is limited to the language description of the dough. If the LLM can get a more accurate/grounded idea of the dough configuration, it might be able to provide higher-quality plans.

Your point about the LLM's input is well-taken. Indeed, a more nuanced understanding of the dough's configuration could potentially enhance plan quality. Our framework is built to be compatible with advancements in perception technologies. We aim to address these perceptual challenges in future work, as our current focus is to establish a robust planning strategy within the confines of present-day perception capabilities.

Can you provide more details on the baseline methods used in the simulated experiments? For example, what is the observation space for the BC and SAC agents?

In our experiments, all baseline methods, including BC and SAC agents, utilize the point cloud of the dough as their observation space, plus the current translation and rotation of the tools. This standardization ensures a fair and consistent comparison across different methods. We use a PointNet-like backbone to encode the input point cloud with embedding dim set to 128. For SAC, the learning rates for actor and critic are set to 3e-4, with the reward decaying ratio $\gamma$ set to 0.99.

Maybe I missed something, but why is the numerator needed in equation 3?

Thank you for your question regarding the numerator in Equation 3. Upon reviewing the equation, we have realized that there was an omission in the manuscript. We apologize for this oversight. The equation indeed requires a summation symbol to accurately represent the aggregation of the terms across all indices $i$. The corrected form of Equation 3 should be: $\mathbf{x}^* = \arg\max_\mathbf{x} \sum_i \frac{\|\mathbf{p}_i' - \mathbf{p}_i\|_1} { \mathrm{sdf}_x(\mathbf{p}_i) + \delta}.$ It depends on the numerator if we write out all the terms and find a common denominator.

We hope this clarifies your concerns and we appreciate any further feedback you may have.

We are grateful for the valuable feedback you have provided on our work thus far. Your insights have been instrumental in helping us refine our approach and presentation. As the deadline for discussion is approaching, we would greatly appreciate it if you could provide any additional questions, comments, or concerns at your earliest convenience.

It appears that the LLM does not undergo any interactions or corrections after generating the point clouds for each stage.

We appreciate your insight on the role of LLM in our framework. The LLM indeed provides a high-level planning blueprint. Our EMD space planning algorithm is designed to be responsive and adaptive, using the LLM's output as a guide while continuously refining the approach based on real-time observations. This dynamic interplay ensures that potential discrepancies between the subgoals and the physical execution are adjusted for at each step. You can also find more visualizations in our updated A.5 and A.6.

The method described is effective for complex tasks with long-term planning, but it can only generate simple shapes using Python code.

Your observation regarding shape complexity is astute. While our current system does indeed focus on simpler shapes, the framework's core contribution lies in its general approach to integrating LLM with spatial planning. The complexity of generated shapes is acknowledged as a limitation and an avenue for future research. Our focus remains on establishing a robust planning methodology, which can later be enhanced to tackle more complex shapes as text-to-3D technologies advance.

Error recovery seems incomplete, and there still appears to be a risk of getting stuck after resetting the tool. There is no comprehensive mechanism for early termination.

Our framework incorporates a continuous correction mechanism within the EMD space planning algorithm, which actively seeks to reconcile deviations from the intended subgoal after each action, which can be further validated by the provided visualizations in A.5 and A.6. We recognize the importance of an early termination condition and have introduced a termination criterion within our algorithm, as detailed in L23 of Algorithm 1 of the revised manuscript, to prevent potential deadlocks and ensure efficient task completion.

Is the human effort required to develop these prompts more costly than defining a traditional task schema?

The effort required to develop prompt templates is minimal when compared to the traditional task schema definition. The use of LLM for planning is grounded in its ability to interpret and act on the context provided by the template. This is a one-time effort, after which users can interact with the system using simple, intuitive prompts. This significantly reduces the cognitive load and technical expertise required from the user.

The paper includes the hyperparameters used for dough simulation in A2, but it lacks analysis or discussion on the selection process and their impact on the results. Given that the physical properties of the dough can vary, how should this issue be addressed to make it more useful for different setups?

The selection of hyperparameters for dough simulation is consistent with established benchmarks, specifically the PASTA framework, to ensure a fair comparison. We also conduct an experiment to demonstrate the robustness of our approach across a reasonable range of material properties.

Q3. How to ensure the intermediate goal is feasible; how to align it with the real world?

Our experimental evidence supports the feasibility of LLM-generated intermediate goals for common bakery items like donuts, baguettes, and pancakes. While there may be limits to the complexity of shapes the current tools can produce, we believe these are reasonable constraints given the scope of this paper. Aligning these goals with the real world is achieved by standardizing the initial frame of reference between the LLM coordinate system and the physical environment.

Q4. How to use parallel grippers to manipulate rolling pins and other tools?

This objective can be attained by installing a grasp-compatible structure on the tools or by adjusting the gripper's opening distance to securely accommodate the tool dimensions. Our configuration is designed to ensure the gripper makes optimal contact with the tools, whether it be clasping the midpoint of a rolling pin or the apex of a cutter. These adjustments are rudimentary and not the primary focus of our research.

We hope that our responses address your queries, and we appreciate any further discussion on our work.

We are grateful for the valuable feedback you have provided on our work thus far. Your insights have been instrumental in helping us refine our approach and presentation. As the deadline for discussion is approaching, we would greatly appreciate it if you could provide any additional questions, comments, or concerns at your earliest convenience.

Thanks for your valuable feedback. Below are detailed clarifications.

The subgoal point cloud is created using the Python code generated by the LLM, with constraints on the shape's complexity or intricacy;

We appreciate your observation regarding the limitations of current LLMs in generating complex shapes. As highlighted in our discussion on limitations (Section 5), our research is not centered on advancing the capabilities of 3D generative models, but rather on proposing a novel planning framework for manipulating deformable objects. The inclusion of Point-E as an illustrative example in Figure 7 demonstrates our commitment to leveraging state-of-the-art Text-to-3D models to aid in intermediate shape generation. As LLMs continue to evolve, we anticipate more accurate and intricate shape generation that will integrate with our proposed framework.

This method necessitates users to submit comprehensive prompts, and the system's effectiveness is closely tied to the quality of the prompt data provided;

We create a standardized prompt template, which simplifies user interaction to a single descriptive line. The effectiveness of this approach is further supported by our introduction of 'chain of thoughts' and 'volume preserving guidance', which are key contributors to achieving high-quality outcomes, as detailed in section 3.1. The system's dependency on prompt quality is a known factor in natural language processing applications [1,2]. The focus of this paper is not to refine the underlying language model but to spotlight our coarse-to-fine hierarchical planning framework that synergizes LLM with EMD space planning for complex manipulation tasks.

[1] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents.

[2] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as i can, not as i say: Grounding language in robotic affordances.

The system appears to operate in a cascading manner, making it challenging to rectify infeasible subgoals generated by the LLM without low-level feedback.

We concur that the LLM may not always generate perfect subgoals. Nevertheless, as shown in our updated supplementary materials (A.6), the EMD space planning algorithm's closed-loop design is robust to such variations, consistently recalibrating based on real-time observations. This adaptability is also exemplified in Figure 4, where even if initial alignment is suboptimal, subsequent steps are unaffected, validating our framework's efficacy in practical scenarios.

Within Table 1, the Baguette demonstrates a volume change exceeding 30%. What does this numerical value signify, and how might it influence the manipulation process?

This question is related to the above. The noted volume change for the Baguette in Table 1 is indicative of the algorithm's resilience to variations in object manipulation. This robustness stems from our framework treating LLM-generated plans as a directional guide rather than absolute instructions. We provide some visualizations in the updated A.5, our model can still accomplish the task because the EMD space gradient is still pointing to the correct direction.

How likely would the LLM generate unachievable subgoals? And for these situations, how would the proposed framework handle the cases?

The potential for an LLM to produce unachievable subgoals is indeed a consideration in our framework. However, our empirical data indicates that the variance in LLM-generated subgoals typically remains within the bounds of what our EMD space planning algorithm can realistically achieve. The EMD space planning algorithm is adept at making incremental adjustments to align with these subgoals, which are often close approximations of the physically possible shapes.

We acknowledge that our method is not a panacea for all, especially those with extreme complexity. This is a point we have transparently addressed in our discussion of the limitations of our study. Nevertheless, the novelty of our contribution rests on the synergy between LLM planning and DiffPhysics-P2P, which enables us to tackle a wider array of complex tasks than previously achievable. This represents a significant step forward, pushing the boundaries of what can be realized in the realm of deformable object manipulation.

Moreover, in the instances where the LLM might propose a subgoal that is beyond the scope of current capabilities, this is not necessarily a failure but an opportunity for further research and development. Our work opens the door for advancements in the interpretation and refinement of LLM-generated plans, setting the stage for future innovations that can handle an even greater spectrum of complexity.

We are happy to participate in any further discussions and we hope the responses address your concerns.