ET-SEED: EFFICIENT TRAJECTORY-LEVEL SE(3) EQUIVARIANT DIFFUSION POLICY

We sincerely appreciate Reviewer kmzf’s recognition and positive feedback on our work, as well as the valuable comments and suggestions. Below, we provide detailed responses to the issues raised:

Interdependence Between Keyposes

We apologize for not sufficiently emphasizing this aspect in the original manuscript. In fact, our method, similar to diffusion policy, operates as a true trajectory-level generative model. Specifically, in our implementation, the input to the denoising network is a noisy action sequence $A^k$, and the output is the predicted noise $\hat{A}^{k \rightarrow 0}$ for that sequence. The keyposes within the sequence are not independent; during the forward pass, they interact and influence each other. We regret any misunderstanding caused by our unclear writing. To address this, we have clarified this point and refined the relevant formulations, particularly in Section 4.3 of the revised manuscript. We hope these revisions offer a clearer understanding of our approach.

Multi-Step Verification of Equivariant Markov Processes

As you correctly pointed out, single-step experiments alone are insufficient to fully validate the benefits of our diffusion process design. Following your suggestion, we have designed multi-step experiments to analyze the effects of different transition function designs on the performance of the diffusion process. Specifically, we compared two types of diffusion processes:

(1) A process where all steps use $p_3$​-type transition functions (aligned with our baseline EquiBot).

(2) A process where the first $K-1$ steps use $p_1$-type transition functions and the final step uses a $p_2$-type transition function (the design used in this work).

Experimental results show that the second design outperforms the first in both convergence speed and final performance. The details of these experiments are provided in Appendix D of the revised manuscript, with the newly added second experiment highlighted in red for emphasis. We hope these additional experiments address your concerns and further validate our design choices.

Dependency on Segmentation

Your understanding is entirely correct: ET-SEED requires segmenting the scene’s point cloud to isolate the points corresponding to the target object. In our experiments, we used the SAM2 model for preprocessing the point clouds captured by the cameras. It is worth noting that many prior works leveraging equivariance for pose estimation or trajectory generation assume access to pre-segmented point clouds$^{[1,2]}$. While the dependency on segmentation is indeed a limitation of our method, as you pointed out, recent advancements in open-vocabulary point cloud segmentation models suggest that isolating the target object from a 3D scene is becoming increasingly feasible. With further progress in point cloud segmentation models, challenges like segmenting non-object entities are also expected to be addressed effectively. To acknowledge this limitation, we have added a "Limitations" section in the last of the revised manuscript, highlighted in red.

Selection of Equiformer

Equiformer V2 is primarily designed with input and structural configurations optimized for atomistic systems, rather than point clouds. Adapting its input design and architecture to effectively handle point clouds is non-trivial, requiring significant modifications to both its data representation and computational framework. To the best of our knowledge, no open-source implementations of Equiformer V2 currently support point clouds as input. Consequently, we have opted to use the SE(3) Transformer as our backbone. In the future, should an open-source implementation of Equiformer V2 for point clouds become available, we plan to integrate it as a replacement for our current backbone, with the expectation of achieving enhanced performance.

We deeply appreciate your thorough review and constructive comments, which have been invaluable in refining the paper's focus and presentation. We have carefully revised the manuscript to address the concerns that lead to your lowered score, and we sincerely hope the changes will allow for a more favorable consideration. If you have any additional questions or concerns, please feel free to reach out to us. Thank you again for your thoughtful review!

[1] Yang, J., Cao, Z. A., Deng, C., Antonova, R., Song, S., & Bohg, J. (2024). Equibot: Sim (3)-equivariant diffusion policy for generalizable and data efficient learning. arXiv preprint arXiv:2407.01479.

[2] Hu, B., Zhu, X., Wang, D., Dong, Z., Huang, H., Wang, C., ... & Platt, R. (2024). OrbitGrasp: $SE (3)$-Equivariant Grasp Learning. arXiv preprint arXiv:2407.03531.

Dear authors,

Thank you for clarifying the keypose interdependence and adding a comparison experiment in Appendix D.2. All of my concerns have been addressed, and I have raised my score and confidence accordingly.

We sincerely appreciate your positive recognition of our work and are grateful for your constructive feedback. Below is our response to your comment:

The Inv. SE(3) Transformer and Eqv. SE(3) Transformer indeed share the same network architecture. The key distinction lies in the types of input and output features they process. As you correctly pointed out, the original representation in Figure 3 might lead to some confusion. To address this, we have added an explanatory note in the figure caption(highlight with red font) to clarify the differences between the two models, while also referencing the detailed explanation provided in the appendix.

Thank you once again for your insightful suggestions, which have highlighted important areas for improvement. Should you have any further questions or suggestions, please feel free to reach out to us.

We sincerely thank Reviewer t1h8 for your insightful comments and suggestions. Below, we provide detailed responses to the points raised.

Core Contribution

We would like to take this opportunity to clarify the core contribution of our work: introducing an SE(3) equivariant diffusion model that leverages spatial symmetries to improve data efficiency and spatial generalization in robotic manipulation tasks while reducing training complexity. Combining score-based modeling and geometric learning has been demonstrated in prior works as an effective approach for generating robot manipulation poses and trajectories. By leveraging the inherent symmetry in manipulation tasks, such methods significantly reduce the number of required demonstrations and improve spatial generalization capabilities [1–4]. Furthermore, this kind of approach has garnered considerable attention from the community. For instance, Equivariant Diffusion Policy [4] was nominated for the Best Paper award at CoRL 2024.

While we are not the first to integrate score-based modeling with geometric learning, our key contribution lies in introducing a novel end-to-end SE(3)-equivariant diffusion policy. Unlike previous methods, our approach optimizes the choice of the transition function in the diffusion process and directly defines the diffusion variables on the SE(3) manifold rather than in Euclidean space. These innovations enable our method to maintain end-to-end equivariance while achieving superior data efficiency and spatial generalization.

Teaser and Visualization of Multimodal Trajectories

We appreciate your feedback regarding the teaser figure. We agree that directly including performance graphs in the teaser is not an ideal approach. Based on your suggestion, we have updated the teaser by adding a set of images(Fig.1(b)) illustrating how our method generates equivariant manipulation trajectories (as our actions are defined as the 6D poses of the end-effector, so the manipulation trajectory is exactly the equivariant representations you were concerned about). Additionally, we showcase its ability to handle multimodal action distributions. To demonstrate this, we selected a fold garment task where multimodality is evident. For the same piece of cloth, one can fold either the left sleeve or the right sleeve. Both modalities are present in the demonstrations. During testing, when given a cloth placed in an arbitrary pose, our method randomly selects one of the two modes to complete the folding task. This example highlights both our method’s ability to handle multimodal task distributions and its spatial generalization capability. We have also revised Fig. 2 to more clearly showcase our core contribution: a more efficient SE(3)-equivariant diffusion policy. Additionally, we addressed the minor points you raised. Specifically, we removed the separate box defining equivariant and invariant functions(L150-L152) to create space for new experiments, and we improved the layout of Fig. 3 and the pseudocode for the algorithm.

Evaluation Criterion

In our paper, we evaluate our method and the baselines using two metrics: the success rate of manipulation tasks and the geodesic distance between the predicted trajectories and ground-truth trajectories. We explicitly point out in the paper (L437-L449,the font is marked in red) that success rate alone is insufficient for evaluating trajectory quality. This is why we additionally report the geodesic distance, which you may have overlooked.

Visualization of Equivariant Trajectories

We agree with your suggestion to include additional visualizations of the generated manipulation trajectories. We have added a new figure (Fig. 5) illustrating the trajectories generated by our method compared to those of DP3 for one of the tasks in our experiments (fling garment). As shown in the figure, when the cloth is placed at a new pose, our method can generate an effective manipulation trajectory, while DP3 generates inaccurate end-effector poses in the first several steps, resulting in the failure of the entire manipulation process. You can find this figure in the revised version of the paper(around L410), with captions in red font.

Limitations

We acknowledge that our method relies on certain assumptions. As you mentioned in the comment, we define the action space as the 6D pose of the end-effector, similar to the formulation of Diffusion Policy$[5]$ and EquiBot$[1]$. We added a “Limitation” section in the last of the revision version, in order to point out these assumptions.

However, we would like to clarify the statement in Weakness.C: "The perception part needs to be a point cloud and a specified coordinate system by experts." This statement is not entirely accurate. Our method does not require a “specified coordinate system by experts.” As outlined in Definition 1, we only assume the existence of a coordinate system attached to the object, but no specific information about it is necessary.

Summary

We deeply appreciate your review and constructive comments, which have been invaluable in refining the paper's focus and presentation. We have carefully revised the manuscript to address the concerns that lead to your lowered score, and we sincerely hope the changes will allow for a more favorable consideration. If you have any additional questions or concerns, please feel free to reach out to us. Thank you again for your thoughtful review!

[1] Yang, J., Cao, Z. A., Deng, C., Antonova, R., Song, S., & Bohg, J. (2024). Equibot: Sim (3)-equivariant diffusion policy for generalizable and data efficient learning. arXiv preprint arXiv:2407.01479.

[2] Hu, B., Zhu, X., Wang, D., Dong, Z., Huang, H., Wang, C., ... & Platt, R. (2024). OrbitGrasp: $SE (3)$-Equivariant Grasp Learning. arXiv preprint arXiv:2407.03531.

[3] Gao, C., Xue, Z., Deng, S., Liang, T., Yang, S., Shao, L., & Xu, H. (2024). RiEMann: Near Real-Time SE (3)-Equivariant Robot Manipulation without Point Cloud Segmentation. arXiv preprint arXiv:2403.19460.

[4] Wang, D., Hart, S., Surovik, D., Kelestemur, T., Huang, H., Zhao, H., ... & Platt, R. (2024). Equivariant diffusion policy. arXiv preprint arXiv:2407.01812.

[5] Chi, C., Xu, Z., Feng, S., Cousineau, E., Du, Y., Burchfiel, B., ... & Song, S. (2023). Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, 02783649241273668.

We sincerely appreciate your positive evaluation of our paper and your valuable feedback. We have carefully addressed the concerns you raised, as detailed below:

• On the mathematical background in Appendix A and B:

We originally included detailed mathematical background in Appendix A and B to provide readers with comprehensive context. However, as you correctly pointed out, such extensive details may not be necessary for the paper's core contributions. In the updated version, we have significantly streamlined Appendix A, retaining only the most relevant mathematical background directly tied to our work. Additionally, we have cited a mathematics textbook to guide readers interested in further exploration of this topic.

• On the definitions of $E_{equiv}$ and $E_{inv}$​:

We apologize for the typo in Equation (5) in the original manuscript, where the left-hand side should have been $E_{inv}$. This has been corrected in the revised version. Both $E_{inv}$ and $E_{equiv}$ represent our modified models based on the SE(3) Transformer, with invariance and equivariance properties under SE(3) transformations, respectively (refer to Equations (4) and (5)). Additional implementation details for these models have been included in the updated Appendix E.

• On the format of observations and their equivariance:

As mentioned in the "Problem Formulation" part of the Method (Section 4), our observations consist of a colored point cloud. We assume the point cloud only contains the object to be manipulated (in our experiments, we preprocess the point cloud to segment object points from the scene). Under this assumption, a change in the spatial pose of the object corresponds to an SE(3) transformation of the input point cloud. This allows us to use a policy equivariant to spatial transformations to handle such inputs effectively.

• On equivariance and invariance in Algorithm 1:

The equivariance and invariance properties of our model are inherently guaranteed by the network architecture. Thus, during the training phase, no special design is required. We simply follow a DDPM-like supervised learning approach, training the model to predict noise as closely as possible to the actual added noise. As you noted, our method's equivariance is guaranteed during the inference phase, where it outputs appropriate action sequences for objects in varying spatial configurations.

We deeply appreciate your thorough review and constructive comments, which have been invaluable in refining the paper's focus and presentation. We have carefully revised the manuscript to address the concerns that lead to your lowered score, and we sincerely hope the changes will allow for a more favorable consideration. If you have any additional questions or concerns, please feel free to reach out to us. Thank you again for your thoughtful review!

Dear reviewer fnNr,

We hope this message finds you well. As we posted our rebuttal to your official review for about 3 days, we kindly request your consideration of our responses to your concerns.

We deeply thank you for your time and reviews. We have addressed many of the comments in your review in the revised version, as hopefully you can agree as mentioned above. We thank you for taking the concern to provide critiques of the paper, and hope that our clarifications sufficiently address your concerns and improve your ratings.

Thank you for your attention and consideration.

Authors

Dear reviewer fnNr,

We hope this message finds you well. As we posted our rebuttal to your official review for about 10 days, we kindly request your consideration of our responses to your concerns.

We deeply thank you for your time and reviews. We have addressed many of the comments in your review in the revised version, as hopefully you can agree as mentioned above. We thank you for taking the concern to provide critiques of the paper, and hope that our clarifications sufficiently address your concerns and improve your ratings.

Thank you for your attention and consideration.

Authors