SE(3)-Equivariant Diffusion Policy in Spherical Fourier Space

We thank the reviewer for their thoughtful feedback. We respond below:

"However, it's not 100% clear how these experiments were performed from the description: "We train all the baselines on progressively tilted environments with 100 demonstrations." Does tilt=30 degrees imply that demonstrations are on environments sampled between 0 and 30 degrees? Some clarification on this experiment would be helpful."

We are sorry for the confusion. Tilt=30 degrees imply the environment (the table top) is sampled between 0 and 30 degrees. We will add the clarification to the paper.

"The paper is a bit densely written at several points, making it not very approachable"

Could the reviewer please point out which points can be improved?

"No testing in cluttered scenes"

For clarification, the (i) Pick Place task and (j) Kitchen tasks in MimicGen (see Figure R3 for visualization) involve multiple objects and can be viewed as slightly cluttered scenes. Unfortunately we don’t have time to test densely cluttered scenes.

"No statistical analysis of the results or inference timing data reported" "The authors claim that their work is more “computationally efficient” than competing approaches, but fail to provide any inference-time comparisons."

The step-wise success rate for physical experiments is reported in Table A.1. Moreover, in Table R1 we add the inference timing data.

"Grammatical Issues"

Thank you for pointing out the grammar issues and typos, we will fix these errors in the revision.

"EquiBot (Yang et al., 2024a) are limited to degree l = 1 representation that suppress rich information. Can the authors elaborate on this point and give some examples of problems for which that would be a limiting constraint?"

Please see the response to ""Although demonstrated in Sec. 4.2, the practical advantages of the proposed ... remain unclear. Specifically, what does it mean that "Vector Neuron only supports up to l=1"?""

"It was not clear to me why equation 5 was chosen in particular. It’s clear from the proof in A.2 that it preserves equivariance, but it seems other expressions (e.g. without normalization) would achieve the same goal."

We agree other expressions (without normalization or using tensor products, etc) could achieve the same equivariance. The proposed method stems from the nonlinearity operation in Vector Neurons. We find Equation 5 works well and we haven’t compared other expressions.

We thank the reviewer for their thoughtful feedback. We respond below (link to figures):

"... the point clouds and robot arm states can be further aligned within gripper's local coordinate frame by applying a known rotation. If the authors target to achieve SO(3) equivariance to object's absolute rotation, then it is equivariant when the scene is under a global rotation to a demonstration."

Leveraging SE(3)-invariant representations is a valuable baseline suggestion, and we have included it as an ablation. Specifically, we implemented a baseline called DP3-cano, which achieves SE(3)-invariance by canonicalizing (i.e., normalizing or transforming) both the point cloud and the action into the gripper frame. However, experimental results show that DP3-cano significantly underperforms the equivariant method SDP.

Table R2. Additional Ablations.

*Currently, these baselines have been trained for 300 out of the planned 500 epochs. We expect the final average success rate to improve by approximately 4%.

We hypothesize that local equivariance in SDP plays a crucial role. Furthermore, Benjamin et al. [1] demonstrate that using equivariant features results in significantly lower prediction errors compared to invariant features in molecular property prediction tasks.

[1] Benjamin Kurt Miller, Mario Geiger, Tess E. Smidt, Frank Noé, Relevance of Rotationally Equivariant Convolutions for Predicting Molecular Properties, Machine Learning for Molecules Workshop at NeurIPS 2020

"Second, the translation equivariance is achieved by explicit alignment of point clouds ... Normalization is a standard preprocessing in point cloud networks, ... I prefer to have SO(3) in the title instead of SE(3) to be more focused and reduce confusion."

For clarification, our method is SE(3)-equivariant. Specifically, SDP achieves SO(3) equivariance through the use of spherical Fourier representations, and T(3) (translation) equivariance via canonicalization. It is important to note that canonicalization differs from normalization in that the generated action is also transformed into the canonical frame.

"it would be beneficial to discuss it and correspondingly validate the maximum degree of spherical harmonics in ablation studies."

We agree that truncated spherical harmonic coefficients approximate the underlying spherical function, and using a low maximum frequency can lead to a loss of important details. As shown in Table A3 of the paper, setting $l = 1$ results in a noticeable performance drop. However, performance saturates at $l = 2$ and $l = 3$, suggesting that higher frequencies provide diminishing returns. Due to space limitations, this table is currently located in the appendix, but we will strengthen the connections in the main text to highlight this finding.

"... it worth to conduct experiments on more demonstrations, such as the settings in EquiDiff to see how the performance saturate."

This is a great question—we have plotted data scaling curves in Figure R1. Each point represents the average performance across four tilted-table tasks (with tilt angles in $[0, 15^\circ]$) for SDP, EquiDiff (EDP), and DiffPo (DP). Notably, SDP trained with $10^2$ demonstrations outperforms EDP trained with $10^3$, while EDP with $10^2$ demonstrations achieves comparable performance to DP trained with approximately $10^{2.5}$ demonstrations.

"... no further elaboration about the detailed settings, such as wether the control signal is directly from the policy network or converted from relative control signals."

The absolute position control does not have translational equivariance while relative position control does, as explained in Section 4.1 and proof in Appendix C.2.

The ablation study in Section 5.3, which uses absolute action control, highlights the importance of translational equivariance in our method.

"The architecture of the encoder"

See Figure R2.

"Any motivation to have physical experiments with only one ray-fin finger ..."

We use the shared robot in our lab and didn’t make any changes on the hardware.

"Is that better to remove e_i^T−e_i^T"

Yes, it's for the convenience of presentation in the paper.

We thank the reviewer for their thoughtful feedback. We respond below:

"Could authors tune some parameters of this baseline to see, like the longer prediction horizon? Besides, it is good to report the parameters for all baselines."

We use the DP3 results reported in the CoRL paper Equivariant Diffusion Policy. Additionally, we implemented a new baseline, DP3-canonicalization, which improves upon DP3 but still significantly underperforms SDP (see Table R2). Regarding DP3's poor performance on the MimicGen tasks, we hypothesize that this is due to its use of an MLP as the vision encoder. The MLP may struggle to capture information about multiple objects or their orientations—factors that are critical in the MimicGen tasks.

"More ablation studies would be helpful to understand the method."

In Table R2, we add DP3 (no equivariance) and DP3-cano (global SE(3)-equivariance) as additional baselines for ablation. We find that SDP, which has local equivariance, outperforms the global SE(3)-equivariance used in DP3-cano. We also introduce a new baseline, Discrete SDTU, which enforces discrete equivariance using the Octahedral or Cubic group, which discretize SO(3)—similar to EquiDiff, which uses the cyclic group C8 that discretize SO(2). Our results show that Discrete SDTU leads to an approximate 10% drop in performance, highlighting the advantage of SDP’s use of continuous spherical Fourier representations.

"Detailed parameters for baselines for fair comparison."

We provide detailed parameters for baselines. SDP generally adopts Diffusion Policy’s hyperparameters, except for batch size, because SDP is heavier.

Table R3. Hyperparameters for baselines.

"report the failure mode of the baselines/their own method."

Detailed failure modes are reported in Section B.3, Table A1, and Figure A4. The major failure mode is inaccurate action prediction when the end effector is about to make contact. We will strengthen the connection between the failure modes analysis section and the main text.

We thank the reviewer for their thoughtful feedback. We respond below:

"Some SE(3) Diffusion Models should be cited as the related work, such as: ..."

Thank you for highlighting these relevant works. We will include citations to these papers in the revised related work section.

"Although demonstrated in Sec. 4.2, the practical advantages of the proposed ... remain unclear. Specifically, what does it mean that "Vector Neuron only supports up to l=1"?"

Both Vector Neurons (VN) and our method can be interpreted as representing features in spherical Fourier space, up to a specified maximum frequency $l$. While our method supports arbitrary spherical harmonic types (we use $l_{max}=2$ in our paper), VN is limited to using only scalars ($l=0$) and vectors ($l=1$) as features. The scalars in VN are mathematically equivalent to type-0 features, which are invariant to rotation. The 3D vectors $V = [V_x, V_y, V_z]$ used in VN correspond to type-1 features $c_1 = [c^{-1}_1, c^{0}_1, c^{1}_1]$, as both are three-dimensional and transform under rotation via a rotation matrix. Thus, VN only supports spherical features up to type-1. However, type-1 features have limited representational capacity—for example, they are incapable of capturing spherical distributions with two distinct modes.

" could you clarify how the "truncated spherical Fourier coefficients ..." in contrast to the "computationally heavy SO(3) irreps used in ET-SEED"?"

Both SDP and ET-SEED achieve SE(3) equivariance by assigning SO(3)-steerable features to each point in the point cloud. While both methods use irreducible representations (irreps), SDP employs lightweight, order-wise linear layers—each order $m$ only connects to itself. In contrast, ET-SEED (based on the SE(3)-Transformer) utilizes heavier and more redundant fully connected linear layers that connect all orders $m$ across all types $l$.

"...comparisons regarding computational efficiency are currently missing."

Empirically, although ET-SEED employs a two-stage diffusion process, its inference time is 60× slower than that of SDP (29.4s vs. 0.44s; see Table below). Moreover, SDP's inference time is on the same order of magnitude as that of Diffusion Policy (SDP is approximately 5× slower than DP).

Table R1. Practical statistics for SDP and SOTA baselines.

"Table 1 indicates that as the degree of SE(3) initialization increases, SDP continues to exhibit significant performance degradation, .... Could you address this point explicitly?"

Authors agree that SE(3) equivariant methods should perform consistently no matter how the task is rotated or translated in 3D, provided the transformations are of the full environment. However, in Table 1, these tasks are not perfectly SE(3) transformed, since the gravity direction is not transformed, the camera view (occlusion) is not transformed, and the robot (kinematics) is not transformed. All of these factors could add complexity to the task (e.g., greater table tilt leads to increased object instability), thus SDP continues to exhibit performance degradation. Despite the increasing complexity, we find that SDP outperforms all baselines on all levels of SE(3) initialization of the tilted table tasks in Table 1.

"Please clarify the fundamental differences ... in Spherical Fourier Neural Operators."

Both methods leverage the spherical Fourier (SF) transform to process signals on the sphere. However, our method is purely based on SF, while SFNO combines an SF neural network for global convolution with a pointwise MLP for local non-linearity. Additionally, our proposed SDTU enables SO(3) and temporal equivariant convolution, and SFiLM provides equivariant spherical conditioning—capabilities not present in SFNO.

"Why not directly consider SE(3)-invariant representations?"

Leveraging SE(3)-invariant representations is a valuable baseline suggestion, and we have included it as an ablation. Specifically, we implemented a baseline called DP3-cano in Table R2, which achieves SE(3)-invariance by canonicalizing (i.e., normalizing or transforming) both the point cloud and the action into the gripper frame. However, experimental results show that DP3-cano significantly underperforms the equivariant method SDP.