SE(3)-bi-equivariant Transformers for Point Cloud Assembly

We thank the reviewer for the time and effort. We address the concerns below.

1. I feel "shape registration" is a more proper description of the task in the paper.

We use the word "assembly" following [7], where two pieces of point cloud are matched together. We avoid using the word "registration" because we are afraid this word will cause some confusion: registration is a classic task which seeks to align overlapped point clouds using correspondence, so aligning non-overlapped point clouds seems not well-defined for this task.

2. Should be compared to prior works on shape registration, including classic optimization-based methods like ICP, and more recent learning-based methods such as [1, 2] below.

We have already included two state-of-the-art registration methods [21] and [39] as baselines (they are more recent than the papers mentioned by the reviewer). These two methods are learning based and have an optimization-type finetuning process. ICP is not compared in the main text because it generally fails when the pose is randomly initialized (e.g. when the initial rotation error is larger than 45 degrees). We have included ICP in table 5 and table 6 in the appendix for clarity.

3. with less training data or less pose augmentation, the proposed framework can have less performance drop than these non-equivariant prior works...

We showed that one important advantage of our method brought by equivariance is that it can be used when correspondence does not exist (Fig.4 when $s < 0.5$). Of course, you are correct about the data efficiency, because we do not use pose augmentations as noted in line 296 (they have no influence on our method), but they are necessary for learning-based methods like [21].

4. [3] is also a related work, especially to the robot manipulation task in Sec. 6.6 (and Suppl. Sec. D.9).

Yes, this is a related paper. We have now cite this paper, but we are not able to conduct quantitative comparisons, because the correct solution is not unique in this task (as noted in line 889). For example, for a correct assembly, the bowl is allowed to rotate horizontally in the plate, so $\Delta r$ can be very large even for the correct assembly. A proper metric is the success rate of the manipulation in physical hardware experiments as in [3], but measuring this metric is beyond the scope of this work.

We thank the reviewer for the time and effort. We address the concerns below.

1. Robotic manipulation papers including Diffusion-edfs, Tax-pose and Sim3

Yes. Those are related papers. We have now cited them, but as noted in line 889, we are not able to conduct quantitative comparisons, because the correct solution is not unique in this task. For example, for a correct assembly, the bowl is allowed to rotate horizontally in the plate, so $\Delta r$ can be very large even for the correct assembly. A proper metric is the success rate of the manipulation in physical hardware experiments as in [1,2], but measuring this metric is beyond the scope of this work.

We have now added the following sentence (bold) in line 88: "... modelling 3D data, and recently they have been used for robotic manipulation task [26 , 27, 38] ". Since the Tax-pose paper does not use equivariant network, we feel that it is more suitable to mention it in line 16 (robotics [27, 20]) in our introduction. (27 is the Diffusion-edfs paper and 20 is the Tax-pose paper, and 38 is the Sim3 paper)

2. The definition of point cloud assembly... is a bit imprecise. .. Presumably what aligned means depends on who generates the dataset.

Yes. Depending on the specific dataset, it can mean reconstructing the shape, robotic manipulation, or even protein binding. We feel the word "align" is general enough to cover these meanings, and we provided some examples and citations in Line 16 to make the meanings concrete.

3. ... most correspondence-based approaches use Aruns method which is not sensitive to initial poses.

Although Arun's method is robust to initial poses, most of the correspondence-based methods are sensitive because the features they use are not invariant (or equivariant). We mentioned this in line 69 in Sec.2.

4. ...“SE(3)-bi-equivariance does not rely on correspondence”; do you mean your BTIR method does not?

Sorry for the confusion. We mean the SE(3)-bi-equivariance prior (the equation in definition 3.1) does not rely on correspondence. We have now made this sentence clearer (the added content is marked bold): "$SE(3)$-bi-equivariance prior does not rely on correspondence, i.e., it can be used to handle PCs with no correspondence."

5. The results on real data and visual manipulation...should be included in more detail in the main paper... Also, more discussion should be included.. why is it that BTIR without ICP is relatively bad on 7Scenes but better on outdoor scenes of ASL?

We agree that the experiments are important, but due to the 9-page limit of the main text, we have to place them in the appendix and refer to them in the main text to keep the overall structure complete.

On the other hand, we have now made it clearer that BITR+OT should be compared with GEO and ROI, because they also include an OT-type refinement process. The result of BITR is used to show that the model can generate results that are close to the optimum (the errors can vary in different datasets).

Specifically, we re-organize the paragraph starting at line 848 as follows: "We report the results in Tab. 5. We observe that BITR can produce results that are close to the optimum ($\Delta r \approx 25$) from a random initialization ($\Delta r \in U [0, 180]$), and extra refinements like ICP and OT can further improve the results ($\Delta r \approx 10$). This observation is consistent with that in Sec. 6.3.1. In particular, BITR with the OT refinement is comparable with GEO and ROI, which use highly complicated features specifically designed for registration tasks and an OT-like refinement process. On the other hand, ICP and OMN fail in this task due to their sensitivity to initial positions. An example of an assembly result of BITR is presented in Fig. 11."

6. ...More baselines should be included ... and more challenging or varied datasets used. For instance, compare performance across multiple shapenet classes...

• As for the registration baselines, we have included the state-of-the-art methods GEO [21] and ROI [39] as two strong baselines. We also considered the classic method ICP [41], and a recent correspondence-free method OMN [36] in Sec 6.5 for completeness. We think the comparisons with these methods are sufficient to support the argument that our method is at least comparable with the existing registration methods.

• On the other hand, as for the dataset, we applied our method to different types of datasets, including indoor (7Scene), outdoor (ASL), objects (ShapeNet), and manipulation. We think these experiments can show the versatility of our method in practice. However, we are not able to train on larger datasets due to high computational cost as discussed in Appx.E. For example, training on one class of ShapeNet takes several days as noted in Appx.D.1, so we do not have enough time to train on multiple classes at this moment. We leave the task of scaling BITR to future research. Probably this can be done with the aid of the techniques mentioned by reviewer fq5D.

7. ...the method is deterministic. How does it solve the wine bottle task in Table 1?

The input pieces of a bottle are not symmetric (there is no restriction on the complete bottle shape), so our method can be used without difficulty. Actually, as noted in line 910, we do not need to worry about the symmetry problems for the Lidar type data because strict symmetry never exists due to noise. (Symmetry is a problem when modelling molecules, such as a benzene ring, which is strictly symmetric.)

8. About the degree of features.

Our layer works for higher order features. We did not use higher order features due to efficiency consideration. We will add a sentence below Eqn 12 to make this clearer: "here we only include degree-0 and degree-1 features, and higher degree features can be used similarily"

9. Is this the first instance of bi-equivariant filters being used to process two inputs simultaneously?

Yes. We have stated it in line 92.

Thanks for the careful reading of our paper. We are happy you like the extended U-BITR model which is not even included in the main text due to space limitations. Thanks for bringing up [4,5], we have read through these papers, and they indeed seem useful for accelerating our method. We now address your concerns as follows.

1. The paper may be challenging for readers without a background in representation theory and SE(3)-equivariant neural networks. The proposed method is complicated and might be difficult to implement and extend.

We agree that some non-trivial preliminaries are required to read this paper. To alleviate this, we included a brief introduction of SE3-transformers in Appx.A to make the material self-contained. We also (briefly) introduced representation theory in Sec.3.2. [5] is cited and readers can find more complete information there. As for the implementation, we will release the code upon accept to facilitate the research in this area.

2. Experimental validation is rather confined to simple problems in which object parts are clipped by a random plane. It isn’t sufficiently verified for more general and realistic assembly tasks in which the parts are not cleanly cut by a plane.

We have tested our method on datasets which are not generated by cutting. For example, the BB, 7Scene and ASL datasets from Sec 6.4 to Sec. 6.6, where BB is generated by physics simulation and the other two are real datasets. Also, the data in the manipulation experiment is not generated by cutting.

3. Why include type-1 features in the last layer before softmax in Eq (12)? Why not use only type-0 irreps? Why not use type-2 or higher?

We only use degree-$\\{0, 1\\}$ features in our network due to efficiency considerations. In the merging layer, we merge these features. We will add a sentence below Eqn 12 to make this more clear: "here we only include degree-0 and degree-1 features, but higher degree features can be used similarly"

4. How is locality (neighborhood) defined for 6D point clouds? Directly using KNN for 6D point clouds to construct a neighborhood graph does not make sense to me.

We use KNN in 6D space. Our rationale is that a small distance in 6D space means small distances in the first and last 3D components ($\tilde{X}$ and $\tilde{Y}$), i.e. the key point coordinates of X and Y. So KNN in 6D is similar to doing KNN for $\tilde{X}$ and $\tilde{Y}$ and then taking the intersection of the edges of these two graphs, and it has the advantage of avoiding empty edges for all points.

5. The key points of X and Y are ordered and thus not permutation-invariant. Could this have any negative consequences?

No, they are permutation invariant. The coordinate of each key point $k_j$ is computed as $k_j = \sum_{i} F_{ji}X_i$, where $F_{ji}$ is an invariant feature at node $i$ (after softmax normalization). So if we permute $i$, we still get the same $k_j$ for all $j$.

We thank the reviewer for the time and effort. We address the concerns below.

1. The presentation quality could be improved. For example, figure 2 ... The description of the toy experiment..

We apologize for the lack of clarity. We kept all descriptions concise to fit in the limited 9-page constraint. However, we have now added more descriptions to the caption of Fig 2 and revised the description of the toy example. We have now cited [37] to provide more visualization of the generated data.

We have made the following modification (the added content is marked bold):

• (The caption of Fig.2) "The input 3-D PCs X and Y are first merged into a 6-D PC Z by concatenating the extracted key points $\tilde{X}$ and $\tilde{Y}$"....
• (Line 298)..We train BITR on the bunny shape. We prepare the dataset similar to [37]: In each training iteration.....We train BITR to reconstruct S using randomly rotated and translated $\\{X_P, Y_P\\}$...

2. Evaluating the benefits of bi-equivariance. An alternative to bi-equivariance, is to train a regular equivariance network on the task of shape completion and then align the two partial inputs according to the predicted completed shape..... Does the suggested bi-equivariance architecture generalize better than a single-equivariance network trained on shape completion?

We have evaluated the benefits of bi-equivariance against non-equivariant methods [21] and SE(3)-equivariant method [35]. We observed some improvement against these methods in our experiments.

On the other hand, we are not aware of any assembly/registration method that is based on shape completion, nor do we think this can be implemented easily. Because the two generated completed shapes may not be consistent (there is no correspondence between the generated part), unless both of the inputs are taken into consideration in the generation. But that will require assembling the inputs first, or at least using SE(3)-bi-equivariant features, which falls back to the assembly task. In summary, we do not think point cloud completion based methods can be used in this task.

We thank the reviewer for the time and effort. We address the concerns below.

1. The setup of zero overlap is not addressed properly when equivariant constraints are enforced...

The uniqueness and overlap ratio are completely different concepts. There can be non-unique assembly even in the fully overlapped case. Consider, for example, aligning a sphere to another sphere of the same size. The uniqueness is already discussed in Appx.E.

2. ...permutation of the points in X,Y would result in permuted keypoints in Eq.(10).

This is not true. Eqn 10 is a quite common technique to get invariant key points, see the definition of $y_{1k}$ and $y_{2k}$ in Independent SE(3)-equivariant Models for End-to-end Rigid Protein Docking, which is a paper mentioned by the reviewer.

3. Extracting the same amount of keypoints from both point clouds might be restrictive when sizes differ by a lot.

We do not see any restriction. If the reviewer thinks there is a restriction, evidence should be provided.

4. standard in the literature. Also the scale of the datasets is much smaller in number and sizes.

Our method is an assembly method where the input is not necessarily overlapped, not just a registration task for overlapped input. Those datasets might be standard for registration, but not for assembly. If the reviewer thinks the scalability of our method is an issue, then we insist a more scalable baseline method on this task must be provided.

5. References

We have now added citations "Coarse-to-Fine Point Cloud Registration with SE(3)-Equivariant Representations" and "Independent SE(3)-equivariant Models for End-to-end Rigid Protein Docking" in line 88 (..modelling 3D data []). We will not cite other papers because they are not related to the subject.

6. Line 28: "Sensitive to the initial poses of PCs". Is there any reference in the literature for that?

It's a well-known fact. For example, the last sentence of the 2nd paragraph in [36] "All these methods are sensitive to the initial positions."

7. In practice we do not know id the point clouds have zero overlap or some overlap. How can you discriminate such cases?

We don't think there is an easy way to discriminate this. That's actually an important reason why our method is more flexible than registration methods: the registration methods will fail silently when the point clouds are not overlapped.

8 and 9. The zero overlap case could in principle involve many solutions.

See 1.

10. In the method the authors extract the same amount of keypoints for the X,Y point clouds. Is that restrictive in cases one point cloud is larger than the other?

See 3

10. Shouldn't there be some sorting of the keypoints before concatenation? See Weaknesses too.

See 2.

11. Why is BITR+ICP bi-equivariant?

Because ICP is distance-based.

12. How does the method scale to large point clouds with complicated distributions as in 3DMatch?

For a "complicated" indoor dataset, see experiments on 7Scenes. See Appx E for computational cost.

13. Are normals given as features in this and the rest of the methods? Are the normals computed before or after the cut?

14. Normals are use in ROI, not other mehtods. 2. After.

15. Is the method trained per-class or for the whole dataset?

per-class

Thanks for your reply. It is quite easy to see that the key points are permutation invariant, simply because the set of features of neiborhood points are always the same under permutation. (It has nothing to do with optimal transport) In other words, the whole network does not rely on the specific index of the node.