Learning Deep O($n$)-Equivariant Hyperspheres

Thank you for your review and assessment of our work!

Below we address the Weaknesses you pointed out:

1. • Our intuition as to why spherical neurons and equivariant hyperspheres in $n$D offer advantageous modeling is as follows: the relation between a point and a spherical decision surface (as the generalization of planar decision surfaces represented by the vanilla MLPs) is isometric (Melnyk et al., 2021), which allows for an O($3$)-equivariant construction of the steerable spherical neuron (Melnyk et al., 2022a), preserving the geometry of the input space. This was previously demonstrated only for $n=3$ and only for the input layer (Melnyk et al., 2022a,b).

   • In our manuscript, we extrapolate these ideas: Melnyk et al. 2022a show that the output of an O(3) layer is a subgroup of O(4), therefore again an isometry.

   • Thus, similar to the argument in Melnyk et al. 2021, a 4D sphere is more appropriate than hyperplanes. Setting an O(4)-equivariant layer on top produces an isometric representation in 5D. We show that we can do this in arbitrary dimensions.

2. Indeed, the proposed neurons are the generalization of the 3D neurons. The correctness of the generalization to $n$D, however, is non-trivial and requires the rigorous proof that we provide in the manuscript (Theorem 4).

3. • We focused on the single point case to verify our hypothesis, which we still believe is confirmed. This verification is necessary for any higher-order extension. If the absolute level of performance is considered to be the most important aspect, we can include our suggested two-point version (and add all the other details). For this, please see the “Low performance” part of the general comment as well as the results (the updated Figure 2 in the manuscript) for the two-point version of our model, showing the superiority of our method over all the models, including both VN and GVP, in the O(5) Convex Hulls task, as well as in the O(3) Action recognition task, including CGENN.

   • In relation to competing work, the experimental setup and results for O(5) Regression and O(5) Convex Hulls are used as-is from Ruhe et al. (2023), as we stated in Sections 4.2 and 4.3, as well as the general comment. We do not have access to the code of the specific models used in those experiments other than those provided in https://github.com/DavidRuhe/clifford-group-equivariant-neural-networks. We will, however, provide all the code for the experiments and models that we had access to and ran ourselves.

   • If the time before the rebuttal deadline allows and it is suggested to be necessary, we will try to conduct additional experiments for some of the competing models, which we will need to create ourselves (since no code is available for further experiments except for those we already included).

UPD: We have now conducted additional experiments with VN. We will include them and update the manuscript before the rebuttal deadline.

Please do let us know if there is anything we need to clarify further.

Thank you very much for your detailed feedback and assessment of our work! We appreciate that our contributions are accurately identified.

We first address the Weaknesses:

W1.

• Why our framework is superior:

  • as we state in the last paragraph on p. 1, “Similarly to the approach of Ruhe et al. (2023), Deep Equivariant Hyperspheres directly operate on the basis of the input points, not requiring constructing an alternative one, ..., which is a key limitation of many related methods (Anderson et al., 2019; Thomas et al., 2018; Fuchs et al., 2020).”, as well as the

  • inference speed: below we provide a comparison of the inference time of the models in the O(5) Convex Hulls task (the models available to us), for which we used the NVIDIA A40:

, where DEHs are our models (see the updated Figure 2).

• Including baselines in the experiments:

  • O(5) Regession and O(5) Convex hulls: the experimental setup and results are used as-is from Ruhe et al. (2023), as we stated in Sections 4.2 and 4.3, as well as the general comment. We do not have access to the code of the specific models used in those experiments other than those provided in https://github.com/DavidRuhe/clifford-group-equivariant-neural-networks. We will, however, provide all the code for the experiments and models that we had access to and ran ourselves. UPD We have conducted additional VN experiments, and will include them in the manuscript before the rebuttal deadline.

  • O(3) Action recognition: we have now included a permutation-invariant version of CGENN (please see the updated Figure 2).

• Theoretical comparison: in addition to the theoretical comparison presented in the last paragraph on p. 1 and the first paragraph on p. 2, we add that the EMLP is a representation method building equivariant feature maps by computing an integral over the respective group (which is intractable for continuous Lie groups and hence, requires coarse approximation); the E(n)-MLP version of EGNN in the comparison is similar to our method in that it operates on scalars: it updates the vector information by learning the parameters conditioned on scalar information and multiplying the vectors with it; the vector neurons, in contrast to our equivariant scheme, operate on latent features of each point in an equivariant manner, and the invariant features are obtained by taking the scalar product of the latent features per point; the vanilla MLP representing planar decision surfaces is, in fact, a particular case of spherical decision surfaces (Perwass et al., 2003; Melnyk et al., 2021), and thus, can be seen as a particular case of the proposed hyperspheres.

W2.

• In general, the presented model enjoys PointNet-like point-wise feature extraction (but with our equivariant hyperspheres). In addition to the architecture description in the second paragraph in Section 4.1, we will provide (UPD the illustrations in the updated manuscript before the rebuttal deadline) and the code, which should facilitate the reader’s understanding.
• We appreciate the suggestion of making our paper more self-contained. We will add more background to it and adjust the relative sections accordingly.
• The guiding intuition behind the hypersphere framework is as follows:
  - the relation between a point and a spherical decision surface (as the generalization of planar decision surfaces represented by the vanilla MLPs) is isometric (Melnyk et al., 2021), which allows for an O(3)-equivariant construction of the steerable spherical neuron (Melnyk et al., 2022a), preserving the geometry of the input space; - we extrapolate these ideas: Melnyk et al. 2022a shows that the output of an O(3) layer is a subgroup of O(4), therefore again an isometry. Thus, similar to the argument in Melnyk et al. 2021, a 4D sphere is more appropriate than hyperplanes. Setting an O(4)-equivariant layer on top produces an isometric representation in 5D. We show that we can do it in arbitrary dimensions.

W3. Experiments and practical applications for $n>3$:

• the choice of the experiments for $n>3$ is motivated by the related work (Finzi et al., 2021; Ruhe et al., 2023);

• a potential practical application for $n>3$ would be for the cases when 3D points (e.g., a point cloud) contain invariant features with $d$ dimensions (e.g., color or density information). Then one could consider the input signal to be $(n+d)$-dimensional and apply the O(n)-equivariant framework. However, there are no results or code for competing methods available for that case and the effort for re-writing the original code is too substantial to be done during the rebuttal.

W4. Please see the “Low performance” part of our general comment and the updated Figure 2.

Q2. We attribute the suboptimal performance not to the equivariant hyperspheres (i.e., the neurons in equation 13) themselves, but to the absence of higher-order interactions between the extracted equivariant features of the points:

• As indicated by the performance of the two-point version of our model (which we will add with all the details if advised; see the “Low performance” part of our general comment and Figure 2), which maintains the original equivariant feature extraction scheme, our model outperforms the related methods in the O(3) Action recognition and O(5) Convex Hulls tasks.

Q1. Given the time constraints, we can only provide an answer based on the experimental verification. The expressiveness of the original architecture is limited due to the absence of higher-order interaction modeling (see "Limitations" on p.9 and our answer to Q2). The two-point version of our model outperforms, among other models, GVP, which has been proven to be a universal function approximator for O(3) (Jing et al., 2021), and other expressive equivariant architectures like VN (Deng et al., 2021) and CGENN (Ruhe et al., 2023).

Q3. O(3) Action recognition experiment without transforming test set (“I”) vs. with O(3)-transformation (“O(3)") for the maximum number of training samples:

, where all the models are permutation-invariant, and DEHs are our models; please see the general comment.

(Since the models DEH_PI, DEH_$\Delta$_PI , and CGENN_PI produce O(n)-invariant predictions, the results are identical to those presented for randomly transformed test data in Figure 2 in our manuscript.)

Q4. We have now added the equivariant CGENN, which initially outperformed all other models in the two O(5) tasks (please see Figure 2), to the experiments. If the time before the rebuttal deadline allows, we will try to add another.

Q5. Relation to the irreducible representation of O$(n)$:

• we appreciate the interesting question! We are looking into it and will respond shortly.
• UPD: Our $**M**$-$***R***_{O}$ decomposition (14) appears to be unrelated to the Clebsch-Gordan decomposition, and hence, our method is not based on the obtained irreducible representations.

Q6. Please see the second bullet in our answer to W3.

We sincerely hope that our answers address the concerns. We are working on incorporating the suggested changes into the manuscript.

Please let us know if we need to provide additional clarification.

First of all, thank you for your positive review and assessment!

Your summary accurately lists our main contributions.

We begin by addressing the Weaknesses you pointed out:

• We learn a model consisting of deep equivariant hyperspheres $B(S)$ defined in equation (13): that is, each $B(S)$ is a matrix $\in \mathbb{R}^{(n+1)\times d}$ and has only one row $S \in \mathbb{R}^d$ as a learnable parameter vector – the rest adhere to the n-simplex construction. Therefore, we don’t need to copy the trainable spherical decision surfaces after training – it is already encapsulated in $B(S)$.

• It is not necessary to have training data aligned:
  (For the 3D case, it’s been demonstrated in “TetraSphere” by Melnyk et al. (2022), in which the first layer consists of 3D equivariant neurons $B(S)$.)

  • To demonstrate this in our case, we conducted the Action recognition experiment, in which we both trained the model on the transformed (“O(3)”) training data and tested it on the transformed test data (“O(3)”, as before), resulting in an O(3)/O(3) train/test augmentation setup (as compared to no-rotation augmentation during training, “I”, in the original experiment, I/O(3)).

  • Here are the test accuracies of our DEH_PI model (for the maximum number of training data samples):

• We train all the models end-to-end, using Adam optimizer. In this regard, our model is no different from the vanilla MLP or any other model in the experimental comparison presented in the paper.

Addressing your Questions:

• In general, the main limitation is the required compactness:
  • Our concept can be generalized, but with a finite number of basis functions, we can only cover a compact group (or compact interval of a non-compact group). O(n) (presumably even U(n)) is a very powerful group that covers many other groups as subgroups.

We sincerely hope that this clarifies our method. Please let us know should you have additional questions.

Thank you for the detailed review and positive assessment of our work!

We first address the Weaknesses:

1. We will update the manuscript and create a separate related work section.

2. Please have a look at the Limitations paragraph on p. 9 in our manuscript.

3. The major hyperparameters of our model are named in Sections 4.1, 4.2, and 4.3, and are summarized as follows:

• the depth and width of the model (just like in a vanilla MLP),

• the bias (True/False),

• sphere representation (normalized/non-normalized (default), see the last paragraph on p. 2),

• invariant feature computation (norm/sum).

  • Initially, we conducted only a quick hyperparameter search for each task. If the time before the rebuttal deadline allows, we will conduct an ablation study. To this end, what setting specifically is required to be experimented with?

4. Please see our general comment (the “Low performance” part), the updated Figure 2 in the manuscript, and our responses to the Questions below.

Q1. We appreciate the suggestion to reduce CGENN for a fairer comparison, but as an extension of our approach was requested by other reviewers anyway, we compare instead the original CGENN with our extended method modeling higher-order interactions --- the two-point scheme that we mention in our general comment. This add-on does not change the original equivariant feature extraction scheme. It allows our model to get significantly closer to CGENN in the O(5) regression task and outperform it in the two other tasks (see the updated Figure 2 in the manuscript).

Q2 and Q3. Please refer to our answer to Q1. The experimental setup and datasets for the O(5) Convex Hulls experiment (and O(5) regression) are borrowed from Ruhe et al. (2023), as we stated in Sections 4.2 and 4.3 as well as the second part of the general comment. We only have access to the results of the models, not the full training code, so we have no option to show the competing methods for bigger datasets (except CGENN).

Q4. The right-most value on the x-axis is the maximum number of training samples in that dataset. As we present in the updated Figure 2, our two-point model outperforms both MLP+Aug and CGENN.

Q5. As we addressed in the Limitations paragraph (p.9) in the manuscript,

"More specifically, in the proposed feature propagation, we did not model higher-order interaction between the equivariant features explicitly, which limits the expressiveness of our model". We have now addressed it in the general comment (please see the “Low performance” part).

Q6. Below we provide a comparison of the time complexity of the models in the O(5) Convex Hulls task (the models available to us):

, where DEHs are our models (see the updated Figure 2).

(For this, we used the NVIDIA A40.)

We sincerely hope this provides clarity and addresses the weaknesses and questions.

Please let us know should we need to provide any additional details or clarification.