On the utility of Equivariance and Symmetry Breaking in Deep learning architectures on point clouds

Dear reviewer, we thank you for your positive comments regarding the detailed background section and for recognizing the importance of the relationship between task complexity and equivariance. We also appreciate your constructive feedback and questions which we address as follows:

1. Regarding novelty: We agree that a better exposition of the novelty would improve the paper, as it may seem that we provide new theoretical results. Although we do not claim new theoretical results, we understand the confusion and will more explicitly state the experimental value of our paper.
   • a. We primarily organize the results that the equivariant deep learning field has developed over the years and put them in a thorough benchmarking structure based on theoretically motivated hypotheses. We thereby aim to constructively contribute to debates surrounding the utility of equivariance in DL. This is our main contribution and we will emphasize it as such in the main paper.
   • b. Additionally, as a contribution, we provide a new general-purpose architecture for point clouds, Rapidash, that allows for handling various forms of inputs (scalars and vectors) with multiple degrees of equivariance constraints.
   • c. To further clarify Eqs 8 and 9. These are used to describe our Rapidash model. These are common definitions of group convolutions, but specifically, we follow the approach taken by (Bekkers et al, 2024) as cited in the paragraph preceding the equations.
2. We thank you for pointing out the insufficient explanation of the benefits of symmetry breaking. We have added the following text to the paper:
   • a. From Tables 1, 2, and 3, we see that models that break SE3 symmetry perform well which supports the notion that having more information like a global pose, helps in performance despite not being exactly equivariant (Hypothesis 5). More explicitly, one can take a look at all the models that only have effective equivariance T3 or none, and compare them against their closest counterparts that have still SO(3) or SE(3) equivariance. For example in Table 1, one could compare model 7 (84.41%) to model 8 (85.60%). Both take normal vectors as input, but 8 additionally has access to a global pose which makes the model only conditionally invariant (the pose should rotate according to the point cloud). Or Compare model 7 (84.41%) to model 13 (85.41%), which both take the same amount of information as input, but 13 takes the normals as scalar inputs and this breaks invariance since then they are expressed in a global coordinate system. We will update the paper with additional details on symmetry breaking in the background section, and add an appendix with further clarifications. The results sections will more precisely refer to the results tables.

Dear reviewer, we thank you for the time you put into reviewing our work and appreciate the positive remarks about the paper writing, motivation, and theoretical insights presented in the paper. Additionally, thank you for your constructive criticism, we answer the concerns and questions below:

1. We agree with your comment about the lack of practical implications and thus are working towards doing classification experiments on the ModelNet40-C dataset as suggested. We hope that the insights derived from these experiments can translate well to the real world. We will add the validity of our hypothesis to the discussion section as soon as the results come in.
2. We will expand our discussion section with practical implications following the hypothesis. To give an example, on Shapenet segmentation the gain in group convolution methods is negligible, but requires ~5x more compute. Then it is advised to use simple and fast non-equivariant methods. However, if equivariance is required for a task, equivariant architectures may be essential. See e.g. QM9, but also see Shapenet segmentation results which heavily drop for non-equivariant models when the input is not aligned with the axis).
3. Regarding additional explainer figures, we will add one that explains the utility of symmetry breaking when tasks do not strictly require this (s.a. shapenet segmentation).

Answering the questions:

• Q1. Taking note of this, we are working towards having a table for ModelNet40 C classification experiments to have a dataset that can represent real-world point clouds more accurately.

• Q2. From the experiments, we plan to add the insights and validity of the presented hypotheses in the discussion section.

Dear reviewer, we appreciate the time taken to review our work and for carefully going through the text and giving constructive feedback. We are grateful for your feedback and will answer the concerns/questions mentioned in the review, below:

1. Limited discussion on symmetry breaking: We appreciate that you pointed out that the current version of the paper does not clearly emphasize the importance of symmetry breaking as a clarification would greatly improve the paper. We address this issue by adding details in an additional appendix and in the background section (an updated pdf will follow shortly):
   • a. In short, we now distinguish two types of symmetry breaking: internal and external. With internal we mean that if the layers are intrinsically equivariant (which ours are) then the layers themselves can be made non-equivariant (symmetry breaking) by relaxing equivariance constraints, recent literature explored such type of symmetry breaking [1, 2, 3]. By external we mean that architectures—that are intrinsically equivariant—can be broken to be non-equivariant by external factors, such as providing inputs that themselves are not invariants/covariants. For example, when providing input coordinates as triplets of scalars in a global reference frame, then this breaks equivariance as now we have an orientation/pose-dependent input. When providing coordinates as vectors, however, the inputs will internally be described relative to local reference frames (the orientation grids) and symmetry is preserved.
   • b. We further clarify that in contrast to related works on symmetry breaking, our study primarily focuses on external symmetry breaking (cf the red exclamation marks in the tables).
   • c. We will add an explanation to the main text, and include the following references for the context of internal symmetry breaking:
     • [1] Wang, R., Walters, R., & Yu, R. (2022, June). Approximately equivariant networks for imperfectly symmetric dynamics. In International Conference on Machine Learning (pp. 23078-23091). PMLR.
     • [2] van der Ouderaa, T., Romero, D. W., & van der Wilk, M. (2022). Relaxing equivariance constraints with non-stationary continuous filters. Advances in Neural Information Processing Systems, 35, 33818-33830
     • [3] Kim, H., Lee, H., Yang, H., & Lee, J. (2023, July). Regularizing towards soft equivariance under mixed symmetries. In International Conference on Machine Learning (pp. 16712-16727). PMLR.
2. Insufficient Evaluation of Model Complexity and Computation Costs: We are working towards classification experiments on ModelNet40-C and plan to add the findings to the discussion section. We hope that adds to the understanding of how these models perform on real-world-like data. Regarding computational complexity, we already reported epoch time per model, but now further discuss in the main text that for non-equivariant tasks (such as e.g. Shapenet segmentation), increased computational costs of group convolutional methods (~5x slower) do not outweigh performance gains. In strictly equivariant tasks (such as QM9), the performance improvement is rather significant in which case it would outweigh the costs.
3. Limited Benchmark Comparisons: The decision to add fewer benchmarks was purely made due to space constraints. To address this, we will keep a short list of reference methods in the current tables but additionally provide separate tables for an exhaustive comparison to the literature in the appendix. Additionally, we will cite the relevant papers in the main text associated with each experiment. We hope this answers your concerns.
4. Lack of Practical Application Discussion: We thank you for bringing this to our attention and we agree that the practical applications of this work are not thoroughly discussed. We will expand our discussion section and are currently running experiments on ModelNet40C to gain additional insights towards this end.

Dear reviewer, We thank you for the time and effort you have put into providing a detailed review of our work. We value your constructive feedback and try our best to address your concerns and questions below:

1. On prior work: We recognize this as an error on our part and thank the reviewer for listing important papers that we missed. We have updated our background section with suggested and additional citations, and added a short appendix that explains our view on symmetry breaking, see also our response to the reviewer regarding symmetry breaking [Wenf].
2. On the generalization of results beyond convolutional architectures: Thank you for raising this important point. We agree that demonstrating generalization across different architectures would strengthen our findings, and we appreciate the opportunity to clarify our approach and expand our discussion on this topic.
   • a. To systematically evaluate our hypotheses, we focused on a single architectural class that enables precise control over equivariance constraints. This allowed us to conduct detailed ablation studies across 58 model variations and five tasks, isolating the effects of varying equivariance constraints on model performance.
   • b. While the literature includes various equivariant architectures (from convolutional to transformer-based), Rapidash's unique flexibility in handling different forms of equivariance constraints was essential for our systematic study. Extending this analysis to other architectures, such as transformers, would require developing a new model class with similar flexibility - an interesting direction beyond our current scope.
   • c. We believe our findings about equivariance effects should in principle generalize across model classes since our ablations isolate fundamental properties of equivariance constraints. We’ll add a concise discussion to the paper, referencing several other works (including those on equivariant transformers) that suggest our findings might generalize, such as e.g. EquiformerV2 [1] which shows that fewer constraints (higher degree irreps) improve performance on equivariant tasks (our hypothesis 4). Also, that paper shows that for strictly equivariant tasks, symmetry breaking is not beneficial, which is something we also observe.
   • d. However, we are very interested in your perspective on this. Are there specific mechanisms that might cause different behaviors in other architectures that prevent generalization of our results? Either way, in the text, we will delineate the scope of our findings to be limited to the convolutional class.
   • e. Finally, to facilitate future work in the direction of generalizing our findings, we have provided our complete codebase and benchmark implementations in the supplementary materials (anonymized .zip file; upon acceptance, we will publish our code in a public repo). This enables direct verification and extension of our findings to other architectural classes.
3. Missing error bars: We agree this is necessary and we are currently running our models with multiple seeds and will update the tables and pdf shortly.

• Q1: We plan to update our paper with additional citations in the related work and discussion.
• Q2: See our response above. We consider doing analysis for different architecture types to be beyond the scope of this work and emphasize that these results hold within the convolutional neural network class paradigm. We agree that the generalizability of the hypotheses remains to be tested outside of the discussed architecture.
• Q3: We are currently collecting those results and they will be uploaded with the updated pdf in due time.

[1]: EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representations, Liao et al, ICLR 2024.