Probing Equivariance and Symmetry Breaking in Convolutional Networks

Dear reviewer, we sincerely thank you for the time and effort you put into providing feedback on our work. We deeply appreciate your insightful questions and respond to the concerns and questions below:

Data augmentation [W1]

Thank you for pointing out that we do not emphasize in the main paper that the experiments are performed with augmentations. We will update the main paper with details on augmentations. All models for equivariant tasks are trained with SO(3) augmentation. For ShapeNet segmentation, we use standard augmentation (20° z-rotations + scaling), as full SO(3) augmentation hurts in-distribution performance. In the code added as supplementary material, we have config files with the augmentation parameter (set as true) for equivariant tasks, which can be used to easily reproduce the results in the paper.

Alternate architectures [W2]

Generally, for this work, we refrain from further comparison with additional model classes, as we believe our findings about equivariance effects should, *in principle, generalize across model classes since our ablations isolate fundamental properties of equivariance constraints. Only within a single architectural class can we do proper ablation studies, and thus, we can attribute performance gains/drops to changes in equivariance constraints. If we want to explore whether or not the results generalize to other architectural classes, we have to develop such a new class of models in addition to our already proposed flexible architecture (Rapidash) and redo the entire study (which includes 58 model variations across 5 tasks). We consider this to be beyond the scope of the current paper and decide to emphasize that these results hold within the convolutional neural network class (both in $\mathbb{R}^3$ and $\mathbb{R}^3 \times S^2$) paradigm. Related works like [1], answer a few questions about the role of equivariance with a transformer-based model and share similar trends as our findings in scaling models for equivariant tasks.

Response to Q1: Augmentations

Yes, the presented results are with augmentations. We will add results without augmentations in the Appendix in our updated version. Also see W1.

Response to Q2: Alternate architectures

Related works like [1] answer a few questions about the role of equivariance with a transformer-based analysis and share similar trends as our results in scaling models for equivariant tasks. We believe the trends of our findings hold true across the mentioned architectures, ESCNN and EGNN, although it can be said with certainty only with a detailed analysis of each model class. Having extensive analysis for each available equivariant architecture/class would be very useful for the community. We leave that as future work.

We once again thank you for taking the time to review our work. We believe the results in this paper offer relevant insights for both theory and practice in convolutional architectures, and we would appreciate your reconsideration of the evaluation in light of these clarifications. If any concerns remain from your initial evaluation, we’d be happy to clarify further.

Reference:

[1] Does equivariance matter at scale?, Brehmer et al.

Dear reviewer,

We thank you again for a detailed and constructive review of our work. We would like to kindly invite you to take a look at our rebuttal, in which we have addressed each of your comments thoroughly. In particular:

• Added data-augmentation in our discussion, as well as clarified that the experiments have data-augmentation.
• Motivated our focus on a single architecture class for this work and added discussion about additional architectures.

We think your feedback has helped us improve our work, and we thank you again for your valuable time and feedback! Kindly let us know if you have any further questions or concerns.

Dear reviewer, could you please spend some time discussing with the authors or commenting on their responses? This paper receives diverging scores, and it would be helpful to have more discussions. To note, in this year's NeurIPS, it is required to have reviewer-author discussions before submitting the mandatory acknowledgement. Thanks!

Dear reviewer,

Thank you for your thoughtful engagement with our work. We appreciate your feedback and address your concerns below.

Q1

We will add experiments for non-equivariant models without augmentations. Regarding T3 augmentations, we did not see them necessary for QM9 datasets, as we work on the Center of Mass frame for the entire task. Regarding CMU motion and ShapeNet, we agree with the reviewer and will add experiments with T3 augmentations in addition to the SO(3).

Q2

We agree with the reviewer on the focus of the paper on point clouds ( also see our response to reviewer KUZW. In the updated version of the paper, we will make sure the scope of the paper is clear both in the introduction as well as the abstract. Regarding the point about G-convolutions, we thoroughly investigate SE(3) convolutions both in $\mathbb{R} ^3$ as well as $\mathbb{R}^3 \times S^2$ across various datasets and tasks. Additionally, we also provide a flexible architecture that allows for fair comparisons across different equivariance as well as symmetry breaking.

Additional points: The authors do not perform an extensive capacity study of the models in the paper, which can have a significant effect on the outcome of results - e.g. see Figure 4 in [1].

Could you please let us know what additional experiments you propose? It would be great to add it to the updated version.

It still remains unclear to me what the exact contributions of this paper are. The architecture proposed is mostly derived form prior work. There are then 5 hypotheses investigated in a short amount of space, without a significant common thread between these hypotheses beyond equivariance, and in my opinion with not enough data to draw broad conclusions that the paper claims. I believe the paper would be better suited to focus on a smaller number of related questions, or take the time and space to investigate each of the questions properly, such as [1]. [1] Does equivariance matter at scale?, Brehmer et al.

The main contributions of the paper are the empirical study answering important research questions to understand the utility of the equivariance, a theoretical formulation to understand symmetry breaking through additional information, as well as a flexible architecture that allows for comparison across different equivariance and symmetry breaking. This is not possible in the prior work [1] as it is limited to fixed SE(3)/T3 equivariance and cannot handle larger point clouds.

We would like to politely point out to the reviewer that while [2] is a great contribution in understanding the role of equivariance in scaling data and scaling models, as the title suggests, they do not consider other important aspects of model design while asking this question. They also present limited experiments and a lack of variety of tasks.

We are working towards adding a larger dataset, Cosmobench [3], as suggested by reviewer KUZW, which we believe to be beneficial in extending the insights to large point clouds.

We would like to thank the reviewer again for the useful feedback, which has helped us improve our work. We thank you again for your valuable time. Kindly let us know if you have any further questions or concerns.

References:

[1] Fast, Expressive SE3 Equivariant Networks through Weight-Sharing in Position-Orientation Space, Bekkers et al

[2] Does equivariance matter at scale?, Brehmer et al.

[3] CosmoBench: A Multiscale, Multiview, Multitask Cosmology Benchmark for Geometric Deep Learning, Huang et al.

Dear reviewer,

We thank you for the time and effort you have put into providing a review of our work. We sincerely appreciate your thoughtful feedback. We are grateful that you find our work an admirable job in constructing a unifying framework. We appreciate your kind words about the background as well as the theoretical results.

In the text below, we try to address your concerns. We appreciate the careful suggestions that have helped improve our work and will make the changes in the updated version.

Scope of the paper

• We agree with the reviewer that the focus of our work is on point clouds, and we will emphasize this in the introduction as well as in the contribution list. We will revise the wording in the paper to emphasize the point clouds aspect. Thank you for pointing us to [1], and will add this to the related work/discussion to share how some insights do not translate across fields, i.e, images to point clouds.
• In the scope of this paper, we do not tackle regularization. We will add this to our limitations and consider this in future works to understand the role of regularization in equivariance learning.
• We would like to point out that Rapidash architecture, unlike PONITA [1], allows for processing a larger number of points per sample. Due to various different input options, Rapidash is a flexible architecture that allows for detailed empirical analysis of different group equivariance as well as symmetry/equivariance breaking. In the updated version of the paper, we will add how Rapidash differs from [1] explicitly.

Connecting research questions presented in the paper

The five core research questions in our work focus on kernel constraints, representation capacity, dataset size, symmetry, and equivariance breaking. We think these are crucial when it comes to understanding the role of equivariance.

• RQ1 addresses the role of constraining the kernel and where that helps or hurts model performance.

• RQ2 and RQ3 help understand if increased model size or increased dataset size can help learn equivariance.

• RQ4 and RQ5 connect to understand how to incorporate more information in data, especially if that leads to symmetry breaking or equivariance breaking, and which of these can be useful for better performance.

Presentation

• We fully agree with the reviewer that the detail of Rapidash should be in the main text. Due to lack of space, the details of the Rapidash architecture are only discussed in the Appendix. We will incorporate the suggestion and add it to the main text of the paper.
• We will rewrite parts of the introduction to better describe the related works in a way that is more focused on the theme of the work.
• We will also provide the necessary background in Section 2 to help understand both the theoretical contribution and research questions cohesively.

Clarification on theoretical contributions

Proposition 3.2 [main text], Corollary 3.1 [main text], Proposition D.1 [Appendix], and Proposition D.2 [Appendix] are the theoretical contributions in this work, while the rest of the propositions/corollaries are presented from earlier works for completeness.

Choice of datasets

With the focus of the paper being on point clouds, we picked the most commonly benchmarked small molecule dataset QM9 (with generation/property prediction task). For adding a different task, learning dynamics, we picked CMU Motion dataset. Given that it is a very small dataset, we wanted to understand the role of equivariance in a small data regime. For the rest, we pick the commonly used standard pointcloud datasets, i.e., ShapeNet3D (for segmentation and generation tasks) and ModelNet40 for classification. These datasets collectively give a good range of tasks along with additional inputs (i.e. global pose, global frame, etc ) allowing for different types of symmetry/equivariance breaking on point cloud datasets of varied size.

Regarding large-scale datasets

In response to the point raised regarding large-scale datasets, we are working towards including the COSMO-bench dataset [1] (comprising of 5000 point clouds, Quijote, coarse-grained per sample as well as a physics baseline in addition to a GNN baseline for velocity prediction tasks for ease of comparison). We are working on having Rapidash + variants for prediction tasks and will add the results in the updated version of the paper. If the reviewer has other suggestions, kindly let us know.

Response to minor comments

• We will update our discussion to include [2] as well as other works in the position-orientation space.
• We will update the AlphaFold version in the introduction and weight-sharing citation.
• With regards to the figure, we will add the split version of Figure 2 to the Appendix.

We once again thank you for taking the time to review our work, and we would appreciate your reconsideration of the evaluation in light of these clarifications. If any concerns remain from your initial evaluation, we’d be happy to clarify further.

References:

[1] Fast, Expressive SE(n) Equivariant Networks through Weight-Sharing in Position-Orientation Space, Bekkers et al.

[2] The Lie Derivative for Measuring Learned Equivariance, Gruver & Finzi et al.

[3] CosmoBench: A Multiscale, Multiview, Multitask Cosmology Benchmark for Geometric Deep Learning, Huang et al.

Dear reviewer,

We thank you again for a detailed and constructive review of our work. We deeply appreciate the time and expertise you invested in evaluating our work. We would like to kindly invite you to take a look at our rebuttal, in which we have addressed each of your comments thoroughly. In particular:

• Re-defined the scope of our work to focus on point clouds.
• Clarified our theoretical contributions.
• Connected the presented research questions.
• We are currently working towards adding a large-scale dataset and will add those results in the Appendix.
• Answered your concerns regarding augmentations as well as made changes to the presentation of our work.

We think your feedback has helped us improve our work, and we thank you again for your valuable time and feedback! Kindly let us know if you have any further questions or concerns.

Dear reviewer,

Thank you for your thoughtful engagement with our work. We appreciate your feedback and address your concerns below.

Scope of the paper: If the paper is now specific to point clouds and doesn't transfer...

We will update the paper's abstract as well as introduction (as previously mentioned) to clarify the scope of the paper to point clouds. (Not sure if we are allowed to change the title after submission, regardless, we will ensure this comes through.) Regarding unifying self-contradictory literature, our goal is not to contrast equivariant and non-equivariant models in isolation, but also to consider relevant symmetry breaking and the task at hand before making a selection; for this, the presented architecture will serve as a useful tool. With regards to images, for instance, we would like to emphasize [2], which highlights the lack of genuine invariance learning while also focusing on true local equivariance loss in various models as given in [1]. We would like to highlight that there is still much to uncover about the inductive biases of model classes like transformers, and that could play a role in the results presented in [1, 5].

Regarding not tackling regularization..

While responding to your questions about regularization, we particularly meant explicit regularization loss like that in [3,4], which we have not explored. To best of our knowledge, there is no general equivariant regularization loss (if there is one, and we have missed that, please do suggest so we can add this to our model in the future), so we focused only on weight decay. We tune weight decay as part of the hyperparameter optimization, and we did not find a significant impact on either model class (equivariant or not). For our experiments, we have weight decay as 1e-8 for QM9 and 1e-12 for the rest. We observe that weight decay does not compensate for a potential loss of equivariance.

IMO, all of these questions could constitute independent papers with deeper analyses instead of the (necessary due to the conference format) more superficial analyses....

We politely disagree with the reviewer on our work as superficial analysis. We think this work provides useful insights about equivariance and symmetry breaking for point clouds while giving a unified architecture that allows for a fair comparison. As noted in the rebuttal, we will update our background section to have a better flow, including a clearer introduction for Lawrence et al. We consider the proposed research questions under the broader question about the utility of equivariance (in models for point clouds), and while doing so, we capture aspects of building equivariant models through kernel constraints, strictness of equivariance, as well as symmetry-breaking aspects. We’ll also provide a more cohesive overview of relevant work on equivariance learning. As suggested by reviewer mQdo, we will add explicit takeaways (see our response to mQdo).

Large-scale datasets: I appreciate the proposed experiments on COSMO-bench, but I don't know what to make of this argument if the results on that dataset aren't presented in the discussion period...

If the results of large-scale datasets do not show the same trends as the datasets presented in the paper, we believe that this will give us insight into how learning equivariance can be dependent on the scale of the dataset. So either way, we believe it would be useful. From our preliminary results, we see that the equivariant model outperforms the given (non-equivariant) baseline from the Cosmo-bench [6], and we hope to include all the Rapidash variants in the final version of the paper, as this is not possible in the short discussion period.

Misc:

Apologies for not highlighting our response to reviewer mQdo previously.

All models for equivariant tasks are trained with SO(3) augmentation. For ShapeNet segmentation, we use standard augmentation (20° z-rotations + scaling), as full SO(3) augmentation hurts in-distribution performance. In the code added as supplementary material, we have config files with the augmentation parameter (set as true) for equivariant tasks, which can be used to easily reproduce the results in the paper.

We thank you again for your valuable time and feedback! Kindly let us know if you have any further questions or concerns.

References:

[1] The Lie Derivative for Measuring Learned Equivariance, Gruver and Finzi et al.

[2] On genuine invariance learning without weight-tying, Moskalev et al.

[3] Latent Space Symmetry Discovery, Yang et al.

[4] Improving Transformation Invariance in Contrastive Representation Learning, Foster et al.

[5] The Importance of Being Scalable: Improving the Speed and Accuracy of Neural Network Interatomic Potentials Across Chemical Domains, Qu et al.

[6] CosmoBench: A Multiscale, Multiview, Multitask Cosmology Benchmark for Geometric Deep Learning, Huang et al.

Dear reviewer, we thank you for the time and effort you have put into providing a review of our work.

Firstly, we sincerely thank you for your kind words and appreciation of our work. We are grateful that you found the paper to be well-written and recognized the extensive study spanning theoretical analysis, empirical evaluation, and practical architecture . We especially appreciate your recognition of our proposed architecture, Rapidash, as a concrete and useful contribution for understanding geometric priors in deep learning. We hope the community finds our work insightful as well.

In the text below, we address questions and weaknesses/limitations.

Train error vs test error [W1, L1]

We would like to mention that for all tasks, we train with CosineAnnealing and select the best checkpoint based on validation performance. For ShapeNet, which lacks a standard validation set, we use the final model; no overfitting was observed. As we consider the validation error to decide the best model, we did not report the val (nor train) error, but only the error on the designated test set.

Readability of large tables/ Takeaways [W2, L2]

We appreciate your feedback regarding the readability of large tables. In the revised version, we will:

• Add split (smaller) versions of the large tables in the Appendix for improved readability.

• Include clearer takeaway messages (as shown below) in the discussion section to enhance accessibility:

  • The performance gap between equivariant and non-equivariant models increases with geometric task complexity. *For simpler tasks like classification and regression, using non-equivariant models performs somewhat similarly to equivariant models.
  • Symmetry breaking, for example, adding positional information, can be useful and give a performance boost.
  • For equivariant tasks like molecular generation, using equivariant models is important, as non-equivariant models perform poorly.

Response to Q1

For the CMU dataset, we report test MSE loss and use this to compare models. We will add a table with train and test errors for the CMU dataset in the Appendix in the updated version.

Response to Q2

We mention ‘the limits of their architecturally defined hypothesis space for geometric expressivity’ with respect to Table 3 for inflated models vs regular models segmentation task on ShapeNet (both rotated, non-rotated samples). We indeed see that making the models larger does not improve test error and will report that in the updated version of the paper.

Once again, we thank the reviewer for the insightful feedback. If any concerns remain from your initial evaluation, we’d be happy to clarify further.

Dear reviewer mQdo, thanks for your time reviewing this paper. Although your opinion might be clear, according to this year's NeurIPS policy, it is required to have reviewer-author discussions before submitting the mandatory acknowledgement. Could you please take some time to comment on the authors' responses? Thanks!

Dear reviewer, We thank you for the time and effort you have put into providing a review of our work. We sincerely thank you for your thoughtful and encouraging feedback. We are glad that you found the topic of the paper to be timely and relevant to ongoing discussions in the geometric deep learning community, particularly regarding the understanding of the benefits and trade-offs of equivariant networks. Your comments on the flexibility of the proposed architecture are greatly valued. We appreciate that you found our experimental evaluation extensive.

In the text below, we try to address your concerns and respond to the questions:

Data Augmentation [W1]

We would like to emphasise that all model variations for tasks that are supposedly equivariant have rotation augmentations for training- we will explicitly mention this in the experiments section.

Equivariance beyond E3/SE3 [W2]

The reviewer raises an important point about understanding the effects of equivariance beyond SE(3) and its subgroups. We would like to first justify our choices made in the paper by mentioning that we focus on standard point clouds in the real world that have mainly E(3) symmetries and permutation symmetries. Hence, we primarily end up focusing on E(3)/SE(3) and its subgroups.

Response to Q1: Role of augmentation

Generally, models without equivariant layers are trained with augmentations. So, we train with SO(3) augmentation for all models on tasks that are equivariant. For ShapeNet segmentation, we use standard augmentation (20° z-rotations + scaling), as full SO(3) augmentation hurts in-distribution performance. This helps us understand if models improve performance with augmentation, and they do. (This effect is less in equivariant models than in non-equivariant models.) We don’t see equivariance and augmentations as exclusive but complementary. With regards to symmetry breaking and augmentations, the non-equivariant method generally benefits from augmentations.

Response to Q2: Generalization to other groups

See W2. The reviewer raises an important point about the effects of equivariance and symmetry breaking for groups apart from SE(3) and its subgroups. We believe this is an important question that we aim to address in future work. As far as the usefulness of equivariance goes, as long as the task is equivariant, having equivariant layers is an effective way to learn symmetries for any group. Although the performance gap between non-equivariant and different equivariant groups depends on the complexity of the group [1].

Once again, we thank the reviewer for the insightful feedback. If any concerns remain from your initial evaluation, we’d be happy to clarify further.

References:

[1] On the hardness of learning under symmetries, Kiani et al.

Dear reviewer Bpqw, thanks for your time reviewing this paper. Although your opinion might be clear, according to this year's NeurIPS policy, it is required to have reviewer-author discussions before submitting the mandatory acknowledgement. Could you please take some time to comment on the authors' responses? Thanks!