Response:

Dear reviewer, we thank you for your time and valuable comments, which we address in the following.

• On the use of element-wise activations: Regular group convolutions offer the benefit of having scalar feature field co-domains, permitting the use of any point-wise activation function without compromising equivariance [1, 2]. In contrast, steerable methods employ irreducible representations, which do not necessitate domain discretization [2]. However, these methods restrict the use of general point-wise activation functions, requiring specialized ones instead, which can be computationally expensive. This may additionally introduce discretization artifacts that disrupt exact equivariance, ultimately constraining the method’s expressivity [2].
• Regarding computational analysis, we have included such an analysis comparing our weight-sharing layer to a conventional group convolution of comparable size in the additional results, please consider point 5 in the general rebuttal, and Fig. 2 and 3 in the attached PDF.
• Regarding our empirical results and the usefulness of our method: we present additional experiments that demonstrate our method matches the performance of an unconstrained CNN with the same amount of effective but free kernels (128-channels) at a lower parameter budget (please also consider point 4 in the general rebuttal). In addition, we can outperform group convolutional methods for the discovery of partial and unknown symmetries. We also address this in points 1 and 2 of the general rebuttal, and Fig. 1a, 1b and 4 in the attached PDF. We hope this sufficiently addresses your concerns about experimental validation.
• If we initialize the weight-sharing schemes as permutation matrices implementing a specific group transformation, we indeed recover a standard group convolution. A fair comparison of baselines would be comparing to methods that use weight-sharing for images, such as reference [31]. However, they report difficulties with end-to-end training and require meta-learning, making a direct comparison with our approach less straightforward. Following your comments, we have added the row-normalized baseline from [30] on rotated MNIST in the additional results (please consider point 3 in the general rebuttal, and Table 2 in the attached PDF). Visual inspection of the weight-sharing schemes seem to imply it is less effective in picking up meaningful group structures.

Regarding your explicit questions:

• (Question 1) Regarding the learned representations in the CIFAR-10 experiments, we highlight that our model operates without applying specific algebraic or group constraints to the learned representations, resulting in a more abstract and adaptable weight-sharing scheme. As a result, the patterns in weight-sharing are not as readily interpretable through traditional group-theoretic approaches, and it also makes direct visual interpretation of these patterns more nuanced.
• (Question 2) While we do not put any explicit regularization on the diversity of the learned representations, we observed in our experiments that our method learns distinct weight-sharing patterns.

We thank you for your insightful comments and would appreciate you considering raising your score if you believe they have been addressed sufficiently, or follow up with any questions we may help clarify further.

References

[1] General E(2)-Equivariant Steerable CNNs. Weiler et al. NeurIPS 2019

[2] Fast, Expressive SE(n) Equivariant Networks through Weight-Sharing in Position-Orientation Space. Bekkers et al. ICLR 2024

[30] Equivariance Discovery by Learned Parameter-Sharing. Raymond A. Yeh, Yuan-Ting Hu, Mark Hasegawa-Johnson, Alexander Schwing. AISTATS 2022

[31] Meta-Learning Symmetries by Reparameterization. Allan Zhou, Tom Knowles and Chelsea Finn. ICLR 2021

Dear reviewer, we thank you for your time and comments, which we address in the following.

Regarding the motivation for double stochasticity and its comparison to prior works, we point out that there are key benefits about employing double stochasticity over single stochasticity:

• Theoretical motivation: We establish a link between regular representations which forms the basis of group convolution actions, and their manifestation as soft permutation matrices. Using double stochastic matrices allows for an analysis of learned representations through the lens of partial equivariance, a connection we elaborate on in Section 4.2. We are happy to motivate this in more detail in the camera-ready version.
• Practical Implications: As outlined in Section 4.2, our methodology employs doubly stochastic matrices for weight-sharing, which facilitates direct calculation of expectations from the implicitly learned distribution. This setup avoids the need for Monte Carlo approximations and is adaptable to unrecognized symmetries that may not conform to standard group structures. This approach to handling partial equivariance offers a novel angle compared to traditional symmetry discovery methods, such as those proposed by Yeh et al. [30] and Zhou et al. [31].

Hence, we propose that there is a well-grounded preference for employing double stochasticity over single stochasticity in the context of group convolutions. Following your comments, we have added the row-normalized baseline from [30] on rotated MNIST in the additional results (please consider point 3 in the general rebuttal, and Table 2 in the attached PDF).

• As per author guidelines on dual submissions (section on Dual Submission, https://neurips.cc/Conferences/2024/CallForPapers) it states that “Papers previously presented at workshops are permitted, so long as they did not appear in a conference proceedings (e.g., CVPR proceedings), a journal or a book.” The work referenced in your review was submitted as an extended abstract at the ICML workshop “Geometry-grounded Representation Learning and Generative Modeling” (https://gram-workshop.github.io/cfp.html) which is non-archival, and hence does not violate dual submission policy.
• Regarding the stated limitation, we appreciate the remark but do not focus on initial and boundary conditions as we do not use PDEs as a use case for our method in this work. If we were to extend our proposed method to PDE solving, we would require updating our architecture to accommodate boundary conditions and initial conditions to solve any PDEs efficiently. As far as the scope of this current work is concerned, we learn the symmetries or partial symmetries directly from the data through weight-sharing schemes, which do not require explicitly defining any boundaries.

We thank you for your insightful comments and would appreciate you considering raising your score if you believe they have been addressed sufficiently, or follow up with any questions we may help clarify further.

References

[30] Equivariance Discovery by Learned Parameter-Sharing. Raymond A. Yeh, Yuan-Ting Hu, Mark Hasegawa-Johnson, Alexander Schwing. AISTATS 2022

[31] Meta-Learning Symmetries by Reparameterization. Allan Zhou, Tom Knowles and Chelsea Finn. ICLR 2021

Dear reviewer uLBU, we believe to have addressed any weaknesses raised in your original review, an acknowledgment or response to our rebuttal would be much appreciated. We are very much open to address any concerns in the remaining discussion period.

Dear reviewer, we thank you for your time and valuable comments, which we address in the following.

Regarding employed regularizers, we acknowledge that the motivation could have been more explicitly mentioned, and we provide some added details:

• Entropy Regularizer: The primary motivation for using an entropy regularizer is to encourage sparsity in our weight-sharing schemes, which helps the matrices approximate true permutation matrices rather than “simply” satisfying double stochasticity. This approach stems from our initial intuition that weight-sharing schemes should mimic soft permutation matrices. The effectiveness of the induced sparsity depends on the specific group transformations relevant to the task. For example, $C_4$ rotations are typically represented by exact permutation matrices. In contrast, $C_N$ rotations or scale transformations might require interpolation, thus aligning more closely with soft permutation matrices. Our experimental results indicate that the utility of this regularizer varies with the underlying transformations in the data. For instance, as anticipated it is of lesser use for scale transformations in the MNIST dataset.
• Normalization Regularizer: Empirically, we have found the normalization regularizer essential for reducing the number of iterations needed by the Sinkhorn operator to ensure the matrices are row and column-normalized. Without this regularizer, the tensors either fail to achieve double stochasticity or require an excessively high number of Sinkhorn iterations to do so.

We will outline our motivation behind these two regularizers and the impact of varying $\lambda$ values for the entropy regularizer more explicitly in the camera-ready version.

• Regarding concerns about scalability: while we agree that quadratic scaling in $|X|$ can be impractical, potentially leading to polynomial $O(s^2)$ scaling in terms of image width $s$, our method exploits the fact that convolutional kernels have limited localized supports in practice. Thus, for a kernel size $k$ we scale with $|X| = k^2$ rather than the entire signal domain. This approach significantly reduces computational demands and makes our approach practically scalable. Please also consider point 5 in the general rebuttal for this aspect.
• Regarding the interpretation of the weight-sharing patterns learned on CIFAR-10, it is important to note that our model does not impose algebraic or fixed group constraints on the learned representations. This design choice enhances the flexibility and abstraction of the weight-sharing schemes. As a result, the weight-sharing patterns may not be easily interpreted using conventional group-theoretic approaches, which could affect the direct visualization clarity of the learned weight-sharing.

We thank you for your insightful comments and would appreciate you considering raising your score if you believe they have been addressed sufficiently, or follow up with any questions we may help clarify further.

Dear reviewer 3Y5V, we believe to have addressed any weaknesses raised in your original review, an acknowledgment or response to our rebuttal would be much appreciated. We are very much open to address any concerns in the remaining discussion period.

Response: Dear reviewer, we thank you for your constructive feedback and engaged questions, which we address in the following.

• We thank the reviewer for pointing out the connection of our proposed method to low-rank approaches. It is possible to make our doubly stochastic matrices converge to permutation matrices with an appropriate regularization technique. In those cases we obtain sparse matrices that could be represented via low-rank approximation methods. However, we note that the degree of regularization is task- and domain-specific.
• Regarding experimental claims, our claim is indeed that the inductive bias introduced by equivariance is beneficial, and can furthermore be discovered automatically [Ref Fig 1 a,b and Fig 4 in the PDF] For natural images, the bias of translation equivariance may be sufficient in many cases, in particular when enough channels are present. This is not the case for other datasets, where the data itself may contain group symmetries beyond translation, as shown by [20, 27]. Our main claim is the ability to uncover such symmetry groups and employ them in different downstream tasks. While a look-up table may match the existence of an a priori specified symmetry group, this is not the case for partial symmetries or incomplete ones. In such cases, we demonstrate that we can discover the symmetry in the sense of a useable shared representation without requiring to point out the exact subgroup (if such a subgroup even exists).
• We acknowledge that conventional CNNs with an equivalent channel count inherently possess a higher expressive capacity due to a lack of kernel constraints. Therefore, we have opted to compare overall parameter counts rather than the number of kernels, a more comprehensive assessment in our opinion. For completeness, we have additionally measured the performance for an unconstrained 128-channel CNN (please consider point 4 in the general rebuttal).
• We thank the reviewer for the suggestion on experimenting with the low-data regime. We agree that the size of the dataset plays an important role, and we have added the suggested experiments for both cases of full and partial symmetry discovery (please consider point 2 in the general rebuttal and Fig. 2 in the attached PDF).
• Regarding additional literature review, we will expand the related works section and further clarify the position of our work within the literature for the camera-ready version. We would like to stress that we do not explicitly impose any group structure in our representation stack, since that would defeat our goal of symmetry learning and enables us to potentially learn a weight sharing scheme that violates some of the group axioms if the group symmetry is not present in the data. Therefore, we refrain from making any theoretical assumptions that make use of more complex statistical group theory results. We agree that we could have been more precise regarding these claims in the paper, and will clarify this. Our method primarily aims to learn symmetries from data explicitly through a weight sharing scheme which may or may not be an exact group symmetric structure (if the data does not possess a full group structure). Then, we implicitly assign several probability measures of finite support over this learned symmetry scheme. We emphasize that our method is therefore more flexible than a strict group equivariant network due to its ability to learn partial symmetries.

Regarding your explicit questions:

• (Question 1) Yes, this parameterization can represent arbitrary group equivariances for compact Lie groups. For strict or partial equivariance, we employ a permutation matrix.
• (Question 2) We hope that our first point in the response above addressed this question.
• (Question 3) In the low-data regime, the model might overfit to a limited set of group transformations. This can be mitigated with a proper regularizer. We would argue that this is an advantage for learning partial or incomplete symmetries, as we make no assumptions on completeness or uniform distribution over group transformation in the data. In those cases, strict equivariance can hurt performance and may also be an overconfident bias lacking the data evidence to support it. On the other hand, our method allows us to address this issue by learning directly from any symmetries that are present in the data, without the need of such assumptions on, e.g., completeness of a group structure or uniform distribution over group transformation.
• (Question 4) Zhou et al. [31] essentially learn a parameter sharing scheme without constraints on the sharing pattern except norm regularization. However, they report that they require meta-learning to pick up on symmetries, while our method does not. We presume that the added constraint on double stochasticity is beneficial in that regard.
• (Question 5) To briefly summarize in two points: (1) The model can recover group convolutional structures when clear symmetries, including partial symmetries, are present in the data (see our experiments on Rotated MNIST); and (2) The model can pick up on useful weight sharing patterns when there are no a priori known symmetry structures. This is shown in our CIFAR-10 experiments, where we match the CNN’s model performance and outperform a model with pre-specified group equivariance. This is additionally underlined in Point 1, 2 and 4 in the general response.

We thank you for your insightful comments and would appreciate you considering raising your score if you believe they have been addressed sufficiently, or follow up with any questions we may help clarify further.

References

[20] Learning equivariances and partial equivariances from data. David W. Romero and Suhas Lohit. NeurIPS 2022

[27] Approximately equivariant networks for imperfectly symmetric dynamics. Rui Wang, Robin Walters, Rose Yu. ICML 2022

[31] Meta-Learning Symmetries by Reparameterization. Allan Zhou, Tom Knowles and Chelsea Finn. ICLR 2021

Dear reviewer 1myY, we believe to have addressed any weaknesses raised in your original review, an acknowledgment or response to our rebuttal would be much appreciated. We are very much open to address any concerns in the remaining discussion period.

We acknowledge that conventional CNNs with an equivalent channel count inherently possess a higher expressive capacity due to a lack of kernel constraints. Therefore, we have opted to compare overall parameter counts rather than the number of kernels, a more comprehensive assessment in our opinion. For completeness, we have additionally measured the performance for an unconstrained 128-channel CNN (please consider point 4 in the general rebuttal).

This is a really helpful addition. Interesting, so the main claim now is that for a parameter-constrained model, you obtain better performance? Size of the model clearly does come into this, as the CNN also gets better as more filters are added. This is fine, and a clear claim. However this does mean that the current experiments only show that the invariance learning properties work in the underfitting regime! This is different from what is wanted from learning equivariances, where we may want to show the ability to obtain improved performance, once other simpler methods like making the model larger stop working.

This is why other methods e.g. [31, 26] consider second order information (meta-learning, marginal likelihood approximations): to distinguish between inductive biases even when the training loss cannot. This was also discussed in [*] and back in 2018 [**]. This is what the low data experiment could have shown, if it was verified that the model was large enough to sufficiently fit the training data. Also the low-data experiment is of limited usefulness, since it does not contain a baseline of a non-invariant model. The interesting question here is to see how much of an improvement you can get by learning invariances over a model that cannot do this. (Also relevant to your answer to my question 4.)

We would like to stress that we do not explicitly impose any group structure in our representation stack

This was clear from the paper, and is certainly valuable! However, I don't see how this makes the literature I pointed to any less relevant?

Overall

Either way, while I don't think this is the end of the question of how to learn equivariance automatically, I do believe it is an interesting paper, and I will continue to argue for acceptance.

[*] Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximations 2022 (not cited in the paper)

[**] Learning Invariances using the Marginal Likelihood 2018 (not cited in the paper)