A Characterization Theorem for Equivariant Networks with Point-wise Activations

We thank the reviewer for the positive review, appreciation, and constructive feedback.

Response to weaknesses:

The paper is rather dense in representation theoretic notions. While I believe the paper is still self-contained, it may be beneficial for readers if the authors can expand on some intuition on the main theorems, maybe even in the appendix.

Regarding the first point, we are currently enhancing the paper's readability and comprehension for a broader audience. In particular, we will incorporate examples and strengthen the appendices.

It doesn’t look like the technique in this paper can generalize to infinite dimensional representation, which the author acknowledged in Page 4.

Concerning the issue of infinite-dimensional representations: these play a significant role in the current theoretical discourse on equivariant models and our framework does not cover this approach. We acknowledge that this is one of the main limitations of our work, and we will address the matter in the manuscript with more precision. It is however remarkable to note that, while the descriptive models employed generally involve infinite-dimensional representations and concepts of harmonic analysis, the final deployed model is essentially a discretization of their theoretical formulation. In certain instances [1, 2], these models maintain exact equivariance solely through finite-dimensional representations and finite groups, conditions addressed in our framework as mentioned on page 4, though perhaps too briefly.

Minor: I think the paper would also greatly benefit from a further discussion of "what's next?", what is the loss if we give up point-wise activations for rotational equivariant network but replacing them with other kinds of activation.

To address another problem raised by the reviewer, we will expand the conclusions. Despite the heterogeneous literature on non-pointwise activations, where each approach may vary for different tasks and little theoretical clarity exists, we will provide a more in-depth discussion.

For permutation-equivariant network, it may also be interesting to work out another nontrivial architecture that is neither DeepSets nor IGN (higher order tensor representation). I'd also understand if the authors deem these as being out-of-scope.

Regarding the reviewer's last request, we can readily provide another example of a permutation-equivariant network that is not an IGN. For instance, considering the alternating group $A_n$ as the subgroup of even permutations in $S_n$, this subgroup cannot be isomorphic to any of the subgroups $S_\lambda$ presented in the paper. The module $\mathbb{R}^{S_n / A_n}$ will be a two-dimensional representation compatible with point-wise activations. Exploring such spaces, although they may seem pathological and of dubious practical utility, could aid in the understanding of permutation equivariance in its generality and hence in constructing better performing networks.

Response to questions:

We thank the reviewer again for the help in improving the manuscript. We acknowledge the errors and inaccuracies and want to assure that we are already incorporating the suggested corrections.

References:

[1] Taco S. Cohen and Max Welling, Group Equivariant Convolutional Networks. Proceedings of the 33 rd International Conference on Machine Learning, https://arxiv.org/pdf/1602.07576.pdf

[2] Taco S. Cohen, Maurice Weiler, Berkay Kicanaoglu, Max Welling, Gauge Equivariant Convolutional Networks and the Icosahedral CNN, Proceedings of the 36 th International Conference on Machine Learning, https://arxiv.org/abs/1902.04615

We thank the reviewer for the comprehensive assessment.

Response to contribution:

The main result looks like a relatively simple add-on...

We agree with the reviewer, and explicitly acknowledge in the paper, that our proposed results are largely built on top of the pioneering work by Wood and Shawe-Taylor [1]. However, we introduce the following extensions and improvements:

• We generalize the theorem to non-finite groups and continuous activations, as new classes of networks emerge from this extension.
• We note the importance of identifying certain network classes by considering representations up to isomorphism. Note that two isomorphic representations are equivalent in terms of learning capabilities and this aspect was not covered by the original paper. Indeed, we solve an issue raised in Section 4.2 of Wood and Shawe-Taylor where they provide an example of a positive monomial representation that is not a permutation representation. That representation is actually isomorphic to a permutation representation. With our approach we fix all issues like this in a general and complete setting.
• We integrate the theory developed by Wood and Shawe-Taylor with several adjustments that make the classification effective. For example, our Lemma 1 ensures the maximality of the families of matrices and activations presented. This aspect is not discussed by Wood and Shawe-Taylor, and it is necessary for the rigor of the characterization.
• We want to highlight that despite acknowledging the authorship of some technical lemmas presented at the end of the appendices to Wood and Shawe-Taylor, our proposed proofs place a greater emphasis on algebraic aspects, especially grounding our characterization on the classification of multiplicative subgroups of $\mathbb{R}$, diverging by Wood and Shawe-Taylor. This approach not only improves the readability of the proof but also highlights the potential to generalize these results to complex numbers as a plausible solution to certain limitations of real equivariant networks as suggested in the Conclusions.

In the manuscript that we will upload, we will further update and refine these proofs. When compared to the original ones, they will show their kinship, but also their novelty. We believe that the impact and implications of these innovations are significant for the machine learning community, especially considering the evolution and growth of the use and study of equivariant representation learning during the twenty years passed since the publication of Wood and Shawe-Taylor's paper.

Though the paper includes a discussion...

We thank the reviewer for pointing out the difficulty in understanding the focus and significance of Sections 5 and 6. We intend to refine these sections in the updated version of the manuscript to enhance comprehension. This will involve introducing fundamental concepts essential for understanding, along with the inclusion of simple examples. In instances where space constraints may limit these improvements, we will ensure to adequately strengthen the appendices and incorporate appropriate references in the main text. The reviewer rightfully questions how our proposed results may influence the design of equivariant models. We indirectly address this concern by highlighting how our findings show gaps in understanding specific phenomena. For instance, in Section 5.2, we present a list of permutation-equivariant neural networks which, to the best of our knowledge, are not found in the existing literature. The existence of these networks show an inadequate understanding of the permutation equivariance, a gap stemming from the scarcity of theoretical results in the field. In the long term, addressing this gap appropriately may lead to the development of new, more efficient, and better-performing models. With this work, we aim to contribute to this endeavor.

Response to writing:

We thank the reviewer for their comments, particularly appreciating the precision of their remarks. We acknowledge the errors identified in Lemma 5 and will promptly rectify them in the manuscript. We will also revise the proof of Lemma 1 to enhance its clarity. Furthermore, we acknowledge and will rectify the errors and ambiguities noted in Lemma 7.

Response to questions:

As mentioned earlier, we are committed to enhancing the readability of the paper by improving the quality of writing and incorporating illustrative examples. Regarding the suggestion to focus on fewer applications but delve into them more thoroughly, we acknowledge this as a viable and effective approach. Secondly, maintaining a comprehensive overview of all applications is essential to convey the significance and ubiquity of the implications of Theorem 1 within the entire research field of geometric deep learning.

References: [1] Wood and Shawe-Taylor. Representation theory and invariant neural networks https://www.sciencedirect.com/science/article/pii/0166218X95000753

We thank the reviewer for the thorough comments.

Response to weaknesses:

Readability:

1. I find this paper quite difficult to read at times, and I think that it would benefit from more explicitly defining certain concepts and writing out certain arguments. In the proof of Corollary 3, I do not quite understand the application of Theorem 1; why does the representation have to be trivial? Also, Section 6.2 about invariant graph networks is very dense. See also questions.

As suggested, we will provide more explicit definitions of concepts and arguments. In particular, we will modify the proof of Corollary 3 to emphasize how it derives from Theorem 1 and how those representations are trivial. Subsequently, we will proceed to clarify and expand Section 6.2.

Applicability to equivariant ML:

2. The applicability of these results to empirical equivariant machine learning is unclear.

This theorem may be of interest even to the ordinary machine learning practitioner as it provides the complete list of cases where the application of point-wise activations is viable and those can generally lead to increased computational efficiency.

Result on SO(3):

Already practitioners do not use pointwise nonlinearities for $SO(3)$-equivariant networks.

We agree that many different approaches to non-pointwise activations for equivariant $SO(3)$-equivariant networks are employed in practice but they in general require more computational resources, present a slow convergence rate, or both. For this reason, it may be favorable to employ pointwise activation to mitigate those shortcomings, but we prove in the paper that this is not a viable option. Indeed, we believe that this result is of greater significance for machine learning researchers aiming to construct new equivariant models. In fact, in some relevant cases (e.g. exact rotation equivariance), this theorem suggests that natural research directions (e.g., pointwise activations coupled with any $SO(3)$-representation) might require more careful thinking.

Implication for IGN:

Also, are there any expectations of potential benefits from using other representations of $S_n$ for invariant graph networks?

Regarding the list of other presented $S_n$-equivariant models, we have no strong results about their general properties. We deemed it significant to highlight their existence not only for sake of completeness but also because this manifests a lack of understanding of the broader concept of permutation equivariance, a deficiency arising from a scarcity of theoretical work in the field, a lacuna that, if strongly addressed, could lead to the construction of more efficient and expressive models. With this work, we hope to contribute positively to the long-term effort of filling this gap.

Response to questions:

1. In Section 6.1 / Section 6.3, why is the following case ruled out? Say I let $f$ be linear, so that $\tilde f$ is just scaling and is hence equivariant to $SO(3)$. Then the output representation is the same as the input, and is in particular not invariant. This can be done in the $SO(3) \times S_n$ case as well.

We thank the reviewer for catching our oversight formulating Corollary 3; indeed, an hypothesis was missing from the statement and the statement holds true only for nonlinear activations which rules out the provided example.

Typos:

We appreciate the reviewer for pointing out these typos. The corrections will be promptly incorporated into the manuscript.

In the following, we present the responses to the questions posed in the second part of the review.

1. On page 3 the symbol $\mathbb{R}^*$ is used. What does the asterix indicate?

These are the multiplicatively invertible elements in $\mathbb{R}$. The definition is initially presented in Lemma 6 in the Appendix. We appreciate the reviewer for bringing this to our attention; we will introduce the definition in the main body of the text before referencing this set.

2. I did not understand the introduction of the main result "Although employing techniques analogue to already known ones, we highlight how hypothesis can generalize ... of the basis" Apart from this sentence being too long, I really do not understand what is being said here, please rewrite.

This segment highlights how our work contributes to advancing the state of the art, specifically building upon the work of Wood and Shawe-Taylor [6]. Notably, we want to emphasize how we achieve more refined results by considering representations up to isomorphism, despite employing more general assumptions. We will ensure to enhance the presentation of this section in the manuscript.

3. Theorem 1 could benefit from some subsequent, explicit examples. Take ReLU and e.g. a cyclic permutation group. Or any example you see fit. I think the paper remains abstract throughout and I think it could be helpful to see some common examples.

We appreciate feedback on Theorem 1 and agree that providing explicit examples, such as employing ReLU with a cyclic permutation group, would enhance the clarity and applicability of the paper. We will certainly integrate these examples into the paper to offer a more concrete understanding of the theoretical concepts presented.

4. Section 5.1 is also a bit abstract still, dispite being in the section for practical scenarios. Some laymans explainations of the quotient structure could be helpful. In particular I struggled with "...an equivariant isomorphism between ... which is the standard representation space for permutations equivariant networks on sets".

We will provide detailed explanations and examples of quotient groups and spaces induced by those groups. Although I believe we may need to move these additional elements to the appendices due to manuscript length constraints.

5. Section 5.2 makes sense, but as discussed it is limited in scope regarding the use of irreducible representations exclusively.

We concur. This section introduces a standalone corollary that does not impact the subsequent results we present. Nevertheless, given the role of disentangled networks in the literature, we deem it relevant to include this presentation.

6. Could you clarify Corrolary 3. It is hard to understand what is about. In particular what is the significance of the connected component (are we talking about subgroups here?)

In this scenario topological groups, such as $SO(3)$, come into play. These groups exhibit topological properties such as connectedness and compactness, which we employ in Corollary 3. We will ensure to provide additional details and examples in both the main text and the appendices.

7. "... but not more general equivariant tasks such as segmentation or detection" again this is an artifact of sticking to irreps.

This point stems from the finiteness of the matrix group induced by the representations. This statement is general and not specific to disentangled networks alone. We believe that our misalignment on this point can be traced back to the discussion in Answer 3 Discretization.

References:

[1] Kuipers, T.P., Bekkers, E.J. (2023). Regular SE(3) Group Convolutions for Volumetric Medical Image Analysis. https://arxiv.org/abs/2306.13960

[2] Cohen, T. S., Geiger, M., Köhler, J., & Welling, M. (2018). Spherical CNNs.

[3] Gasteiger, J., Becker, F., & Günnemann, S. (2021). Gemnet: Universal directional graph neural networks for molecules.

[4] Fast, Expressive SE(n) Equivariant Networks through Weight-Sharing in Position-Orientation Space, Preprint, https://arxiv.org/abs/2310.02970

[5] Taco S. Cohen and Max Welling, Group Equivariant Convolutional Networks, https://arxiv.org/pdf/1602.07576.pdf

[6] Jeffrey Wood and John Shawe-Taylor. Representation theory and invariant neural networks. https://www.sciencedirect.com/science/article/pii/0166218X95000753

[7] Nadav Dym and Haggai Maron. On the Universality of Rotation Equivariant Point Cloud Networks.

[8] Taco S. Cohen, Maurice Weiler, Berkay Kicanaoglu, Max Welling, Gauge Equivariant Convolutional Networks and the Icosahedral CNN, https://arxiv.org/abs/1902.04615

We thank the reviewer for the valuable comments and suggestions.

In the following, we present responses to the comments organized in a manner analogous to the structure of the review.

1. Readability and accessibility:

1. The paper is a theoretical contribution and emphasizes that the results are of practical value. [...]

We acknowledge the importance of improving the paper's accessibility and readability. In our revisions, we ensure to present the main results in a more straightforward manner to enhance clarity for a broader audience.

2. Regular representations and Irreps:

2. Although partially covered by clarifying the scope of the paper, I do think the notion of regular representations is insufficiently covered. [...]

If we correctly understand the concerns regarding regular representations and irreps, we suppose that there may be confusion due to the adoption of Wood and Taylor's notation for regular representations. We will explicitly clarify in the manuscript that despite this notation choice, our results address arbitrary finite-dimensional representations, which encompass regular and, more in general, permutation representations. Additionally, we will emphasize that the specific treatment of disentangled representations (hence the introduction of irreps) is confined to Section 5.2 and does not affect other parts of the paper. The invariance result for compact connected topological groups (e.g. $SO(3)$) is not implied by disentanglement but it follows from the finiteness of the matrix groups underlying admissible representations. Indeed, note that hypotheses of Corollary 3 do not require disentanglement. For further clarification, the last paragraph in Section 6.3 exploits the existence of decomposition in irreducible components and Schur's Lemma only to prove triviality, but it does not require disentanglement of the layer, meaning that the basis chosen in the definition of the pointwise activation (see Section 3.2) does not need to agree with some irrep decomposition (see the first paragraph of Section 5.2). Note that irrep decompositions exist for each representation, regular ones too, (see Appendix A.2, Theorem 2) the stringent constraint is disentanglement that we are not assuming in this instance.

3. Discretization:

3. By the previous item, remarks such as (see abstract) "we prove that rotation-equivariant networks can only be invariant" is in practice more nuanced, and I consider the statement actually incorrect. [...]

In agreement with the reviewer, we acknowledge the necessity for a broader contextualization of Theorem 1 within the current literature covering all nuanced takes on equivariance. For brevity in this comment, we categorize these nuances as approximate equivariance compared to exact equivariance, and finite-dimensional frameworks compared to harmonic analysis approaches. Adopting this simplified view, our restriction to exact equivariance and finite-dimensional representations should be pointed out earlier and more clearly in the text, highlighting how the analysis conducted under these constraints also affects practices derived from harmonic analysis approaches. We proceed to elaborate on the matter in greater detail: We agree, as pointed out by the reviewer, that a substantial portion of the current literature on equivariant models relies on harmonic analysis approaches [1, 2, 4]. Processed inputs are continuous finite-energy signals defined on homogeneous spaces (e.g., $L^2(G)$ functions) which are infinite-dimensional spaces and thus fall outside the scope of our study. However, as pointed out at page 4 of the paper, in practice continuous models are often discretized, relaxing the equivariance condition to an approximate notion. This process is often obtained by truncating finite Fourier transforms of the signal or discretizing the symmetry group and domain. This discretization frequently boils down to considering equivariant models with finite-dimensional representations of finite groups [5, 8], which is dealt with by our framework and, in line with our findings, allow pointwise activations and non-trivial representations. Contrary to our approach, other methods suggested by the reviewer (such as GemNet [3]) employ tools akin to Tensor Field Networks [7], featuring non-pointwise activations that, as implied by the paper's title, go beyond the scope of our work (utilizing tensor representations, a nonlinear function, formula 4 in Dym and Maron [7]). These are however very interesting directions that we will address as future research.

4. Conclusion:

4. In conclusion, I think the paper could improve on impact if the relation to regular group convolution methods is further clarified and if the main results are introduced in simpler terms. [...]

We agree, and in light of the observations made, we will incorporate your suggestions and compare our work to the proposed references to enhance the overall impact and clarity of the paper.

We have uploaded a new version of the draft, aiming to incorporate the changes suggested by the reviewers. First and foremost, our effort has been directed towards enhancing the overall accessibility of the work. We provided a clearer description of the results in plain English, incorporating examples, expanding the appendices with the requested introductory elements, and relocating more technical results to the appendices.

In order to enhance text comprehensibility, we opted to split the statement of the main theorem into two theorems. The first theorem is the generalization of the work by Wood and Shawe-Taylor to continuous activation functions and arbitrary groups. The second theorem refines the outcome of the preceding theorem, up to isomorphisms of representations.

Secondly, we improved the readability of the proofs of the theorems in the appendices. It should now be more evident what our contribution is in comparison to Shawe-Taylor's work. We have divided the proofs into multiple lemmas, aiming to achieve results that are simpler to comprehend, and striving to explain each aspect thoroughly. As elucidated in the text, the results we propose from Shawe-Taylor are only Lemma 11 and Lemma 12, while we believe the remainder of the results between pages 17 and 21 to be original.

Thank you, I think the extra context and detail provided in the current manuscript are valuable additions. Based on it, I decided to raise my score.