Rethinking the Benefits of Steerable Features in 3D Equivariant Graph Neural Networks

Q14. Conclusion. Two key findings are claimed: (1) Propagating steerable features of type $\geq 1$ equips geometric GNNs with the inherent ability to automatically capture the geometry between local structures. (2) When the feature dimension remains constant, increasing the use of steerable features up to type-L cannot essentially improve the performance of equivariant GNNs But I think that (1) is vague and already known. For instance from (16), which clearly distinguishes invariant ($L=0$) from equivariant $L\geq 1$ networks. And that (2) is known as oversquashing and was known before. I note that the term oversquashing is absent from the manuscript, although it is mentioned several times in [16]. I think the term should appear.

Reply: We respectfully disagree with these comments and fear that there are substantial misunderstandings of our contribution. Please allow us to clarify our contribution below.

Concern about (1), we would like to clarify our results not only cover cases like ComENet -- as outlined in the limitation section of [16] but also emphasize the significance of faithfulness in effectively capturing geometry between local neighborhoods during message passing. It's noteworthy that [16] suggests that incorporating pre-computing non-local features, as in ComENet, may offer a straightforward approach to overcoming the limitations of invariant GNNs. However, our results, both theoretical and empirical, indicate that this approach may not be as effective as suggested. Please refer to Theorem 1 and Table 2. Moreover, [16] asserts that their setup can be extended to higher-order tensors. However, we found that similar results for this extension might not hold if the geometric objects, which correspond to the steerable features, do not lie on faithful representations. In particular, our framework highlights that significant geometric information between local neighborhoods could be lost during message passing if the representations lack faithfulness. Please refer to Theorem 1 and Appendix B in the revised paper for details.

Concern about (2), we did not include a discussion of over-squashing since it is not the problem we are studying and unrelated to our contributions. [16] does mention the over-squashing phenomenon for equivariant features in geometric GNNs, using the example of EGNN, which utilizes a single vector feature to aggregate and propagate geometric information. Specifically, these models are constrained to learn growing information within a fixed-length vector, potentially failing to propagate long-range information and learning only short-range signals from the training data [1].

Let us clarify the empirical evidence supporting the two contributing components of our work. Table 2 presents empirical evidence for first part of our theory involving $k$-hop models like ComENet – this was previously unsupported in [16]. In Table 2, we observe the over-squashing in some models as in [16] and we have included a discussion on over-squashing in Appendix F.2.

The ablation studies in Figure 2, Table 3, and Table 4 support the second part of our contribution which is aimed at answering: When the feature dimension remains constant, does the performance of equivariant GNNs employing steerable features up to type-$L$ increase as $L$ grows?

We study the information-carrying ability of different types of steerable features by investigating their corresponding invariant features. In Table 3 and Table 4 we fix the steerable feature dimension while varying the steerable feature type $L$. We select the steerable feature dimension based on the best-performing model (see Table 1). We find that the performance may not necessarily increase when increasing $L$. We do not investigate the phenomenon of over-squashing in this ablation study and we do not observe any effects of over-squashing in our results. Moreover, our theory on the information-carrying ability of steerable features is not limited to GNNs.

We believe that Section 5.2 in [16] is more relevant to the problem we study. As discussed in Section 4, [16] has conducted a comparison of the utilization of different types of steerable features. However, that comparison is not explicitly designed for invariant classification.

In summary, our paper addresses the limitation mentioned in [16] and is the first research highlighting the significance of faithful representations in capturing geometry between local neighborhoods during message passing. Moreover, we provide theoretical reasons supporting the idea that when the feature dimension remains constant, the performance of equivariant GNNs employing steerable features up to type-$L$ may not increase as $L$ grows.

[1] Alon, U., and Yahav, E. On the bottleneck of graph neural networks and its practical implications. ICLR 2021.

[16] Joshi, C. K., Bodnar, C., Mathis, S. V., Cohen, T., and Lio, P. On the expressive power of geometric graph neural networks. ICML 2023.

Q12. IS2RE. Table 3. Here I enjoyed a lot the idea of comparing models with comparable feature dimension. However, several points: 1) why not report also the total number of parameters? They do not grow as fast as feature dimension since weights are shared for a given $L$ (type order). 2) although I understand it's somewhat hardware dependent, please also report the training time for these models. Maybe even the memory footprint (on GPU). 3) Figure 3 gives an idea of how things perform over a reasonable number of epochs (and I salute the attempt to not go to an exceedingly large number of epochs), but it seems that a more thorough study of variations with $L$ and $c$, reporting only the epoch=12 validation performances, would be useful to the reader (I did not have time to look at the content of ref [5])

Reply:

Thank you for the suggestion. Details for # Param, Run Time, Memory, and validation results for $L=1,3,5$ have been incorporated into Appendix F.2 in Table 6 and Table 7. Below is a reproduction of these tables for the reviewer's convenience.

Note that for EquiformerV2 L=1, the model will fit onto the GPU. However, during training the data and the model will exceed the GPU memory capabilities. We denote the out of memory phenomenon OOM in the tables above and report the model parameters and size.

Q13. S2EF. Table 4 is commented, but no conclusion stands out. What is the point of that table ?

Reply: Table 4 is used to support our theory that no model stands out under a suitable ablation study on $L$ and $c$. This is directly motivated by the analysis in Table 1. Both eSCN[L=6,c=256] and eSCN[L=2,c=824] achieve strong results in comparison to eSCN[L=2,c=256]. However, the fact that eSCN[L=6,c=256] and eSCN[L=2,c=824] themselves are indistinguishable provides empirical support for our theory. In particular, there is no significant gain by increasing $L$ when $c$ is proportionately increased.

Q9. The remark below corollary 3 is a very strong, original conclusion. I think I understand Corollary 3 (although I feel like in practice lambda is the norm of the representation, and in that case it's kind of trivial..). In any case I do not see how the remark "the expressiveness of learning steerable features is primarily characterized by the feature dimension -- independent of the highest type utilized" follows from corollary 3.

Reply: We have clarified the non-triviality of $\lambda$ in the above responses. Here we would like to address your concern about the implication of Corollary 3. First, we would like to acknowledge that there are two main aspects related to the concept of expressiveness: the capacity of features to carry information and the model's ability to extract that information -- the latter is commonly referred to as universality.

Our discussion on the utilization of different types of steerable features focuses on the former, given that the latter is subject to the model’s architecture. Therefore, asserting that “the expressiveness of learning steerable features is primarily characterized by the feature dimension -- independent of the highest type utilized” might lack precision without considering the model's universality. We have refined the assertion to state that: “the invariant features carried by steerable features are primarily characterized by the feature dimension -- independent of the highest type utilized.”

Additionally, we have rephrased the relevant sections and included a new section in Appendix B of the revised version to delve into universality and potential methods. We believe these revisions address your concerns and provide more precise implications of our results.

Q10. In table I, I can do the maths for line 1 and 2, but not line 3. (5+3+1)*256 = 2304 (5+3+1)*824 = 7416 but then, (1+3+5+7+9+11+13)*256 =12544, not 7424

Reply: Two hyperparameters are used to control the degree $l$ and order $m$ of the spherical harmonics when generating steerable features. The degree $l$ is directly determined by the steerable feature type $L$. However, the maximum absolute order $m$ is set to $|m|\leq 2$ by default [1] in both eSCN and EquiformerV2. The order $m$ of the spherical harmonics can range from $-l$ to $l$, and when $|m|\leq l$ it serves to restrict the feature space. When $|m|\leq 2$ the equation becomes (1+3+5+5+5+5+5)*256 =7424.

[1] https://github.com/Open-Catalyst-Project/ocp/tree/main

Q11. experiments are a bit disappointing. 7a. First, Table 2 shows a number of red (interesting) cases, but they turn out to be related to eSCN being SO(3)-equivariant, when the task is O(3). This is not making the point of the paper, and instead is rather confusing, at first sight. Most importantly, it's not clear to me in which sense table 2 is different from table 1 of ref [16] (which by the way seemed more readable and dense). Please clarify this.

Reply: The inclusion of eSCN in Table 2 is used to provide a robust experiment section for all models considered. However, to avoid confusion with the limitations of eSCN, namely the O(3)-equivariance and the quasi-equivarianc, we have included additional discussion in Section 5 and Appendix F.1.

Table 1 of ref [16] considers only chain lengths of $k=4$. This makes it difficult to discern the impact of incorporating $k$-hop information. This is not a concern for the authors of [16] because they do not provide a theoretical understanding of models like ComENet and LEFTNet. Table 2 and Table 5 incorporate a more robust analysis of different models. To improve the clarity of the paper, we have included a more detailed description of the figure in the caption.

Q6. gothic{g} appears in the r.h.s. of Eq (4) or in the last line of the previous est of equations (let me call this last line Eq. 3b is) (actually the gothic-style has been forgotten in Eq. (4), this is a typo I believe). How can we identify the right term in the r.h.s. of Eq. 3b is as a lambda (invariant), when $g^-1$ appears in it? I don't see why $g^{-1}\cdot x_{ij}$ should be invariant, on the contrary, it seems to me like an equivariant feature.

Reply: Thank you for pointing out the typo. Notice that $g$ is a group element satisfying $g\cdot c(G \cdot{\bf X})={\bf X}$. In other words, we have $g^{-1}\cdot{\bf X}=c(G\cdot {\bf X})$. It is crucial to note that $c(G \cdot{\bf X})$ is invariant as it only depends on the orbit. The term $g^{-1}\cdot x_{ij}$ is a linear combination of the columns in $g^{-1}\cdot{\bf X}= c(G\cdot{\bf X})$. From this, we deduce its invariance.

Q7. In Eq. 7, first line, why isn't it $2l+2$ instead of $2l+1$ (in the r.h.s.)? I understood that representing O(3) required one additional component? Could you provide an intuitive explanation?

Reply: The construction of O(3) representations has been thoroughly documented in existing literature, and we have provided a concise review in Section 2. Notably, the irreducible representations of O(3) arise from the product of the irreducible representations of $SO(3)$ and those of ${\pm {\bf I}}$ (rather than a direct sum). Hence, the dimension remains $2l+1$.

Q8. You write, in page 6: Does any $(2l+1)$-dimensional invariant feature $\lambda({\bf X})$ correspond to a type-l steerable feature $f({\bf X})$? Which sounds like an interesting question (although ideally I'd like to know the mapping, not just know about its existence). Note that this question is essentially asking whether the space of all type-l steerable features $f({\bf X})$ has a dimension of $2l+1$ since $D^l (gX)$ is invertible. But that sounds like a known fact: using Spherical Harmonics, it is known that type-l steerable features need components to be represented. That is, they need $2l+1$ numbers (dimensions)

Reply: We believe the reviewer has a misunderstanding regarding the definition of steerable features in our context. It is true that SO(3)-steerable vector spaces are spanned by Spherical Harmonics, and thus, a type-$l$ steerable vector space $V_l$ has a known dimension of $2l+1$. However, the space of steerable features is a subspace of $V_l$, consisting of type-$l$ steerable vectors that are generated by some equivariant function from the input ${\bf X}$. More formally, it can be expressed as {$f({\bf X}) \mid \text{ G-equivariant }f:\mathbb{R}^{3\times m}\to V_l$}. In Fig 4 of the revised version, we provide an illustrative example to emphasize that the space of type-$1$ features can be 1-dimensional rather than 3-dimensional, showing that they might not be equal.

The map sending $f({\bf X})$ to $\lambda({\bf X})$ is defined by the composition $\lambda({\bf X}):=f(c(G \cdot{\bf X}))$. The function $c$ is highly related to the notion of alignment in reference [1] listed below. We have included relevant discussions in Appendix A of the revised version.

[1] Winter, R. et al. Unsupervised learning of group invariant and equivariant representations. NeurIPS 2022.

Q4. You prove that the decomposition of Lemma 1 is unique. But, concretely, what is lambda? Isn't it almost always simply the norm of the irrep? Like, for a vector, its norm, or for a higher-order tensor, also its norm? And then $\rho(g_X)$ is just the usual (matrix) representation of $g_X$. Can it be otherwise? Or is it more complicated? Am I missing the point? If in practice things are simple, in most of the usual settings, saying so is helpful to the reader. OR, maybe $g_X$ simply encodes the orientation of $X$ (say if it's a vector)?

Reply: There seem to be a few points of misunderstanding.

Firstly, the output of the G-invariant function $\lambda$ is a vector of invariant scalars, while the norm is just a scalar. They cannot be the same, especially when the dimension $d$ is greater than $1$. Concretely, $\lambda({\bf X}) = f(c(G \cdot{\bf X}))$ is equal to evaluating the equivariant function $f$ at the representative $c(G \cdot{\bf X})$. Further details can be found in the proof of Lemma 1 (refer to Appendix C in the revised version).

Regarding $\rho$, it represents the group representation on the steerable vector space $V$. On the other hand, $g_{{\bf X}}$ is a group element that satisfies $g_{{\bf X}}\cdot c(G \cdot {\bf X})={\bf X}$. It is true that $g_{{\bf X}}$ encodes the orientation of ${\bf X}$ in the sense that applying rotations or reflections to ${\bf X}$ may lead to a change in $g_{{\bf X}}$ while $c(G\cdot{\bf X})$ remains unchanged. We have provided additional clarification on this point in Remark 1.

It is worth mentioning that $g_{\bf X}$ and $\lambda({\bf X})$ share similarities with local frames and the scalarization proposed in [1], respectively. To provide a more comprehensive understanding, we have included an additional section on related works to discuss the similarities and distinctions between them. Please refer to Appendix A in the revised paper for details.

[1] Du et al. SE(3) Equivariant Graph Neural Networks with Complete Local Frames, ICML 2022.

Q5. In ref [16], page 3, it says: At initialisation, we assign to each node $i\in V$ a scalar node color $c_i\in C’$ and an auxiliary object $g_i$ containing the geometric information associated with it i.e. they already separate (factorize) the scalar (feature only) contributions from the geometric ones (vectorial one v + previous color). In this respect, it is not obvious how contribution (1) (your lemma 1) is new. Furthermore, I lack an intuitive view of what may concretely go into your scalar function lambda(X).

Reply: We believe this is a misunderstanding. In the initialization of the GWL test in [16], the scalar node color only considers the node features. As iterations progress, the color $c_i^{(t)}$ and the auxiliary object $g_i^{(t)}$ incorporate not only the node features but also the geometric information and the information about the graph structure. In contrast, Lemma 1 exclusively focuses on the coordinates (geometric information) and remains independent of node features and the graph structure. In other words, both $\rho(g_{{\bf X}})$ and $c(G \cdot {\bf X})$ are unaffected by node features and the graph structure.

The separating technique used in Lemma 1 separates ${\bf X}$ into the representative $c(G \cdot{\bf X})$ and the associated group element $g_{{\bf X}}$. Using this technique, we can effectively separate any steerable feature $f({\bf X})$ into its equivariant part $\rho(g_{{\bf X}})$ and its invariant part $\lambda({\bf X})$. It provides us with a framework to investigate steerable features by examining their corresponding invariant features. While we acknowledge that this separating technique might share similarities with the one you mentioned, it is important to emphasize that our framework is more flexible -- it is not limited to GNNs but can be applied to other neural networks or algorithms.

Concretely, $\lambda({\bf X}) = f(c(G \cdot{\bf X}))$ is equal to evaluating the equivariant function $f$ at the representative $c(G \cdot{\bf X})$. Further details can be found in the proof of Lemma 1 (refer to Appendix C in the revised version).

Thank you for your thoughtful review and valuable feedback. We fear the reviewer has some misunderstandings of our work which we have worked to clarify in this rebuttal and in our revision.

Regarding your concern about the novelty, we would like to clarify there are some misunderstandings. In particular, our results cover $k$-hop geometric GNN, such as ComENet, which is mentioned in the limitation section in Joshi et al.’s work. Moreover, we investigate whether, when the feature dimension remains constant, the performance of equivariant GNNs employing steerable features up to type-$L$ increases as $L$ grows.

In what follows, we provide point-by-point responses to your comments.

Q1. In the definition of $k$-hop geometric GNNs, $k$ is not defined (its role is not defined). Actually, the definition of $k$-hop GNN is a bit overlooked (although it's rather intuitive what the meaning of $k$-hop is), and more importantly, it is not clear to me how these are not essentially equivalent to regular GNNs with more layers (since a single layer of a 2-hop GNN aggregates information with a similar receptive field as a 1-hop GNN with 2 layers). Probably authors should elaborate on that or cite appropriate references.

Reply: We present the architecture of $k$-hop geometric GNNs in Eq (1), where $k$ denotes the size of neighborhoods from which a node aggregates information -- or equivalently, the maximum distance over which it aggregates information in a single message-passing step. Additionally, we reference ComENet as an instance of 2-hop invariant GNNs in Section 1.1.

Moreover, it is crucial to note that $k$-hop geometric GNNs are not necessarily equivalent to regular 1-hop geometric GNNs with $k$ layers. As proved in Appendix D of the revised version, under some assumptions on UPD and AGG, k-hop invariant GNNs can distinguish any two $k$-hop distinct geometric graphs but fail to distinguish any two $k$-hop identical geometric graphs. This indicates that 1-hop invariant GNNs might be less powerful than their 2-hop counterparts, and this trend continues.

The reason lies in the fact that the value of $k$ influences the size of local neighborhoods, consequently affecting the input of the aggregation process. Utilizing a smaller $k$ may cause $k$-hop invariant GNNs to overlook changes in geometry between small local neighborhoods, whereas using a larger $k$ can capture these variations.

However, it's noteworthy that, for any finite $k$, we can always construct two $k$-hop identical but $(k+1)$-hop distinct geometric graphs. In this scenario, $k$-hop invariant GNNs cannot differentiate them even with thousands of layers, while 1-hop equivariant GNNs, due to the faithfulness of features, can achieve this with $k+1$ layers. This finding sheds light on the advantages of learning steerable features up to type-$L\geq1$. We have substantiated this point through empirical results for ComENet. In the revision, we have bolstered these observations by considering the LEFTNet model in Appendix F.1 Table 5 of the revision.

To clarify this implication, we have refined the concluding remarks. In particular, we have added the following remark: relying solely on propagating invariant features confined to specific k-hop neighborhoods is insufficient for enabling geometric GNNs to capture the global information of geometric graphs precisely.

Q2. About Lemma 1, page 4. When you define the function $c$, why can't it simply be always the identity? One could always choose the neutral element from the group $G$. I do understand why the set {$g\in G | g\cdot c(G\cdot {\bf X})={\bf X}$} is not empty. I do not understand what is non trivial about $c(G\cdot X)$

Reply: The function $c$ is defined on the quotient space $\mathbb{R}^{3\times m}/G$ and maps to $\mathbb{R}^{3\times m}$. Therefore, it cannot be a trivial identity, which is a function mapping from a space to itself. The composition that sends ${\bf X}$ to $c(G \cdot {\bf X})$ is also not an identity because, for any two ${\bf X}$ and ${\bf X}'$ lying in the same orbit, we have $c(G\cdot{\bf X}) = c(G\cdot{\bf X}')$.

Q3. Also, $V_0^{\oplus d}$ is not explicitly defined early enough. What is $d$? Does it relate to the maximal rotation order $L$?

Reply: We appreciate your feedback. However, we have introduced $d$ in the first sentence of the statement of Lemma 1, where we mention, "Let $V$ be a $d$-dimensional $G$-steerable vector space…". The parameter $d$ represents the dimension of the $G$-steerable vector space $V$ and is not directly related to the maximal rotation order $L$.

Regarding $V_0$, we acknowledge that we overlooked mentioning in Lemma 1 that we use $V_0$ to denote the trivial 1-dimensional representation of an arbitrary group $G$. We have made a revision to address this.

Q3. Why did you use $L=2$ and higher in your experiments in Tables 1, 2, and 3?

Reply: Table 1 rows 1&3 report the baseline hyperparameters for the eSCN model. Table 1 row 2 illustrates what we propose as a suitable comparison. In particular, we fix $L=2$ to maintain the steerable feature type in the baseline, while adjusting the hidden channels $c$ in order to maintain the dimension of the steerable feature. Table 2 reports results for different models which use $L=0,1,2$. We report results for $L=2$ to demonstrate the effects of $k$-Chains when using higher-order steerable features. Table 3 reports results for $L=2,4,6$ for eSCN and $L=2,4$ for equiformerV2. This serves as an ablation study over the steerable feature dimension $L$. We have enhanced the robustness of this study in the revision which now includes results for $L=1,3,5$.

Q4. Concern about exact message in the paper. In particular, elaborate on the last comments in the weaknesses section?

Reply: We would like to address each of your concerns individually.

For your first concern, we want to clarify that there is no inconsistency. While Lemma 1 allows us to study equivariant features by examining their corresponding invariant features, it is essential to consider the message-passing mechanisms, especially the local aggregation, when investigating geometric GNNs. We observe that, due to the faithfulness of representations, equivariant GNNs can effectively capture the geometry between local neighborhoods. However, hindered by trivial representations, invariant GNNs struggle in this regard significantly affecting their ability to precisely capture global geometry and obtain global invariant features from local invariant features. Basically, there are scenarios where equivariant GNNs can learn certain global invariant features that invariant GNNs cannot. In Appendix B of the revision, we have added a detailed explanation of why faithfulness is important.

The response to your second concern is contained in our response to Q3.

For the last concern, we would like to acknowledge that there are two main aspects related to the concept of expressiveness: the capacity of features to carry information and the model's ability to extract that information -- the latter is commonly referred to as universality. Our discussion on the utilization of different types of steerable features focuses on the former, as the latter is subject to the given model architecture. It is not sufficient to conclude that $L=1$ is enough without considering universality. We have added a section in Appendix B of the revised version to discuss universality and potential methods. Nevertheless, our results still show that when the feature dimension remains constant, the performance of equivariant GNNs employing steerable features up to type-$L$ may not increase as $L$ grows.

We have revised the relevant sections in our paper to enhance clarity and eliminate potential misunderstandings. We believe these revisions address your concerns and provide more precise implications of our results.

Q5. Do you know if your results were somehow already known in (geometric) invariant theory?

Reply: We have included an additional section in the revised version to discuss the similarities and distinctions between Lemma 1 and existing results. While we have conducted an extensive literature review, we have not identified an exact identical result in the existing literature.

However, if there is the same result already presented in the existing literature, we would appreciate it if the reviewer could provide a specific reference. This information will be crucial for us to acknowledge and appropriately credit prior contributions.

Thank you for your thoughtful review and valuable feedback. In what follows, we provide point-by-point responses to your comments.

Q1. Can you contrast your results with Villar et al. (2022)? How do your results differ/improve upon theirs? Let's take a basis vector $e_1 \in \mathbb{R}^{3 \times 1}$ and $f: \mathbb{R}^{3 \times 1} \rightarrow \mathbb{R}^3$ with $e_1 \mapsto \alpha e_1, \alpha \in \mathbb{R}$. This is clearly an equivariant function. What would the corresponding invariant function be such that Lemma 1 holds?

Reply: We have integrated the work by Villar et al. (2022) into our related work, including a discussion in Appendix A of the revision on the distinctions between their research and ours. Here, we provide a brief discussion.

Our results exhibit clear distinctions from theirs, mainly due to our consideration of steerable features of higher types $l>1$ and the different approaches we have taken. Instead of investigating the spanning space, we leverage the concept of "representatives" of orbits and apply the equivariance of the function to derive Lemma 1.

Our form also differs from Villar et al. (2022) in terms of the number of basis vectors and the uniqueness of invariants. In our form, the number of basis vectors aligns with the dimension of (steerable) features, rather than the number of input coordinates. Taking your example into account, where the function $f$ sends $e_1$ to $\alpha_1 e_1$. In the form provided in Villar et al. (2022), $f$ would be expressed as a linear combination of $e_1$, resulting in one basis vector. However, in our form, the number of basis vectors corresponds to the number of column vectors in $\rho(g_{{\bf X}})$ (which is three in this case). Moreover, $\lambda({\bf X})$ is directly determined by $\rho(g_{{\bf X}})^{-1}f({\bf X})$, using the invertibility of $\rho(g_{{\bf X}})$. This not only ensures the uniqueness of $\lambda$ but also highlights a key distinction. In Villar et al. (2022)’s framework, the invariants may not be unique, especially if the number of input coordinates exceeds the rank of their spanning space.

Let’s consider the example you provided, where the function $f$ sends $e_1$ to $\alpha_1 e_1$. In this case, the input embedding contains only one vector, namely $e_1$. There exists a rotation that transforms $e_1$ to the vector $(||e_1||, 0, 0)$, which lies on the $x$-axis. Here, we choose the representative $c(G\cdot{\bf X})$ to be the vector $(||e_1||, 0, 0)$ and the associated group element $g_{{\bf X}}$ to be the transformation that rotates $(||e_1||, 0, 0)$ to $e_1$. Then, $\lambda({\bf X}) = f(c(G\cdot{\bf X})) = f((||e_1||, 0, 0)) =(\alpha_1 ||e_1||, 0, 0)$ is equal to evaluating $f$ at the representative.

Additionally, we have established different correspondences between equivariant functions and invariant functions. Our results also involve a detailed investigation into the message-passing mechanisms of equivariant GNNs and invariant GNNs. We believe our results contribute to a more comprehensive understanding of the benefits offered by learning steerable features in geometric GNNs.

Q2. Let's take three basis vectors and with $e_1,e_2,e_3\in\mathbb{R}^{3 \times 1}$ and $f: \mathbb{R}^{3 \times 3} \rightarrow\mathbb{R}^3$ with $e_1,e_2,e_3\mapsto \alpha e_1 + \alpha e_2 + \alpha e_3, \alpha \in \mathbb{R}.$ 1) Isn't the set stabilizer now only the trivial rotation? 2) What would, in this case, be the corresponding unique?

Reply: The stabilizer is defined based on a given input embedding ${\bf X}$. Additionally, the size of stabilizers is contingent on the rank of ${\bf X}$, as detailed in the proofs of Theorems 2 and 3. Notably, when the rank of ${\bf X}$ is one, the stabilizer is formed by rotations around the line spanned by the column vectors in ${\bf X}$.

As mentioned earlier, the corresponding $\lambda({\bf X})$ is equal to $\rho(g_{{\bf X}})^{-1}f({\bf X})$, while we define $\lambda({\bf X})$ to be $f(c(G\cdot{\bf X}))$ in the proof of Lemma 1. Therefore, the explicit value of $\lambda({\bf X})$ may vary depending on the choice of the representative $c(G\cdot{\bf X})$ and the group element $g_{\bf X}$. It is challenging to explicitly work through your example due to the difficulty in explicitly constructing $c(G\cdot{\bf X})$ or $g_{{\bf X}}$ for any ${\bf X}$. In particular, this construction is highly related to the notion of alignment in reference [1] listed below, where they apply an autoencoder to learn $c(G\cdot{\bf X})$. (We also added a few comments to discuss this in Appendix B of the revision)

Nevertheless, we have performed computations for the associated group element $c(G\cdot{\bf X})$ and the corresponding invariant function $\lambda({\bf X})$ using another example you provided in Q1. We hope this addresses your concerns.

[1] Winter, R. et al. Unsupervised learning of group invariant and equivariant representations. NeurIPS 2022.

Q6. Concern about notations mentioned in weakness?

Reply: Thank you for your feedback. We have improved the presentation of the argument above Lemma 1.

Regarding your concern about the group action in Lemma 1, we have provided further clarification in Remark 1, particularly emphasizing that the group action is absorbed by $\rho(g_{\bf X})$. An alternative perspective is to consider the product $\rho(g_{\bf X})\lambda({\bf X})$ as a linear combination of the column vectors of $\rho(g_{\bf X})$ with the coefficients given by the scalars in $\lambda({\bf X})$. Induced by the definition of group homomorphism $\rho(h\cdot g) = \rho(h)\cdot \rho(g)$, the group action then acts on the column vectors of $\rho(g_{\bf X})$.

Furthermore, the well-defined mapping in Lemma 1, which associates equivariant functions with invariant functions, allows us to use the corresponding invariant function $\lambda({\bf X})$ to understand an equivariant functions $f({\bf X})$. In the proof of Lemma 1, we define $\lambda({\bf X})$ as $f(c(G\cdot{\bf X}))$, evaluating $f$ at the representative $c(G\cdot{\bf X})$. The key reason $\lambda({\bf X})$ sufficiently represents $f({\bf X})$ is that we can use the associated group element $g_{\bf X}$ to recover $f$ from $\lambda({\bf X})$ by computing $\rho(g_{\bf X})\lambda({\bf X})$. It is essential to note that $g_{\bf X}$ is pre-selected and is independent of $f$. In fact, $g_{\bf X}$ shares similarities with local frames in reference [1] listed below. To provide a more comprehensive understanding, we have included an additional section on related works to discuss the similarities and distinctions between $g_{\bf X}$ and local frames. Please refer to Appendix A in the revised paper for details.

[1] Du et al. SE(3) Equivariant Graph Neural Networks with Complete Local Frames, ICML 2022.

We have updated our submission based on the reviewer's feedback, with the revision highlighted in blue. We are happy to address further questions on our paper. Thank you for considering our rebuttal.

Q10. In Table 2 the invariant equiformerV2 actually does reasonably well, seemingly contradicting your theorem?

Reply: While the equiformerV2 model obtains improved accuracy over other invariant architectures in Table 2, it is difficult to assume that this is a desirable performance feature. Due to the enhanced robustness of our dataset, which is obtained by augmenting the data with rotations, reflections, and translations, it is difficult to determine if the improvement in accuracy is due to sensitivity to the numerical precision in our augmentation. In particular, we observe that the best model is still three standard deviations away from perfectly classifying the $k$-chains. For this reason, we do not draw any strong conclusions about the invariant equiformerV2 architecture.

Small Comment List:

We have clarified that stabilizers are subgroups.

We have corrected the typo in the definition of k-hop distinct.

In response to your concern about group action in Lemma 1, we have added further explanation in Remark 1, particularly emphasizing that the group action is absorbed by $\rho(g_{\bf X})$.

The definition of $V_{l, aug}$ is presented in Section 2, where we introduce $O(3)$-steerable vector spaces.

We have refined our phrasing to improve clarity and precision.

We have updated our submission based on the reviewer's feedback, with the revision highlighted in blue. We are happy to address further questions on our paper. Thank you for considering our rebuttal.

Thank you for your thoughtful review and valuable feedback. In what follows, we provide point-by-point responses to your comments.

Q1. Regarding (a): This correspondence has an important limitation which I believe the authors should mention: Update and Aggregation functions which are actually useful for learning are reasonably ‘nice’: for example, differentiable almost everywhere, continuous, etc. The correspondence suggested by lemma 1 uses a mapping of every orbit to a canonical element which indeed exists, but is not in general continuous or very well behaved. As a result an equivariant ‘nice’ aggregation function will correspond to a ‘wild’ invariant aggregation function.

Reply: The point you raised is accurate. In the revised version, we have provided additional explanations and potential methods to address this concern. For detailed information, please refer to Appendix B.

Q2. The paper's stand on invariant vs equivariant features seems inconsistent. On the one hand the paper maintains that there is a one-to-one correspondence between invariant and equivariant features, and that “propagation of steerable features can be effectively understood as propagation of invariant features, therefore analyzing the message passing using the corresponding invariant features is a reasonable approach (page 4)” on the other hand once this analysis is carried out the paper maintains that it points to an advantage of equivariant over invariant.

Reply: We appreciate the reviewer's careful examination of our paper. We want to clarify that there is no inconsistency in our stance. Corollary 1 establishes one-to-one correspondences between invariant and equivariant features. However, when investigating geometric GNNs, it is essential to consider the message-passing mechanisms. We observe that, due to the faithfulness of representations, equivariant GNNs can effectively capture the geometry between local neighborhoods, while invariant GNNs -- hindered by trivial representations -- struggle in this regard. This distinction significantly affects their ability to precisely capture global geometry and derive global invariant features from local invariant features. There are scenarios where equivariant GNNs can learn certain global invariant features that invariant GNNs cannot.

To enhance clarity and mitigate potential misunderstandings, we have revised all relevant contexts in our paper. We believe these revisions address your concerns and provide more precise implications of our results.

Q3. Page 4: Steerable features: It would make more sense to me to define f to be the steerable features. What does it mean to say that a vector is a steerable feature? Say you have a vector $x=[1,2,5]$ in $\mathbb{R}^3$. Is it a steerable feature or not?

Reply: Features, in the context of our discussion, refer to values obtained through a parameterized function applied to the input. That's why we refer to the steerable vector $f({\bf X})$, generated by the equivariant function $f$ from the input ${\bf X}$, as a steerable feature. It's important to note that not every arbitrary (steerable) vector, like $x=[1,2,5]$, qualifies as a steerable feature. There may not be an equivariant function such that $f({\bf X}) = [1,2,5]$ for some ${\bf X}$ with a rank less than 3. We have provided an example illustrating this in Figure 4 of the revision.

Q4. Page 5: "without loss of generality we may assume the group reps are all the same" why?

Reply: To enhance clarity, we have replaced the phrase “without loss of generality” with “in practice.” In practice, models are typically designed to update hidden features in the same space before the last layer. Consequently, we can reasonably assume that the group representations are all the same.

Q5. Remark 3 is vague and out of context. Corollary 3 seems related, is much more formal and accurate, but does not require the action to be faithful. Can you formally explain why faithfulness is important? This could be the critical missing link explaining why in terms of the irreducible order $\ell$ we claim to have $0<1=2=3=4=\ldots$ which is the message you are trying to convey but I don't think the theory currently supports.

Reply: We have added a detailed explanation of why faithfulness is important in Appendix B of the revised paper. Here, we elaborate on the significance of the faithfulness of representations to address your concern. The faithfulness of representations ensures that the data $\rho\big((\mathfrak{g}^{(t+1)}_i)^{-1}\mathfrak{g}^{(t)}_j\big)$ must originate from $(\mathfrak{g}^{(t+1)}_i)^{-1}\mathfrak{g}^{(t)}_j$. Without faithfulness, $\rho\big((\mathfrak{g}^{(t+1)}_i)^{-1}\mathfrak{g}^{(t)}_j\big)$ might stem from different group elements. For instance, consider the worst case where the representation is trivial. Then different $(\mathfrak{g}^{(t+1)}_i)^{-1}\mathfrak{g}^{(t)}_j$ are all mapped to the same matrix $\rho\big((\mathfrak{g}^{(t+1)}_i)^{-1}\mathfrak{g}^{(t)}_j\big) = {\bf I}$, the identity matrix. In other words, the faithfulness of representations ensures that our models do not lose this geometric information.

In practice, equivariant GNNs are designed to learn steerable features up to type $L>0$, including the learning of type-1 steerable features, which lie on faithful representations. Therefore, as we have explained in Remark 3, they exhibit the same ability to capture the geometry between local neighborhoods even as $L$ varies.

Q6. Page 6: “This question is essentially asking whether the space of all type-$l$ features has a dimension of $2l+1$” This is well defined once $f$ is fixed I believe and $X$ is free? Perhaps clarifying this will also help me understand how you define steerable features in general.

Reply: In this question, ${\bf X}$ is a fixed input and $f$ represents any equivariant function. Therefore, we are examining the space {$f({\bf X}) \mid \text{ G-equivariant } f:\mathbb{R}^{3\times m}\to V_l$}, consisting of all possible type-$l$ steerable features generated from ${\bf X}$.

Q7. Page 6: Could you specify v=f(x_1,x_2) or whatever the relation between v and x_1,x_2 is?

Reply: According to our definition of steerable features, $v = f(x_1, x_2)$ is a type-1 steerable vector that is generated by an equivariant function $f$ from the input coordinates $x_1,x_2$. The relation between $v$ and $x_1,x_2$ can be explained as follows: applying a transformation $g$ to $x_1$ and $x_2$ will result in a changed vector $g\cdot v$ due to the equivariance of $f$.

In the revision, this example has been moved to Appendix G.

Q8. I think a similar observation appears in [Villar, Lemma 3] I suggest you cite them

Reply: Thank you for bringing this to our attention. We have incorporated [Villar] into the additional related work section and provided a thorough discussion outlining the differences between their work and ours. For more details, please refer to Appendix A in the revised version.

Q9. Table 1: what are the significance of the specific three values of (L,c) chosen in Table 1? Presumably you can run eSCN with any choice of the c-s?

Reply: Table 1 demonstrates that existing comparisons made between feature type L and channels c are unfair due to the significant decrease in the dimension of the hidden features and the number of parameters. The existing ablation study uses rows 1&3, only adjusting the feature type. This results in vastly different model sizes. By contrast, comparing rows 1&2 would be a more suitable comparison in which case the number of parameters and the feature dimension are similar. In particular, maintaining the feature dimension is crucial for providing empirical evidence supporting our results.

To clarify this we have added the following sentence to the revision at the end of the section “Numerical comparisons in steerable feature types.”: The ablation study in [23] compares rows 1&3, while rows 2&3 provide a more suitable comparison.

Thank you for your continued engagement and feedback. We appreciate your interest in clarifying the theoretical aspects.

• Regarding the 1st statement, demonstrating that equivariant networks can effectively distinguish geometric graphs require strong assumptions about the injectivity of the update function (UPD), the aggregation function (AGG), and the graph-level readout function, as highlighted in Prosition 2 in [20]. The work in [20] has already proven, under these assumptions, that equivariant networks have the same expressive power as the geometric Weisfeiler-Lehman (GWL) test, enabling them to distinguish any two $k$-distinct geometric graphs ($k$ is arbitrary). Please refer to Propositions 2 and 3 in Appendix D of our revised paper for a quick review. It's worth noting that using the GWL test to establish distinguishability is preferable due to its direct connection with the task at hand. However, as the distinguishability of equivariant GNNs has been extensively explored in [20], we tend to focus on $k$-hop invariant GNNs, which cover the cases outlined in the limitation section of [20]. Our framework enables us to identify the underlying issues, specifically non-faithful representations, that limit the distinguishability of $k$-hop invariant GNNs.

• About the 2nd statement, it is easier to show that equivariant GNNs, assuming faithful representations and the injectivity of the update function, aggregation function, and graph-level readout function, have the same capability as the GWL test. Indeed, the proof of Proposition 2 in [20] can be directly applied to demonstrate this. We emphasize the significance of faithfulness in our work because the proof provided in [20] does not hold for equivariant GNNs learning features on non-faithful representations. Please refer to Appendix B in our revision for details.

We hope these responses address your concerns, and we are happy to address further questions regarding our paper.

[20] Joshi, C. K., Bodnar, C., Mathis, S. V., Cohen, T., and Lio, P. On the expressive power of geometric graph neural networks. ICML 2023.

Dear Reviewer WxNP,

Thank you for your positive response to our contribution and the increased score. We sincerely appreciate your recognition of our efforts. Regarding the proof of Theorem 2, we acknowledge its origin in [20]. We would also like to take this opportunity to commend the work done by [20], which has undoubtedly laid a solid foundation in this domain.

However, it is crucial to emphasize that the faithfulness of representations ensures the existence of injective functions, an aspect not explored in [20]. Additionally, as highlighted in our contribution, we would like to stress the novelty of Theorem 1, which conflicts with certain claims in [20], instead of that of Theorem 2.

In response to your valuable feedback, we have added a sentence at the beginning of Section 2 in the revised version to guide readers to the review of the Geometric Weisfeiler-Lehman test (GWL) and its relevant results in Appendix D. We believe this addition enhances the accessibility of the paper, especially for those unfamiliar with [20].

We are open to addressing any further questions regarding our contributions and remain committed to improving the comprehensibility of the background and the overall quality of our paper.

Q4. I find the $k$-hop contribution new but less interesting since it seems natural given the result in [20] re 1-hop. Also most geometric GNNs I believe, use 1-hop. In particular in the appendix where you discuss this all the methods you discuss are 1-hop methods.

Reply: While the findings in [20] extend easily from 1-hop to $k$-hop, it is crucial to note that [20] identified these works as limitations and suggested that integrating non-local features, like ComENet, might overcome the limitation of propagating invariant features.

However, our work fills this gap and emphasizes the significance of incorporating global features. Notably, our results suggest that the integration of non-local features may not adequately address the limitations of invariant GNNs. Our analysis indicates that considering global features is necessary, as evidenced by the performance of ComENet and LEFTNet, both classified as 2-hop methods.

It is noteworthy that designing multi-hop methods is more challenging than 1-hop methods, and they have been considered a potential solution to mitigate the limitations of invariant GNNs. Nevertheless, our work emphasizes that relying solely on multi-hop methods might not be as effective a solution as previously anticipated.

Q5. Re invariant equiformer2: From reading the paper without going into the detail of this method, it seems to be two orders of magnitude better than the 50% I would expect from your theorems. I think you should add this to the discussion and explain why you think this is (numerical errors, whatever).

Reply: The EquiformerV2 and eSCN architectures both rely on the spherical activation function introduced in eSCN which is not truly equivariant but quasi-equivariant. The details of the quasi-equivariance introduced by the architectures are discussed in Appendix D of [1] listed below. These quasi-equivariant architectures introduce error into the equivariance. This error is larger when the grid size of the spherical activation function is small as evidenced in Figure 10 of [1]. The grid size scales with $L$ and $m$ as $(2L+1)\times(2m+1)$, therefore the error is largest and the grid is smallest when $L=m=0$. We have provided more discussion on the error introduced by quasi-equivariant architectures; see Section 5 and Appendix F.1 of the revised paper for details.

[1] Saro Passaro and C. Lawrence Zitnick. Reducing SO(3) Convolutions to SO(2) for Efficient Equivariant GNNs. ICML 2023.

We hope these responses address your concerns and we are happy to address further questions regarding our paper. Thank you for considering our rebuttal.

Q4. Intuitively, where is the multi-hop aspect used in this work? The intuition about invariant features and moving frames seems true for 1-hop networks too. What is the takeaway regarding “multi-hop” architectures from this work, in contrast to 1-hop networks?

Reply: The concept of "multi-hop" in our work specifically refers to how far each message-passing step can propagate features. We present the architecture of $k$-hop geometric GNNs in Eq (1), where the variable $k$ denotes the size of neighborhoods from which a node aggregates information. In other words, it represents the maximum distance over which a node aggregates information. Additionally, we reference ComENet as an instance of 2-hop invariant GNNs in Section 1.1 since it utilizes dihedral angles.

Importantly, $k$-hop geometric GNNs are not equivalent to regular 1-hop geometric GNNs. As demonstrated in Appendix B (refer to Appendix D in the revised paper), under certain assumptions on UPD and AGG, $k$-hop invariant GNNs can distinguish any two $k$-hop distinct geometric graphs but fail to distinguish any two k-hop identical geometric graphs. This indicates that 1-hop invariant GNNs might be less powerful than their 2-hop counterparts, and this trend continues.

The reason lies in the fact that the value of $k$ influences the size of local neighborhoods, consequently affecting the input of the aggregation process. Utilizing a smaller $k$ may cause $k$-hop invariant GNNs to overlook changes in geometry between small local neighborhoods, whereas using a larger k can capture these variations.

However, it's noteworthy that, for any finite $k$, we can always construct two $k$-hop identical but $(k+1)$-hop distinct geometric graphs. In this scenario, $k$-hop invariant GNNs cannot differentiate them even with thousands of layers, while 1-hop equivariant GNNs, due to the faithfulness of features, can achieve this with $k+1$ layers. This finding sheds light on the advantages of learning steerable features up to type-$L\geq1$. We have substantiated this point through empirical results for ComENet and the addition of the LEFTNet model in Table 5 of Appendix F.1 in the revision.

To clarify this implication, we have refined the concluding remarks. In particular, we have added the following remark: relying solely on propagating invariant features confined to specific $k$-hop neighborhoods is insufficient for enabling geometric GNNs to capture the global information of geometric graphs precisely.

Q5. eSCN is an architecture evaluated in the experiments section. But is this a multi-hop architecture?

Reply: No, the eSCN architecture is a 1-hop architecture. However, to avoid confusion, we have limited the eSCN experiments to performing the ablation studies on the steerable feature type $L$ and dimension $c$.

We have updated our submission based on the reviewer's feedback, with the revision highlighted in blue. We are happy to address further questions on our paper. Thank you for considering our rebuttal.

Q3. Questions raised by weaknesses (2) and (3).

Reply: We appreciate the reviewer's feedback and concerns. In response, we have made several revisions to enhance the precision and clarity of our statements. Additionally, we have incorporated a new section (refer to Appendix A in the revised paper) to discuss additional related works, including Puny et al.'s frame averaging paper and the works by Du et al. on ClofNet and LEFTNet, and the distinctions between these existing notions and Lemma 1.

Indeed, within our framework, the function ${\bf X}\mapsto g_{\bf X}$ is exactly a singleton-valued equivariant moving frame defined in [1], if we restrict the function’s domain to where the action of $G$ is free. While Puny et al.'s frame averaging paper studies set-valued equivariant moving frames, the domain of these equivariant moving frames is still restricted to guarantee the equivariance. In contrast, our defining $g_{\bf X}$ remains well-defined even in cases where the action of $G$ is not free, making it not necessarily equivariant.

The scalarization proposed in Du et al.'s papers, while similar to how we define $\lambda({\bf X})\coloneqq \rho(g_{\bf X})^{-1} \cdot f({\bf X})$, differs in that the local frames used for scalarization are generated from two coordinates rather than all the coordinates in ${\bf X}$. In contrast, our $g_{\bf X}$ involves a consideration of all input coordinates in ${\bf X}$ since it is related to the orbit $G\cdot{\bf X}$.

To augment the paper's comprehensiveness, we have included the empirical results of ClofNet and LEFTNet on $k$-chains. In particular, we believe the invariant design for LEFTNet belongs to the class of 2-hop invariant GNNs. So it's important to highlight that LEFTNet may still face the issues discussed in the last part of Section 3.1. Please refer to the table below for further details.

Lastly, we have cited Smidt et al. (2021) to acknowledge and reference the well-established fact about the output symmetry of an equivariant function being at least the input symmetry, as noted at the beginning of Section 3.2. We believe these revisions address your concerns and provide a more precise and well-referenced discussion of related works and concepts.

K=2 & K=3

K=4

[1] Olver, P. J. (2009). Lectures on moving frames.

Thank you for your thoughtful review, valuable feedback, and endorsement. In what follows, we provide point-by-point responses to your comments.

Q1. Questions related to a few minor typos and writing note.

Reply: We appreciate your feedback, and we have made revisions to address the noted typos and have provided the requested clarifications. Specifically, we have added the definition of $V_0$, representing the 1-dimensional trivial representation of a given group $G$. Regarding the use of the calligraphic g, we acknowledge the convention that it is commonly associated with the Lie algebra. In this context, we have chosen it to distinguish from the regular g, which is used as a function in the proof of Lemma 2.

Q2. Answer the questions raised by weakness (1) and the question: Could the authors comment further on Corollary 3, regarding point (1) from the Weaknesses section? For example, doesn’t the choice of internal representation affect the universality of a given architecture (beyond just the dimension)? .**

Reply: We acknowledge two main aspects related to the concept of expressiveness: the capacity of features to carry information and the model's ability to extract that information. We intended to focus on the first aspect for the exact reason you mentioned in the passage on weakness (1) -- the latter is subject to the architecture of the model given, while the former is not. We want to clarify that our intention is not to directly conclude that the choice of equivariant representation is irrelevant up to dimension. We appreciate your effort in pointing out the lack of precision of some claims in the paper, especially the misleading conclusion derived from Corollary 3. We have carefully revised our statements to convey the implications of our results more precisely. Additionally, we have added a section in Appendix B to discuss the universality and potential methods. Here, we present a part of that discussion.

The relationship between internal representation and universality is delicate. At first glance, the topological properties of the Lie group $G=SO(3)$ (or $G=O(3)$) prevent us from constructing a continuous function ${\bf X}\mapsto g_{\bf X}$. That is, the poor regularity of $\lambda({\bf X})$ is inevitable, making any general analytical tool that relies on regularity assumption unsuitable for studying universality. However, a potential solution to this regularity issue is to extend the function ${\bf X}\mapsto g_{\bf X}$ to a function from a suitable covering space of the spatial embedding to $G$. To put it simply, lifting the spatial embedding to the covering space makes it possible to select a continuous extension of ${\bf X}\mapsto g_{\bf X}$. Once we have that, we naturally obtain an extended version of Corollary 3 with well-behaved correspondence between two representations on the covering space. As a result, we suspect that under any architecture with universality, changing the internal representation while fixing the dimension does not affect the class of function the intrinsic invariant feature $\lambda({\bf X})$ can approximate. However, we cannot deny the possibility that certain choices of representation could benefit the model, allowing it to approximate the target function with fewer layers. To demonstrate this idea will require extensive work, and given the page limitation, we leave this to future works.

Dear Reviewer nbPE,

Thank you for your further feedback and invaluable comments. We highly appreciate your endorsement and have taken your suggestions into careful consideration.

Regarding the extended version of Corollary 3 and the concept of lifting the spatial embedding to the covering space, we have included the relevant details in Appendix B of the revision. We aim to provide a clear understanding of the key concepts involved, and we hope the additional information addresses your questions. It's important to note that while we believe this extension can be proven, we have not delved into all the details and solved all technical challenges. As such, we have identified it as a potential direction for future work. We appreciate your understanding and are open to any further questions you may have.

Q1. First, the authors said, "Once we have that, we naturally obtain an extended version of Corollary 3 with well-behaved correspondence between two representations on the covering space.". Where is this extended version of Corollary 3? I am also not sure what "lifting the spatial embedding to the covering space" means, and Appendix B (unless I am looking in the wrong place) did not further clarify. What is the covering space -- could the authors give a concrete example?

Reply: We apologize for the lack of precision in our previous statement. In response to your suggestion, we have incorporated Question 1 and Question 2 into Appendix B of the revised version. Notably, Question 2 complements Corollary 3 by considering the universality aspect. Let us briefly explain it as follows. Consider two steerable vector spaces, $V$ and $W$, of the same dimension. Suppose we have a continuous $\mathfrak{G}$-equivariant function $f_V:\mathbb{X}_3 \to V$. According to Corollary 3, which implies the existence of a $\mathfrak{G}$-equivariant function $f_W:\mathbb{X}_3\to W$ with the same corresponding invariant functions as $f_V$. However, the continuity of $f_W$ cannot be guaranteed, and hence it may not be approximated by a given architecture. Therefore, in Question 2, we ask if there exists a continuous $\mathfrak{G}$-equivariant function $f'_W:\mathbb{X}_3\to W$ such that $f'_W$ and $f_W$ coincide on any compact set except an $\epsilon$-small measurable subset. By applying the universality of the given architectures, we can approximate $f'_W$, which has a corresponding invariant function identical to that of $f_V$ on any compact set except an $\epsilon$-small measurable subset. Then we delve into a detailed explanation of a potential method to address Question 2 in Appendix B. In particular, we describe how we adopt a similar approach used to obtain the equivariant moving frame in Puny et al.'s frame-averaging paper to construct the concrete covering space. Intuitively, the covering space $\widetilde{\mathbb{X}_3}$ of a space $\mathbb{X}_3$ is a topological space that locally looks like multiple copies of $\mathbb{X}_3$. We show that we have the desired continuity and equivariance of the function $g_X$ on these copies. Leveraging these properties, we should be able to construct the desired $f'_W$ for any given $f_W$. For a more in-depth understanding, please refer to the details provided in Appendix B.

Q2. Generally speaking, I stand by my endorsement of the paper, but I agree with reviewer WxNP that it is rather important to make these central claims formal theorems for a final version (including those added newly to Appendix B)

Reply: We appreciate your valuable feedback. To enhance the formality of our claims, we have introduced formal results, namely, Theorem 2 and Proposition 8 (originated from Remark 3 in the old version), to elucidate the significance of faithful representations. Additionally, we have presented formal statements, namely, Questions 1 and 2, to delve into the universality aspect and have proposed potential methods to address Question 2. It's important to note that we've used the term "Question" instead of "Theorem" or "Proposition" as we have not yet verified all the details and have decided to leave this as future work. Nevertheless, we hope these improvements in the formality of our claims and statements address your concerns.

We hope these responses address your concerns, and we are happy to address further questions regarding our paper. Thank you for considering our rebuttal.