Thank you for your comments! We provide the following responses to your concerns:

W1: More detailed background knowledge and conceptual explanations are needed to improve clarity.

Thank you for your valuable suggestions! We will further summarize the formal definitions of steerable representations at the end of Section 3.1, and introduce the concept of CG tensor product in Section 3.1 and compare its difference with Eq.(6) in Section 4. Furhter eblations will be added in Examples 3.2 and 3.3. We sincerely thank you once again, and will further enhance the clarity of our paper by providing more explanations related to geometric GNNs, Euclidean symmetries, and Examples 3.2 and 3.3.

Q1: The significance of re-scaling on spherical harmonics.

Thanks for your comments! The re-scaling part corresponds to the radial function, which is widely used in previous papers such as TFN. Here, for convenience, we combine the radial function into the formulation of the spherical harmonics to improve the readability of our paper.

Q2: Origin of O(3) group representation.

Nice suggestion! This representation method is conventional and can be found in many physics-related literature, such as [a,b]. More general and theoretical content of representation theory can be found in some pure mathematical books, such as [c,d]. We will add the necessary citations around Eq.(2) in the revised version.

[a] Landau, L. D., and E. M. Lifshitz. Quantum Mechanics: Non-Relativistic Theory.

[b] Chen, J. Q., Jialun Ping, and Fan Wang. Group Representation Theory for Physicists.

[c] Weyl, Hermann. The Classical Groups: Their Invariants and Representations.

[d] Fulton, William, and Joe Harris. Representation Theory.

Q3: It is helpful to explicitly point out that Eq. (3) refers to graph-level features.

Thank you for raising this point, which is very important for improving the readability of our paper. We will clearly mention this in subsequent revisions.

Q4: What information is lost in the scalarization trick compared to the CG tensor product?

Thanks for this valuable question! Our HEGNN exclusively passes invariant quantities (inner products of high-degree representations) between features $\tilde v^{(l)}$ of different degrees $l$, unlike the expensive CG tensor product used in TFN, which considers all possible interactions between different frequencies. Our model can be seen as a generalization of the scalarization trick in EGNN to high-degree representations. While the scalarization trick might somehow sacrifice model expressivity in theory, it has shown significantly better efficacy and efficiency in practice compared to conventional high-degree models, as demonstrated by EGNN paper and also our experiments here. Additionally, Theorem 4.1 indicates that passing inner products of full degrees is sufficient to recover the information of all angles between each pair of edges, affirming the theoretical expressivity of our HEGNN in characterizing the geometry of the input structure. As suggested, we will add a new paragraph for the comparision between Eq.(6) and CG tensor products in Section 4.

Q5: Explanation of the results in Table 2.

Thank you for your comments. The experiment setup in Table 2 is designed from the GWL paper [a]. It requries to first embed the input graphs through an equivariant neural network (e.g. EGNN, HEGNN, TFN), and then classify them through a simple classifier. As for the phenomenon that HEGNN does not achieve 100% in some cases, it is probably because the classifier is not trained well. We observe that accuracy can be improved to 100%, if increasing the number of training epochs to a sufficiently large value (e.g. 2,000).

The high std is due to the settings used in the GWL paper. There are only 10 test groups in total, and each group is to classify two graphs into two classes. The classification accuracy can only be 0%, 50%, and 100% for each group, which might exlain why the std will be relatively large.

[a] Joshi C K, Bodnar C, Mathis S V, et al. On the expressive power of geometric graph neural networks.

Q6: The reason for the performance gap between EGNN and HEGNN$\_{l\leq 1}$.

Very insightful comment! We speculate that why HEGNN with $l\leq 1$ outperforms EGNN is mainly owing to HEGNN initialization (Eq.(5)), which is equivalent to an additional massage passing layer, and increases the depth of the neural network. To show this, we additionally test the effects of EGNN-4layer, HEGNN-3layer, and HEGNN-4layer on the N-body dataset, and the results are as follows. All HEGNNs only used 1-st steerable feature. It can be found that the performance of EGNN-4layer is very close to HEGNN-3layer.

Table S8: Comparison between EGNN and HEGNN on N-body

Q7: How to achieve reflection equivariance with HEGNN?

Thank you for this valuable question. Since the steerable features are initialized by spherical harmonics in Eq.(5), reflection equivariance is associated with the transformation $(-1)^lD^{{l}}(\mathfrak r)$, where $D^{{l}}(\mathfrak r)$ is the rotation representation. In other words, the sign of the steerable feature changes if $l$ is odd, and keeps unchanged if $l$ is even. Besides, during the message passing in our model, all steerable features only interact with the scalar (i.e. the inner product), without changing their original equivariance. Hence, reflection equivariance is always represented by $(-1)^lD^{{l}}(\mathfrak r)$ throughout our model.

Q8: It is suggested to move Table. 5 to the main text.

Thank you for your suggestion. We will include it in the main text in the revised version.

Minor: Some typos.

Thank you very much. We will proof-read our paper again and have the typos fixed.

Thank you for your comments! We provide the following responses to your concerns:

W1 & Q2: HEGNN should be tested on conventional dataset splits and compared with new baselines such as ClofNet.

Nice suggestion! We have additionally conducted experiments on standard N-body benchmarks (train/valid/test = 3k/2k/2k), and compared our method with more state-of-the-art methods including ClofNet as well as its variant ClofNet-vel [a], MACE [b] and SEGNN [c], thanks to the availability of their open-source code. The results on standard split and our original protocol are reported in Table S5. It is observed that our HEGNN clearly outperforms existing methods in both cases, indicating the general effectiveness of our model.

W2 & Q1 & Q4: Is it possible to test the performance of HEGNN in different prediction targets and different application fields to further illustrate its versatility?

Thanks for your valuable suggestion. We choose the prediction of future atoms as our task, as it is equivariant, indicating that the input and output spaces share the same coordinate system. With this task, we are capable of evaluating if the output of the model can retain the full geometry including orientation information of the input after multi-layer message passing. We understand that additional experiments on different tasks or domains (e.g. force field prediction or molecular property prediction) are helpful, which, yet, are not the main focus of this paper and are better left for future exploration.

W3: It is necessary to supplement baselines such as GMN and GMN-L.

Thanks for the reminder. As suggested, we have further included the results of GMN-L in the experiments on MD17 in Table S6, and added the comparisons with GMN in the N-body experiment in Table S5. Our HEGNN-6 generally outperforms GMN, and achieves comparable performance to GMN-L in most cases. Given that GMN-L requires careful handcrafting of constraints for chemical bonds into the model design, our model's ability to derive promising results without such enhancements supports its competitive performance. These results will be added to the revised paper.

Table S5: Results of N-body dataset under two partitions

Table S6: Results on MD-17 with GMN & GMN-L

W4 & Q3: Comparison of various indicators at runtime between HEGNN and other high-degree steerable models.

Thanks! The main difference between our HEGNN and traditional high-degree models is that we employ inner products for message exchange between the representations of different degrees, while TFN resorts to CG tensor products to consider all possible interactions between different representations. By denoting the maximum degree as $L$, the complexity of our inner products is equal to $\textstyle\sum_{l=0}^L(2l+1)=(L+1)^2=O(L^2)$, whereas the complexity of CG tensor products is derived as $O(L^6)$ by [a]. The following tables further report the inference times and model sizes of the high-degree models in the 100-body case of the N-body dataset. It is verified that our HEGNN obtains better performance with lower computation cost, compared to TFN, SEGNN, and MACE.

Table S7: Parameters and inference times of models

[a] Passaro S, Zitnick C L. Reducing SO (3) convolutions to SO (2) for efficient equivariant GNNs.

Thank you for your comments! We provide the following responses to your concerns:

W1. Differences and connections with GWL.

Thank you for raising the comparison with the GWL paper [1]. Here, we would like to further highlight the difference between [1] and our paper:

1. Different Motivations: the GWL paper aims to investigate the expressivity of different geometric GNNs from the perspective of WL test, whereas our paper focuses more on exploring the necessity of involving high-degree representations. The GWL test paper has discussed rotational inner-symmetry, but the analyses are only derived experimentally. Upon the formal definition of symmetric graphs, we are able to derive rigorous and theoretical results to explain the expressivity degeneration of equivariant GNNs on symmetric graphs in Table 1. Besides rotational inner-symmetry, we have also investigated regular polyhedra, the derivations of which are not trivial and rely heavily on our derived Theorem 3.6.
2. Different Evaluation Scopes: while the GWL paper only discusses the phenomenon of rotational inner-symmetry degradation in a single section, we thoroughly explore more general kinds of symmetric graphs in our experiments. In addition, to evaluate the practical effectiveness of our proposed model, we conduct experimental comparisons on the N-body and MD-17 tasks. The performances align with our theoretical findings in Theorem 3.6 and Theorem 4.1, verifying the strong expressive power of our model.

W2. Stability of symmetric structures under perturbations.

Thanks for your question. We agree that it is unlikely to encounter exactly symmetric inputs in practice, but this does not indicate that our theoretical analyses are NOT practically meaningful. Even though symmetry-breaking factors will make the geometric graph deviate from the symmetric state, the deviated graph is still roughly symmetric. In other words, the outputs of equivariant GNNs on the derivated graphs keep close to zero if the degree value is chosen to be those in Table 1 according to Theorem 3.5 and 3.6, which wil still lead to defective performance. To explore this further, we take the tetrahedron as an example and compare the cases of EGNN, HEGNN$\_{l= 3}$, and HEGNN$\_{l\leq 3}$ when adding noise perturbations.

Table S3: Resluts under perturbations

Here, $\varepsilon$ represents the ratio of noise, and the modulus of the noise obeys $\mathcal{N}(0,\varepsilon\cdot\mathbb{E}[||\vec x-\vec x_c||]\cdot I)$. It can be observed that the performance of EGNN is slightly improved in the presence of noise (from $50$% when $\varepsilon=0.01$ to $60$% when $\varepsilon=0.5$), while HEGNN demonstrates better robustness. The symmetry-breaking factors you mentioned are very interesting, and the results above will be included in our revised paper.

W3. The difference between Theorems 3.5, 3.6, and Theorem 4.1.

We are sorry for any potential confusion. Since our proposed HEGNN has involved high-degree representations, it does directly address the original problem given in Theorem 3.5 (3.6). The purpose of introducing Theorem 4.1 is to further demonstrate the enhanced expressivity of our HEGNN by using the inner products of full degrees. In other words, Theorem 3.5 (3.6), and Theorem 4.1 discuss different aspects of the benefits by our model.

W4. The performance of MACE on the N-body dataset and the MD-17 dataset should be provided.

Thanks for your reminder. We have additionally tested MACE on the N-body dataset. Due to the expensive running cost, we reduce the channel number of MACE from 64 to 8 (but out of memory still occurs in the 100-body case). Please note that MACE also resorts to CG tensor products similar to TFN, and it is thus consistent that its performance is worse than our model.

Table S4: Results of MACE and HEGNN on N-body dataset

Thank you for your comments! We provide the following responses to your concerns:

Q1: The performance gap between scalarization-based models (e.g. HEGNN) and high-degree steerable models (e.g. TFN, SEGNN).

It is true that our HEGNN exclusively passes invariant quantities (inner products of high-degree representations) between features $\tilde v^{(l)}$ of different degrees $l$, unlike the expensive CG tensor product used in TFN or SEGNN, which considers all possible interactions between different frequencies. Our model can be seen as a generalization of the scalarization trick in EGNN to high-degree representations. While the scalarization trick might somehow sacrifice model expressivity in theory, it has shown significantly better efficacy and efficiency in practice compared to conventional high-degree models, as demonstrated by EGNN paper and also our experiments here. Additionally, Theorem 4.1 indicates that passing inner products of full degrees is sufficient to recover the information of all angles between each pair of edges, affirming the theoretical expressivity of our HEGNN in characterizing the geometry of the input structure.

Q2: Comparison with SEGNN is important.

Nice suggestion! We have additionally compared our model with SEGNN in the table below, using the default settings from the public code. It is observed that SEGNN performs significantly worse than our model (in Table S1). We conjecture that the equivariant non-linear functions applied to the CG tensor products in SEGNN make it difficult to converge to a desirable solution during training, resulting in suboptimal performance. We will include these results in the revised version.

Q3: The relationship between runtimes and model degree.

Thank you for this valuable observation. While the model size does scale quadratically with the maximum frequency $L$, this does not mean that the exact runtimes in Table 3 follow the same pattern. By leveraging PyTorch, which is based on CUDA toolkit, computations for different frequencies can be largely parallelized. As a result, the actual implementation cost is lower than the quadratic complexity.

Q4: SOTA on N-body and MD-17.

Thanks for the reminder. For the N-body dataset, we follow the settings in FastEGNN [a], which was recently published and can be regarded as the SOTA model. As shown in the table below, in all cases from 5-body to 100-body, our HEGNN models generally outperform FastEGNN, verifying the effectiveness of our model's design.

For the MD-17 dataset, we apply the evaluation protocol from the GMN paper [b]. We thoroughly checked the methods in the citations of this paper and found that the GMN-L method proposed in [b] generally performs the best, thus consider it the SOTA method. Our HEGNN-6 achieves comparable performance to GMN-L in most cases. Given that GMN-L requires careful handcrafting of constraints for chemical bonds into the model design, our model's ability to derive promising results without such enhancements supports its competitive performance.

[a] Zhang Y, Cen J, Han J, et al. Improving Equivariant Graph Neural Networks on Large Geometric Graphs via Virtual Nodes Learning.

[b] Huang W, Han J, Rong Y, et al. Equivariant graph mechanics networks with constraints.

Table S1: Results on N-body

Table S2: Results on MD-17

Q5: The relationship between Theorems 3.5 and 3.6.

Thanks for your comment. In fact, Theorem 3.5 addresses the more general case, while Theorem 3.6 is a special case for practical convenience (since $I-0=I$ is full-rank).

For practical purposes, we consider only the finite symmetric group because only geometric graphs with one or two nodes can exhibit infinite symmetric groups, representing single atoms or diatomic molecules in physics [a]. We do not consider these cases due to their trivial topological structure in realistic tasks. Generalization to compact subgroups can be easily achieved by replacing summation with integration, with volume elements chosen to be normalized and invariant to group action.

From a mathematical perspective, Theorems 3.5 and 3.6 essentially calculate the dimension of the invariant space of a group. This can be achieved by calculating the trace of the projection map of the subgroup [b]. Generalization to compact subgroups is more naturally done by introducing the projection map of compact subgroups, as directly found in [b]. However, for readability, we have not delved deeply into representation theory in this paper. In our future work, we may present our results and further findings using the language of representation theory.

[a] Landau, L. D., and E. M. Lifshitz. Quantum Mechanics: Non-Relativistic Theory.

[b] Fulton, William, and Joe Harris. Representation Theory.

Q6: Efficiency of operators in Line 226.

Yes, this is a more efficient way. In this form, we are able to apply e3nn, which is the most commonly used library for processing spherical harmonics.

We sincerely appreciate your further comments. We provide more explanations to address your concerns below.

Q1: The relationship between model time consumption and degree.

Your question is both valuable and inspiring! We sincerely apologize for our previous unconsidered response. Upon re-evaluating the complexity of our model, we now believe that your insight is correct—the model's bottleneck lies not only in the value of the degree $L$ but also in the number of channels. As illustrated in the following table, for our models with different $L$ values, the number of channels for type-0 features is fixed to 65, while it is no more than 3 for higher-type features. In other words, the computations for type-0 features account for the majority of the overall running cost. This could explain why the runtimes in Table 3 do not scale quadratically with respect to $L$. Additionally, there are other factors that affect the model's time consumption, including the initialization in Eq. (5) and the coefficient calculations in Eqs. (5) and (8).

Thank you once again for your constructive comment! We will definitely include the above explanations around Table 3 in the revised paper.

Table S9: Total dimensions used in HEGNN of different degrees

Q2: The gap in performance of SEGNN between our results and original paper.

Yes, the performance discrepancy of SEGNN can be attributed to the use of a different dataset than the one employed in the original SEGNN paper [21]. In our paper, we utilized the dataset constructed by [a] to provide a more thorough evaluation of the compared methods. This dataset includes a wider range of scenarios, spanning from 5 bodies to 100 bodies, beyond the 5-body case studied in [21]. It is important to note that [a] employed different preprocessing and dataset splitting compared to [21], which may explain why SEGNN's performance differs even in the same 5-body scenario as [21].

In addition to the dataset in [21], here we have conducted additional experiments using the SEGNN dataset, following the official code from the SEGNN paper for dataset construction. We then re-evaluate the performance of EGNN, SEGNN, and our proposed HEGNN models across various scenarios, ranging from 5 bodies to 100 bodies. The results are presented in the following table.

Table S10: Comparison between EGNN, SEGNN and HEGNN on N-body from [21]

As observed, SEGNN's performance closely aligns with the results reported in its original paper for the 5-body case (0.50 vs. 0.43), supporting the reliability of our implementation. However, SEGNN shows significantly higher losses in scenarios with a larger number of bodies, indicating a decline in performance as task complexity increases. We conjecture that the steerable nonlinear functions used in SEGNN could make the learning process more difficult when the number of bodies increases.

Although our proposed HEGNN model performs worse than SEGNN in the 5-body case, it consistently exhibits lower loss across various configurations, demonstrating its superior ability to handle increasing task complexity.

We appreciate your valuable attention to this detail and will include the above results and analyses to further strengthen the validity of our findings.

[a] Huang W, Han J, Rong Y, et al. Equivariant graph mechanics networks with constraints.

Thanks for the reply, I think that cleared most of my doubts! I encourage the authors to include these details and results in the final version of the manuscript.

I still feel like the novelty is limited, but the paper includes some interesting insights and the proposed method seems effective and seems to be thoroughly evaluated. For these reasons, I maintain my positive recommendation.