Thank you for your valuable reviews. If you have any additional comments or concerns, please do not hesitate to let us know.

Responses to weekness.

We clarify that ID, OOD Ads, OOD Cat, OOD Both are four tasks in OC20 (different data), not metrics. Our original sentence in abstract was that " demonstrating its state-of-the-art performance on several tasks". We did not say that we achieved SOTA results on all tasks. The average scores in Table 1 can comprehensively evaluate the models. In Table 1, we achieve the best average scores and achieve 4.4% improvement. As for the subtasks, although our method did not achieve the best results on the ID and OOD Cat tasks, they both achieved the second best results and were very close to the first place. Note that OC20 is a representative data because it represents a large swath of chemical space. In QM9 tasks, we also achieved second place in multiple indicators, very close to first place.

Responses to question 1.

We discuss relevant theorem and experiments in Appendix A.3. Our discussion is based on two theoretical works [1] [2]. The theorem 2 in [1] asserts that any equivariant function can be approximated by a CG tensor product with infinite degree, though practical constraints limit it to a finite low degree. Therefore, some complex equivariant pattern can not be fitted by CG produt with low degree (please refer to Table 2 in [2]). Concurrently, based on the approximation properties of MLP, we recognize MLP's capability to approximate any continuous function, which includes equivariant functions (please refer to figure 1 in [1]). Our intention is to leverage MLP to overcome the limitations of CG tensor products with low degrees. Table 6 further illustrates MLP's proficiency in learning higher-degree patterns when embedding is in a low-degree representation. Due to space constraints and the non-original nature of the aforementioned theorem, we provide a more detailed discussion in the Appendix.

[1] On the Universality of Rotation Equivariant Point Cloud Networks. Nadav Dym, Haggai Maron. ICLR 2021

[2] On the Expressive Power of Geometric Graph Neural Networks. Chaitanya K. Joshi, Cristian Bodnar, Simon V. Mathis, Taco Cohen, Pietro Li.

Responses to question 2.

Firstly, we did not assess the MD17 dataset. Given its small size, it poses challenges for HDGNN to learn the correct equivariance (other methods use inherent equivariance, expect for SCN). Our work is designed to enhance expressive power by relaxing equivariance, enabling networks to comprehend intricate physical interactions between atoms. Consequently, HDGNN is expected to have good performance across a large swath of chemical space. OC20 stands out as representative due to its substantial size (O(100M)) and diversity (55 elements). Additionally, Equiformer presented some results involving data augmentation on OC20 experiments, such as noisy nodes and IS2RS. This may be attributed to Equiformer's competitive nature (OC20 challenge), where the authors tend to achieve optimal MAE. In contrast, our study focuses on only comparing the performance of pure models. The basic training environment (without augmentation) can provide an accurate evaluation.

Since part of the equivariance of HDGNN comes from learning, SO(3)-transformation augmentation of MD17 may allow HDGNN to have a good performance. But this will lead to unfair comparisons (the results of other methods use the original dataset). In addition, we are sorry that we cannot complete the data augmentation experiments on OD20 during the rebuttal time due to limited resources (OC20 experiments take up a lot of resources). We will publish these results in the public community in the future. Nevertheless, outcomes derived from data augmentation may not precisely represent the true performance of the model, as it remains unclear which specific component contributes to the observed improvement. We believe that the pure model comparisons in our Table 1 and Table 2 can provide a significant evaluation for HDGNN.

We appreciate your acknowledgment of our work. If you have any additional comments or concerns, please do not hesitate to let us know.

I raised my score a bit (from 3 to 5) due to a lacking validity of the mentioned weakness 1 from my side. Even though the results are great, I cannot recommend accept because I am still not convinced about the motivation behind the method, it still seems like the design is flawed but somehow this flaw is turned into a feature. In particular the notion of "approximate equivariance" is not convincing, since there seems nothing equivariant about the method left after all the "approximations". I elaborate my concerns below. I highlighted four statements [A], [B], [C], [D], which I hope you could still respond to.

Regarding weakness 1

Thank you for your patience with me, and I apologize for my rushed review. My statement that equation 9 was incorrect, was incorrect. The identity is clearly true. My listed weakness number was not valid, furthermore, I think it has been appropriately addressed in the new text. However, stating that randomness does not affect equation 9, does not solve the following issue.

Regarding weakness 3

Apologies for the poor latex formatting, somehow it wouldn't render the subscripts correctly

What I was concerned about were actually the next steps, after the decomposition using equation 9. My comment number 3 addresses this concern of mine, and it is still not fully resolved. To clarify my thoughts, firstly, let us validate when the operations are equivariant and when not, and check the validity of changing the original tensor product $\mathbf{D}(\mathbf{R}_{ij})^{-1} (\mathbf{D}( \mathbf{R}{ij} ) \mathbf{x}_i) \otimes \mathbf{S}(\vec{\mathbf{C}})$ by $\mathbf{D}(\mathbf{R}{ij})^{-1} NN(\mathbf{D}(\mathbf{R}{ij}) \mathbf{x}_i)$.

For now assume a well-defined unique rotation $\mathbf{R}{ij}$ such that $\mathbf{r}{ij}=\mathbf{R}{ij}^{-1} \vec{\mathbf{C}}$, as defined in the paper, then a global rotation action on the point cloud would lead to

and thus a global rotation leads to $\mathbf{R}{ij} \mapsto \mathbf{R}{ij} \mathbf{R}^{-1}$. If all layers are equivariant, then the corresponding transformation to feature vectors $\mathbf{x}i$ would be via $\mathbf{x}i \mapsto \mathbf{D}(\mathbf{R}) \mathbf{x}i$. The corresponding transformation to the NN layer would thus be

and we nicely have that the output of such a transformation is still steerable via a Wigner-D matrix applied on the left, namely we have $\mathbf{D}(\mathbf{R}{ij})^{-1} NN(\mathbf{x}') \mapsto \mathbf{D}(\mathbf{R}) \mathbf{D}(\mathbf{R}{ij})^{-1} NN(\mathbf{x}')$. The equivariance is due to the fact that $\mathbf{x}'$ is invariant to rotations.

Let us see what happens if $\mathbf{R}{ij}$ is not well defined, as we have estabilshed in the first round of this rebuttal. Then we in fact have that $\tilde{\mathbf{R}}^{-1} \mathbf{R}{ij}$ would be valid for any $\tilde{\mathbf{R}}$ that is a rotation around the z-axis. Then effectively any $\mathbf{D}(\tilde{\mathbf{R}}^{-1})\mathbf{x}'$ could have been obtained. The $NN$ is then no longer receiving an invariant input and now $\mathbf{D}(\mathbf{R}{ij})^{-1} \mathbf{D}(\tilde{\mathbf{R}}) NN(\mathbf{D}(\tilde{\mathbf{R}}^{-1}) \mathbf{x}') \mapsto \mathbf{D}(\mathbf{R}) \mathbf{D}(\mathbf{R}{ij})^{-1} \mathbf{D}(\tilde{\mathbf{R}}) NN(\mathbf{D}(\tilde{\mathbf{R}}^{-1})\mathbf{x}') \, ,$ and since the $NN$ is not equivariant the Wigner-D matrix will not compensate for the arbitrary rotation $\tilde{\mathbf{R}}$.

Thus when stating "replacing with a MLP will break the equivariance", it is not due to mere use of the MLP, since as just shown above this is actually no problem, but it only becomes a problem when a well-defined rotation matrix $\mathbf{R}_{ij}$ is lacking. [A] The paper is still a bit suggestive in it's presentation as it gives the impression that the use of the MLP breaks equivariance, but it is rather the free rotation $\tilde{\mathbf{R}}$.

Responses to weekness 7 in detailed comments.

We first clarify the equivariance of each term and purpose of each equation. Equation 10 is strictly equivariant, and $NN(\mathbf{x})$ and $NN(\mathbf{x}'')$ in equation 11 are both non-equivariant. Equation 9-11 are expansion of Equation 7 which is based on equivariant CG tensor product. We mentioned that Equation 7 is widely used in other equivariant models (the sentence after Equation 7). However, finite degree limits expressiveness. In equation 11, we aim to replace CG product by MLP to improve expressiveness.

Second, we introduce the idea of decomposition. If there is no decomposition, then Equation 9 will be approximated by $\mathbf{D}_{-1}(\mathbf{R} _{ij})NN(\mathbf{x}')$. This will cause two problems: 1. The randomness in $\mathbf{x}'$ will affect the learning of MLP. 2. We hope that MLP can capture directional features of edge. In equation 9, these features are stored in the wigner-D matrix ($\phi$ is random, but $\theta$ is well-defined). However, the input of MLP ($\mathbf{x}'=\mathbf{D}\mathbf{x}$) is a whole, and we cannot expect MLP to capture the information in $\mathbf{D}$ separately from this whole. Through the decomposition of equation 10 and equation 11, one of the branches $NN(\mathbf{x} _{i})$ is not affected by randomness. Moreover, we feed $(\mathbf{D}-\mathbf{I})\mathbf{x}$ and $\mathbf{x}$ to the MLP, making it possible to extract the directional features in $\mathbf{D}$. We discussed this decomposition in the sentences following Equation 11.

We add some content here to facilitate understanding the role of MLP. We use MLP to learn the CG product process $f(\mathbf{x}) = (\mathbf{x} \otimes \mathbf{S}(C))$. Due to MLP's ability to approximate any function, it can even extend the learned function to higher-degree CG tensor products. Table 6 verified that MLP can learn features in higher-degree representations.

Responses to weekness 8 in detailed comments.

$\mathbf{a} _{ij}$ is the attention coefficient, they are generated by invariant branches. The formula is $MLP(\mathbf{f} _i, \mathbf{f} _j, \mathbf{c} _{ij})$, which corresponds to the green module in figure 1. Thank you for pointing this out, we have added a description of it in the new version. The attention coefficients are used to automatically adjust the weights of non-equivariant and equivariant operations. Due to limited space, the details of invariant branch $\mathbf{f}$ are put in Appendix B.1.

Responses to weekness 9 in detailed comments.

Equations 13 and 14 represent the Fourier transform on the sphere. Spherical harmonic can be regarded as frequency representation of Fourier transform on a sphere (Appendix A.1.2). The CG tensor product can be regarded as a special convolution process. Based on the convolution theorem, we can use point-wise operations in the time domain to approximate the convolution process in the frequency domain. Therefore, in equation 13, we first convert the signal in the frequency domain to the time domain, and then perform a point-wise MLP in the time domain. Finally, we convert it to the frequency domain through equation 14. We do this to reduce the amount of computation. The purpose of the update block is to calculate the interaction between different channels of each node embedding. If we reference the CG product to each channel pair, the computational cost will become very high. This approach also cannot consider the interaction between three or more channels. Although the point-wise MLP in the time domain is only an approximation, its complexity is much smaller and interactions between mutiple channels are considered.

Responses to weekness 10 in detailed comments.

When we perform point-wise operations in time domain, the point refers to each direction vectors $\mathbf{r}$. Taking the convolution theorem as an example, assume that node embedding in the frequency domain has only two channels, denoted by $m _1$ and $m _2$. We use $p _1$, $p _2$ to denote their signals in time domain. We can use point-wise multiplication $p _1 \circ p _2$ to approximate convolution ($\circ$ is the Hadamard product). For each point $\mathbf{r}$, the output is $p _1(\mathbf{r}) * p _2(\mathbf{r})$. Now, we have C channels and use MLP as point-wise operation. The calculation at point $\mathbf{r}$ can be expressed as $MLP([p _1(r), p _2(r), ..., p _C(r)])$. Moverover, we extend to correlation so also include $\mathbf{-r}$.

Responses to weekness 11 in detailed comments.

We regard CG tensor product as a special convolution because some paths in CG tensor product are similar to the form of convolution. However, the paths outside of the convolution will be missing. We supplement the missing path through correlation. For example, if we calculate CG product between two irreducible representation $m_1$, $m_2$. We assume $L=4$, and $m_1$, $m_2$ and $m'=m_1 \otimes m_2$ contain type-0, type-1, $\cdots$, type-4 vectors. (type-$l_1$, type-$l_2$) denotes a path based on $m_1$ and $m_2$. Taking type-2 vector of $m'$ as an example, it is the aggregation of paths (1, 1), (0 , 2), (2, 0), (1, 3), (3,1), (4,2), and (2,4) (refer to equation 5). However, convolution only contains (1, 1), (0, 2), and (2, 0) paths (refer to equation 32). This mehtod will weaken the equivariant interactions they learn. We introduce correlation whose corresponding paths are (0, 2), (2, 0), (1,3) , (3,1), (4,2), and (2,4), including paths that are not belong to convolution. In equation 5, convolution/correlation corresponds to $(l_{1}+l_{2}= l)/|(l_{1}-l_{2}|=l)$ paths. The conversion of correlation to the time domain is an operation between ($\mathbf{r}$) and ($\mathbf{-r}$). Therefore, we concatenate the ($\mathbf{r}$) and ($\mathbf{-r}$) of all channels in the time domain.

Although the difference between convolution and correlation is only one negative sign, they correspond to different paths of CG tensor product in the frequency domain. We discussed this in appendix B.3. Table 11 also proves that introducing correlation can improve performance.

Responses to weekness 12 in detailed comments.

Thanks for your reviews, we have updated this sentence.

Responses to weekness 13 in detailed comments.

Your understanding is correct. We have updated this sentence to make a rigorous expression. This sentence is used to compare with SCN. What we want to express is that we introduce equivariance operations while learning equivariance, improving expressiveness while also preserving equivariance (the equivariance of SCN is mainly from learning). We are glad that we have similar insights as you on the limitations of CG tensor product (band-limit degree). In our paper, this band-limit degree is represented by "finite degree". We mainly discussed in the second paragraph of the introduction and section 2.4.

Responses to weekness 14 in detailed comments.

Thanks for your review. we treat it as a conjecture in new version to make a rigorous expression. Note that the previous sentence of this sentence is a comparison of OOD. OOD represents unseen distribution (we briefly introduced OOD in section 5.1). Atomic energy is related to some physical mechanisms, such as Coulomb and van der Waals forces. However, due to the properties of machine learning, what the model learns may not be the general physical mechanism, but some special mechanisms in training data. HDGNN performs better on unseen distribution, so we infer that what it learned may be closer to the real physical mechanism.

Responses to weekness 15 in detailed comments.

These scores ​​are the average of OOD tasks (OOD ads, OOD Cat, and OOD Both). The average scores in the Table 1 is the average of all tasks (ID, OOD ads, OOD Cat, OOD Both).

Responses to weekness 16 in detailed comments.

We have observed overfitting in all the tasks of QM9. For example, The final training MAE is much lower than the validation MAE (in the strictly equivariant model, they are similar, you can refer to the open training log of Equiformer). The reason for overfitting is that the dataset of QM9 is relatively small, and HDGNN cannot learn the equivariance under SO(3) transformation. If we perform SO(3) transformation augmentation, this overfitting may be alleviated. However, this method will bring an unfair comparison (other methods use the original dataset). Thanks for your suggestion, we will evaluate rMD17 in the open source community in the future (sorry that the available resources are currently limited). QM9 and IS2RE are also purely invariant, and lead a comprehensive comparison to SOTA molecular models. Many advanced methods have published code and results on them.

Responses to weekness 17 in detailed comments.

The MLP in equation 12 introduces learned equivariance. The METP introduces inherent equivariance.

Responses to weekness 18 in detailed comments.

Thanks for your suggestion. We have updated.

Responses to weekness 19 in detailed comments.

Please refer to our response to 4-th main comments.

Responses to weekness 20 in detailed comments.

Thank you for your suggestion. The experiments assessing speed efficiency are presented in Table 13 in the Appendix. Furthermore, we have added the complexity analysis of our operations in Appendix B.2 and B.3. This update includes a comparative assessment of the efficiency between our proposed structure and the CG tensor product.

Responses to weekness 4 in main comments.

The results in Table 1 and Table 2 represent the generalization ability on validation, which reveal whether models can accurately decipher the underlying physical or chemical principles inherent in molecular models. In our ablation experiments, we systematically examine how specific sub-modules in HDGNN influence the generalization ability. In essence, generalization ability is influenced by both expressive power and equivariance. The modules in HDGNN impact these two aspects. When equivariance is severely damaged, generalization ability will also be severely reduced ("Unrestricted" in Table 3). Additionally, insufficient expressive power also constrains generalization ability. In Table 3, we employ strictly equivariant, approximately equivariant, and unconstrained operations to compute messages within our model framework. The results confirm that approximate equivariance is able to enhance generalization. Table 4 assesses our attention method, which is designed to dynamically adjust the ratio of non-equivariant to equivariant operations ($a _{ij}$ in Equation 12). This outcome illustrates our method's endeavor to strike a balance between expressive power and equivariance for improved overall generalization. Table 5 is dedicated to validating the detail structure or design idea of the core module. A more extensive evaluation is provided in the Appendix. These experiments serve to facilitate the reproducibility and customization of HDGNN.

Responses to weekness 1 in detailed comments.

Thank you for your review, we have updated this sentence to make a rigorous expression. We are sorry that our expression caused you misunderstanding and hope our rebuttal can clarify it. What we want to express is that the constraints in equivariant operation (sband-limit degree) can impact the model's expressive capacity, leading to inefficient learning. In this context, inefficient learning refers to the model's inability to capture accurate mathematical relationships between atoms.

Responses to weekness 2 in detailed comments.

We have included the analysis of expressive power in the Appendix for two main reasons: 1. Given the constraints on text length, our primary aim is to maximize the introduction of our main contribution—HDGNN. While the analysis of expressive power serves as a crucial motivation, the intricate details of this aspect are complex. Consequently, we can only provide a concise overview of the core ideas. 2. The theorems related to expressive power in equivariant models are not our contributions. Excessive description in the main text may lead to misunderstanding of our contribution. Therefore, we only put the most important conclusions in the introduction and section 2.4.

Responses to weekness 3 in detailed comments.

Thanks for your suggestion, we have updated.

Responses to weekness 4 in detailed comments.

Thank you for your suggestion. We have updated additional details and discussions concerning $\mathbf{R} _{ij}$. In practice, we first produce normalized vector of $\vec{\mathbf{r}} _{ij}$, denoted as $\vec{\mathbf{a}}$. We then randomly sample an orthogonal vector $\vec{\mathbf{b}}$ of $\vec{\mathbf{r}} _{ij}$ on the sphere, and calculate another orthogonal vector $\vec{\mathbf{c}}$ through cross product. $\mathbf{R} _{ij} = [\vec{\mathbf{b}}^{T}; \vec{\mathbf{c}}^{T}; \vec{\mathbf{a}}^{T}]$. This approach ensures orthogonality of $\mathbf{R} _{ij}$ and is similar to produce $\mathbf{R} _{ij}$ by random $\varphi$.

Responses to weekness 5 in detailed comments.

MLP breaks equivariance because its functional form is non-equivariant to SO(3)-transformation. MLP does not conform to the definition in Equation 1, when $T(g)$ denotes SO(3)-transformation. In simpler terms, when a 3-D vector $\mathbf{x}$ (or an irreducible representation) is input into MLP, the outcome violates the equality $NN(\mathbf{R} \mathbf{x}) \not= \mathbf{D} NN(\mathbf{x})$. Here, $\mathbf{R}$ represents any rotation and $\mathbf{D}$ represents corresponding Wigner-D matrix.

For example, we assume that the input $\mathbf{x}$ is based on the spherical harmonic basis. We can use $\mathbf{D}^{-1}(\mathbf{R} _{ij})$ to rotate it back only when $NN(\mathbf{D}(\mathbf{R} _{ij}) \mathbf{x}) = \mathbf{D}(\mathbf{R} _{ij}) NN(\mathbf{x})$. However, an untrained MLP cannot satisfy this condition (its output is not based on the spherical harmonic basis).

Randomness will amplify the impact of non-equivariance of MLP. When we replace $\mathbf{x}_i \otimes \mathbf{S}(C)$ with a untrained MLP in equation 9, different $\mathbf{R} _{ij}$ lead to different results. In this case, the equality of equation is not true, and the equivariance on the right side breaks down.

Responses to weekness 6 in detailed comments.

Yes, the randomness is caused by free rotation angle.

Thank you for your valuable reviews. We have incorporated many of your suggestions to make the article clearer and more rigorous (Modified parts are highlighted in red). To streamline your review process, we offer guidance on locating specific responses. Answers concerning randomness can be found in response 1 of the main comments, and responses 4-6 of the detailed comments. Insights into breaking equivariance operations are addressed in response 3 of the main comments, along with responses 5 and 7 of the detailed comments. Clarification on the Fourier transform on the sphere is provided in responses 9-11 of the detailed comments. If you have any additional comments or concerns, please do not hesitate to let us know.

Responses to weekness 1 in main comments.

While acknowledging the existence of multiple rotation transformations $\mathbf{R} _{ij}$ capable of rotating $\vec{\mathbf{r}} _{ij}$ to [0,0,1], the equality in Equation 9 remains valid. In detail, we use two rotation parameters, namely $\theta$ and $\varphi$ (changes in the polar and azimuth angles), to denote $\mathbf{R} _{ij}$. $\varphi$ is random when $\mathbf{R} _{ij}$ rotate $\vec{\mathbf{r}} _{ij}$ to [0,0,1]. We randomly select one parameter configuration and consistently apply it throughout the subsequent processes. With the choosen $\mathbf{R} _{ij}$, we get $\mathbf{D}^{-1}(\mathbf{R} _{ij})\mathbf{D}(\mathbf{R} _{ij})=\mathbf{I}$. Consequently, we initially convert the left side of Equation 9 to $(\mathbf{D}^{-1}(\mathbf{R} _{ij})\mathbf{D}(\mathbf{R} _{ij})\mathbf{x} _{i}) \otimes (\mathbf{D}^{-1}(\mathbf{R} _{ij})\mathbf{D}(\mathbf{R} _{ij}) \mathbf{S}(\vec{\mathbf{r}} _{ij}))$. Based on the definition of $\mathbf{x}' _{i}$ and the Wigner-D matrix, this formula transforms into $(\mathbf{D}^{-1}(\mathbf{R} _{ij})\mathbf{x}' _{i}) \otimes \mathbf{D}^{-1}(\mathbf{R} _{ij})\mathbf{S}(\mathbf{C})$. Additionally. The CG product is equivariant for all rotations (refer to Equation 26 in the Appendix). Consequently, the above formula can be transformed into the right side of Equation 9.

In summary, equation 9 can not be affected by randomness because of the equivariance of CG product. The $\mathbf{D}^{-1}$ is the transpose of $\mathbf{D}$ in the implementation (They are based on the same $\mathbf{R}_{ij}$). In this case, the outputs of the right side of equation 9 is the same when we choose different $\mathbf{R}_{ij}$. We deliberately break equivariance in Equation 11, incorporating the non-equivariant MLP (You can refer to Responses to 7-th detailed comment).

Reading reponse to the 5-th detailed comment together may better address your concerns.

Thank you for your valuable suggestions. We updated two parts in the new version: 1.We provide a more rigorous reference and description to the local reference frames. 2.We will add the disscusion that Equation 9 will not influence by randomness.

Responses to weekness 2 in main comments.

Figure 1 may appear intricate for two main reasons:

1. We show most of the details to improve its reproducibility. Analogues can be found in other MPNNs, such as the Figure 2 in GemNet [1] and Figure 1 in Equiformer [2]. While the design of many MPNNs may seem complex, it's important to note that most of modules are fixed. Therefore, we can bypass the complex hyper-parameter design in engineering and easily apply models to various molecular tasks. Our approach is similar. In our ablation (Appendix E.4), we show that HDGNN mainly replies on four hyper-parameters ($K$, $C$, $L$, $L'$).

2. Some operations depicted in our Figure 1 share the concise high-level scientific ideas. For instance, the neural network paths in Multi-TP share the similar non-equivariant design. The modules in Figure 1 can be summarized to two scientific ideas: 1.the design of approximately equivariant modules; 2.the combination between equivariant modules and non-equivariant modules. We discuss the motivation of ideas in introduction and section 2.4, the evolutions process of ideas in section 3.2 and 3.3. We have emphasized the core scientific ideas in new version and indicated their corresponding building blocks.

[1] Gasteiger J, Becker F, Günnemann S. Gemnet: Universal directional graph neural networks for molecules. Neurips 2021.

[2] Liao Y L, Smidt T. Equiformer: Equivariant graph attention transformer for 3d atomistic graph. ICLR 2023

Responses to weekness 3 in main comments.

Thank you for the suggestion. We will address this in the new version. We clarify that two components in Equation 11 break equivariance. This is because the non-equivariance of MLP (You can refer to Responses 5 to detailed comments). The equivariant component in message block is METP operation. Note that the final message is calculate by Equation 12. We use equations 8 and 11 to introduce the equivariant and non-equivariant parts of equation 12, respectively.

Thank you for your prompt reply. We are delighted to discuss equivariance with you. We update paper again based on your review. Please forgive our lengthy discussion, we hope that all concerns can be addressed well.

Response to statement [A]

Thanks for pointing this out. As you say, although we mentioned randomness in the paper, we did not elaborate on its impact on equivariance. We have updated expression in the paper. In short, our modification clarifies that the destroy of equivariance arises from both ill-defined rotation and the non-equivariance of the MLP. As you noted in the last formula, ill-defined transformation $\mathbf{D}(\tilde{\mathbf{R}})$ lead a term $\mathbf{D}(\tilde{\mathbf{R}}) NN(\mathbf{D}(\tilde{\mathbf{R}}^{-1}) \mathbf{x}')$, and the non-equivariance of MLP prevents $\mathbf{D}(\tilde{\mathbf{R}}^{-1}) NN(\mathbf{x}') = NN(\mathbf{D}(\tilde{\mathbf{R}}^{-1}) \mathbf{x}')$. This ultimately leads to the breakdown of equivariance.

Responses to statements [B] and [C]

Let us assume that through training, MLP can learn perfect equivariance, that is, $NN(\mathbf{D}(\mathbf{R})\mathbf{x}) = \mathbf{D}(\mathbf{R}) NN(\mathbf{x})$. We will discuss whether MLP can learn equivariance in Responses to statements [D].

For your statement [C]. If we replace $\mathbf{x}''$ with $\mathbf{x}'$ in Equation 11, the equality of Equation 11 does not hold, and the right hand side is not invariant. In this case, the right side is expressed as $\mathbf{D}^{-1}(\mathbf{R} _{ij}) NN (\mathbf{x}') + \mathbf{D}^{-1}(\mathbf{R} _{ij}) NN(\mathbf{x}) = NN(\mathbf{x}) + \mathbf{D}^{-1}(\mathbf{R} _{ij} ) NN(\mathbf{x})$. The first term is equivariant but the second term is not.

We clarify that Equation 10 is a identity transformation of Equation 9 and the original CG tensor product. Tensor product satisfies $(A + B) \otimes C = A \otimes C + B \otimes C$, so we can decompose Equation 9. We noticed that you mentioned "equation (10) is a reparametrization of the original tensor product, and the identity holds due to linearity of the tensor product". If what you want to express is the same as our understanding, please forgive our repeated explanation. Based on this fact, we cannot replace $\mathbf{x}''$ with $\mathbf{x}'$ for the first MLP in Equation 10.

For statement [B]. In ideal case, $NN(\cdot)$ is strictly equivariant, and two conditions will be satisfied. First, $NN((\mathbf{D} - \mathbf{I}) (\mathbf{R})\mathbf{x}) = (\mathbf{D}- \mathbf{I})(\mathbf{ R}) NN(\mathbf{x})$. Second, $\mathbf{x}''$ is still an equivariant representation when $\mathbf{x}$ is an equivariant representation. This is because addition operation is equivariant. These two conditions ensure that ill-defined $\mathbf{R}_{ij}$ can be offset in the equation 11.

In reality, it is difficult for MLP to learn perfect equivariance. For the analysis of this imperfect equivariance in equation 11, you can refer to response to statement [D].

We appreciate your suggestion "split the expression into an non-equivariant and equivariant part". In fact, we attempted to substitute one of the terms in equation 11 with a CG product. However, any changes resulted in a degradation of performance. You can observe the outcomes of our previous experiments in the table below. Here, $\mathbf{x}$ denotes the replacement of $NN(\mathbf{x}{i})$ with $\mathbf{x}{i} \otimes \mathbf{S}(\mathbf{C})$. We employed a small HDNN (K=6, L=4) on the IS2RE dataset (MAE on ID task).

Response to statement [D]

The function form of last term (METP) in equation 12 is strictly equivariant. We first discuss whether MLP can learn equivariance, and secondly, we discuss the performance of the last term (METP) in an approximately equivariant system.

MLP is able to approximate continuous functions, including all equivariant functions (refer to figure 1 in [1]). Let's consider a simple case where we let MLP learn the length of a vector. If training set contains all SO(3) transformations of vectors, it is possible for MLP to successfully learn the function for calculating the length. In real, the size of the training set is very important. The more SO(3) transformation types the MLP receives, the more likely the MLP is to learn the correct equivariance.

Equation 11 expands the SO(3) transformation through another method. Note that the MLP remains the same in the equation. The desired output is achieved only when the MLP exhibits equivariance for both $\mathbf{D}-\mathbf{I}$ and $\mathbf{I}$ transformations. To measure equivariance, we introduce an error term. Let $\mathbf{\lambda}$ be a constant vector. For all transformations and a given MLP, there exists $\mathbf{\lambda} > | NN(\mathbf{D} \mathbf{x}) - \mathbf{D}NN(\mathbf{x}) \|$. Considering Equation 11 and the version without decomposition ($\mathbf{D}^{-1}(\mathbf{R} _{ij}) NN(\mathbf{x}')$), when the input undergoes a random transformation, the upper limit of error for the version without decomposition is $\mathbf{D}^{-1}(\mathbf{R} _{ij}) \mathbf{\lambda}$. Using the L2 Norm provides a stable upper limit, denoted as $e _{1} = | \mathbf{D}^{-1}(\mathbf{R} _{ij}) \mathbf{\lambda} | = | \mathbf{\lambda} |$. However, the upper limit of Equation 11 is $e _{2} = | 2 * \mathbf{\lambda}|$. The training process aims to reduce the upper limit of error through the loss function between output and label. If equation 11 and the version without decomposition achieve the same upper limit ($e _{1} = e _{2}$), the equivariant error of MLP needs to satisfy ($|\mathbf{\lambda} _{1}| = 2 | \mathbf{\lambda} _{2} |$). In summary, the decomposition in equation 11 reduces the equivariant error of each MLP after training. When $\mathbf{\lambda} _{2}$ is small enough, we can refer to it as approximately equivariant. Our MAD in Table 4 demonstrates the equivariance error of the overall model.

The essence of METP is CG tensor product. It remains strictly equivariant. You may worry that $\mathbf{x}$ is approximate spherical harmonics basis based. However, the equivariance on low-degree representations will be influence Slightly. For example, the CG tensor product on type-1 vectors is equivalent to the cross product of 3D vectors. No matter what value the type-1 vector takes. The output of CG tensor product is strictly equivariant. Additioanlly, combination with attention coefficients is an equivariant process, since they are invariant.

For more rigorous expression, we have updated the explanation of equation 11. Furthermore, we also indicate that approximately equivariance requires an efficient training. Please check section 3.2 in updated paper.

Finally, we provide our macro-level expectations for MLP. In our equation, MLP replaces the function CG tensor product with a certain degree. But in fact, what it approximates may be a more complex function, such as a higher-order tensor product. Some operations can promote effective learning of MLP. For example, it is also possible to learn CG tensor product using $NN(\mathbf{x} _{i}, \vec{\mathbf{r}} _{ij})$. However, transforming to the z-axis can lead to a more efficient learning, because it can transform the tensor product into a simple matrix multiplication. Similarly, we expect that the MLP in Equation 11 can learn some equivariant information in $\vec{\mathbf{r}} _{ij}$. This information may be useful for embedding angular information (such as bond angles, dihedral angles). We consider that separated output is more conducive to parsing out direction information.

[1] On the Universality of Rotation Equivariant Point Cloud Networks. Nadav Dym, Haggai Maron. ICLR 2021

Thanks again for your prompt reply.

Response to the comment "Thank you for the added details. One last question regarding ..."

Yes, we have tried using equation 9, as evidenced by our ablation experiments ($NN(\mathbf{x}'')$ in Table 5). The experiment involved replacing $NN(\mathbf{x}'') + NN(\mathbf{x})$ with $NN(\mathbf{x}')$ and showed $NN(\mathbf{x}'') + NN(\mathbf{x})$ method achieves better result. Noticeable gaps were observed in our previous experiments, particularly on small datasets (like qm9).

It is worth noting that learning how to deal with randomness is essentially learning about equivariance. Since only equivariance ensures that the ill-defined $\tilde{\mathbf{D}}$ inside the brackets can compensate for the $\tilde{\mathbf{D}}^{-1}$ outside the brackets, thereby eliminating randomness (like what you say in your second review). In either case, whether it is $NN(\mathbf{x}')$ or $NN(\mathbf{x}'') + NN(\mathbf{x})$, equivariance of $NN(\cdot)$ is common learning goal. From this perspective, starting from Equation 9 does not increase the learning facilitation.

I am sorry that we made some errors when submitting our responses to openreview, which may have resulted in multiple emails you receive. We have responded to your two new concerns. You can check in the webpage.

Response to the comment "Thank you also for clarifying the approximation accuracy. Firstly ..."

Your understanding is correct. Equivariance is just a learning goal, and the process of approaching equivariance must be data-driven. As I mentioned in the second round of reponse to your statement [D], if the training set contains all possible SO(3) transformations, the output of the neural network is likely to satisfy $NN(\mathbf{R}x)=\mathbf{R}NN(x)$. This is based on the knowledge that continuous functions contain equivariant functions.

We now have two MLP terms, but they share weights, representing the same function. Under the equal performance of overall model, the equivariant error of MLP structure will be better if we use $\mathbf{x}''$ and $\mathbf{x}$. However, in our experiments, the training process tends to make the equal equivariance bound of individual MLP, thereby making the equivariance bound of the overall model (using $\mathbf{x}''$ and $\mathbf{x}$) better.

Let’s briefly discuss the benefits of introducing MLP. MLP brings an advantage and a disadvantage. The advantage is that MLP can fit more complex equivariant functions compared to low-degree CG tensor products. The disadvantage is that its equivariance is data-driven, which needs sufficient molecular data. In engineering, this disadvantage may be alleviated by SO(3)-transformation augmentation of the training set. Satisfactory models is possible in a variety of molecular tasks. In our paper, we do not introduce SO(3) augmentation as it leads to an unfair comparison.