Higher-Rank Irreducible Cartesian Tensors for Equivariant Message Passing

We thank the reviewer for their positive assessment of the manuscript and have addressed each point they raised below. All numerical results for the experiments conducted for this review are presented in Tabs. 1, 2, and 3 of the attached PDF. We also include Fig. 1, illustrating inference time and memory consumption of ICTP and MACE as a function of $L$ and $\nu$.

W1: To demonstrate the advantage of a Cartesian approach we can consider computational complexities for equivariant convolutions ($\mathcal{O}(E N_\text{ch} 9^{L} L!/(2^{L/2}(L/2)!))$ and $\mathcal{O}(EN_\text{ch}L^5)$ for ICTP and MACE, respectively) and the product basis ($\mathcal{O}(M N_\text{ch} K (9^{L} L!/(2^{L/2}(L/2)!))^{\nu-1})$ and $\mathcal{O}(M N_\text{ch} L^{\frac{1}{2}\nu(\nu + 3)})$, respectively). For more details on the asymptotic computational complexity, see W2.1 and Q1 by the Reviewer 69Cm and Q3 by the Reviewer qF29. Furthermore, Tab. 1 and Fig. 1 assess the inference time and memory consumption varying rank of messages $L$ and tensors embedding atomic environments $l_\text{max}$; we also vary the number of contracted tensors $\nu$. Our results in W2.1 by the Reviewer 69Cm and Q3 by the Reviewer qF29 demonstrate that MACE scales worse than ICTP when increasing $\nu$. Particularly, for MACE with $L=3$, we could raise $\nu$ to a maximal value of 4 since, for larger values, we obtained an OOM error on an NVIDIA A100 GPU with 80GB. Spanning the $\nu$-space more efficiently may be important to improve the model's expressive power for tasks requiring correlations of higher body orders. These correlations become more important when, e.g., environments are degenerate with respect to lower body orders, and higher accuracy is required [B, G].

Apart from the theoretical complexity and obtained inference times, as seen from an implementation perspective, symmetric tensors allow for more efficient implementations and algorithms for the general matrix-matrix multiplications (GEMM), which PyTorch has not yet provided. Finally, our approach can exploit the symmetry of tensors when computing forces and stresses, omitting the transpose calculation.

Q1: Our approach includes TensorNet and CACE as special cases. A TensorNet-like architecture could be defined with $\nu=2$ and $L = l_\text{max} = 2$, though with equivariant convolution filters. We provide the corresponding results in Tab. 3. We found model configurations with $\nu=3$ and $L=2$ (and $L=1$) outperform the model configuration with $\nu=2$ and $L = l_\text{max} = 2$ by factors of $\leq$ 1.4 and $\leq$ 1.2 in energy and force RMSEs, respectively. Tab. 2 in the manuscript demonstrates a better accuracy for ICTP by $\leq$ 2.3 compared to CACE. We will add this discussion to the revised manuscript.

More specifically, TensorNet uses rank-2 reducible Cartesian tensors to embed atomic environments and decomposes them into irreducible ones before computing products with invariant radial filters, i.e., before computing messages. It includes explicit 3-body features in a message-passing layer since it computes a matrix-matrix product between node features and messages. CACE uses reducible higher-rank Cartesian tensors to embed local atomic environments and their full tensor contractions (see also MTP or GM-NN) to build invariant many-body filters. Our approach uses exclusively irreducible Cartesian tensors for embedding environments, equivariant convolutions, and product basis. Thus, we do not mix irreducible representations during our many-body message passing. We use irreducible tensor products for the equivariant convolution and go beyond invariant filters. Finally, we systematically construct equivariant many-body messages.

Q2: We added results for ICTP and MACE for the large-scale Ta-V-Cr-W data set [C], which is diverse and includes 0 K energies, forces, and stresses for 2-, 3-, and 4-component systems and 2500 K properties in 4-component disordered alloys [C]. It contains 6711 configurations with sizes ranging from 2 to 432 atoms in the periodic cell. We were experimenting with this data set before the rebuttal and performed a hyperparameter search for both models to obtain suitable relative weights for energy, force, and virial losses. No configuration for MACE provides competitive accuracy for energies and forces simultaneously. Tab. 3 shows that MACE at most matches the accuracy of ICTP on forces but is typically outperformed by a factor of $\leq$ 2.0 on energies.

Results are obtained by averaging over 10 independent runs. Best performances are highlighted in bold. Inference time and memory consumption are measured for a batch size of 50. Inference time is reported per atom in μs; memory consumption is provided for the entire batch in GB.

All values are obtained by averaging over five independent runs. Best performances are highlighted in bold. Inference time and memory consumption are measured for a batch size of 10. Inference time is reported per structure in ms; memory consumption is provided for the entire batch in GB.

All values are obtained by averaging over five independent runs. Best performances are highlighted in bold. Inference time and memory consumption are measured for a batch size of 100. Inference time is reported per structure in ms; memory consumption is provided for the entire batch in GB.

We thank the reviewer for their constructive assessment of the manuscript. We have addressed each point they raised below. All numerical results for the performed experiments are presented in Tabs. 1–3 and Fig. 1 of the attached PDF.

W1: We added a large-scale data set that aims to assess the model's performance on a varying number of atom types/components and relaxed (0 K) as well as high-temperature structures. The Ta-V-Cr-W data set is diverse and includes 0 K energies, forces, and stresses for 2-, 3-, and 4-component systems and 2500 K properties in 4-component disordered alloys [C]. It contains 6711 configurations with sizes ranging from 2 to 432 atoms in the periodic cell. We were experimenting with this data set before the rebuttal and performed a hyperparameter search for both models to obtain suitable relative weights for energy, force, and virial losses. No configuration for MACE provides competitive accuracy for energies and forces simultaneously. Tab. 3 shows that MACE at most matches the accuracy of ICTP on forces but is typically outperformed by a factor of $\leq$ 2.0 on energies.

We decided not to use MD22 and QM9 since they do not include variations in atom types or MD trajectories, respectively.

W2: We agree that we could better motivate our approach regarding the computational advantages compared to spherical tensors. Considering results from Q3 and computational complexities (Q1 by the Reviewer 69Cm) for equivariant convolutions ($\mathcal{O}(E N_\text{ch} 9^{L} L!/(2^{L/2}(L/2)!))$ and $\mathcal{O}(EN_\text{ch}L^5)$ for ICTP and MACE, respectively) and the product basis ($\mathcal{O}(M N_\text{ch} K (9^{L} L!/(2^{L/2}(L/2)!))^{\nu-1})$ and $\mathcal{O}(M N_\text{ch} L^{\frac{1}{2}\nu(\nu + 3)})$, respectively), employing irreducible Cartesian tensors offers more than simply replacing mathematical expressions. Indeed, ICTP is more efficient in covering larger $\nu$ (and larger $L$ if $\nu$ is large) than MACE; see also Q3.

We expect our approach to inspire the development of computationally efficient models and frameworks using strategies other than those employed for spherical ones. We also demonstrate improved efficiency by leveraging the symmetries of tensor products and coupled product features; see Tab. 2 in the manuscript. Apart from the above, symmetric tensors allow for more efficient implementations and algorithms for the general matrix-matrix multiplications (GEMM), which PyTorch has not yet provided. Finally, our approach can exploit the symmetry of tensors when computing forces and stresses, omitting the transpose calculation.

Q1: Our approach includes TensorNet as a special case with $\nu=2$ and $L = l_\text{max} = 2$, though with equivariant convolution filters; for results, see Tab. 3. Please see also W1 and Q2 by the Reviewer 69Cm. TensorNet uses reducible Cartesian tensors to embed atomic environments and decomposes them into irreducible ones before computing products with invariant radial filters. It includes explicit three-body features since it computes a matrix-matrix product between node features and messages. Our approach uses exclusively irreducible Cartesian tensors for embedding atomic environments, equivariant convolutions, and product basis. Thus, we do not mix irreducible representations during message passing. We use irreducible tensor products for the equivariant convolution and go beyond invariant filters. Finally, we systematically construct equivariant many-body messages.

Q2: We see no hurdle to applying our approach to $N$-center properties such as single-particle Hamiltonian matrices; see [D]. Our approach also includes all operations required for [E] and [F].

Q3: We agree that an ablation study on $L$ and $\nu$ would improve our work; see also W2.1 and Q1 by the Reviewer 69Cm. Tab. 1 and Fig. 1 show the inference time and memory consumption for varying ranks $L$ (messages) and $l_\text{max}$ (tensors embedding environments); we also vary the number of contracted tensors $\nu$. Indeed, ICTP outperforms MACE for most parameter values. Particularly, ICTP allows spanning the $\nu$-space more efficiently and, thus, improves models' expressive power for tasks requiring correlations of higher body orders. These correlations become more important when, e.g., environments are degenerate with respect to lower body orders, and higher accuracy is required [B, G].

ICTP is also more computationally efficient if $\nu = 3$ and $L \leq 4$. Note that $L \leq 4$ is sufficient for most applications in physics; see, e.g., [A]. For neighborhood orientations with $L$-fold symmetries, however, at least rank-$L$ tensors may be required [B]. These symmetries are typically lifted in atomistic simulations. Fig. 1 shows that for $L > 4$, Cartesian models will also be advantageous if $\nu > 4$. These results agree with our complexity analysis in Q1 by the Reviewer 69Cm.

Q4: When we mention an interatomic potential's transferability, we refer to its ability to accurately predict energies, forces, and stresses for crystal structures, temperatures, and stoichiometries on which it was not trained; see [77] in the manuscript.

Q5: We agree that spherical and irreducible Cartesian tensors (also reducible ones) should have comparable expressive power for a fixed $L$ since both tensors are related through a linear transformation; see [40, 49, 59-61] in the manuscript. However, our results in Q3 demonstrate that MACE scales worse than ICTP when increasing $\nu$, i.e., when increasing the expressive power of the model. Particularly, for MACE with $L=3$, we could raise $\nu$ to a maximal value of 4 since, for larger values, we obtained an OOM error on an NVIDIA A100 GPU with 80GB. We also conducted a careful hyperparameter tuning for MACE and ICTP to ensure a fair comparison, expecting similar energy and force errors due to the comparable expressive power of spherical and Cartesian tensors.

Results are obtained by averaging over 10 independent runs. Best performances are highlighted in bold. Inference time and memory consumption are measured for a batch size of 50. Inference time is reported per atom in μs; memory consumption is provided for the entire batch in GB.

All values are obtained by averaging over five independent runs. Best performances are highlighted in bold. Inference time and memory consumption are measured for a batch size of 100. Inference time is reported per structure in ms; memory consumption is provided for the entire batch in GB.

All values are obtained by averaging over five independent runs. Best performances are highlighted in bold. Inference time and memory consumption are measured for a batch size of 10. Inference time is reported per structure in ms; memory consumption is provided for the entire batch in GB.

Sorry for the oversight. I was not paying so much attention to the general response. That addresses most of my concern. But I am still worried that the performance/efficiency improvements is not significant and stronger baselines (such as Equiformer V2) or more standardized datasets (such as MD22/OCP as mentioned by Reviewer 69Cm) are needed.

Dear Reviewer,

We thank you for your prompt response. We noticed that other reviewers cannot read your comment. Therefore, we will add it to allow them to follow our discussion:

Thank you for your response.

I am curious on how you trained the Ta-V-Cr-W systems. Did you train them jointly or separately? If separately, can model train > on low temperature extrapolates to high temperature? How long does it take to train the system? Also, how do you handle the > heterogeneity in the system for predicting the energies and forces? Can I find a reference for the dataset?

We have addressed each of your questions below:

• We train ICTP and MACE using all Ta-V-Cr-W subsystems simultaneously. Particularly, as already stated in the general response, all models are trained using 5373 configurations (4873 are used for training and 500—for early stopping), while the remaining 1338 configurations are reserved for testing the models' performance. The performance is tested separately using 0 K binaries, ternaries, quaternaries, and near-melting temperature four-component disordered alloys.

• Models trained exclusively on 0 K subsystems are not expected to generalize to near-melting temperature four-component disordered alloys. The 0 K subsystems span: (i) different atomic combinations for relaxed binary, ternary, and quaternary systems; (ii) different low-temperature ordering in the Ta-V-Cr-W family (B2 ordering, B32 ordering, random binary solid solution, BCC interface); (iii) all possible phase separations on the TaVCrW lattice (B2/B2 ordering, B2/B32 ordering, B32/B32 ordering, B2/random binary ordering, B32/random binary ordering, random binary/random binary ordering). None overlaps sufficiently in local environments with high-temperature (2500 K) disordered structures. For more details on the data set, we refer to the "Description of the data set" section of the original publication [C].

• Training a single model requires up to 12 hours on an NVIDIA A100 GPU with 80GB.

• We did not implement any specific step for handling the heterogeneity in the Ta-V-Cr-W data set. We only increased the mini-batch size to 32 for both models to account for energy statistics and reduced the relative weight of the force loss.

• For the dataset, we have referenced [C] in our previous response (K. Gubaev, V. Zaverkin, P. Srinivasan et al.: Performance of two complementary machine-learned potentials in modelling chemically complex systems. npj Comput. Mater. 9, 129 (2023)). The data set can be accessed via the link: https://doi.org/10.18419/darus-3516.

We hope we could properly address your questions and awaiting on your response.

Dear reviewer,

We again noticed that other reviewers cannot read your comment. Therefore, we will add it to allow them to follow our discussion:

Sorry for the oversight. I was not paying so much attention to the general response. That addresses most of my concern. But > I am still worried that the performance/efficiency improvements is not significant and stronger baselines (such as Equiformer > V2) or more standardized datasets (such as MD22/OCP as mentioned by Reviewer 69Cm) are needed.

We want to point out that the official review does not mention comparing to an additional baseline, such as EquiformerV2. Besides that, baselines, such as MACE, Allegro, NequIP, TensorNet, and CACE, which we used in our work, are current state-of-the-art models. Therefore, we do not see how adding experimental results for EquiformerV2 could further contribute to demonstrating the performance and efficiency advantages of our approach.

We evaluated ICTP using rMD17, 3BPA, and Acetylacetone, which other state-of-the-art models commonly use. Also, as requested by the reviewer, we included another, more challenging data set (Ta-V-Cr-W) and motivated our choice. Including yet another benchmark data set, such as MD22 or OC20/22, would not improve the value of our work. As we already explained, the suggested MD22 data set does not include variations in atom types. Thus, it would not provide additional insights beyond those we acquired with rMD17; the models' performance would again be tested on vibrational degrees of freedom of a single molecule. Furthermore, and contrary to the Ta-V-Cr-W data set, OC20/22 does not allow systematic evaluation of the models' performance across different crystal structures, temperatures, and stoichiometries.

We would appreciate further clarification on the reviewer's concerns regarding the models and data sets used in the manuscript and the response to the official review. We are eager to better understand the specific reasons why our current evaluation is not convincing so far.

We thank the reviewer for their valuable feedback and comments. We have addressed each of their points below and included all new results in Tabs. 1–3 and Fig. 1 of the attached PDF.

W1: We agree that discussing these points would improve our work. Indeed, our approach includes TensorNet and CACE as special cases. A TensorNet-like architecture could be defined with $\nu=2$ and $L = l_\text{max} = 2$, though with equivariant convolution filters; see Tab. 3. For the computational complexity, see W2.1 and Q1.

Unlike TensorNet and CACE, which use invariant radial (both) and many-body (CACE) filters, we propose equivariant convolution filters based on irreducible Cartesian tensors. Additionally, while TensorNet and CACE embed atomic environments using reducible tensors, we use exclusively irreducible ones and ensure that irreducible representations are not mixed during message-passing. TensorNet decomposes reducible rank-2 tensors before computing messages and builds explicit 3-body features. In contrast, our approach introduces the product basis using irreducible Cartesian tensor products and systematically constructs equivariant many-body messages. The limited message-passing mechanisms in TensorNet and CACE restrict their architectures and expressive power. Our approach enables the systematic construction of O(3) equivariant models and enhances their expressive power.

W2.1: We agree that an ablation study on $L$ and $\nu$ would improve our work. Tab. 1 and Fig. 1 show the inference time and memory consumption for varying ranks $L$ (messages) and $l_\text{max}$ (tensors embedding environments); we also vary the number of contracted tensors $\nu$. Indeed, ICTP outperforms MACE for most parameter values. Particularly, ICTP allows spanning the $\nu$-space more efficiently and, thus, improves models' expressive power for tasks requiring correlations of higher body orders. These correlations become more important when, e.g., environments are degenerate with respect to lower body orders, and higher accuracy is required [B, G].

ICTP is also more computationally efficient if $\nu = 3$ and $L \leq 4$. Note that $L \leq 4$ is sufficient for most applications in physics; see, e.g., [A]. For environments with $L$-fold symmetries, however, rank-$L$ tensors may be required [B]. These symmetries are typically lifted in atomistic simulations. Fig. 1 shows that for $L > 4$, Cartesian models will also be advantageous if $\nu > 4$. These results agree with our complexity analysis in Q1.

W2.2: We added a large-scale data set that aims to assess the model's performance on a varying number of atom types/components and relaxed (0 K) as well as high-temperature structures. The Ta-V-Cr-W data set is diverse and includes 0 K energies, forces, and stresses for 2-, 3-, and 4-component systems and 2500 K properties in 4-component disordered alloys [C]. It contains 6711 configurations with sizes ranging from 2 to 432 atoms in the periodic cell. We run a hyperparameter search for both models to obtain suitable relative weights for energy, force, and virial losses. No configuration for MACE provides competitive accuracy for energies and forces simultaneously. Tab. 3 shows that MACE at most matches the accuracy of ICTP on forces but is typically outperformed by a factor of $\leq$ 2.0 on energies.

We decided not to use OC20/22 as it does not allow systematic evaluation of models' performance across different crystal structures, temperatures, and stoichiometries, which can facilitate further method development.

W2.3: Tab. 2 shows ICTP/MACE results with $\nu=1$, i.e., using only two-body interactions. ICTP has accuracy comparable to or better than that of MACE and inference times smaller by 1.13.

Q1: For irreducible Cartesian tensors, the computational complexity of a tensor product is $\mathcal{O}(9^{L} L!/(2^{L/2}(L/2)!))$; see Section B3. Thus, for two-body interactions, we get $\mathcal{O}(E N_\text{ch} 9^{L} L!/(2^{L/2}(L/2)!))$ ($E$: number of edges; $L$: tensor rank; $N_\text{ch}$: number of features). The Clebsch-Gordan (CG) tensor product in spherical models has the complexity of $\mathcal{O}(L^5)$. Thus, for an equivariant convolution, we get $\mathcal{O}(EN_\text{ch}L^5)$. For many-body interactions, we obtain $\mathcal{O}(M N_\text{ch} K (9^{L} L!/(2^{L/2}(L/2)!))^{\nu-1})$ and $\mathcal{O}(M N_\text{ch} K L^{5(\nu-1)})$ for ICTP and MACE, respectively ($M$: number of nodes; $\nu$: number of contracted tensors; $K = \text{len}(\eta_\nu)$: all possible $(\nu-1)$-fold tensor contractions). Spherical models can use generalized CG coefficients, resulting in $\mathcal{O}(M N_\text{ch} K L^{\frac{1}{2}\nu(\nu + 3)})$. The factor $K$ is removed in MACE by restricting the parameterization to uncoupled features, i.e., we have $\mathcal{O}(M N_\text{ch} L^{\frac{1}{2}\nu(\nu + 3)})$. However, this choice of the product basis makes MACE more computationally efficient than ICTP only for large $N_\text{ch}$ and small $\nu$; see also W2.1. Tab. 2 in the manuscript also shows that leveraging the symmetry of tensor products and coupled features improves the computational efficiency of ICTP.

Q2: ICTP includes TensorNet as a special case with $\nu=2$ and $L = l_\text{max} = 2$, though with equivariant convolution filters; see also W1 and Tab. 3. TensorNet uses reducible Cartesian tensors to embed atomic environments and decomposes them into irreducible ones before computing products with invariant radial filters. It includes explicit three-body features since it computes a matrix-matrix product between node features and messages. Our approach uses exclusively irreducible Cartesian tensors for embedding environments, equivariant convolutions, and product basis. Thus, we do not mix irreducible representations during message passing. We use irreducible tensor products for the equivariant convolution and go beyond invariant filters. Finally, we systematically construct equivariant many-body messages.

Results are obtained by averaging over 10 independent runs. Best performances are highlighted in bold. Inference time and memory consumption are measured for a batch size of 50. Inference time is reported per atom in μs; memory consumption is provided for the entire batch in GB.

All values are obtained by averaging over five independent runs. Best performances are highlighted in bold. Inference time and memory consumption are measured for a batch size of 100. Inference time is reported per structure in ms; memory consumption is provided for the entire batch in GB.

All values are obtained by averaging over five independent runs. Best performances are highlighted in bold. Inference time and memory consumption are measured for a batch size of 10. Inference time is reported per structure in ms; memory consumption is provided for the entire batch in GB.

Thank you for your clarifications. Most of my concerns have been addressed. I choose to increase my rating to 6.