We thank the reviewer for your reading of our work and feedback. We appreciate that the reader finds our explanation of differences between TPOs clear.

Regarding the weaknesses:

Weaknesses

1. We would like to clarify that the main point of the paper is understanding how alternatives to CGTP are achieving their performance gains. Crucially, we want to clarify that direct speed comparisons between GTP and CGTP are unfair. Instead, GTP is performing a different operation from CGTP. This realization was a major motivation for our work and in fact through our work we see that GTP does not give speed ups compared to CGTP in a fairer comparison.

   The key point is our framework provides a systematic way to analyze the differences between any TPOs (such as GTP, MTP, CGTP) and compare speedups in a fairer setting. We believe that this is extremely valuable for designing new TPOs.

   We do not claim that our measure of expressivity has direct correlation with actual training performance. In fact, combining the previous observation that GTP can achieve similar performance as CGTP and our result showing it provably has fewer degrees of freedom is already interesting. The true benefit of GTP should not be understood as a speedup but rather the elimination of many degrees of freedom with minimal impact. These observations illuminate potential avenues for further improvement.

2. By identifying that we can directly use a S2Grid, we highlight a possible asymptotic runtime improvement over the original implementation of GTP by using a S2FFT. This is the first algorithmic improvement which cannot simply be explained by a reduction in degrees of freedom. We believe this is quite interesting and can potentially have a significant impact in the future.

   However, we emphasize that the current $\ell$s used are far too small to see these asymptotic benefits from the complicated S2FFT algorithm. Despite this, using a S2Grid and a seminaive S2 Fourier transform (instead of S2FFT) gives a much simpler implementation of GTP which already sees performance gains over the original.

3. This is a good point, and we did try to mitigate implementation differences as much as possible (e.g. using JAX for everything). We would like to highlight that it is not the results themselves but the method of benchmarking that is important.

   Our microbenchmarks provide valuable insight on how different implementations can be improved. In particular, the discrepancy between FLOP counts (which are invariant) and walltime indicates potential for significant acceleration (eg. custom kernels) by better utilizing the GPU.

Questions

1. This is a great question! Our experiment in 6.2 gives a simple demonstration highlighting this failure. However, we actually strongly suspect most common molecular datasets are minimally impacted. Loosely speaking, this is because irrep types of commonly predicted quantities such as energy or forces can be constructed purely from symmetric tensor products of the input irrep types. We are actively exploring the impact of antisymmetric tensor products in future work.

   Further, we have come up with a new GTP-like tensor product which does not suffer the antisymmetry issues and has the same potential asymptotic benefit from S2FFT. However, we feel it distracts from the main message of this paper and have omitted it.

2. Thanks for the question! We chose the highest batch size (10,000) that we could fit onto the GPU without going out of memory. To offset any profiling overheads we chose to measure our walltime/FLOPs/throughput metrics using hardware instruction counters at the GPU driver level which is based on Roofline Toolkit Framework for Deep Learning (https://arxiv.org/abs/2009.05257) and has been successfully used in FourCastNet (https://dl.acm.org/doi/10.1145/3592979.3593412).

3. We are providing a framework for systematically analyzing how different TPOs achieve efficiency gains. Theoretically, we provide a way to see whether these came from a clever reduction in degrees of freedom or from actual algorithmic improvements. Our microbenchmarks provide a way to see runtimes of TPOs in practice and avenues for implementation improvements.

   Many previous works simply state their method is faster and achieves similar training results. Our work provides a way to analyze why their method was faster and whether there may be possible improvements or limitations. Our experiments are designed with this goal in mind. As such, the comparison of different methods in training results is not the central focus of this work (those results can be found in the original papers introducing the specific TPOs). However we understand the importance of training performance and agree that evaluation of training performance is also crucial for new TPO proposals.

We thank the reviewer for your careful reading of our work and positive feedback. We appreciate that the reviewer finds our evaluation criteria and experiments well aligned with the problem, the theoretical claims well supported, and discussion on expressivity a clear explanation for why certain TPOs fail to capture specific interactions.

Regarding the comments and suggestions

1. We thank the reviewer for bringing our attention to recent work in the Cartesian basis. This is a very interesting body of work which we will definitely mention in our revised manuscript.

   We would like to point out that our focus is on irrep based frameworks as in that case maximally expressive linear layers are easy to parameterize (as briefly explained in section 2.1). For the case of Cartesian tensors, it is easy to perform tensor products and remain a Cartesian tensor. However, especially at higher rank, it becomes difficult to parameterize fully expressive equivariant linear layers. Regarding the specific sources:

   It seems the interaction in https://arxiv.org/abs/2306.06482 is limited to rank 2 tensors and is a special case of MTP. https://arxiv.org/abs/2412.18263 seems to have significant work on decomposing Cartesian tensors to irreps in order to construct equivariant linear layers.

   It seems https://arxiv.org/abs/2405.14253 introduces a Cartesian tensor product which is also a true tensor product. However, this method has poor asymptotic scaling for 2-fold tensor products which was the focus of this work. Instead, their speedups are for small $\ell$ or large $\nu$-fold tensor products (performing multiple tensor products in succession like in MACE). We believe our framework can also be adapted for analyzing large $\nu$ by replacing the fixed bilinearities with fixed $\nu$-linearities and defining expressivity as the dimension of constructible $\nu$-linearites by inserting equivariant linearities in the inputs and output.

   We will add a discussion of these points in our final manuscript.

2. This is a good question. We would expect the same optimizations that were accelerating the forward pass to work for the backward pass. We were able to confirm this hypothesis by running a short experiment on the different input/output irreps settings and are able to see similar trends as the forward time. The plots and experiment details are included below. In particular, we create a random $z$ and then benchmark how long it takes to compute $\nabla_x(T(x,y)-z)^2$.

https://anonymous.4open.science/r/PriceofFreedom-A835/benchmarking/plots/png/walltime_bwd_gpu_MIMO_10000_RTX-1.png https://anonymous.4open.science/r/PriceofFreedom-A835/benchmarking/plots/png/walltime_bwd_normalized_gpu_MIMO_10000_RTX-1.png https://anonymous.4open.science/r/PriceofFreedom-A835/benchmarking/plots/pof_legend.png

3. We will definitely mention other basis choices in our revised manuscript and discuss Cartesian tensors in particular. The content in Section 2.1 motivates why we focus on an irrep basis.

We thank the reviewer for your careful reading of our work and helpful feedback. We appreciate that the reviewer finds our measure of expressivity reasonable and our runtime evaluation rigorous and comprehensive.

Regarding the weaknesses

1. Contribution of the new GTP implementation appears limited

   We very much agree that spherical harmonic transforms (which we refer to as $S^2$ fourier transforms) have been extensively studied and we cited Healy et. al. 2003 which improves upon the seminal work by Driscoll and Healy 1994 (which we will add as a citation). However, much of the work on S2FFTs involves numerical precision issues for super high $\ell$s when using asymptotically fast algorithms. This is beyond the scope of our work as most equivariant networks are still limited to small $\ell$. Instead, we just point out asymptotic improvements exist if equivariant networks scale to super high $\ell$ in the future. Importantly, we emphasize that this connection gives the first asymptotic runtime improvement over sparse CGTP that is not simply a consequence of reduced expressivity.

   Our actual grid implementation uses a much simpler seminaive algorithm described in Healy et. al. 2003 and Dricoll and Healy 1994.

2. Our experiments in Section 6.1 (Figure 6) mainly serve to show a drop-in replacement of Fourier GTP with Grid GTP leads to speedups. These are done on standard benchmarks of 3BPA and rMD17 and with an equivalent number of GPU hours. If the reviewer is concerned about actual training performance, comparisons against CGTP can be found in the original GTP paper (https://arxiv.org/abs/2401.10216).

   Our experiment in Section 6.2 (Figure 7) is a simple demonstration that lack of “antisymmetric” interactions can have tangible limitations on model performance. If the reviewer is concerned about how the lack of such interactions affect real world datasets, which is an excellent question, we actually strongly suspect most common molecular datasets are minimally impacted. Loosely speaking, this is because irrep types of commonly predicted quantities such as energy or forces can be constructed purely from symmetric tensor products of the input irrep types. We are actively exploring the impact of antisymmetric tensor products in future work.

Other Comments or Suggestions

Your understanding is correct, the $c_*$ are meant to separate the multiplicity of irreps. For CGTP, we wrote $c_Z$ as a tuple to try to highlight which inputs contributed, however in practice, this tuple is simply embedded as an integer. We will explain this more clearly in our revised manuscript.

We thank the reviewer for the thorough feedback on our work. We appreciate that the reviewer finds our work well-organized and presented, the experiments thorough, and provides important insights to practitioners.

Regarding the weaknesses

Weaknesses

1. This is a great question. The motivation is to highlight the difference in runtime improvements caused by cleverly reducing degrees of freedom from improvements caused by actually making tensor products faster. In particular, we also emphasize that the output spaces of these algorithms are different, making a direct comparison unfair.

2. We provide a framework to analyze the expressivity-vs-runtime tradeoffs in popular tensor product operations (TPOs) in equivariant neural networks. These tradeoffs are non-obvious, and can help other practitioners realize that these operations are not necessarily equivalent to each other. Importantly, we highlight that our work has revealed that most existing TPOs actually only get improvements from reducing degrees of freedom. Our connection to $S^2$ fast Fourier transforms provides the first (though for now impractical) algorithm which has an asymptotic improvement on the runtime/expressivity ratio.

   Our microbenchmarking efforts help highlight the different algorithmic tradeoffs. MTP focuses on maximising GPU utilization with a more matrix-multiplication friendly algorithm which ends up costing more FLOPs and hence higher overall runtime. CGTP on the other hand is FLOPs efficient but suffers from poor GPU utilization. GTP offers a balance between both.

3. This is a great question as indeed, different implementations could have drastically different walltimes in practice and we tried to mitigate implementation differences as much as possible (e.g. implementing all of the algorithms in JAX). We would like to highlight that it is not the results themselves but the method of benchmarking that is important.

   Our microbenchmarks provide valuable insight on how different implementations can be improved. In particular, the discrepancy between FLOP counts (which are invariant) and walltime indicates potential for significant acceleration (eg. custom kernels) by better utilizing the GPU.

Other comments and suggestions

1. Good catch! Indeed we do need an algebraically closed field for the maps to be identity. In the case of $SO(3)$, the irreps over complex vector spaces are real irreps.
2. Thanks for catching this! We meant to add 'AMD EPYC'.
3. Thanks for catching this!

Questions

1. As discussed above, the normalization for expressivity accounts for the fact that the output spaces of the different TPOs are not identical. We agree that there can be many ways to account for this difference.
2. Good question! The algorithm we used for CGTP-sparse (Appendix G) while being able to exploit the sparsity, introduced a lot of overhead due to the conditional logic leading to poor GPU utilization. As is seen in recent efforts (https://developer.nvidia.com/blog/accelerate-drug-and-material-discovery-with-new-math-library-nvidia-cuequivariance/, https://arxiv.org/abs/2501.13986v2), a more GPU friendly algorithm can improve the runtime.
3. This is an excellent question! In most of the popular datasets, we strongly suspect antisymmetric operations to have minimal impact. Loosely speaking, this is because irrep types of commonly predicted quantities such as energy or forces can be constructed purely from symmetric tensor products of the input irrep types. We plan to explore the impact of antisymmetric tensor products further in future work.