Equivariant Graph Self-Attention Transformer for Learning Higher-Order Interactions in 3D Molecular Structures

1. The writing and formatting of the paper are somewhat rough. The size of Figure 1 and the font of Figure 2 both require adjustments. The caption of the table should be placed above the table. Additionally, Section 4 contains only a portion of the content, yet it is labeled with a subsection title '4.1,' which seems redundant. The first sentence of the abstract uses 'their' without a clear referent, among other issues. These problems make the article difficult to read and do not meet the standard of a top conference paper.
2. This paper resembles a review of equivariant graph neural networks, and its actual contributions are not aligned with what is claimed in the introduction. The article devotes a significant amount of space to background knowledge and related work, only presenting the proposed method towards the end of page 7. Given the structure, it would be more appropriate to submit this as a review paper.
3. The experimental performance of the proposed method is not promising, and the baseline methods used for comparison are somewhat outdated, mostly from before 2021. Several well-known methods in the field, such as Equiformer[1], are missing from the comparison. As a result, the experiments do not effectively demonstrate the validity of the proposed method.
4. There seems to be an error in Equation 5, where $d_{ij}$ should be $d_{jk}$.
5. The innovation of the proposed method is rather limited. Introducing the attention mechanism into graph neural networks is not particularly novel, and I am curious about the differences and connections between the proposed method and existing approaches like GAT[2].

[1] Equiformer: Equivariant graph attention transformer for 3d atomistic graphs.

[2] Graph attention networks.

Lack of Novelty:

1. Equivariance Claim: The proposed method, Equivariant Graph Self-Attention Transformer (EG-SAT), is supposed to be equivariant, which the authors aim to achieve by encoding the geometric information of 3D graphs in Euclidean space. However, this is a commonly used approach seen in models such as SchNet, DimeNet, and SphereNet, which are not only equivariant but also invariant. Therefore, the claim of "equivariance" is insufficient as a primary contribution of this method. Given that the model is invariant, introducing the concept of irreducible representations (irreps) in Section 3 is unnecessary. The irreps of SO(3) are typically not used in invariant graph neural networks (GNNs).

2. Graph Transformer: The model is based on the graph transformer architecture, which is also widely used in the field. This, again, is not sufficient to be considered a significant contribution of the work.

3. Scalability of ACSFs: The authors claim to address scalability issues in Atom-Centered Symmetry Functions (ACSFs) using attention-based mechanisms through GRU blocks to approximate interactions between atoms. However, attention mechanisms for interaction have already been considered within the graph transformer framework. The authors should further explain the necessity of introducing this specific mechanism in their approach.

Insufficient Experiments:

1.Outdated Comparisons: The experiments compare the proposed method with multiple other approaches. However, many of these comparison methods are outdated. The authors should benchmark their model against more recent methods such as SphereNet and Molformer. Moreover, since the method is theoretically invariant, there is little need to compare it with numerous equivariant methods.

• The specific research problem addressed by the paper is unclear. While it provides comprehensive background information on molecular learning, symmetry, invariance, irreps, and ACSFs, the research problem is not clearly defined. In Sec. 4.1, the authors highlight the limitations of ACSFs but do so without detailed discussion or analysis to clarify the issue.
• The paper contains excessive information that may obscure its primary focus. For instance, the authors introduce irreps, but there is limited mention of its relevance within the method or analysis.
• Compared to ACSFs, the proposed EG-SAT still faces challenges in computational complexity. While the paper provides a complexity analysis for the proposed method, it lacks a direct comparison with ACSFs. Equations 7 and 8 do not offer computational savings relative to Equations 4 and 5.
• Overall, the content does not sufficiently support the contribution claims in Sec. 1. The paper lacks novelty and significant contributions.

1. The paper omits several recent works in molecular property prediction, making the comparisons less relevant.
2. The authors didn't conduct ablation studies to evaluate the contribution of different components in the framework.
3. There is no computational efficiency analysis, what's more, the claims of improved scalability are unsupported by any experiments.
4. The motivation for incorporating angular information lacks clear examples where angular information provides benefits.
5. There's no proof that the attention mechanism preserves chemical validity

1. Incomplete baselines for all the datasets presented in the paper. For QM9 dataset, authors didn't include recent works such as Spherenet [1], Equiformer(V2) [2,3], LEFTNet [4], SaVeNet [5], and Geoformer [6] to name few. Although the authors cited Equiformer, they didn't compare the results on QM9.

2. The baselines are on all small molecular tasks on QM9 and MD17 datasets. Therefore, limiting the applicability of the proposed methods.

3. Complexity Analysis: While the authors claim linear complexity with respect to the number of edges, this seems inconsistent with the use of angular information ($\beta_{ijk}$) which typically involves triplet interactions. The current analysis doesn't adequately justify how the model maintains linear complexity despite considering all possible triplets.

4. Empirical Validation of Efficiency Claims: Despite emphasizing computational efficiency and suitability for high-throughput screening, the paper lacks empirical evidence comparing computational costs with baseline methods.

[1] Liu, Y. et al. (2022) ‘Spherical Message Passing for 3D Molecular Graphs’, in International Conference on Learning Representations. Available at: https://openreview.net/forum?id=givsRXsOt9r.

[2] Liao, Y.-L. and Smidt, T. (2022) ‘Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs’. Available at: https://openreview.net/forum?id=_efamP7PSjg.

[3] Liao, Y.-L. et al. (2024) ‘EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representations’, in The Twelfth International Conference on Learning Representations. Available at: https://openreview.net/forum?id=mCOBKZmrzD.

[4] Du, W. et al. (2023) ‘A new perspective on building efficient and expressive 3D equivariant graph neural networks’, in Thirty-seventh Conference on Neural Information Processing Systems. Available at: https://openreview.net/forum?id=hWPNYWkYPN.

[5] Aykent, S. and Xia, T. (2023) ‘SaVeNet: A Scalable Vector Network for Enhanced Molecular Representation Learning’, in Thirty-seventh Conference on Neural Information Processing Systems. Available at: https://openreview.net/forum?id=0OImBCFsdf.

[6] Wang, Y. et al. (2023) ‘Geometric Transformer with Interatomic Positional Encoding’, in Thirty-seventh Conference on Neural Information Processing Systems. Available at: https://openreview.net/forum?id=9o6KQrklrE.