$E^3$former: An Adaptive Energy-Aware Elastic Equivariant Transformer Model For Protein Representation Learning

Q1: "The paper’s primary motivation is to address noise in structural data from both experimental and AlphaFold-predicted sources. However, the paper lacks a detailed quantitative or qualitative analysis of the specific types and impacts of noise present in these datasets. Without a thorough analysis of how noise in these data affects current models, it is difficult to assess whether the E3former’s adaptive mechanisms effectively target the underlying issues."

A1: Based on the causes and property of noise, we categorize it into three types:

1. Perturbation Noise: These are approximated as Gaussian-distributed noise fluctuations, primarily caused by measurement inaccuracies of experimental instruments, molecule dynamics or environmental factors [1].
2. Biased Noise in Experimental Structures: Examples include noise introduced by resolution limitations or reconstruction algorithms in cryo-EM or NMR data [2].
3. Biased Noise in AlphaFold Predictions: These arise from structural tendencies embedded during AlphaFold’s training rather than inherent biological conformational variability. Notably, this type of noise is particularly prominent in low-confidence regions [3], and AlphaFold2 predictions often capture only one experimentally determined conformation while omitting others [4].

Perturbation noise can be mitigated using conventional methods such as regularization and data augmentation.

For biased noise in experimental structures, Figure 4 shows that noise in lower-resolution interval significantly impacts model performance. However, $E^{3}$former continues to demonstrate robust performance, showcasing its ability to adapt the type of noise.

Similarly, for biased noise in AlphaFold predicted data, results in Figure 3 and Table 6 reveal that $E^{3}$former achieves more comparable performance in low-confidence regions, showcasing our model 's ability to noise tolerance.

Additionally, the cross-dataset evaluations we provided in [A3] further quantify this issue. As the confidence of predictions decreases and the proportion of biased noise increases, $E^{3}$former exhibits a more pronounced performance improvement, reveal its effectiveness to handle different types of noise.

Q2: "While the authors introduce two new datasets (CATH-AF and PPBS-AF) based on AlphaFold-predicted structures, these datasets appear to be relatively simple conversions of existing datasets into AlphaFold’s predicted coordinates"

A2: Thank you for this important question, which addresses a significant challenge we encountered during data processing. To clarify, CATH-AF and PPBS-AF are not simple coordinate transformations of existing datasets. Rather, they represent careful mappings between the original protein sequence and the predicted structure corresponding to its UniProt sequence, which naturally leads to some variation in amino acid counts. While ideally AlphaFold-predicted structures should perfectly match their original protein sequences (differing only in atomic coordinates), the computational demands of folding tens of thousands of protein sequences exceeded our available resources. As detailed in the appendix, we implemented a pragmatic and scalable solution: leveraging the RCSB PDB database to map UniProt IDs to their corresponding AlphaFold Database predictions. However, this approach has a minor limitation, as the AlphaFold Database does not contain predictions for every UniProt ID.

In PDB files, due to experimental conditions (e.g., X-ray crystallography, cryo-EM) or technical limitations, not all regions of a protein sequence are resolved into structures. Consequently, the sequences recorded in PDB files are typically truncated versions of the corresponding UniProt sequences. Furthermore, proteins in PDB files may have undergone experimental modifications or intentional truncation, resulting in discrepancies from their original UniProt sequences.

Looking ahead, we plan to investigate open-source protein folding models with lower computational requirements (e.g., Protenix, Chai-1, Boltz-1) to enable direct structure prediction from sequences. This will allow us to evaluate model performance under conditions of greater sequence consistency.

I sincerely appreciate your careful revisions and constructive feedback on this paper. I believe your suggestions are truly inspiring and will greatly help me enhance the quality of this work. I will now provide detailed responses to your questions.

Q1: "My primary concern with this work is with the authors' premise that robustness to noise is a fundamental limitation of existing works. Often, works in this area have shown that adding noise to crystal structures provides a form of regularisation that generally boosts performance -- sometimes quite substantially. I think the issue that the authors are highlighting is not so much "noise" in experimental structures, but with prediction errors which can substantially alter the input graphs and affect model training. I'd suggest being clearer with this distinction"

A1: We agree that the inherent biases or systematic errors in both experimental techniques (e.g., cryo-EM or NMR)[1] and AlphaFold-predicted structures reflect non-random, structured deviations[2,3]. However, from the model's perspective, since $E^{3}$former does not introduce the above information as inductive bias, all types of deviations can be generalized as "noise" without explicit prior knowledge. This assumption allows our framework to treat these biases and errors in a uniform manner, enabling robustness without detailed priors about the specific data source or noise characteristics. To handle such diverse noise sources, $E^{3}$former introduces Molecular Dynamics Signals(Energy-aware radius graph) and utilizes high signal-to-noise ratio information extraction method(Elastic selective mechanism). These innovations enable $E^{3}$former to generalize effectively across a wide range of noise types.

To further clarify this point, we conducted additional cross-dataset evaluations: We trained $E^{3}$former on PPBS experimental data and tested it on PPBS-AF (AlphaFold-predicted data). Despite the systematic differences and the lack of noise-specific inductive biases in the training data, the model achieved comparable performance in the test set. This highlights the model’s ability to generalize to unseen types of noise and prediction errors. Conversely, training on noisy AlphaFold-predicted data and testing on experimental structures also demonstrated strong predictive ability.

We greatly appreciate your constructive comments and the valuable questions you raised. We are also grateful for your recognition of the innovation and value of our two core modules: the hybrid Transformer-SSM and the Energy-aware radius function.

Q1: "The captions of the figures and tables are too short (i.e., fig. 2), it is too difficult to understand and follow. The terminologies and notations are also not clarified, which is very confusing."

A1: We have revised and clarified the captions in the manuscript. All updated captions are colored in blue.

Q2: "While the model demonstrates efficacy on AlphaFold and crystal structures, this reliance could constrain its generalizability to other types of protein data with different structural features."

A2: To clarify, $E^{3}$former can generalize to other types of structural data beyond the datasets explored in this study. In our work, we focus on two representative types of structural data: (1) experimentally measured data, such as crystal structures, and (2) model-generated data, such as AlphaFold-predicted structures. However, the inductive biases of our model are not limited to these two data types. From molecular dynamics perspective, the positional information of atoms within macromolecules is interdependent, providing latent unsupervised signals for self-correction(Energy-aware graph). Noise in structural data often propagate and intensify within equivariant neural networks. By selectively leveraging these signals, we effectively reduce the impact of such noise(Equivariant high-tensor-elastic SSM). These assumptions are grounded in the properties of biological macromolecules rather than limited to any specific data type.

Q3: "In Section 4.2, the rationale for selecting specific baseline models and configurations lacks clarity. The reasons of such implementation should be explained and introduced with sufficient motivation."

A3: We provide additional details and rationale for our choices as follow. Our choices based on a recent benchmark for evaluating models in protein representation learning [1]. To ensure fairness in our comparisons, we use the default settings from this work for most baselines and common configurations, such as learning rates, optimizers, and architectural parameters. Additionally, we selected baseline models based on their representativeness and characteristics. The chosen baselines include a mix of classical and state-of-the-art approaches: SchNet (Classical Invariant GNNs) [3], TFN (Classical Spherical Equivariant GNNs) [4], EGNN (Equivariant GNNs) [5], Equiformer (Spherical Equivariant GNNs with Transformer blocks) [6], GearNet (Powerful protein-specific SE(3) Invariant GNNs) [7], GCPNet (Powerful protein-specific Equivariant GNNs) [8]. We hope this additional information helps clarify our choices.

Q4: "The results analysis and discussions are insufficient, like comparison across baseline models. For example, in Tables 2 and 3, the authors should discuss why the baseline results differ from the proposed model and what factors may contribute to these discrepancies.

A4: Our model demonstrates superior performance compared to baselines due to two key factors:

1. Stronger Fitting Capability with the Hybrid Transformer-SSM Architecture: The hybrid architecture enhances the model’s fitting ability, enabling it to achieve more significant performance improvements in challenging tasks. When fewer features are used (+Seq features) or in situation with large training-test set difference, such as the "none" split in the PPBS dataset, the model’s advantages are more pronounced (Tables 2 and 3).

2. Noise Resilience through Selective SSM and Adaptive Energy Functions: By incorporating Selective SSM and an Adaptive Energy function, the model effectively enhances its robustness to noise. In noisy datasets, such as the AlphaFold-predicted data (Table 2) and in low-resolution interval with significant noise effects (Figure 4), these components allow the model to demonstrate more comparable performance.

Q1: "A kind suggestion is that the split interval is a little narrow (80-85,90-95…), instead using 0-60, 60-70, 70-80…. may be more appropriate. And the readers may also want to know the performance under noisy structures (e.g. with confidence below 70, 60, etc)"

A1: Thank you for raising this valuable question. While using broader intervals (e.g., 0–60, 60–70, 70–80) seems reasonable, we decided to use narrower intervals (e.g., 80–85, 90–95) due to the specific data distribution. Our primary aim is to ensure that each interval contains a sufficient number of samples for meaningful statistical evaluation. The sample distribution across intervals is as follows:

As seen, the number of samples in very low-confidence intervals is sparse. This makes it challenging to conduct reasonable evaluations.

Q2: "There are typos and incorrect spaces in the method section, in Eq,3,4, 10,11. I kindly suggest the authors to keep a clean format."

A2: Thank you for pointing out the typographical errors. The corresponding sections have been carefully revised, and all changes are visible in the revised manuscript.

Q3: "The connections between equivariant elastic selective SSM and rotation noise are week for me. It’s hard for me to get the logic from rotation noise to SSM. I kindly suggest the authors to introduce more details about this part."

A3: Thank you for highlighting the need for a clearer explanation of the connection between equivariant elastic selective SSM and rotation noise. Below, we provide a more detailed explanation.

Equivariant Models and Susceptibility to Rotation Noise: Equivariant models, particularly those handling high-order tensors, are more sensitive to rotation noise [1,2]. This is because rotation noise introduces distortions in the geometric relationships captured by these high-dimensional tensors, significantly impacting the model's performance. As a result, addressing rotation noise is critical for maintaining the accuracy of equivariant architectures.

Role of Elastic Selective SSM: The elastic selective SSM serves as a sparse alternative to traditional Transformer attention mechanisms [3]. In our framework, it acts as a filter for high-dimensional tensor signals, selectively focusing on stable, informative components while omitting noisy or less relevant information. By increasing the signal-to-noise ratio, the elastic selective SSM effectively mitigates the impact of rotation noise on the model's performance.

Q4: "Performance improvement on some tasks are a little marginal and the improvement introduced by the key component Energy-aware radius function is marginal. The motivation for using SSM is missing."

A4: The performance gains introduced by our model are more pronounced in situation with significant noise. In low-Confidence AlphaFold-predicted samples, as shown in Figure 3 and Table 6, the model demonstrates notable improvements, with performance increases exceeding 20% in the PPBS-AF Homology dataset for samples with 0–80% confidence. While the performance gains are relatively smaller in experimental datasets, the model consistently maintains state-of-the-art performance, demonstrating its robustness across diverse data conditions. The motivation for using the SSM module lies in its ability to process high-order tensors with a higher signal-to-noise ratio. By selectively focusing on critical geometric features, the SSM module reduces the influence of noise and enables more effective representation learning. This has been further detailed in our response to [Q3].