Functional Geometry Guided Protein Sequence and Backbone Structure Co-Design

We appreciate the reviewer’s valuable and insightful suggestions, which are very helpful for us to improve our paper. We have clarified all your concerns, added the experiments and updated the paper accordingly. We address the specific concerns as follows:

Weakness 1: Regarding the consistency metrics reported, I noticed that the consistency metric of NAEPro in Table 1 (as the main results) underperformed half of the baselines. Such key observation greatly weakened the method. The authors can elaborate on this point.

Ans: Yes, we agree with the reviewer’s opinion. We would like to clarify our task more clearly. Our goal is to design functional proteins with novel sequence and structure. As shown in Figure 5(b), the designed myoglobin has a RMSD of 3.943\AA with the most similar one in PDB and 26.7% AAR with the most similar one in Uniprot, but it can still bind the corresponding metallocofactor heme in the AlphaFill simulation [1]. In our paper, we use ESMFold to predict the structure of the designed sequence. However, as pointed out by reviewer VH6U, ESMFold might not accurately predict the structure for a novel sequence. To further clarify this point, we calculate the scRMSD of motifs between the designed structure and the predicted structure from the designed sequence in the following table. As the results show, our NAEPro achieves smaller RMSD than RFDiffusion+ProteinMPNN on myoglobin and comparable results on beta-lactamase.

[1] Alphafill: enriching alphafold models with ligands and cofactors. Maarten L Hekkelman et al. Nature methods. 2023.

Weakness 2: For the experiments, how is the model trained to obtain the evaluation digits in Table 1 for each baseline. This is very important.

Ans: We are sorry for the confusion. We got the codes of all baselines from their GitHub repos, and then re-trained all the models separately on myoglobin and beta-lactamase datasets. The evaluation procedure among all the baselines and our model follows the same process, and thus we believe the comparisons in Table 1 are fair.

Weakness 3: For the dataset, I found that in section 4.1, the author(s) mentioned that both beta-lactamase and myoglobin datasets are split randomly. This is worrying because the entries in PDB can usually be redundant, especially for such large and well-studied families that the authors selected for evaluation. The author(s) can elaborate on this point. Moreover, in the Table 3 (in appendix), I am curious about the column name “PDB”, does it mean the number of PDB entries (with unique ID) or the number of single chains?

Ans: Yes, this is a very good question. Since we focus on sequence and structure co-design task, we aim to design both functional sequence and structure. Therefore, our input data is the <sequence, structure> pair. From this point, even though the sequence might be the same, the structure is somehow different. Therefore, our paired input data are different and would not be redundant. We appreciate the reviewer for carefully reviewing our paper and also reading the appendix. In Table 3, PDB means the number of different PDB entries. To eliminate the reviewer’s concerns, we re-cleaned the test set by filtering pairs whose sequences have over 30% sequence identity to any sample in the training set after doing pairwise alignment for myoglobin. We compare our NAEPro and ProtSeed (the most competitive baseline) in the following table. As the results show, our model can design proteins with higher binding affinity (by Gnina [2]) on samples that have lower similarity to training data.

[2] GNINA 1.0: Molecular docking with deep learning. A McNutt et al. Cheminformatics. 2021.

Corrected statement: But it can still bind the corresponding metallocofactor heme in the AlphaFill simulation. For the claim “binding rates of 100% and 90%”, I wonder how is the criterion/threshold of “binding” determined? This success rate is rather unrealistic and probably fails to reflect the real performance. Thus, the statement in Section 5.2. saying “It is evident that our method can generate proteins that express the basic and important functions.” can be quite misleading for readers who are not familiar with the task.

Ans: Sorry for the misleading statement. We mean through the AlphaFill prediction, the designed proteins are highly potential to bind the corresponding metallocofactors. We have corrected all the claims in our paper. We provided the process of how to decide if the designed protein can “bind” the metallocofactors in Appendix B.3. We would like to further explain it here. First of all, we use AlphaFill to do the first-step prediction. AlphaFill will return a complex based on the input protein structure. If the complex includes the corresponding metal ions, we think the designed protein is potential to have functions. Then to guarantee the results convincing, we further use some additional constraints to ensure the designed proteins are potential to express functions. Specifically, we assume if a designed protein share the similar active site environments with their natural counterparts (target protein in our dataset), then it’s highly potential to bind the corresponding metal ions. For myoglobin, we calculate the distance between axial histidine ligand to Fe ion. Besides, we also detect the presence of distal histidine within the active sites, which plays an important role in the oxygen molecule binding function of myoglobin . If the distance is between 2.0Å and 2.5Å, and the distal histidine exists, we think the designed myoglobins share similar active site environments to natural ones. For $beta$-lactamase, we detect the residues that directly contact with zinc ion. If they show similar chemistry properties with natural $beta$-lactamases, we think they can highly potentially bind the corresponding metallocofactors. For example, in Figure 5(a), the active site possesses one zinc atom coordinated to three histidines, while the second zinc atom is coordinated to one histidine, one cysteine, and one aspartate, demonstrating a high degree of amino acid analogy to the natural protein.

Could the author intuitively describe how does the NAEL deal with the reflection case so as to be simply SE(3)-equivariant instead of E(3)?

Ans: Intuitively, applying reflection to the input structure, the output structure will also be reflected. The conformation of the original backbone will be changed from L-amino acid to D-amino acid (We provide an illustration in Figure 6 in appendix). Therefore, the reflection will change the chirality of the protein which may cause the deficiency of binding to the ligand and eventually its function. However, the protein sequence in our model will not be influenced. Therefore, the 3D structure of the designed sequence which is L-amino acid and the output structure which becomes D-amino acid will be inconsistent. From this aspect, our model doesn’t satisfy reflection equivariance.

The authors are encouraged to encompass appropriate comparison with EGNN/SE(3)-transformer in their writing to better contextualize the proposed NAEL.

Ans: We provide the comparison with EGNN+ESM2 on myoglobin in the following Table and also updated the results in Table2 accordingly.

It is surprising to see scaling the model does not yield consistent performance gain. Could the authors help explain this behavior? Also, I noticed that the authors merge the two myoglobin and lactamase data for training newly in the rebuttal response. Is there any insight for doing such multi-task learning?

Ans: Enlarging the model hasn’t improved the performance too much. Our interpretation is that the training data size is moderate, and thus a model with moderate size can fit the data well. To validate if this guess holds, we merge the two protein data and then train a unified model. As we can see in the previous table, our model did achieve obvious better results on $\beta$-lactamase, i.e., TM-score from below 0.5 (0.4919) to over 0.5 (0.5055).

We thank the reviewer for the positive reviews as well as the suggestions for improvement. We have clarified all your concerns, added the experiments and updated the paper accordingly. Our responses to the reviewer’s concerns and questions are provided below:

Weakness 1: Both EGNN and ESM2 are existing models, and there are many existing works on antibody structure and sequence co-design. The combination may limit the novelty of this method.

Ans: We would like to point out that our NAEPro is not a combination of ESM2 and EGNN. Instead our key innovations are (1) interleaving layers of sequence-level attention and local neighborhood equivariant sub-layer (2) accelerating the local sub-layer with the k-nearest neighbors. On the contrary, EGNN is a fully-connected graph and updates message, coordinates and atom features based on all other atoms in 3D space, which is not efficient for long proteins. Besides, EGNN+ESM2 is a sequential process and can not cross-condition on sequence and structure. We compare our method with EGNN+ESM2, NAEPro w/o ESM2 initialization on myoglobin in the Following Table (Table 2 in the paper). As the results show, EGNN+ESM2 performs much worse than our NAEPro, and removing the ESM2 initialization will not influence the performance too much, demonstrating that our model is much more superior than EGNN+ESM2.

Weakness 2: It is likely that previous works can be readily applied to the motif-conditioned setting, and the authors can easily adapt their method to antibody design tasks. It would be beneficial if the authors further elaborate on the significance of their work

Ans: We would like to clarify our goal more clearly. Our task is to co-design functional and novel protein sequence and structure. Previous work which is most similar to ours is Inpainting [1]. However, [1] provides the functional sites manually while we automatically mined the functional sites (plus the conserved sites as suggested by reviewer VH6U) by MSAs. Previous methods considering only sequence design constrained by fitness value [2,3] or sequence design based on given backbone structure (inverse folding) [4] is not suitable for our task. For antibody design, antibody is always Y-shaped, while one of our goal is to design novel and diverse protein structure. For example, in Figure 4, the designed beta-lactamases have different fold categories and belong to three different sub-classes. In Figure 5 (b), the designed myoglobin has a RMSD of 3.943\AA with the most similar one in PDB, but it can still bind the corresponding metallocofactor heme in the AlphaFill simulation [5]. From this aspect, we think antibody design models can not be directly applied to our task without any modification and vice versa.

[1] Scaffolding protein functional sites using deep learning. Jue wang. et al. Science. 2022.

[2] Biological Sequence Design with GFlowNets. Moksh Jain et al. ICML 2022.

[3] Proximal Exploration for Model-guided Protein Sequence Design. Zhizhou Ren et al. ICML 2022.

[4] Robust deep learning based protein sequence design using ProteinMPNN. J. Dauparas et al. Science. 2022.

[5] Alphafill: enriching alphafold models with ligands and cofactors. Maarten L Hekkelman et al. Nature methods. 2023.

Weakness 3: The authors do not provide code for checking the soundness of the methods.

Ans: We have provided the code and data in the supplementary material. Due to the memory limitation, we couldn’t upload the model checkpoints. However, the training process is highly efficient, and the reviewer could train the model if he/she is interested.

Thanks for the authors' efforts. I have additional concerns:

• Page 1, "Despite their great potential for novel structure design, such sequential design policy fails to cross-condition on sequence and structure, which might lead to inconsistent proteins and inefficient design process". I can not agree with this statement. RFDiffusion's authors say that they also considered simultaneously designing structure and sequence within RFdiffusion, but combining ProteinMPNN with the diffusion of structure alone provides the excellent performance, as shown in your Table.1. Unfortunately, I observe that you deleted the consistency metric in Table.1,which seems like dishonest behavior to me.

• Page 4, "Through this way, amino-acid combinations frequently occur in the same context would draw higher attention scores". Have you checked this statement? Could you provide some visualizations of the attention map and correlated frequences of amino-acid combinations?

• Page 4, "Updating residue representations and coordinates in 3D space with only nearest neighbors enables more efficient and economic message passing compared to prior approaches which compute messages on the complete pairwise residue graph". To my knowledge, most of the previous methods such as proteinmpnn adopt knn for constructing sparse graph in the 3D spcace. You should provide evidence to support the significance of your statement.

• Novelty: I do not observe enough novelty from the perspective of machine learning in algorithm design.

Thanks for the responses! Our answers to your follow-up questions are provided as follows:

1. About Weakness4 & Q1 Page 2, "We carry out experiments on two metalloproteins, including β-lactamase and myoglobin." I would like to say that the experimental setting is the main concern to me. Since the authors have only evaluated the proposed method on two structures, I am not sure if the method can be extended to generalized situations. I know that you are doing protein sequence-structure inpaintning problem, in which case a comparison with a previous baselines [1,2,3] on stantard datasets may help to reveal the generalization capabilities of your algorithm.

Ans: We agree with reviewer Wq6M and additionally compare our method with PROTSEED on B12 protein. Due to the approaching ddl and limited time, PROTSEED has only been trained for 100 epochs, the same as our method. (suggested training epoch is 2000 on their GitHub)

The results are reported as follows:

Binding affinity score for designed B12

2. About Q2 Your methods are not carefully designed for protein inverse folding. However, the presented results outperformed current SOTA by a large margin (curent SOTA is about 50%-60% AAR), which could not convince me. I doubt the veracity and correctness of the experimental results.

Ans: Sorry for the confusion. We evaluate the inverse-folding task on our own two datasets instead of the original CATH. For our datasets, the proteins are from the same family, and thus they may have much overlap on the sequence level, which definitely will lead to high AAR. Besides, our model is initialized with ESM2 weights, which will also improve the AAR.

Although the sequence prediction based a fixed backbone structure is not our objective, we adapt our method to CATH setting by keeping all the CA coordinates and masking all residues. Due to the approaching ddl and limited time, our model has only been trained for 10 epochs, while usually the suggested training epoch would be 100 like GVP, PiFold, etc. (See below: ESMIF, ProteinMPNN, PiFold results are quoted from PiFold paper).

CATH 4.2 results

Again, we did not claim the superiority of our method on inverse folding task on CATH dataset.

3. Page 1, "Despite their great potential for novel structure design, such sequential design policy fails to cross-condition on sequence and structure, which might lead to inconsistent proteins and inefficient design process". I can not agree with this statement. RFDiffusion's authors say that they also considered simultaneously designing structure and sequence within RFdiffusion, but combining ProteinMPNN with the diffusion of structure alone provides the excellent performance, as shown in your Table.1. Unfortunately, I observe that you deleted the consistency metric in Table.1, which seems like dishonest behavior to me.

Ans: We do not intend to hide any information. As suggested by reviewer MgyU, the binding affinity is a better metric for functional protein design rather than the accuracy metrics (consistency, AAR, RMSD, etc.). Therefore, we replaced all the accuracy metrics in Table 1 and 2 in our paper according with the binding affinity scores calculated by Gnina. The full table with all scores are listed below:

$\beta$-lactamase

Myoglobin

We thank the reviewer for the insightful questions. We have clarified all your concerns, added the experiments and updated the paper accordingly. Answers to specific points are provided below:

Weakness 1 & weakness 2 & Q5: The identified problem or the motivation for designing such a method is not supported by existing literature. Some designed modules counter the intuition in biology, such as "motif mining"

Ans: We follow the design insight proposed by [1], which starts from the functional site and fills in additional sequence and structure to create a viable protein scaffold in a single forward pass. Different from [1], we provide the functional sites which are automatically mined from MSA as previous methods [2, 3] while [1] provides the functional sites manually. Besides, we design a new architecture called NAEPro to generate the whole protein sequence and structure based on the given partial protein fragments, while [1] achieves this goal by finetuning the RoseTTAFold. However, we agree with the reviewer’s opinion in question 5 that MSA discovers not only protein functional sites as addressed in [2,3] but also finds conversed sites which may not directly relate to the protein function. Therefore, we change all the term “motif/functional sites” in our paper to “meaningful protein fragments”.

[1] Scaffolding protein functional sites using deep learning. Jue wang. et al. Science. 2022.

[2] Emerging methods in protein co-evolution. David de Juan et al. Nature Review Genetics. 2013.

[3] Evolutionary information for specifying a protein fold. Michael Socolich et al. Nature. 2005.

Weakness 3 & Q1: The method is not 'extensively evaluated' as claimed by the authors (they only selectively analyzed 2 proteins), and many results do not suggest that the proposed method is SOTA among baselines. Moreover, the evaluation does not include all relevant methods, which makes it too aggressive for the authors to claim in the Introduction that their method "achieves the HIGHEST...among ALL competitors" and "is faster than the FASTEST method". Empirically, It has been proven by several recent research that the de novo method can generate new sequences for fixed backbone or desired functions [4-6].

Ans: Sorry for the confusion. Our method achieves the best performance on 4/5 metrics on myoglobin and 3/5 metrics on beta-lactamase among all the compared representative baselines in our paper. Our model is faster than all baselines compared in our paper. We have revised all these statements in our paper (highlighted in red). For other relevant methods as the reviewer suggested in reference [4-6], we have already compared with [4]. [5] was posted on Oct. 3, 2023, which is later than the ICLR submission ddl, and [3] was posted on Aug. 14, 2023, which is within 3 months before ICLR submission ddl. We appreciate the reviewer’s suggestion, and we tried to compare with [3], but they haven’t released their code now. We emailed the authors of [3] to ask for code and they said their work is under review now and refused to provide code currently. We are trying to reimplement the model now and will update the results in our revised version.

[4] Watson et al., De novo design of protein structure and function with RFdiffusion (2023).

[5] Sumida et al., Improving protein expression, stability, and function with ProteinMPNN (2023).

[6] Zhou et al., Conditional Protein Denoising Diffusion Generates Programmable Endonuclease Sequences (2023).

Thanks for your responses. However, I remain unconvinced regarding the issues I raised earlier.

Response to Weakness 3 & Q1: My point in mentioning these three papers is not to introduce additional (and possibly meaningless) in-silico comparisons to the authors. Instead, they validated their designs with solid wet-lab experiments demonstrating that it is possible to generate novel protein sequences with fixed backbones.

Response to Weakness 4: I completely agree that all the existing evaluation metrics we used in deep learning are only indirect indicators, and they only imply a higher chance that the assessed protein will perform its function. However, I have trouble following the logic that "Therefore, a low TM-score cannot indicate the designed protein doesn’t have the corresponding function" suggests "proteins with TM-scores that are lower than 0.5 (which is a wildly-used standard) are still reliable in terms of their functionality in general." Similarly, if we all agree that the existing metrics are suboptimal and none of them provides 100% accurate evaluations, it is then meaningless to criticize the widely applied standards with one or two counter-examples. After all, you don't have any better evidence (such as wet-lab results) that is more reliable than the existing criteria.

Response to Q6: Because ESMFold is a less reliable tool for evaluating novel proteins (as mentioned in their paper), it should not be used here to assess your designs at all -- If the evaluation metric itself is unreliable, the whole evaluation makes no sense.