Fast Uncovering of Protein Sequence Diversity from Structure

We first want to thank the reviewer for carefully reading the manuscript and for the questions raised, which is allowing us to improve the manuscript. We will address the questions point by point, reporting the original question in italics followed by the answer.

1. It would strengthen the paper to benchmark against other methods as well.: We thank the reviewer for raising this point, which has also been brought to our attention by another reviewer. To improve the manuscript we are hence also comparing InvMSAFold-AR/PW with ProteinMPNN, which will then be added to the manuscript.
2. I highly recommend adding a plot that shows RMSD versus sequence recovery, as these metrics would provide valuable insights into the model’s performance: We thank the reviewer for this comment, which allows us to clarify some of our experiments and improve the paper. Indeed we agree that assessing the relationship between sequence recovery and predicted RMSD of generated sequences is a key metric to evaluate the different models analyzed. As originally conjectured, and provided evidence for in Figure 7 of the paper, ESM-IF1 and our InvMSAFold architectures sample generally at different hamming distances from the native sequence; indeed the former focuses too narrowly on the native sequence, while both InvMSAFold-PW and InvMSAFold-AR explore sequences whose distance from the native is more consistent with those observed in true MSAs. Given this fact, to make more of an "apple to apple" comparison between sequences from the the different models we also needed samples from ESM-IF1 at higher hamming distances, which can be achieved by leveraging the built-in temperature parameter of ESM-IF1 sampler. This is precisely what we do in Section 4.5 and is reported in Figure 8. To highlight the dependence between sequence recovery and RMSD for the different models we decided to bin sequences based on their distance from the native. As can be seen from Figure 8, the RMSD of predicted structures from sequences generated from ESM-IF1 deteriorates much more rapidly compared to those from both InvMSAFold-PW and InvMSAFold-AR, which we feel addresses the comment raised by the reviewer. For more details on the experimental procedure we refer the reviewer to appendix A.2.2.
3. In Section 2.2.1, the explanation of Equation 7 and how it maintains linear scaling isn’t entirely obvious, at least to me: We thank the reviewer for raising this point, which allowed us to better clarify a crucial aspect of the architecture. The reason why Eq.(7) proves linear scaling, is that on the left side of Eq.(7) we have a double sum over the amino acid positions $\sum_{i<j}$, resulting in a quadratic cost, while through careful computations on the right side of Eq.(7) we only have single sums over the position $\sum_i$, resulting in a linear cost. In the updated version of the manuscript we are making sure to make clear how linearity is established and why Eq.(7) achieves linearity.
4. To make the manuscript even stronger, it would be useful to include (1) an analysis of how the method scales with very large sequences or structures, and (2) a discussion of how the size of the MSA impacts model performance.: We thank the reviewer for raising this point, which allows us to make the manuscript clearer. During training we capped the domains length to $512$ as this included almost all domains in the CATH dataset, and resulted in a significant reduction in memory requirements to run the model. In any case, the built-in attention layers of PyTorch are capped to a maximum input length of $1024$, hence we could not consider sequences with length higher than that. We interpret the second point of the reviewer as the size of the MSA used during training, as during testing the MSA is not needed to run the model. In this case, we agree with the reviewer that assessing the relationship between the MSA size and model performance is of great interest. Indeed, for this specific reason we included the MSA size as one of the parameters to be tuned in hypertuning reported in Appendix A.1.2. From there we observed that as soon as the MSA size is large enough, where this threshold heuristically seems to be around $32$, then the observe comparable performance for the different models trained with all other parameters equal. We have highlighted more this aspect in the revised version of the manuscript.
5. In Section 2.2, I recommend including the formula that shows the normalization constant, as it is referenced in the text but not explicitly provided. : We will add the normalization constant in the updated manuscript as suggested by the reviewer
6. We further thank the reviewer for reporting the typos/grammar above, which allowed us to polish the manuscript. We are re-reading the manuscript with great care and correcting all grammatical errors to make sure the updated manuscript will not have any.

We first want to thank the reviewer for carefully reading the manuscript and for the questions raised, which is allowing us to improve the manuscript.

We will address the questions point by point, reporting the original question in italics followed by the answer.

1. The authors only compare their method with ESM-IF1, and do not compare their method with other state-of-the-art inverse folding methods: We thank the reviewer for raising this point, which is allowing us to significantly improve our manuscript. At the time on this answer we are comparing InvMSAFold-AR/PW with ProteinMPNN, which will then be added to the manuscript.
2. In many places such as in section 1, "ESM-IF" was wrongly typed as "ESM-1F". This may lead readers to perceive the authors as lacking expertise.: We thank the reviewer for spotting this inaccuracy, which allowed us to polish the manuscript. We diligently re-read the text and correct all typos and especially any possible misspelling if "ESM-IF".
3. The article contains too many grammatical errors.: We thank the reviewer for raising this point, which is allowing us to polish the manuscript. We are re-reading the manuscript with great care and correcting all grammatical errors to make sure the updated manuscript will not have any.
4. The symbols of Eq.(5) is not consistent with that of Eq.(3). It would be better to use consistent symbols.: We thank the reviewer for raising this point, as it allowed to improve the clarity of notation in these equations. In Eq.(5) we are replacing all occurrencies of $p(\sigma_p| \sigma_{\setminus p})$ with $p^{pw}(\sigma_p| \sigma_{\setminus p} H, J)$ to have consistent symbols with Eq.(3). We are, however, not sure if the parts changed correspond to what the reviewer pointed to, so it would be kind if the reviewer could confirm or be more precise.
5. The proof in section 2.2.1 is incoherent. What is the function of Eq.(5)?: We thank the reviewer for raising this point, which is allowing us to clarify a critical point of the proof mentioned. The role of Eq.(5) is to show that the computation of the different elements of Eq.(3) can be carried efficiently, as the numerator of Eq.(5) is shared for all terms in Eq.(3). The rest of the proof then hence shows how one can compute efficiently the terms in the denominator. We are making sure to better highlight the role of Eq.(5) in the updated version of the manuscript.
6. In Eq.(7). It would be better to clarify that Eq.(7) is the L2 regularization term: We will make sure to clarify this fact in the updated manuscript.
7. In section 3, it would be better to list the number of entries in each dataset.: We have added this information regarding the entries of each dataset in the updated manuscript.
8. In section 4.1, what is the necessity of tuning the hyper-parameters of InvMSAFold-AR: We tuned those parameters because we felt that they played a crucial role in the performance of the model, and hence we wanted to find the combination of them that yielded the most accurate and efficient results. It might seem that we only did this for InvMSAFold-AR, which would completely justify the question of the reviewer, we report in Appendix A.1.2 we also performed such a tuning for InvMSAFold-PW, yet we found that then in many application the tuned model slightly underperformed the original model we were previously using, and therefore we kept that one. We believe this could be due to the fact that the pseudo-likelihood can be not well correlated with generative properties of the model.
9. It seems that InvMSAFold-PW performs better than InvMSAFold-AR at larger hamming distance. What is the probable cause?: We thank the reviewer for raising this point, which we also noted and gives us the opportunity to discuss our view on this matter. We do not have a strong explanation for this phenomenon and believe that a more in- depth analysis is needed. However, we speculate that this is linked with what we observe in Figure 7 of the manuscript, i.e. that InvMSAFold-PW generates sequences at higher hamming distances that InvMSAFold-AR, hence it is more accurate in that region which is more on the tail for InvMSAFold-AR.

We first want to thank the reviewer for carefully reading the manuscript and for the questions raised, which is allowing us to improve the manuscript.

We will address the questions point by point, reporting the original question in italics followed by the answer.

1. There is not a specific example taken through to the conclusion that the model preserves "structural and functional integrity". Functional integrity is what you want when you're designing new proteins/doing virtual screening. The authors should consider including such an example or clarifying this statement since that is a major claim of their paper: We thank the reviewer for raising this crucial point. While direct wetlab validation of the generated sequences is not in the scope of our work, we believe that the structural consistency results provide some evidence that functionality is conserved, given the tight relationship between structure and function. We agree, however, that this point should be addressed in a wetlab setting in the future.
2. It was not clear on the InvMSAFold-AR/-PW. I understand that PW requires MCMC and AR does not but I wonder are there cases/tasks in which a PW vs AR model is more appropriate?: We thank the reviewer for pointing out this possible source of confusion. It has been observed in a different context Trinquier, J. et al. (2021) that both models tend to result in very similar performance on different tasks when trained directly on MSAs. However, we found in our paper that the autoregressive model is outperforming the pairwise model when generated by a neural network in our setting. Nonetheless, the pairwise model has a stronger theoretical foundation and has been validated as a good model for protein sequence variability over several decades of research. We strongly suspect that this is due to the fact that the autoregressive model allows for an exact computation of the likelihood and does not need pseudo-likelihoods for training. We therefore strongly feel that this is a result of the training procedure As a result, this situation could change by more exploring other training procedures in the literature of pairwise models Barrat-Charlaix, P (2018).
3. What would be an example in which you could demonstrate preserved functional integrity that is not directly related to structural integrity in your model's generation of diverse protein sequences? It seems an important question because when you want to design a protein to do some specific function (bind some small molecule or interact with another protein) you only care about structure to the extent that it acts as a proxy for function. But maybe it doesn't have to be? Do you think your models could get at function outside the restraint of the specific structure that is your input?: We thank the reviewer for raising this crucial point, similar to the question 1 reported in this answer. We refer to the answer there.

We first want to thank the reviewer for carefully reading the manuscript and for the questions raised, which is allowing us to improve the manuscript.

We will address the questions point by point, reporting the original question in italics followed by the answer.

1. How is sampling of InvMSAFold-PW achieved? Which MCMC algorithm do you use?: We thank the reviewer for raising this question, which allowed to clarify a point on which we had been too vague. Sampling for InvMSAFold-PW is achieved by leveraging the bmDCA sampling library. Such a library implements a standard Monte Carlo algorithm with Metropolis-Hastings proposal as default, yet more advanced proposals as z-sqrt and z-barker from Livingstone S. & Zanella G. (2019) are also available. The main advantage of such a library is the computational efficiency of its implementation and parallelization features. We will highlight more these aspects in the updated manuscript.
2. By using a PCA projection you show that sequences generated by InvFoldMSA have a better coverage of sequence space. But why do you restrict the analysis to the first two principal components?: We thank the reviewer for raising this point, which gives us the chance to explain better our experimental procedure, especially the connection between plots which might seem independent from one another. The choice of using only the first two principal components was driven by different considerations. First of all, for visualization the first two components are most convenient. Moreover, the choice of reporting the first two PC components is consistent with other relevant works on Potts models in the literature as Trinquier, J et al (2021). Lastly, the PCA plot as Figure 6 in the manuscript have to be interpreted in conjunction, and not independently, from Figure 5. Indeed the latter, by computing the correlations between the synthetic and true covariances, gives a global result, while the former allows for a local and clear interpretation of consequences of the results of Figure 5. In turn, Figure 5 ensures that the results observed in Figure 6 are not restricted to the first two principal components. We will underline this connection between Figure 5 and 6 in the updated manuscript.
3. Have you tried AlphaFold3 to validate the sequences generated by InvFoldMSA?: We agree with the reviewer that such an experiment would be very interesting, unfortunately currently we did not run AF3 on the generated sequences. For the scale of our experiments, we need several thousand forward passes which we were not able to do with the web server available during the conception of our work. The code and weights of AF3 were only released very recently and we do not have sufficient time to run it locally.
4. We further thank the reviewer for reporting the typos/grammar above, which allowed us to polish the manuscript.