MSA Generation with Seqs2Seqs Pretraining: Advancing Protein Structure Predictions

Answer 1

Certainly, this is an insightful query. Indeed, the seqs2seqs pretraining method can be applied in protein design. A following study, titled "Protein generation with evolutionary diffusion: sequence is all you need," showcases this application. This research involves a diffusion framework that has been pretrained on extensive MSA datasets. It focuses on a sequence-driven design strategy, moving away from the conventional structure-function approach in protein engineering.

Answer 2

In our study, we strategically chose CASP14 and CASP15 as they represent the latest and most renowned benchmarks in protein structure prediction. Our decision was guided by the prominence of these benchmarks in the field. To the best of our knowledge, CASP16 has not been released at the time of our research. However, we are confident in the applicability of our method to future benchmarks, as evidenced by our encouraging results on CASP14 and CASP15. It's important to note that our current model is relatively modest in scale, with limitations in both parameter size and the extent of the pretraining dataset. We anticipate significant enhancements in performance with the expansion of our model, both in terms of its complexity and the breadth of training data.

Answer 3

AlphaFold2 indeed utilizes an attention mechanism; however, its application differs from ours as it is not designed for generative purposes and thus does not incorporate a cross-attention mechanism. Our proposed MSA-Generator, in contrast, is specifically tailored to leverage various forms of attention to enhance generative capability.

1. Tied-Row Attention: This component, whose efficacy is supported by research in the MSA Transformer field, is critical in our framework. It helps in accurately capturing the evolutionary information present in MSAs, a key aspect that drives our model's performance.
2. Cross-Row Attention: This feature plays a vital role in reducing computational complexity. It is essential for the decoder to efficiently process the entire MSA input, enabling our model to handle large-scale data more effectively.
3. Self-Column Attention: Consistent with prevailing insights from MSA-related studies, we recognize self-column attention as crucial for accurately modeling MSAs. Our framework incorporates this to better understand intra-column relationships, thereby enhancing prediction accuracy.
4. Cross-Column Attention: We have specifically designed this to uphold the inductive bias related to the conservation property of MSAs. It adds a layer of sophistication to our model, allowing it to capture subtle but crucial evolutionary patterns.

Regarding the possibility of conducting ablation studies, it is important to note that our framework's components are deeply integrated. Removing any one of them would not only diminish the model's effectiveness but also require extensive pretraining on large datasets. Hence, conducting such studies is not feasible within our limited computation resource in this period. We do, however, acknowledge the importance of these studies and plan to undertake them in our future work. This will allow us to further validate the individual contributions of each component to the overall effectiveness of our model.

Answer 4

In Fig. 4(b), created using GraphPad Prism 9, the solid line represents the linear regression equation, while the dashed lines indicate the 95% confidence intervals, automatically plotted by the software.

Regarding your observant question about the performance in certain cases, it's true that sequences with a baseline depth of 5 occasionally outperform those with a real depth of 15, as seen in examples like T1035-D1, T1074s1-D1, T1048-D1 (refer to Tables 5 and 6 in the Appendix). However, generally speaking, there is a proportional relationship between depth and lddt on average.

Answer 5

Certainly, we release our codebase including inference and pretraining for future research. Here is the link to the anonymous codebase: https://anonymous.4open.science/r/MSA-Augmentor-E2D7/README.md

Answer 1

We appreciate the reviewer's suggestions and will include them in our revision. Our work differs from the VAE-based methods mentioned in key ways:

1. Our method is tailored for MSA generation, emphasizing unsupervised generative pretraining for zero-shot downstream tasks. This is a capability not present in the cited VAE-based methods, which demonstrate a limited capacity in producing effective sequences across diverse domains.
2. Unlike VAE works focused on evaluating and simulate protein mutations and their functional implication, our focus is on novel MSA generation for improved protein structure prediction. We have included Potts model which can generate MSAs in our comparison.
3. Our main contribution is innovative pretraining tasks and a large-scale pretrained model ready to use for enhancing AlphaFold2 and others, distinct from prior research.
4. Learning and sampling from MSA distributions is more complex than it appears within VAE framework, especially with the diversity of protein families and long protein sequences. We did not see improvement on structure prediction task in relevant works. Inspired by NLP advancements, we provide an effective solution for MSA generation, directly confronting these complexities.

Answer 2

The link between MSA depth, quality, and downstream performance is clear. For details, see AlphaFold2's paper "Highly accurate protein structure prediction with AlphaFold", specifically Fig 5 (a), discussing how shallower MSA depth reduces quality and impairs downstream tasks. This was our initial motivation.

Our Fig 4(a), a violin plot, demonstrates that a baseline depth of 5 generally underperforms against real and virtual depths of 15. This is evident from the blue distribution (Baseline depth=5) being more concentrated in lower scoring areas compared to the yellow (Real depth=15) and red (Virtual depth=15).

Regarding random protein sequences, our 'Iterative Unmasking' baseline already employed a similar concept, unmasking random tokens with the MSA Transformer. Yet, as Table 1 shows, this didn't significantly improve results. Addressing your concern, we tested introducing random sequences by replacing 5%, 10%, and 15% of tokens, increasing MSA depth. This worsened downstream performance, confirming our approach's effectiveness.

Answer 3

We would like to clarify that Figure 7 in the Appendix serves purely for visualization purposes of the MSAs generated and is not a basis for analytical evaluation in our study. For visualization, we use Jalview, a standard tool that automatically calculates metrics like quality and consensus. However, these metrics are not the focus of our experiments in the main content of the paper. They are relevant only for single MSA analysis and do not provide an accurate assessment of our methods. We have not employed these metrics in any part of our research presented in the paper. The Diversity and Conservation metrics we use are derived from PSSM and Shannon Entropy, as detailed in section 4.5, and Figure 7 is not connected to this analysis.

These metrics are calculated per column and don't indicate overall MSA quality and conservation. Our model doesn't enforce conservation but learns it via cross-column attention. Fig 7's metrics don't show overall MSA quality, and we didn't target them specifically. We've removed this unclear part in our revision.

In our prior response, we detailed outcomes for sequences generated via random replacement; see those for details.

Answer 4

Section 4.5 showcases the diversity of generated MSAs, focusing on analyzing these MSAs directly instead of downstream tasks like protein structure prediction. Thus, we omitted a comparison of pLDDT scores here, but it's available in section 4.3 and table 1.

To better clarify:

• We describe that 'The results, derived from 10 runs following section 4.3' showing dataset and strategies are consistent with Section 4.3.
• The construction of the Position-Specific Scoring Matrix (PSSM) is detailed in the middle part of Section 4.5 as “we examined the PSSM of both original and MSA-Generator generated MSAs”
• To measure the similarity between generated and original MSAs' PSSMs, we used the Pearson correlation coefficient. Below is the pseudocode for this computation:

gen_pssm = calculate_pssm(gen_msas_seqs[i], bg_freqs)
source_pssm = calculate_pssm(source_msas_seqs[i], bg_freqs)
correlation = np.corrcoef(source_pssm.ravel(), gen_pssm.ravel())[0, 1]

we have revised our draft for clear presentation.

Answer 1

To clarify, our approach is not intended to replace actual multiple sequence alignments (MSAs) with synthetic alternatives. As we highlighted in the introduction of our manuscript, our main focus is on proteins characterized by a limited number of homologous sequences. In these instances, traditional MSA search algorithms often fail to construct comprehensive alignments. Our methodology, therefore, arises out of necessity: we generate the required MSAs when existing data is insufficient. An illustrative example is the protein T1122 from CASP15, identified in the Tipula oleracea nudivirus. The scarcity of homologous sequences for this protein in existing databases presents a significant obstacle in constructing MSAs, which are vital for the analysis of protein structure and function.

Our MSAGenerator is designed to specifically address this gap, generating virtual sequences to effectively supplement real sequences, especially in scenarios where actual sequence acquisition is challenging or not feasible. Notably, our model, pretrained on a diverse array of searched sequences, is adept at learning complex co-evolutionary patterns within MSAs. This learning approach enables it to generalize to a broader domain, similar to the capabilities of a Large Language Model. This functionality significantly enhances downstream structural predictions, a claim substantiated by the experimental results we present.

Answer 2

Thank you for pointing out this typo. We apologize for any confusion caused by the apparent inconsistency in our manuscript. To clarify, we consistently used the UniRef90 database for both the pretraining and testing phases of our study, UniRef50 was not employed throughout our study

Answer 3

The concept of the artificial challenging MSA is introduced in our study to critically assess the efficacy of our generated MSAs in comparison with real MSAs. We wish to clarify that our intention is not to suggest the exclusive use of virtual MSAs over real ones, nor do we claim that virtual MSAs are inherently superior (please refer to response 1). Our primary objective is to demonstrate that virtual MSAs can closely approximate real ones in downstream tasks.

The inclusion of this comparison is vital to establish the utility of our MSAGenerator in situations where real MSAs cannot be obtained, as discussed in section 4.3. Before proposing the use of virtual MSAs in such contexts, it is essential to validate their effectiveness in comparison to real MSAs derived from traditional search methods. We have renamed this section into “Are Virtual MSAs as good as Real MSAs?” for better presentation.

We also want to emphasize that none of the testing sequences were included in any part of our pretraining data. This ensures a fair and unbiased assessment of the MSAGenerator’s performance

Answer 4

Thank you for your advice. We have incorporated the suggestion of providing a negative interval, and this has been included in the revised draft of our manuscript

Answer 5

We appreciate your inquiry regarding the consistency of our reported improvements when using different protein structure prediction methods, such as RoseTTAFold. In Section 4.2 of our paper, our primary goal is to demonstrate the quality of the MSAs generated by our method. This section is not intended to suggest the replacement of real MSAs with virtual MSAs when real ones are available. We chose to focus on AlphaFold2 (AF2) for these experiments because it generally outperforms RoseTTAFold.

However, recognizing the validity of your concern, we have expanded our analysis in Section 4.3, the core results section of our paper. Here, we conduct experiments using both RoseTTAFold and AF2. This section specifically addresses scenarios where no homologues are identifiable using traditional search algorithms. By including both methods in our analysis, we aim to provide a more comprehensive validation of the effectiveness of virtual MSAs in these challenging situations

Answer 6

Thank you for your suggestion to clarify the experimental setup detailed in Table 1. In these experiments, we employed real-world challenging MSAs as introduced in Section 4.1. For MSA construction, we utilized JackHMMER with its default search parameters, specifically targeting the UniRef90 database. Each MSA created in this process contained fewer than 20 homologues, and these MSAs were consistently used across all experiments in this section

To elaborate further:

• For single-sequence-based algorithms like ESMFold and OmegaFold, we directly input the query sequence, bypassing the need for an MSA.
• For MSA-based algorithms, we employed various methods (Potts model, Iterative Unmasking and Ours) to generate MSAs, which were then fed into the folding algorithms.

We ensured a consistent experimental setup across all results reported in Table 1, aligning the benchmarking process closely with the conditions of our model's pretraining.