About Question-1: the explanation of 2D RoPE and the novelty clarification.

2D RoPE Explanations. Rotary Positional Embeddings (RoPE) encode position information of tokens with a rotation matrix that naturally incorporates explicit relative position dependency. First, consider the 1D Rotary Positional Embedding. Given any two-dimensional feature vector $x_i \in \mathcal{R}^2$ at position $m$. The position embedding can be expressed as:

<p> $f_{\{q,k\}}(x_m, m) = \mathbf{R}_{\theta,m}^2 W_{\{q,k\}} x_m$. Such that $$q_m^Tk_n = (\mathbf{R}_{\theta,m}^2 W_q x_m) ^ T(\mathbf{R}_{\theta,n}^2 W_k x_n)=x_m^TW_q\mathbf{R}_{\theta,n-m}^2W_kx_n $$ Here: $\mathbf{R}_{\theta,\{m,n\}}^2$ is the rotation matrix that depends on the position. $W_{\{q,k\}}$ is a learnable weight matrix. The key to RoPE embeddings is to determine the rotation matrix: $$ \mathbf{R}_{\theta, m} = \begin{pmatrix} \cos m\theta & -\sin m\theta \\ \sin m\theta & \cos m\theta \end{pmatrix} $$ After some derivations, given a 2D position $(m,n)$, a solution for the 2D RoPE is obtained as: $$ \mathbf{R}_{\theta,(m,n)} = \begin{pmatrix} \cos m\theta & -\sin m\theta & 0 & 0 \\ \sin m\theta & \cos m\theta & 0 & 0 \\ 0 & 0 & \cos n\theta & -\sin n\theta \\ 0 & 0 & \sin n\theta & \cos n\theta \end{pmatrix} $$ </p> This solution is easy to understand. It is a block matrix composed of two 1D RoPEs, essentially dividing the input vector into two halves, applying the 1D RoPE to $m$ for one half and the 1D RoPE to $n$ for the other half. From this form, we can also easily generalize to RoPE for 3D, 4D, and other dimensions.

The novelty clarification. The principle of multi-dimensional positional encoding has been explored across various domains to address specific challenges inherent to those fields with different intrinsic design proposes. In the MSA generation scenario, incorporating a dual-axis positional encoding scheme is driven by the unique requirements of modeling the complex dynamics of evolutionary patterns in protein homologous sequences, which involves identifying simultaneous mutations across multiple amino acid sites (columns) in different homologs (rows), other high-level interactions. Therefore, a multi-dimensional encoding approach, as compared to a decoupled single-dimensional approach, is both distinct and critical. In light of this, we adapt the RoPE-2D relative positional encoding extended from 1D RoPE to capture these patterns.

About Question-2: the efficiency comparison. We compared the generation speed between MSAGPT and several baseline generative models, including the newly-added EvoDiff. These models were run on a single A100 80G GPU under the direct sequential generation regime, and we report the average tokens per second (toks/s) for generating 2k tokens, averaged over three runs.

From this comparison, we can conclude that MSA-Aug. achieves the lowest inference efficiency due to its encoder-decoder framework. EvoGen and our proposed MSAGPT have similar inference speeds, with the diffusion framework (EvoDiff) showing better inference efficiency. However, EvoDiff shows worse MSA generation quality in the structural prediction tasks (The performance comparison refers to the Table. 1, 2, 3,4 in the attached PDF).

About Question-3: the performance on MSA-abundant conditions. We compare the results of query sequences with abundant natural MSAs to those with abundant natural MSAs augmented by MSAGPT's generated MSAs on CAMEO set. For this comparison, we sample 128, 256, and 512 sequences from both the natural MSAs and the generated MSAs. The results are shown in Table. 6 in the attached PDF. These results indicate that the inclusion of generated MSAs has no significant effect on the performance in MSA-abundant conditions, which is consistent with previous findings that when more than 64 MSAs as input, AF2 predicts a "converged" structure.

About Question-4: the metrics measuring the "prediction accuracy". The prediction accuracy refers to the TM-score, which serves as the golden metric for evaluating the accuracy of predicted protein structures compared with the ground truth structures. Additionally, we provide other metrics such as pTM, GDT_TS, and LDDT in Table 1,2,3,4 in the attached PDF. The results indicate that enhancements in TM-score are consistently accompanied by improvements in these golden metrics, i.e., GDT_TS and LDDT, confirming the robustness and reliability of our predictive method across different evaluation criteria.

About Question-5: the pLDDT selection strategy. pLDDT selection helps identify structurally similar sequences by providing a confidence measure in predicted protein structures without needing ground truth. Higher pLDDT scores highlight regions predicted with greater accuracy, indicating that the corresponding virtual MSA is informative and structurally similar. This confidence-based filtering focuses on the most reliable parts of the predicted structures for more accurate identification. For detailed selection processes, please refer to Appendix E.

About Question-6: the number of cases used in DPO. We construct RLAF preference dataset $\\{Q^{(k)}, m_{w}^{(k)}, m_{l}^{(k)}\\}_{n=1}^{N}$ for the DPO training, where $N=11k$.

Thank you for your valuable feedback and careful assessment of our work. We address your concerns below,

About Weakness-1: the missing important experimental details. The training details and experimental settings, including the processes for Pre-training, Rejective Finetuning, and DPO, as well as the hyperparameter selection, are thoroughly discussed in Appendix Section C.

About Weakness-2: the data leakage prevention and test data split.

Data Leakage Prevention. As outlined in Section 6.1 of our paper, we implemented a thorough filtering process to eliminate any potential data leakage. Specifically, we removed all MSAs of sequences in the test sets (CAMEO, CASP, and PDB) from the pre-training dataset. Furthermore, we ensured that any sequence in the pre-training set with a similarity greater than 0.9 to a sequence in the test set was excluded. To validate this filtering process, we used the HHblits tool to retrieve sequences from the test set and calculate their maximal similarity distribution with sequences in the pre-training dataset. The results, are illustrated in Figure. 1 in the attached PDF, shows that the maximum similarity is 0.89, confirming that there is no data leakage in the pre-training dataset.

Temporal and structural splits. For the pre-training dataset, we used the OpenProteinSet, containing protein sequences collected before December 28, 2021. For structural predictions, we followed AlphaFold2's evaluation settings, using PDB datasets from before January 22, 2024, CASP14 from May to August 2020, CASP15 from May to August 2022, and CAMEO after August 20, 2020. Our primary goal is to improve structural prediction accuracy in low-MSA regimes using generated virtual MSAs. Therefore, our methods need to generalize across different protein families and timelines. We ensured that sequences in the test set were not included in the pre-training set, as AlphaFold2 does, and conducted experiments on three well-benchmarked datasets to confirm the robustness and generalizability of our approach.

About Weakness-3: the clarification of evaluation. Our primary goal is to enhance low-resource structure prediction using generated virtual MSA. The main experiments focus on protein structure predictions in zero-shot and few-shot scenarios on a Natural MSA-scarce benchmark. We also present results on artificially MSA-scarce and MSA-abundant scenarios (see Tables 4 and 6 in the attached PDF). Additionally, we demonstrate our model's transferability across four protein tasks, highlighting MSAGPT’s potential to impact a broad range of protein-related tasks with generated MSA.

About Weakness-4: the statistical significance test of results. We have addressed statistical significance in several ways. For results in Tables 1 and 2, t-test results demonstrating significance against baseline methods are in Appendix Table 6. For Table 3, we conducted 5-fold cross-validation and reported average performance to mitigate random effects, detailed in Appendix Table 7.

Thanks for your insightful feedback and constructive suggestions for our work. We addressed your questions as follows:

About Question-1: add more evaluations. We have adopted both suggestions to confirm the superiority of MSAGPT:

• Statistical Significance of Metrics: We conducted a paired Student's t-test between MSAGPT and other baselines. The results, shown in Appendix Table 6 in the paper, indicate that the virtual MSA generated by MSAGPT significantly improves structure prediction accuracy in cases with limited MSA compared to other baselines.
• Evaluation on a Larger Set and more metrics: We created the Artificial MSA-Scarce benchmark based on the PDB dataset released before 2024-01-22. We collect approximately 8k protein sequences after filtering and perform zero-shot evaluations using these sequences with MSAs containing only virtual sequences. The results are detailed in Table. 4 in the attached PDF. These results demonstrate that the generated MSA significantly improves structure prediction accuracy in cases with limited MSA than other baselines.

About Question-2: the explanation of decreased pLDDT. The predictive metrics estimated by AlphaFold2, such as pLDDT and pTM, measure the confidence level of AlphaFold2's predictions, rather than reflecting the true structural prediction accuracy. pLDDT can be easily influenced by adding more MSAs. We conducted an ideal experiment to compare the performance among original sequences, augmented by randomly generated MSAs, and augmented by MSAGPT MSA as shown in Table. 5 in the attached PDF. We can see even with randomly generated MSAs, the pLDDT and pTM values are higher than with the original sequences, while the TM score decreases due to the introduction of noisy MSAs. Therefore, we adopt the TM score as the primary metric to select high-quality MSAs for supporting the subsequent RFT and DPO processes. Additionally, we provide other oracle metrics including GDT_TS, LDDT in Table 1,2,3 in the attached PDF. The results show that with TM score as the reward signal, other oracle metrics, i.e., GDT_TS and LDDT also improve, while the predicted metric pTM shows the opposite trend. Thus, the reasonable explanation for the decreased pLDDT and pTM metric is that the post-alignment process reduces hallucination scenarios.

About Question-3: correct the clarification on the utilization of MSA in AF2 . Thanks for your constructive feedback. We will incorporate your suggestions to clarify this in the next version of our paper.

About Question-4: the typo of "homogenous". Thank you for pointing out the typo. The correct term should be "homologous," which, in the context of biology, refers to organs or sequences that are similar in position, structure, and evolutionary origin, but not necessarily in function.

About Question-5: the semantics of MSA rows. The MSA obtained by MSA search tools, such as HMM search, inevitably contain noisy co-evolutionary patterns, such as large portions of deletions, insertions, and gaps. Many previous works aim to filter high-quality MSAs by clustering or ranking them based on predefined rules. One primary rule is to find MSA sequences most similar to the main sequence with fewer gaps, as these are more likely to represent informative co-evolutionary patterns. Following this idea, we sample MSA sequences using similarity-based weighted sampling, where sequences more similar to the query protein with less gaps are more likely to be ranked higher and selected. Our ablation study results, shown in Figure 5, confirm that compared to 2D positional encoding, the 1D positional encoding (which only retains the column site position while abandoning the row ordering information) performs worse. This indicates that incorporating row-order semantics through similarity-based sampling improves performance.

About Question-6: the flattening rules. As demonstrated in the overall framework in Figure 2 in the paper, we flatten the MSA along the row axis to ensure we can generate the MSA sequentially during inference. Flattening along the column axis is also theoretically reasonable, as the ordering information is bounded by the 2D evolutionary position embedding. However, when generating the MSA sequentially during inference, we would need to reverse the positional IDs along the row side. This differs from the training pattern and could introduce unforeseen errors, which require further investigation.

About Question-7: the degenerated cases after DPO. The case studies showing the differences in generated sequences before and after DPO are already presented in Appendix Section G and Figures 11 to 15. Generally, we do not observe significant differences in the generated sequences, except for a few degenerate cases, including generated MSAs that do not match the length of the query sequences or MSAs that contain only gaps or a single type of residue. The proportion of these degenerate MSAs is approximately 6% for DPO-generated MSAs compared to 1% for the base model. To address these degenerate cases, online RLHF like PPO, which directly involves AF2 in the training procedure, may be more effective. However, this approach faces efficiency issues due to the lower inference efficiency of AF2, which needs further investigation.

We guarantee that we will include all experimental results and discussions in the next version of our paper. Your feedback has been invaluable in refining our research. If we've addressed your concerns, we hope you might consider raising your score.