Immunogenicity Prediction with Dual Attention Enables Vaccine Target Selection

3. Data Splitting: In the initial submission, we only checked that there are no 100% same sequences exist in the dataset. Following the suggestion, we applied an additional redundancy removal step between all training and test samples with a 90% similarity cut-off to avoid potential data leakage. We test the performance on Immuno-Bateria and use ESMFold2 for structure prediction. After this modification, we observed a reasonable decrease in performance across all models, including the baseline models. Despite this challenge, ProVaccine continues to outperform other models and demonstrates its robustness and effectiveness.

4. Comparison between XGBoost and ProVaccine: (1) Would you mind provide additional explanations regarding “the novel antigens share high similarity with known ones”? We are unsure about the evidence that supports this statement. (2) We respectively disagree with your statement that “Actually XGBoost seems to be perfect and much better than Provaccine” for two reasons, following the observations we concluded in Section 5.3.1, First, the majority of antigens in nature are non-immunogenic, meaning they should be assigned a negative label in our prediction task. Therefore, an ideal predictive model should assign probabilities closer to 0 for most data points, rather than distributing them more uniformly across the 0-1 range (as observed in many other baseline methods) or concentrating them in a very narrow region (as seen in logistic regression). The KDE plot in Figure 4 shows clearly that only ProVaccine manages to approximate this behavior. Secondly, the time and financial costs of biological experiments are significant. Therefore, in addition to capturing true positives, it is preferable for a model to minimize false positive predictions. As shown in Table 10 of the Appendix, our method assigns the fewest positive labels (i.e., the fewest PVCs) while successfully identifying all 11 true positives. This demonstrates that our method can identify positive immunogens at a lower experimental cost.

5. The Existence of Spike Proteins in the Training Dataset?: We examined the dataset and confirmed that spike proteins (as well as all the proteins assessed in the two case studies) are not present in the training dataset. In the revision, we add explicit statement regarding the exclusion of these proteins to avoid confusion.

6. Difficulty of Identifying the Spike proteins and RBD regions and model effectiveness: We would like to clarify two important points in response to your concerns: (1) While the spike protein was successfully predicted as the top candidate, the model had no prior exposure to the SARS-CoV-2 spike protein (as this protein was never seen during training). This highlights the model’s ability to generalize knowledge to new spike proteins. Although the spike protein is a prominent candidate in current vaccines, the model's ability to make this prediction without explicit training on SARS-CoV-2 data underscores its reliability. (2) We designed the analysis of identifying potential RBD regions to demonstrate the explainability and potential usefulness of the attention scores produced by the model. Instead, we assess the model’s effectiveness through previous experiments, such as overall performance comparisons on benchmark datasets.

We believe the above responses clearly address your concerns and clarify the misunderstandings in the previous submission. We look forward to your feedback and are happy to continue the discussion if there are any remaining issues.

Thank you for the review and recognize the novelty of our work. Below we respond to your concerns and questions point-by-point.

Weaknesses:

1. Dataset Construction: We agree with your concerns regarding the construction of negative samples. This method indeed has notable flaws, and assuming that a combination of BLAST and VaxiJen serves as a gold standard is certainly not ideal. However, the lack of negative instances is a well-known issue in many scientific problems, and we chose to adhere to an acceptable, though suboptimal, method that has been used in prior publications (Dimitrov et al., 2020; Sotirov & Dimitrov, 2024; Doneva & Dimitrov, 2024). We have addressed this discussion in the manuscript (see lines 176-179 on page 4). In the revision, following Reviewer G2xM’s suggestion, we explored alternative data construction methods by augmenting negative instances from IEDB database validated more than two negative B cell assays, which make the negative data set more convincing. We evaluated the results of this new construction on both the baselines and our method, as shown below:

2. Negative Label Assignment Using Different Baseline Methods: As explained above, our use of VaxiJen to define negative samples follows existing publications. Introducing an advantageous bias for a baseline method (i.e., VaxiJen) holds no benefit for our work, and empirically VaxiJen did not achieve top performance under this construction method. The new data construction approach we tried during the revision included more negative samples directly collected from IEDB and publications, thus reducing bias toward any model. As shown in the previous results, our method still achieves top performance, demonstrating a clear advantage of ProVaccine over existing methods.

Thank you for acknowledging the novelty and contributions of our work in terms of the framework and benchmark dataset. We notice that we need to first emphasize our contributions and clarify that proposing a new benchmark for immunogenicity prediction is not the only major contribution of our study. To be precise:

1. Novel Algorithmic Approach: Our algorithm represents a significant departure from traditional machine learning in immunogenicity prediction. We introduce a pioneering multimodal deep learning-based model that integrates sequence, structural, and physicochemical data. The proposed model achieves top performance on benchmark datasets and show promising generalizabiltiy and interpretability in case study analyses.
2. Extensive Dataset Curation: We present an extensive, curated dataset of antigens from various sources. To the best of our knowledge, it offers the largest and most comprehensive training and testing benchmarks for the immunogenicity prediction community and serving.
3. Comprehensive Assessment Methodology: We designed the evaluation protocols based on meaningful criteria that are crucial to biologists and significantly relevant to immunological research. In addition to commonly used metrics in computer science (such as accuracy and recall) to assess the model's performance on the test set, we specifically included two case studies to evaluate the practical value of the model for vaccine development. This analytical approach also provides a novel validation framework for future research that may lack experimental conditions.

These concludes were also described in manuscript (line 93-98, in page2). For the other weaknesses and questions you raised, below is our point-by-point response.

Weaknesses:

1. Testing on Existing Benchmark Datasets: We understand that testing on benchmarks used in previous studies is usually an essential validation step when proposing a new algorithm. However, we did not include additional comparison on other datasets for two main reasons: (1) All benchmark datasets used in existing studies correspond to a subset of our dataset, to the best of our knowledge. This is different from other well-defined and largely explored tasks in machine learning (such as developing graph convolutional algorithms and testing on citation networks), As introduced in Section 3 of the manuscript, we incorporated data from a broader range of literature, included more diverse species, and extracted and processed updated records from public databases. (2) All our data is sourced from published studies and adheres to data processing principles from previous research (such as Dimitrov et al., 2020; Sotirov & Dimitrov, 2024; and Doneva & Dimitrov, 2024). A similar example of study that only tested on their new benchmark is ProteinGym (Notin et al., 2024), which consolidates multiple existing benchmarks plus additional benchmarks from different literature. As a result, their testings were also focused on their proposed larger, and more comprehensive benchmark dataset. We thus believe it is sufficient to evaluate baseline methods on our proposed new benchmark dataset.

2. Computational Cost of the Algorithm: To address this problem, we added an analysis of computational cost comparison in Figure 6 (in the Appendix) in the revision. We compared the training curves of ProVaccine and three vanilla PLMs on the VirusBinary dataset. It is evident that ProVaccine exhibits a much faster convergence rate, approximately 2-3 times faster than the corresponding vanilla PLM methods. Meanwhile, the per-epoch runtime of ProVaccine's is comparable to that of vanilla PLMs. For instance, training ESM2-650M for one epoch takes 2 hours and 52 minutes, while ProVaccine-ESM2-650M requires 2 hours and 56 minutes per epoch.

3. Improvement over XGBoost on Virus vs Tumor: (1) Regarding the statement, “The model’s performance is similar to XGBoost on the largest dataset, Immuno-Virus”, we assume this observation is based on Table 1 of the initial submission. We respectfully disagree with this statement because, except for T30 (where our DL methods ranked second), our ProVaccine consistently ranked in the top three for all other metrics. Furthermore, on the updated dataset (with improvements in negative sample construction and the use of AF2-based predicted structures, as suggested by reviewer G2xM), the performance gap between our method and XGBoost has further widened. (2) It is commonly observed that DL methods perform better on larger datasets due to their capacity to learn more parameters, which typically requires a larger volume of data for supervised learning tasks. However, in our case, we first utilize pre-trained PLM modules and a structure tokenization module to learn effective representations of protein sequences and structures. We greatly appreciate your observation of this phenomenon, which may indicate an additional novelty of our approach, i.e., its capability to perform robustly even with relatively smaller sample sizes. Unfortunately, the limited time and computational resources during the rebuttal phase restrict our ability to validate this finding. Nevertheless, this could be a promising direction for future work.

4. Quality and Superiority of the New Benchmark Dataset: We added a paragraph at the end of Section 3 to explicitly summarize the superiority of our new benchmark dataset compared to existing ones. In short, we provide larger and more comprehensive cross-species datasets with more reliable data processing and labeling procedures. For the various sources used to collect raw data, we conducted manual checks by immunology experts to minimize the risk of mislabelling issues. However, it is important to note that we cannot completely eliminate any potential noise originating from original experimental errors and variations in experimental technologies and environments. Such errors are inherent in other datasets derived from experimental sciences, and addressing them falls beyond the scope of our work.

5. Ablation Study with Sequence and Structure Input: The required ablation studies on different modules have been provided in Tables 7–9 in the appendix (uploaded as supplementary material). These studies include experiments with the following inputs: PLM-encoded sequence only, PLM-encoded sequence + atomic tokenization, PLM-encoded sequence + physiochemical descriptors, and the complete version combining PLM-encoded sequence (finer-level information) + atomic tokenization (coarser-level information) + physiochemical descriptors. Furthermore, following your suggestion, we conducted an additional experiment that directly predicted the information from one-hot encoded sequences. The results have been updated in the same tables in the revised appendix.

6. Comparison with Other DL Methods: To the best of our knowledge, there are no existing deep learning-based solutions for immunogenicity prediction, which is a research gap we have identified and mentioned in the manuscript. In our experiments, we included similar comparisons wherever possible. For instance, ProVaccine incorporates different base models (Ankh, ESM2, and ProtBert). Additionally, our ablation study, where we consider inputs only the PLM embedding, can be roughly regarded as a comparison with existing PLM-based models.

Questions:

1. Model Performance on Existing Datasets: See response to weakness 1.

2. Prediction Results on Virus and Tumor: See response to weakness 3.

3. Additional Analysis on Case Studies: Thank you for the suggestion. In the revised manuscript, we have included an analysis of the factors contributing to the model's predictions, such as the characteristics of the antigens, the role of multimodal representations, and the impact of the dual-attention mechanism.

In conclusion, we believe our work makes significant contributions not only in constructing a new benchmark dataset but also in advancing the algorithmic frontier of immunogenicity prediction and establishing robust evaluation protocols. The efficiency of algorithm as well as the quality and superiority of the new benchmark were identified.

We hope this response addresses your concerns and highlights the innovative and comprehensive nature of our research. Thank you again for the review and suggestions. We look forward to your feedback and are happy to continue the discussion if there are any remaining issues.

Questions:

1. homology cut-off: In the initial submission, we only checked the similarity between positive and negative samples during dataset construction. In the revision, we followed your suggestion and applied an additional redundancy removal step between all training and test samples with a 90% similarity cut-off to avoid potential data leakage. We test the performance on Immuno-Bateria and use ESMFold2 for structure prediction due to the limited time of implementation. After this modification, we observed a reasonable decrease in performance across all models, including the baseline models. Despite this challenge, ProVaccine continues to outperform other models and demonstrates its robustness and effectiveness.

Bateria

2. Figure Caption: Thank you for the suggestion. We did not provide more details in the initial submission due to the page limit. In the revision, we have updated the manuscript to include more informative captions.

3. Case Study 1: Thank you for your suggestions on the post-hoc analysis. In the revised manuscript, we modified the description and provide formulation of recall and enrichment to improve clarity and understanding.

4. Interpretatability: We completely agree with your perspective that model interpretability is a crucial consideration, especially in addressing scientific problems like immunogenicity prediction. This was indeed one major motivation for incorporating the attention pooling layer and designing the third experiment in Section 5.3.2. Our experimental design aligns closely with the idea you mentioned here. We assume that for a given protein, each AA has an importance score indicating whether it is immunogenic. In our implementation, this importance score is quantified using the attention scores from the pooling layer. Although this implementation is still relatively simple, it already demonstrates that our model is capable of identifying some critical regions as plausible epitopes, as illustrated in the example of the spike protein visualized in Figure 5. We believe this exploration holds promise for future applications to other deep learning-based methods. Furthermore, it presents a meaningful and worthwhile direction for advancing epitope recognition using deep learning approaches.

5. Generalizability to Other Pathogens: We believe the results from both post-hoc analyses address the generalizability of our model to other pathogens. In both cases, we tested and analyzed the pretrained model on entirely new proteins. In other words, the 1,858 candidates for Helicobacter pylori examined in Section 5.3.1 and the 38 candidates for SARS-CoV-2 tested in Section 5.3.2 were not included in ProVaccine's current training dataset.

Thank you once again for all the suggestions. We believe that the revised manuscript has significantly improved in quality and is now qualified for publication.

2. Folding Model Dependencies: We followed the suggestion and assessed the impact of structure quality on model performance by two additional experiments. We extracted AlphaFold2-predicted structures for all proteins in the constructed Immuno dataset and tested their performance. We retrieved pre-folded protein structures with UniProt IDs from the AFDB and used AlphaFold2 to predict structures for the remaining sequences without UniProt IDs. The test results are as follows:

Bacteria:

Virus:

Tumor:

For the second experiment, we tried to evaluate the prediction performance on a subset of the test dataset derived from the PDB database, which provides publicly available crystal structures. However, this subset consists of only 10 entries, which means the majority of the proteins have no crystal structure. Thus, this experiment could not be performed. We have added the result analysis and discussions in manuscript section 5.2.1.

Weaknesses:

1. Dataset Construction: We agree with you regarding the inherent limitation in constructing negative instances in the dataset, although this method has been used in previous studies (Dimitrov et al., 2020; Sotirov & Dimitrov, 2024; Doneva & Dimitrov, 2024). In the revision, we explored the two alternative approaches to constructing negative samples that you suggested.

• We first attempted to generate random amino acid (AA) sequences as negative samples, ensuring that their sequence identity with any positive samples from the same dataset was less than 30%. However, we identified two significant issues with this implementation. First, the predicted structures generally exhibited low folding confidence, with average pLDDT scores of lower than 50%. Second, when we applied these new negative samples to the training dataset while using the original test dataset (which contains negative samples derived from natural sequences), we observed that across all three datasets, the models achieved 100% accuracy during training but trivially predicted positive labels for all test samples. Our analysis suggests that when negative samples are generated randomly, the model primarily learns to differentiate between "protein" and "non-protein" rather than distinguishing "immunogenic proteins" from "non-immunogenic proteins".
• We also attempted to prepare negative samples from the IEDB database, identifying bacterial and viral non-antigens confirmed through experimental B-cell assays in Homo sapiens. We excluded tumor samples because the tumor-related records in IEDB do not indicate whether they are from human sources. Notably, all positive antigens in IEDB were already included in our existing collection. After preprocessing steps such as redundancy removal and filtering out low-quality data, 149 novel bacterial and 124 viral non-antigens remained. We incorporated these newly labeled negative data into the existing dataset and updated the performance on both our model and the baseline models. The updated results are provided in Table 1 as follows:

ProVaccine continues to demonstrate significantly better performance compared to other baselines. The new results have been updated in the revised manuscript.

Thank you for acknowledging the contributions of our work and for providing detailed and constructive feedbacks to further strengthen it. In response to the weaknesses and questions you raised, we have updated the paper accordingly and provide a point-by-point reply below.

Thank you for your review and for recognizing the novelty of our work in terms of algorithmic construction, dataset preparation, empirical performance, and so on. We noticed that the concerns raised mainly stem from unfound tables and figures, which we have provided in the appendix and uploaded separately as supplementary material. Below, we address each of the mentioned weaknesses and questions in detail.

Weaknesses:

1. Ablation Study: The required ablation studies on different modules have been provided in Tables 7–9 in the appendix (uploaded as supplementary material). These studies include experiments with the following inputs: PLM-encoded sequence only, PLM-encoded sequence + atomic tokenization, PLM-encoded sequence + physiochemical descriptors, and the complete version combining PLM-encoded sequence (finer-level information) + atomic tokenization (coarser-level information) + physiochemical descriptors. Furthermore, following your suggestion, we conducted an additional experiment that directly predicted the information from one-hot encoded sequences. The results have been updated in the same tables in the revised appendix. In summary, encoding sequence information with PLMs is essential for enhanced performance in immunogenicity prediction tasks. Additionally, both atomic tokenization and physiochemical descriptors contribute significantly to the representation of the protein. The combination of all three modules achieves optimal prediction performance.

2. Likelihood values in Figure 4: Thank you for suggesting a clearer way of reading Figure 4. We have now included a detailed likelihood report for the 11 determined immunogens under each model in Table 15 of the revised Appendix. We believe this additional information provides clearer insights into the performance of different models and their ability to identify known protective antigens.

3. Missing Tables and Figures: The appendix has been uploaded separately as the supplementary material, which includes all the mentioned reference Tables and Figures in the comment (Tables 4-6, 10, and 11). Note that Table 11 in the initial submission is now Table 12.

Qestions:

1. Random Splits on the Test Dataset:We repeated the construction of random test sets to obtain the variance in the model’s predictive performance. This repetitive evaluation process is conducted entirely on the test set and is independent of validation or hyperparameter tuning.
   As explained at the beginning of Section 5.2.1, our data is split into training, validation, and test sets in a 7:1:2 ratio. We use 70% of the data as the training set to train the model and compute accuracy on the 10% validation set for hyperparameter optimization. For the 20% test set, we performed 10 random sampling rounds with each round picking up 50% of the test sets as a new “sub-test set” and calculated the scores of the trained model on these sub-test sets.

2. Generalizability Evaluation: In Section 5.2.2, we designed experiments to evaluate the generalizability of the model when the training and test sets come from different species (lines 359-360). For instance, the orange bar in the top left corner of Figure 3 represents a model trained on virus data and tested on bacteria data. Contrary to the question raised in the review, we did not train the model on additional datasets and then test it separately on the virus, bacteria, and tumor datasets. The reason to design such an experiment is that immunogenicity data varies significantly in abundance across different species, making it potentially infeasible to retrain a model specifically for any given species. Therefore, we desire a well-designed model that could generalize and perform well on test data from species different from the training data.

3. Interpretation of Case Study 1: For the two points in the case study, (2) The majority of antigens in nature are non-immunogenic, meaning they should be assigned a negative label in our prediction task. Therefore, an ideal predictive model should assign probabilities closer to 0 for most data points, rather than distributing them more uniformly across the 0-1 range (as observed in many other baseline methods) or concentrating them in a very narrow region (as seen in logistic regression). (3) The time and financial costs of biological experiments are significant. Therefore, in addition to capturing true positives, it is preferable for a model to minimize false positive predictions. As shown in Table 10 of the Appendix, our method assigns the fewest positive labels (i.e., the fewest PVCs) while successfully identifying all 11 true positives. This demonstrates that our method can identify positive immunogens at a lower experimental cost.

We believe the above responses clearly address your concerns and questions. We look forward to your feedback and are happy to provide further clarification on any remaining issues.

Thank you for your response, and my apologies for not having access to the supplementary material while reviewing, I don't know why I did not.

• Ablation studies: Thank you for these. It is interesting that just using the fine and coarse sequence models do better on most tasks, with Recall being an exception (NB: I wish the same color scheme used in Tables 4-6 to indicate top performers would have been used here to make this clearer). Why then also add the physchem descriptors to ProVaccine?
• Likelihoods: Thank you, this is much easier to interpret.
• Splits of test dataset: doing what you've done says more about the distribution of the test set, which is apparent in the similar scales of stdevs reported, than it does about the model, e.g., the effect of random initialization or training on different subsets.
• Generalizability evaluation: Apologies for not being clearer, but I was asking the specific question of why we should expect the model trained on one species to generalize to another. Is it clear that the immunogenic responses should be the same across species?
• Interpretation of Case Study 1: Thank you, this sufficiently answers my questions.

I've raised my score, looking forward to discussing further.

Thank you for your continued engagement and for raising your score. We appreciate your thorough review and address your points below.

1. Ablation Studies:

Following your suggestion, we have updated the tables in the appendix to highlight the best-performing results. Regarding the inclusion of physiochemical descriptors, according to the results of the ablation studies reported in Tables 12–14, using a combination of sequence, structure, and physiochemical descriptors (row 4) generally outperforms using only sequence features (row 1) or sequence + structure features (row 2) within the same PLM encoder. This observation aligns with other machine learning-based immunogenicity prediction studies, which typically rely on these physiochemical descriptors for predictions. It also suggests that while deep learning-based sequence and structure embeddings can be powerful in general representation learning tasks, molecular patterns derived from prior domain knowledge can still provide valuable additional information for specific downstream tasks. For instance, in the Table 12, when use Ankh to find the sequence embedding, we have:

2. Splits of Test Dataset:

We agree with you that the random split of the test set primarily reflects the distribution of the data, which was our primary motivation for designing this test, i.e., to evaluate the stability of the trained model's performance on data with inherent variance.

We also recognize the importance of examining model stability and have therefore conducted additional experiments by repeatedly training and testing the model on randomly split datasets. We conduct the evaluation on the Immuno-Bacteria dataset with an 8:1:1 random split over 5 repetitions. As shown in the table below, our model demonstrates considerable stability, maintaining consistent performance across different dataset partitions, indicating its resilience to random initialization and variations in training subsets. The results have also been added to Table 8 of the revised Appendix.

3. Generalizability Evaluation:

Before giving a detailed justification, we would like to first clarify that our model predicts immunogenicity specifically in humans, regardless of the species origin of the antigens. Thus, we evaluate the generalizability from the perspective of whether a model trained on one species' antigens can predict the immunogenicity of antigens from another species, in the context of human immune responses.

We understand that there are differences in immunogenic responses across species, but our focus is on the shared immunogenic properties that antigens from different species may exhibit when interacting with the human immune system. Proteins from various species can elicit immune responses in humans, suggesting that there are commonalities that can be leveraged for prediction, despite their different origins. In this regard, we desire a model to capture these shared immunogenic features, such as when the training data comes from one species and the test data from another (in the context of human immunogenicity). This approach allows us to evaluate the model's performance and its capacity to generalize across different protein sources while maintaining the focus on human immune responses.

We hope this round of discussion provides solid evidence or justification for the previously unresolved questions, and we look forward to your feedback.