RankNovo: A Universal Reranking Approach for Robust De Novo Peptide Sequencing

We thank you for your reviews and address your concerns as follows.

Q1: The proposed RankNovo relies on the setting of multiple base models, which increases the computational cost.

A1: Your assessment is reasonable. We have now incorporated comprehensive details of the inference speeds for RankNovo in the new section A.5.6. As anticipated, RankNovo's inference speed is indeed slower than single-model approaches. However, the reranking process itself is not the primary time constraint. The majority of RankNovo's inference time is consumed in gathering peptide candidates from base models, as these require sequential autoregression and beam search decoding, while RankNovo's inference involves only a single attention forward pass.

As we scale from 2 to 6 base models in RankNovo, the inference time increases approximately linearly, with inference speed decreasing proportionally. This increased computational cost is an inherent characteristic of reranking frameworks and represents an unavoidable trade-off compared to single-model approaches. However, RankNovo effectively leverages this additional inference time to achieve superior performance levels unattainable by single models. This inference time-performance trade-off can be flexibly adjusted by modifying the number of candidates.

In the context of de novo peptide sequencing, RankNovo's significance lies in introducing a novel approach that allows researchers to optionally scale up inference time in exchange for enhanced performance. This represents the first such option in the field.

Q2: In formula 10, L_coarse and L_fine are not clearly defined. It seems more appropriate to use L_PMD and L_RMD?

A2: We agree with the your suggestion. To enhance clarity and consistency, we have replaced the terms L_coarse and L_fine with L_PMD and L_RMD in Formula 10. These new terms more accurately reflect the nature of the loss functions, aligning with the definitions provided earlier in the paper.

Q3: In formula (1) and formula (10), λ is confused.

A3: We're sorry for the confusion caused by the misuse of notations. To resolve this confusion, we have now changed all λs representing mass-to-charge ratio to μ, while λ is only used in formula (10) representing the weight of aggregating PMD objective and RMD objective.

Q4: Is the λ used in different tasks inconsistent? The author needs to clarify the setting of λ and the impact of λ on performance.

A4: Thank you for pointing this out. We apologize for the oversight in clarifying this aspect. To address this, we have now explicitly stated in line 348 of the revised manuscript that λ is consistently set to 0.5 across all tasks. The parameter λ is used to aggregate the two training objectives, PMD (Prefix Mass Difference) and RMD (Residue Mass Difference). To analyze the impact of λ on performance, we conducted an ablation study to evaluate the effects of using PMD alone (λ=1) or RMD alone (λ=0) as the training objective. The results of this study are provided in Table 4 and discussed in detail in Appendix A.4.2. Our findings show that combining PMD and RMD (λ=0.5) yields superior results compared to using either PMD or RMD individually. This demonstrates that a balanced integration of both objectives is crucial for achieving optimal performance.

We thank you for your reviews and address your concerns as follows.

Q1: The approach heavily depends on outputs from existing peptide sequencing models, which may limit its novelty.

A1: Reranking is a well-established technique across numerous natural language processing tasks, including question answering [1][2], recommendation systems [3], and recent large language model answer selection [4]. This technique involves collecting various candidates for a single query and selecting the optimal one among them. In the context of de novo sequencing, while reranking does utilize outputs from different existing models, this should not be viewed as a weakness or limitation. On the contrary, the reranking framework demonstrates excellent flexibility - regardless of base models and the source of candidates, RankNovo consistently delivers performance improvements (see Fig 3 (A) and Appendix A.5.1). RankNovo has illuminated an alternative pathway for enhancing sequencing task performance, distinct from the traditional approach of improving individual models. We anticipate that, influenced by RankNovo, future algorithmic development in this field will benefit from the synergistic interaction between two approaches: "improving single model performance" and "developing superior ranking strategies." This dual-pathway advancement represents RankNovo's fundamental contribution to the field.

Q2: The conclusion is somewhat simple and could be expanded to discuss the limitations of the current approach and potential future research directions.

A2: Your suggestion is very pertinent. In response, we have revised the Conclusion section to address the current limitations, future development directions, and potential impact on the de novo sequencing field. Specifically, we acknowledge that the primary limitation is the relatively slow inference speed. Future research could investigate efficient candidate sampling techniques, such as implementing base models with partially shared weights to reduce computational complexity. While RankNovo faces speed constraints, it represents a pioneering deep reranking framework that enables flexible balancing between inference time and performance, introducing an innovative approach to performance enhancement. We anticipate that, influenced by RankNovo, future algorithms in this field will benefit from the synergistic approach of simultaneously improving single-model performance and developing advanced reranking strategies.

Thank you for your detailed comments. We re-list and address your concerns as follows. The order of these concerns are rearranged to better express our opinions.

Q1: RankNovo should be evaluated against SOTA baselines, or others may favor a stronger single model instead.

A1: ContraNovo [1], which is referenced in the initial three lines of our manuscript, represents the current published state-of-the-art baseline for Nine-species-V1 and Nine-species-V2 datasets. In our research, we incorporated ContraNovo as one of six base models in our reranking framework, and Tables 1 and 2 present comprehensive comparative analyses between RankNovo and ContraNovo. Therefore, RankNovo has already been evaluated against SOTA baselines, and shows superior performance both on Nine-species-V1 and Nine-species-V2.

Our reranking framework includes not only published models but also four internally developed base models: ByNovo, R-Casanovo, R-ContraNovo, and R-ByNovo. These models demonstrate performance comparable to or exceeding that of ContraNovo, with ByNovo being a notable example of superior performance. Tables 1 and 2 clearly demonstrate RankNovo's performance advantages over all baseline models.

Therefore, we believe that empirical evidence substantiates RankNovo's superior performance relative to state-of-the-art baselines.

Q2: RankNovo's peptide recall improvement compared to base models are modest, which is expected since RankNovo uses ensemble and reranking framework.

A2: We acknowledge your observation regarding the scale of RankNovo's improvements, but we would like to contextualize these results within recent advances in the field. For perspective, the state-of-the-art ContraNovo [1] demonstrated peptide recall improvements of 0.051 and 0.038 over its predecessor, Casanovo-V2 [2]. Subsequently, a recent preprint work PrimeNovo [3] achieved gains of 0.020 and 0.025 over ContraNovo. In this context, RankNovo's improvements of 0.042 (Version 1) and 0.029 (Version 2) over ContraNovo, and 0.037 (Version 1) and 0.023 (Version 2) over the strongest base model ByNovo (also introduced in this work) represent significant advancements in the field. These incremental improvements are particularly meaningful in practical proteomics applications, where each additional correct peptide identification contributes to experimental accuracy. Furthermore, RankNovo's significance extends beyond metric improvements alone. The framework introduces a novel reranking methodology that complements existing sequencing models, establishing an alternative approach to performance enhancement that diverges from conventional single-model optimization strategies. We anticipate that this dual-approach paradigm will influence future algorithmic developments in the field, fostering innovation through the integration of both methodologies.

Q3: RankNovo should be slower than other single model frameworks, and the parameter number, training time, and inference time should be provided.

A3: We have incorporated comprehensive details regarding parameter counts, training time, and inference speeds for our six base models and RankNovo in the new section A.5.6. In summary, RankNovo comprises 50.5M parameters and requires 4 days of training utilizing four 40GB A100 GPUs, which is comparable to the base models. As anticipated, RankNovo's inference speed is indeed slower than single-model approaches. However, the reranking process itself is not the primary time constraint. The majority of RankNovo's inference time is consumed in gathering peptide candidates from base models, as these require sequential autoregression and beam search decoding, while RankNovo's inference involves only a single attention forward pass.

As we scale from 2 to 6 base models in RankNovo, the inference time increases approximately linearly, with inference speed decreasing proportionally. This increased computational cost is an inherent characteristic of reranking frameworks and represents an unavoidable trade-off compared to single-model approaches. However, RankNovo effectively leverages this additional inference time to achieve superior performance levels unattainable by single models. This inference time-performance trade-off can be flexibly adjusted by modifying the number of candidates.

In the context of de novo peptide sequencing, RankNovo's significance lies in introducing a novel approach that allows researchers to optionally scale up inference time in exchange for enhanced performance. This represents the first such option in the field.

We thank you for your reviews and address your concerns as follows.

Q1: The discussion of transformer-based approaches in the introduction is insufficient.

A1: We have significantly enhanced the discussion of transformer-based approaches in both the Introduction and Related Work sections. The revised text provides a more comprehensive review of recent advancements in transformer architectures, with a particular focus on their relevance to de novo sequencing and ensemble methods. These additions aim to contextualize our work more effectively within the broader research landscape. The updated content can be found on lines 74–78 and 136–138 of the revised manuscript.

Q2: In line 270, M represents the residue mass, but the same symbol is also used in line 306 to denote prefix mass. It would be better to add a subscript or identifier to distinguish prefix mass or, alternatively, use a different symbol to represent it.

A2: We sincerely apologize for any confusion caused by the inconsistent notation. In the revised manuscript, we have clarified this issue by ensuring that 'M' consistently represents the residue mass throughout the paper. Additionally, we have added summation notation in line 306 to explicitly and accurately denote the prefix mass, thus eliminating potential ambiguity.

Q3: In the mathematical formula (7) on line 309, should \overline{m}{q\tilde{j}} - \overline{m}{k\tilde{j}} be corrected to \overline{m}{qi} - \overline{m}{k\tilde{j}} to represent the prefix mass difference more accurately?

A3: Thank you for catching this error. You are correct, and we have rectified this typographical mistake in formula (7) on line 309 of the revised manuscript. The correction now accurately reflects the intended representation of the prefix mass difference. We appreciate your careful review and attention to this detail.