Distilling Non-Autoregressive Model Knowledge for Autoregressive De Novo Peptide Sequencing

Thank the reviewer for time and effort in reading our work, we provide detailed responses below to your raised questions and concerns:

The two-step training scheme, while effective, is more time-consuming than previous methods.

Thank you for raising this point. We will include a discussion on the additional computational costs for both training and inference in our revised manuscript. Indeed, as our method involves a two-stage training process and requires two decoders to be trained, it is more time-consuming compared to previous methods.

Specifically, training the AT decoder alone requires 130 epochs, with 1 million data samples per epoch, while the NAT decoder alone requires 150 epochs. When jointly trained, the total number of epochs is reduced to 120, though each epoch takes approximately 1.3 times longer than training a single decoder. The second phase, which involves knowledge distillation, requires an additional 20 epochs. Altogether, the total training time is approximately 30% greater than training a single AT model when using the same setup of 8 A100 GPUs.

However, this modest increase in training time yields significant benefits, with an improvement of over 3% in peptide-level accuracy—a substantial gain in this field, where accuracy improvements are notoriously challenging to achieve.

Regarding inference, our approach incurs minimal additional computational cost. This is because inference does not involve backpropagation, and the NAT decoder operates efficiently during inference, as it does not decode tokens sequentially but instead computes features in a single forward pass. Consequently, the time increase during inference, once the model is fully trained, is almost negligible. This is an important consideration for biological research, where computational efficiency during deployment is critical. We appreciate this constructive feedback and will ensure these points are clearly addressed in our manuscript. Thank you for the opportunity to clarify this aspect of our work.

The proposed method is a straightforward and expected combination of existing techniques (e.g., knowledge distillation is essentially a cross-attention with additional NAT embeddings).

Thank you for this comment. The intention behind proposing this framework was to introduce a clear and effective approach for efficiently improving this critical task in protein research. In protein-based biological research, usability and performance are of paramount importance, often outweighing other considerations. Tools developed for this field are heavily utilized by biologists directly in laboratory analyses and protein discovery workflows. For example, the baseline tool DeepNovo, which employs a relatively simple LSTM architecture, has seen extensive application in such tasks since its development.

Our method demonstrates a significant improvement of approximately 3% in whole-sequence-level accuracy, as shown in benchmark progression. This improvement represents a substantial advance, pushing accuracy to new heights. Such a performance gain is noteworthy in this domain, where even incremental improvements have a major impact. Moreover, our open-sourced software ensures that the benefits of our work are immediately accessible and applicable to the broader biological research community.

We believe that the clarity of our method and its straightforward design are among its strengths. By avoiding overly complex or redundant modules, we have ensured that our approach remains practical and efficient for downstream tasks. This simplicity not only makes our method more user-friendly for biologists but also lays a solid foundation for future developers to build upon. Thank you again for raising this point, and we hope this explanation helps clarify the motivation and value of our approach.

Minor (typos): Figure 2: “Step 2: Cross-Decoder Traning for Distiling Learned Knowledge”, "Probably Matrix" Figure 3 caption: Shouldn’t it be purple lines instead of green lines?

We sincerely thank the reviewer for their careful reading and attention to detail! We have noted these typos and will correct them in our revised manuscript. Thank you again for bringing them to our attention.

How much extra training time does CrossNovo need compared to existing end-to-end methods?

As mentioned in our response to Question 1, CrossNovo requires approximately 30% more training time compared to the baseline AT model. However, the additional computational cost during testing is minimal, with less than a 4% increase compared to existing AT models. Thank you again for this question!

Let us know if there is any other questions, we would love to provide more information! Thank you again for your time!

We first thank the reviewer for recognizing our contribution and providing valuable feedbacks!

Here are our reply to your questions:

While the results are strong, the improvements over baselines are relatively modest in some cases (e.g., 1–3% on some species). The authors could provide more discussion on the practical significance of these improvements.

Thank you for this valuable comment. We appreciate the opportunity to further elaborate on the practical significance of our results. While the improvements may appear modest at first glance (e.g., 1–3% on certain species), we would like to emphasize their importance, both here and in our revised manuscript:

• The Challenge of Peptide Sequencing Accuracy:

  In peptide sequencing, an entire peptide sequence must match the target exactly to be considered correct. This contrasts with other sequence prediction tasks, such as NLP, where predictions do not need to match word-for-word to be considered acceptable. Even a single incorrect amino acid prediction renders the entire peptide sequence invalid, potentially leading to a completely different protein function. This "all-or-nothing" correctness requirement makes improvements in accuracy inherently more challenging.

  As a result, every 1% improvement in peptide accuracy represents a significant number of correctly predicted peptide sequences, especially when applied to large sequencing datasets. For example, a 3% improvement indicates a substantial leap in the number of correctly sequenced peptides, reducing downstream errors in protein-level studies.

• Historical Context and Difficulty of Improvement:

  Previous de novo sequencing algorithms have been published in high-impact venues, including Nature Methods, Nature Machine Intelligence, Nature Communications, ICML, and AAAI. However, progress in peptide-level accuracy has historically been slow. Most prior works have reported only incremental improvements of 2–5%. For instance, the transition from LSTM-based models to Transformer architectures—a significant architectural change—yielded a modest improvement of roughly 4%.

  This slow progression highlights the inherent difficulty of achieving substantial accuracy gains in this task. Nevertheless, even incremental progress, such as our observed 4% average improvement across species, represents a meaningful step forward. Over time, such step-by-step advances can accumulate to achieve high accuracy, bringing us closer to reducing sequencing errors and enhancing protein sequencing techniques for real-world applications.

  Thank you again for raising this important point. We will expand on these discussions in our manuscript to better contextualize the significance of our improvements.

The computational difficulties of CrossNovo compared to baselines is not discussed. Given the additional components, it would be worth clarifying any additional computational costs required in the training.

Thank you for highlighting the importance of discussing computational difficulties. We agree that this is a valuable suggestion, particularly for those aiming to replicate our results. The training process for our model involves two stages, and the additional computational time is a notable cost. Specifically, during joint training, we observed that both decoders converged faster than when trained separately. Training the AT decoder alone typically requires 130 epochs, with 1 million data samples per epoch, while the NAT decoder on its own requires 150 epochs. In joint training, the total number of epochs is reduced to 120, although each epoch takes approximately 1.3 times longer compared to training a single decoder. The second stage, which involves knowledge distillation, requires an additional 20 epochs. Overall, the total training time increases by approximately 30% when using the same setup of 8 A100 GPUs, compared to training a single AT model.

In addition to time, the computational cost also increases with respect to GPU memory usage. With both decoders needing to be accommodated during joint training (without employing CPU offloading), the memory requirement is approximately 40% higher than for a single AT model. We will expand on these computational considerations in our manuscript to provide greater clarity. Thank you again for this constructive suggestion, which we believe will be highly beneficial for future researchers working on this task.

Questionable design choice: the choice of CTC loss is rather uncommon for peptide and protein modeling but not explained or backed by experiments:

Thank you for raising this concern. The use of CTC loss in this task follows the precedent set by previous state-of-the-art model, as briefly mentioned in our paper. The effectiveness of CTC loss for protein modeling has been extensively studied in earlier works on this task. We will expand on this discussion in our manuscript to provide further clarity and context.

To summarize the rationale for using CTC loss (also detailed in our response to Question 3), CTC enables global sequence-level optimization rather than focusing solely on token-level accuracy. This is critical for peptide sequencing, as the task demands the entire amino acid sequence, including the precise order of residues, to be correct. Any error in the amino acid sequence can result in functional and property changes in the peptide. CTC aligns well with this requirement by prioritizing sequence correctness, ensuring that the model captures global dependencies and relationships.

Moreover, CTC loss has been demonstrated to perform exceptionally well in this task by previous state-of-the-art models, and we have adopted it as part of our pipeline to build on their success.

We thank the reviewer once again for their thorough evaluation of our work and for catching these detailed points. We hope our response clarifies the rationale behind this design choice and provides a better understanding of our contributions. If our responses address the reviewer’s concerns, we would greatly appreciate a higher score if the reviewer finds our points valuable. We are happy to engage in further discussions or address any additional questions. Thank you once again for your time and thoughtful review!

It does not seem that there is distillation involved as what the distillation term is generally understood in the ML community. Rather, the proposed approach is a two-stage supervised training where the second stage is a finetuning process by adding the NAT representations as additional input representation for the AT decoder.

Thank you for your thoughtful comment regarding the use of the term "Knowledge Distillation" (KD). We categorized our work under the broader scope of KD because, in a general sense, we understand KD as leveraging a better-performing "teacher" model to guide a "student" model in improving task performance. In our case, this involves utilizing the stronger performance of the trained Non-Autoregressive Transformer (NAT) model in certain species to enhance the performance of the Autoregressive Transformer (AT) model (the "student").

We recognize, however, that our use of the term KD may deviate from the strict definition commonly accepted in the machine learning community. Traditionally, KD involves using the soft labels or logits produced by the teacher model as supervision for the student, rather than leveraging intermediate latent features. We acknowledge that our interpretation of KD might be viewed as an extended or unconventional usage. If this terminology is contentious, we are more than willing to adjust it in our manuscript to align with community standards. Thank you for pointing this out!!

In addition to addressing KD, our work introduces several novel ideas specifically targeting this task, to address your question on novelty:

• The Cross-Decoder Attention Module was designed to capitalize on prior findings that NAT and AT excel in different scenarios due to architectural inductive biases. Unlike approaches that combine methods arbitrarily (e.g., A+B), our design is deeply rooted in biological insights and leverages the complementary strengths of NAT and AT across different species and data distributions.

• Many aspects of our model, such as the encoding of spectra and peptides, shared encoders, and joint training, are informed by biological insights and tailored to improve sequencing performance. For example, joint training was implemented based on our observation that AT and NAT encourage distinct encoding features for input spectra, allowing their strengths to complement each other effectively.

We believe the significance of our work lies not only in the novel design elements but also in its practicality and usability. Specifically, our model delivers impactful results that directly benefit the research community by providing a tool that can be readily applied to real-world biological research (open sourced in provided link)

Application-specific studies: While combining the strengths of NAT and AT have the potential to impact various applications, only mass spectrum based de novo peptide sequencing is studied here which might only be of interest to a small subset of the ML community in this venue.

We thank the reviewer for raising concerns about the domain coverage of our work. To clarify, we would like to provide some context about the scope and significance of the de novo peptide sequencing task.

Peptide sequencing is currently one of the primary methods for obtaining protein sequences from biological samples. For example, when a blood sample is submitted to a hospital or research lab, protein sequences are almost always derived through mass spectrometry. This process involves analyzing the resulting spectra using algorithms like de novosequencing to extract peptide sequences. Such methods are widely used in numerous labs and medical institutions, playing a critical role in proteomics research.

De novo peptide sequencing serves as the starting point for downstream analyses, such as protein structure prediction (e.g., AlphaFold), making it a fundamental task in the field. Over the past 2–3 years, de novo sequencing algorithms have been featured in high-impact venues such as Nature Methods, Nature Machine Intelligence, Nature Communications, ICML, and AAAI, These publications underscore the competitive nature and importance of advancing performance in this area.

Our work builds on this well-established task, setting a new state-of-the-art performance compared to all prior methods, as verified against published baselines. Furthermore, we provide an open-source implementation of our code and model, making our tool accessible for a wide range of proteomics-related research applications.

We hope this additional information on our task can clarify the importance of this task and the domain we work on!

Thank you for the response and clarification. My main concern of lack of novelty and limited scope remains so I will keep my score.

It might be a result of the very short sequences that it works, I would like to see biological evaluations on longer sequences.

We apologize for not providing sufficient context regarding the characteristics of peptide sequencing tasks. To clarify, a peptide is a short protein sequence, and the reason peptides are sequenced instead of entire proteins lies in the mechanism of mass spectrometry. This process involves enzymatically breaking down proteins into charged fragments, effectively splitting long protein chains into very short pieces.

In peptide sequencing, the length of a peptide almost never exceeds 30, with only a small fraction reaching lengths of 35–40. Once the peptides are sequenced, they are assembled back into proteins using separate algorithms, which is a different task entirely. This means that peptide sequencing as a task does not involve handling long sequences.

Consequently, algorithm design for peptide sequencing focuses specifically on shorter sequences, with an average length of around 20. This approach has been standard across all existing algorithms for this task. Long-sequence modeling, on the other hand, is typically employed in different tasks such as protein modelling (using LLM) and de novo protein design. The biological nature of peptides inherently restricts the task to most sequences of length 10–20 in both training and testing datasets.

Although the sequences are short, peptide sequencing presents a unique challenge: every single token must be predicted correctly. Even a single incorrectly predicted amino acid can lead to errors in protein assembly. This stringent accuracy requirement makes the task highly challenging despite not involving long sequences. We hope this context provides additional clarity and helps the reviewer better appreciate the value of our work. Thank you for the opportunity to address this important point.

I'd expect evaluations across diverse biological benchmarks with longer sequences to highlight how deep learning extends to proteomics.

As mentioned in our response to the previous question, the entire field of peptide sequencing is inherently focused on peptides with lengths of up to approximately 40 amino acids. This limit is reflected in the datasets we have evaluated, which include peptides of varying lengths from 10 to 40 amino acids, with an average length of around 20.

Peptides longer than this threshold are rare because they tend to be enzymatically shortened during the mass spectrometry process. This is due to the inherent instability of longer peptide chains, which are fragmented into shorter, charged pieces to facilitate sequencing. We apologize for not clarifying this background information earlier and will ensure that these details are more explicitly explained in our revised manuscript. Thank you again for raising this question—it has helped us identify an opportunity to make our work more transparent and accessible.

I'd like to see a comparison with the size of models for previous work, I want to know that this is not just a result of a larger model capacity.

We thank the reviewer for raising the question regarding the model size. For this task, we have used a model with the exact same size as the baseline Transformer model. Specifically, our final Autoregressive Transformer (AT) network consists of 33 million parameters, as the Non-Autoregressive Transformer (NAT) decoder serves only as an auxiliary component. If the NAT decoder is included, the total model size amounts to 45 million parameters.

Our architecture comprises 9 Transformer layers for both decoders, with a hidden size of 400 dimensions. This is consistent with the baseline NAT model, PrimeNovo (33 million parameters and same number of layers and dimensions). In comparison, the previous best-performing AT models in our baseline—ContraNovo and Casanovo V2—are larger, with model sizes of 42 million parameters (512 dimensions, 9 layers, with additional contrastive module) and 38 million parameters, respectively. Notably, our AT model is smaller in size due to the reduced dimension, making it more efficient while maintaining state-of-the-art performance.

We also confirm that all training data used in our experiments are identical to those used in prior work. The full details of our model configurations, along with the codebase, are publicly available for further verification.

We hope this response sufficiently addresses your concerns and provides clarity regarding the novelty and engineering contributions of our work. Thank you again for your thoughtful review and feedback. We look forward to any additional questions or discussions. We sincerely hope that the additional information provided allows you to reassess our work and offer a revised evaluation on previous scores. Thank you once again for your time and consideration.

I am not familiar with proteins research. However, I have given my best analysis of the work from a methodical perspective.

We sincerely thank the reviewer for their time and effort in reading our manuscript and providing valuable feedback. We appreciate your comments regarding your background and your insights into protein-related tasks. To provide additional context (background) and highlight the significance of our work, we would like to briefly summarize the task of de novo peptide sequencing. Thank you in advance for your time in reviewing this explanation.

Peptide sequencing is currently the primary methods for obtaining protein sequences from biological samples. For example, when a blood sample is submitted to a hospital or research lab, the protein sequences are typically derived through mass spectrometry. This involves analyzing the resulting spectra using algorithms like de novo sequencing to extract the sequences.

Given its role as the starting point for downstream analyses, such as protein structure prediction (e.g., AlphaFold), the task of peptide sequencing is fundamental in proteomics research. Previous de novo sequencing algorithms have been published in high-impact venues such as Nature Methods, Nature Machine Intelligence, Nature Communications, ICML, and AAAI, showcasing significant advancements and highly competitive performance in this field.

Our work builds on same task problem, setting a new state-of-the-art performance benchmark compared to all prior work mentioned above (as verified against published baselines). Additionally, we provide an open-source implementation of our code and model weights (linked in paper), making our tool accessible for a wide range of proteomics-related research applications. We hope this context helps emphasize the importance of our contributions. Thank you once again for your thoughtful review.

Below, we address all reviewer’s concern point-by-point:

This reads like an engineering paper. It largely focuses on ensemble and engineering strategies without introducing a novel methodology. The use of a shared encoder and cross-decoder attention mechanism, while useful, does not represent a fundamental advance in model architecture for peptide sequencing.

We appreciate the reviewer’s question regarding the novelty and engineering aspects of our work, we’d like to further clarify this as follows:

• For protein-based biological research, usability and performance are paramount considerations, often surpassing other factors. Tools developed for this field are heavily utilized by biologists directly in lab analyses and protein discovery workflows. For instance, the baseline tool DeepNovo Nature Methods has seen extensive application in such tasks, since developed.

  Our work demonstrates a significant improvement of approximately 3% in whole-sequence-level accuracy, as evident from benchmark progression. This represents a substantial advance, pushing accuracy to a new heights. Such a performance gain is noteworthy and immediately applicable to biological research through our open-sourced software.

  It is also important to note that engineering in this domain—unlike traditional NLP or CV tasks—requires extensive trial and error due to the specific challenges of bioinformatics. We have dedicated significant effort to ensuring the system works robustly, which we believe is a valuable contribution in itself.

• other two points of this question continues in next thread, thanks for reading!