MeToken: Uniform Micro-environment Token Boosts Post-Translational Modification Prediction

Dear Reviewer GBsm,

Thank you for your thoughtful feedback and constructive suggestions. We appreciate the opportunity to address your questions and comments, and we provide detailed responses below. Your insights have highlighted several areas where we can improve the clarity and depth of our manuscript, and we are committed to incorporating these changes in the revised version.

Q1 PTMs can be summarised into four types according to the modification types: chemical groups (e.g. Phosphorylation, Acetylation), proteins/peptides (e.g. Ubiquitylation, Sumoylation), complex molecules (e.g. glycosylation), and cleavage (proteolysis). These four types could have different structural preferences. How can the proposed model capture the differences between them? Which types work well using the proposed method?

A1 This is an insightful question, and we appreciate the opportunity to clarify how our model captures these differences. MeToken’s micro-environment tokenization approach is designed to capture both the local sequence motifs and spatial context around each residue, which should theoretically help distinguish PTMs based on their structural preferences:

• Chemical group modifications (like phosphorylation) often depend on specific sequence motifs and can occur on exposed residues, which are easier for the model to identify through sequence context and local structural features.
• Protein/peptide modifications (e.g., ubiquitylation) typically require a spatially accessible lysine residue, along with structural features that might allow for binding of modifying enzymes. MeToken’s structural context encoding is particularly beneficial here.
• Complex molecules (e.g., glycosylation) generally have specific sequence motifs and often require certain three-dimensional accessibility, which the model can learn through spatial tokenization.
• Cleavage events (like proteolysis) may have broader sequence motifs but are often associated with regions of the protein that are structurally exposed or flexible, properties that are encoded in the model through local structural information. Since our model is built on the consistency of the lengths of structural data, sequence data, and PTM modification data, we do not consider this type in our dataset.

MeToken performs best on chemical group modifications, achieving the highest F1 score. These modifications often have strong sequence motifs and occur on solvent-accessible residues, making them easier for the model to identify. The model achieves moderate performance for complex molecule modificationsm due to the complex and varied structural contexts in which glycosylation occurs.

Q2 The description "Existing computational approaches predominantly focus on protein sequences to predict PTM sites" in the Abstract is not accurate. There are many graphs and structural-aware methods that have been developed and published.

A2 Thank you for this important observation. If you are aware of additional studies that would further strengthen our discussion, we would greatly appreciate your recommendations. We are eager to learn about and incorporate any relevant advancements that may enhance the context and positioning of our work.

Dear Reviewer GBsm,
Your response is very meaningful and has helped make our work more convincing.
Given that the discussion deadline is approaching, we hope that our additional experiments and explanations have addressed your concerns and improved our work. If you are satisfied with our additions, we sincerely hope you will consider increasing our score. If you believe there are still issues with our additions, we kindly ask you to engage in further discussion with us.
We once again sincerely thank you for your review work, which is of great significance.

Our dataset is therefore uniquely positioned to serve as a foundation for PTM prediction models that incorporate both sequence and structure, with a focus on addressing the long-tail distribution of PTM types.

In our baseline comparisons, we evaluated MeToken against several state-of-the-art PTM prediction models. Musite-Deep, DeepTL-Ubi, and DeepGpgs are all binary classification models specifically designed to predict a single type of PTM—phosphorylation, ubiquitylation, and glycosylation, respectively. On the other hand, models like EMBER and DeepPhos provide more flexibility, as they can predict multiple kinase-specific phosphorylation types. We re-implement these baselines in our multiclass prediction task, and the results are reported in our manuscript.

Q4 How is the imbalance in the samples for no modification class affects the classification? Particularly the merged small PTMs and no modifications.

A4 Thank you for raising this important question about how the imbalance in "no modification" samples and rare PTM classes affects classification performance. We conducted additional experiments to analyze the impact of merging rare PTM classes and including "no modification" sites on model performance. Below, we present a comparison among three configurations: (1) the current MeToken model (which merges rare PTM classes and excludes "no modification" sites), (2) a model that does not merge rare PTM classes, and (3) a model that considers "no modification" sites in addition to PTM classes. The results are summarized below:

In our dataset, there are 106,162,623 "no modification" sites compared to 1,263,935 PTM sites, meaning "no modification" accounts for over 98.8% of the data. This extreme imbalance severely impacts model performance.

Your initial review and encouraging feedback were impressive, and we sincerely appreciate your thoughtful and thorough evaluation of our work. We respectfully inquire if there might be an opportunity for you to consider further raising the score? Your kindness and fairness in assessing our manuscript have been invaluable to us.

Dear Reviewer HxpX,

Thank you for your feedback and constructive suggestions. We appreciate the opportunity to clarify and expand on several aspects of our work. Below, we address each of your points in detail and outline the changes we plan to make in the revised manuscript:

Q1 It is not clear how the sequence similarity or redundancy was mitigated in the train-test-validation sets. Please report how minimal redundancy was ensured among test-train and validation sets.

A1 As shown in Figure 7 (Appendix C), we applied MMseqs2 with a 40% sequence identity threshold, meaning that sequences sharing more than 40% similarity were assigned to the same cluster. After clustering, we assigned entire clusters to either the training, validation, or test set, ensuring that no sequence in one split had more than 40% similarity to any sequence in another split.

Q2 In the rare modification class, how good is the predictor to discriminate the classes is not made clear. In this case authors may consider to report the different metrics for the PTM classes merged.

A2 Thank you for your questions. We recognize the importance of clearly demonstrating the model’s ability to handle rare PTM classes and appreciate the opportunity to address this. Rare PTM types pose a significant challenge due to their extreme data sparsity, which makes it difficult for the model to learn meaningful, class-specific features. In the original implementation, we merged all rare PTM types (those with fewer than 100 samples) into a single "rare classes" category to mitigate issues caused by insufficient data per individual class.

In response to your suggestion, we conducted additional experiments where we retained individual PTM types as separate classes rather than merging rare PTMs. The results are summarized below:

When rare PTMs are treated as distinct classes, the total number of classes increases from 26 to 73, introducing significant class imbalance and making it harder for the model to correctly classify rare PTMs.

Q3 There are datasets and methods which are specifically learned for particular type of PTM and that has been a trend in the literature. Please report how does those methods perform comapred to MeToken.

A3 We summarize the characteristics of our dataset and existing PTM datasets in Table 6 (Appendix C) of the manuscript. Our dataset is one of the most comprehensive resources available for PTM prediction, containing 1,263,935 PTM sites across 187,812 proteins, spanning 72 PTM types. This not only covers a broader range of modification types than most existing datasets but also integrates both sequence and structural information, enabling models to capture both sequence motifs and spatial context. For comparison:

• PTMint contains 2,477 PTM sites from 1,169 proteins and covers only six PTM types, which are subsets of those included in our dataset. While it includes structural information, the dataset size limits its applicability for generalized PTM prediction.
• qPTM provides 660,030 PTM sites across 40,728 proteins, also limited to six PTM types, but lacks structural annotations, restricting its use for models incorporating 3D context.
• PRISMOID and PTM Structure datasets are small-scale datasets containing structural annotations but cover fewer modification types (37 and 21, respectively) and a limited number of PTM sites and proteins.
• dbPTM is a very large-scale sequence-only dataset (2,235,000 PTM sites, 223,678 proteins) spanning 72 PTM types. However, it lacks structural annotations, which are crucial for capturing the 3D spatial context of PTMs.

Dear Reviewer HxpX,
Your response is very meaningful, as it has helped us further clarify certain points in the paper. The discussion deadline is approaching, and we kindly inquire whether our explanation has effectively clarified our paper. If our explanation satisfies you, we would appreciate it if you could consider increasing our score. If not, we welcome further discussion with you.
Thanks for your reviewing work again.

Dear Reviewer HxpX,

Thanks again for your recognition of our work. Since the deadline for the discussion period draws near, we respectfully inquire whether our response has addressed your concerns. If our response echoes your concerns and issues, we would be grateful if you could consider raising our work's scores?

We sincerely appreciate your contribution and work as a reviewer again.

Sincerely,

The Authors

Dear fwnN,

Thank you for your detailed and insightful feedback. We appreciate your recognition of the strengths of our approach, and your constructive suggestions highlight important areas for further improvement. Below, we address each of your comments and questions:

Q1 The reliance on AlphaFold-generated data may limit its generalizability in real-world applications where experimental structures are less available or differ significantly.

A1 Thank you for raising this important concern about the reliance on AlphaFold-generated data. As shown in Appendix C, we prioritized experimental data from the PDB whenever it was available. Specifically, we used experimental structures as the primary source of structural information to ensure the highest level of accuracy and biological relevance in our dataset.

We acknowledge, however, that AlphaFold predictions are generated under certain assumptions and are not always perfect representations of in vivo protein structures, especially for highly dynamic regions or protein-protein interfaces. To assess the impact of using AF2 data, we conducted additional evaluations (Appendix F.3) to compare the predictive performance on subsets of the dataset containing only experimental structures versus those with AlphaFold2-predicted structures. Our analysis showed that the inclusion of AF2 predictions substantially improved the size and diversity of the dataset, enabling more robust training

Q2 Although the authors address the dimensionality reduction challenge, computational complexity remains an issue, especially given the additional steps in codebook learning and temperature-scaled quantization. Further exploration of optimization methods would enhance applicability for high-throughput or large-scale data analysis.

A2 Thank you for raising this important point. We will include a discussion of these limitations, along with the potential optimization strategies mentioned, in the revised manuscript. Additionally, we are exploring the possibility of open-sourcing a lightweight variant of MeToken that trades off some accuracy for faster runtime, making it suitable for high-throughput or resource-constrained scenarios.

Q3 The study focuses primarily on PTM types without including interactions with enzymes or other proteins, which are crucial for PTM formation and regulation in vivo. Extending the approach to consider these interactions could improve the model’s utility in biological applications. Could you consider ways to include these relationships in MeToken’s predictions?

A3 This is an excellent suggestion, and we appreciate the opportunity to expand on this important aspect. You are correct that PTM formation and regulation are often mediated by enzymes (e.g., kinases, glycosyltransferases, proteases) or occur in the context of protein-protein interactions. Incorporating these interactions into MeToken could significantly enhance its biological utility, as the structural micro-environment alone does not fully capture the complex interplay of factors influencing PTMs in vivo.

One potential way to address this is by integrating enzyme recognition motifs or interaction features directly into MeToken’s inputs, such as:

• We could add known enzyme-specific sequence motifs (e.g., kinase binding motifs for phosphorylation) as auxiliary features during training.
• Including protein-protein interaction network data could provide additional context, as PTMs often occur at protein interfaces or are influenced by specific interaction partners.
• Utilizing reaction-specific enzyme pockets generation approaches for enzymes and substrates could augment the micro-environment representation.

Q4 Have you explored options like pruning or adaptive quantization to mitigate these costs?

A4 Thank you for this insightful suggestion. Pruning and adaptive quantization are promising strategies to reduce the computational costs associated with codebook learning and tokenization. While these methods were not explored in the current study, they could significantly improve MeToken’s efficiency without substantial loss of accuracy. We appreciate your thoughtful questions and suggestions, which will help us significantly improve the clarity and scope of the manuscript.

We express our sincere gratitude for your constructive feedback in the initial review. It is our hope that our responses adequately address your concerns. Your expert insights are invaluable to us in our pursuit of elevating the quality of our work. If you find our efforts satisfactory, we kindly request that you consider revisiting and potentially raising the score for our submission. Thank you once again for your time and valuable input.

Dear Reviewer fwnN,
Your response is helpful and be appreciated for us, making us refine and clarify some points in our work.
As the time passing by, the deadline of discussion period is coming soon. If our response addresses your concerns, we hope you will consider increasing the score of our work. We appreciate your time for this.
We would like to express our gratitude once again for your reviewing work.

Dear Reviewer fwnN,

Thank you once again for your review. As the discussion period deadline approaches, we are eager to hear your opinions and suggestions regarding our response. If our reply addresses your concerns, we would greatly appreciate it if you would consider raising the score of our work?

Your recognition and feedback are extremely valuable to us.

Sincerely,

The Authors

Dear sns6,

Thank you for your detailed and constructive feedback on our work. We appreciate your insights into both the strengths and weaknesses of our approach, and we address each of your points below:

Q1 It's beneficial to investigate deeper (and perhaps more ablation studies on only this arm of contribution) to stress the importance of added benefit of structural information.

A1 Thank you for raising this important point. To better illustrate the contribution of structural information in our model, we performed additional ablation experiments where we removed the structural information from MeToken and evaluated the performance. The results are summarized below:

These results clearly demonstrate the significant added benefit of incorporating structural information into MeToken.

Q2 Additionally, ESM-2 baseline seems to be very close the proposed method and beats all other baselines so there's still arguably a lot of sequence information that's useful already, so how to showcase their method is not overfitting to structural features might help reviewers to understand the validity of their method.

A2 Thank you for this thoughtful observation. We agree that ESM-2, as a powerful sequence-based model, does indeed provide valuable insights based on sequence information alone. However, it is important to highlight several key distinctions between our method and ESM-2. ESM-2-650M, which we used as a baseline, contains over 650 million parameters and has been pretrained on 65 million protein sequences. In contrast, our model is lightweight, with fewer than 10 million parameters, and it is trained on a dataset of approximately 140k data. Despite these differences, our method achieves competitive performance with ESM-2 and even outperforms it on certain structure-dependent tasks, demonstrating that structural information adds meaningful context to PTM prediction.

To showcase the added benefit of incorporating structural information, we examined the tokens within our learned codebook, particularly focusing on those that represent amino acid micro-environments. As illustrated in Figure 20 (in Appendix F.4) of our manuscript, the most prevalent amino acids associated with a specific token are valine (Val, V) and leucine (Leu, L), both of which are hydrophobic and share similar structural properties. The distribution also includes three other hydrophobic amino acids, reinforcing this trend. This pattern suggests that our model has successfully learned not only the sequence-specific characteristics of these amino acids but also their associated structural properties, such as their tendency to cluster in hydrophobic regions of proteins.

By explicitly capturing these structural and physicochemical features in the form of learned tokens, our method introduces a level of structural awareness that goes beyond simple sequence-based representations. The structure of the token space reflects the functional and structural context of the amino acid environments, which provides a richer understanding of the micro-environment around PTM sites.

Q3 Uniform sub-codebook strategy and temperature-scaled vector quantization, seems to offer marginal benefit which IMO dilutes the core contribution brought by the Micro-env arm of contribution. In addition, the temperature adjusted approach might necessitate a huge effort for hyperparameter tunning to make it work. But the authors didn't discuss any ramification of it. Further, the uniform sub-codebook strategy might risk embedding redundancy for common PTM types, and this was not discussed either.

A3 Thank you for raising these important points. We appreciate your careful consideration of the uniform sub-codebook strategy and temperature-scaled vector quantization. We understand that these techniques may seem to offer only marginal benefits compared to the core contribution of the micro-environment tokenization. However, we believe that they play an important role in improving the overall effectiveness of the model and in refining the functional token space for PTM prediction.

The uniform sub-codebook strategy was designed to address the challenges posed by the long-tail distribution of PTM classes. By dividing the space into smaller, more manageable sub-codebooks, we were able to prevent the model from overfitting to highly frequent classes while still allowing it to learn generalizable features. While we acknowledge that this strategy could potentially introduce redundancy for common PTM types, it is important to note that this redundancy is mitigated by the way the sub-codebooks are structured. Each sub-codebook is tailored to capture specific structural or functional features relevant to the PTM classes it represents. The redundancy issue you raised is something we will explore further in future work to ensure the sub-codebook strategy delivers its intended benefits without unnecessary overlap. We have added this discussion in the limitation section of our revised manuscript.

We are sorry that the detailed temperature-adjusting implementation is missing. In our implementation, the initial temperature is set to 1 and then linearly decays to 0.0001 over the epochs of training.

We fully recognize that these strategies may appear as incremental or potentially adding complexity, and we appreciate your feedback on this. In future work, we will investigate additional methods to refine these techniques and better assess their true impact on model performance. We really appreciate your insights into their potential limitations.

Q4 They highlighted the long tail nature of the current datasets in the domain but it lacks some deeper analysis for the distribution e.g. rarity analysis, frequency statistics. It's unclear the extent to which the long tail nature is currently at play for the modeling difficulty in the domain, as what they claimed as a challenge.

A4 Thank you for this insightful question regarding the long-tail distribution of PTM classes and its impact on modeling difficulty.

As shown in the top half of Figure 8 (Appendix C) of the manuscript, the dataset exhibits a pronounced long-tail distribution. The most frequent PTM class, Phosphorylation, has a scale of over $10^6$ labeled sites, making it by far the dominant class in the dataset. In contrast, the rarest classes, such as Carboxyethylation, D-glucuronoylation, N-carbamoylation, Octanoylation, Pyrrolylation, and S-carbamoylation, have fewer than $10^1$ labeled instances each. These classes represent extreme sparsity, with fewer than 10 samples per modification type, making it almost impossible for a model to learn reliable representations for these rare classes without additional interventions.

To provide more clarity, we summarize the distribution of the top five most frequent PTM types in the dataset below (total=1,263,935 sites):

• Phosphorylation constitutes 76.74% of the dataset.
• Classes like Acetylation and Ubiquitination follow with significant numbers, but they are still orders of magnitude smaller than Phosphorylation.
• Rare PTMs, as noted above, all of them collectively contribute less than 0.08% of the dataset.

Dear Reviewer sns6,
We appreciate your response, which has helped us further improve our work. Your time and contribution means a lot for us.
At the same time, if our explanation has addressed your concerns, we kindly hope that you would consider increasing the score or confidence of our work. If you still have any questions regarding our work, please feel free to contact us, and we will respond as soon as possible.
Thanks again for your assistance here.

I thank authors for addressing my points. But my main concerns still stands (1) core contribution is diluted, which should've been a heavy study on WHY the structural data works better, as this finding can drive more interesting research, rather than merely cranking up the prediction performance by incorporating the added contribution (the latter two) (2) the smaller model size is another interesting angle but it feels weak to say 'our model is much smaller and at the same time offers few points bump'. In the foundation model era, scaling up the model size to have better generalizability is more important (unless you are in some edge computing domains). Plus the model parameters wasn't a driving factor for contribution in the paper to begin with so mentioning it wouldn't help reviewers to appreciate the core technical novelty. Appendix F.4 offers a tiny snapshot, which covers a limited number of amino acids, thus less convincing

Dear Reviewer nhjP,

Thank you for your thoughtful and detailed feedback. We appreciate your positive assessment of the strengths of our work, as well as your suggestions for areas of improvement. We address each of your comments in detail below:

Q1 A simple baseline of one-hot encoded amino acid sequence representations would be useful to see if certain classes can easily be predicted based on their modification site.

A1 This is an excellent suggestion. We agree that including a one-hot encoded baseline for amino acid sequences would provide a valuable comparison, helping us to understand the baseline predictability of PTM classes based purely on sequence information without complex embeddings. We have now added a one-hot encoding baseline.

For the one-hot encoding baseline, we implemented a simple model with:

• An embedding layer that converts the one-hot encoded amino acid sequence into dense vectors,
• Three fully connected linear layers (hidden dimension=128) with ReLU activation functions in between, and
• A final classification layer to predict the PTM type.

This baseline model represents a straightforward approach to PTM prediction based solely on primary sequence data. Below, we summarize the results of this one-hot baseline in comparison to MeToken, our proposed model.

The results indicate that while the one-hot baseline achieves reasonable accuracy and AUROC scores, it performs significantly worse in terms of precision, recall, and F1 scores. This suggests that one-hot encoding struggles to capture the complex patterns required for effective PTM prediction, particularly for rare or structurally influenced PTM types.

To further explore this baseline’s limitations, we evaluated its performance on specific PTM types. Below are the results for phosphorylation, ubiquitylation, and acetylation, which are among the most biologically relevant PTM types.

These results highlight some important observations:

• Phosphorylation: The one-hot baseline achieves relatively high accuracy because phosphorylation sites often have clear sequence motifs (e.g., serine/threonine/tyrosine residues with flanking sequence preferences). However, its precision and recall are low, reflecting difficulty in distinguishing phosphorylation sites from non-modified residues with similar motifs.
• Ubiquitylation: The performance on ubiquitylation is poor. This aligns with the biological reality that ubiquitylation does not have clear sequence motifs and is strongly influenced by structural factors, such as residue accessibility and enzymatic binding.
• Acetylation: Similar to phosphorylation, acetylation achieves decent accuracy but suffers from poor precision and recall, as sequence alone cannot fully capture the modification’s context.

Thank you again for this valuable suggestion, which has helped us better contextualize and strengthen the evaluation of our method.

Q2 The authors could compare their work to other structure-aware representations. For example, SaProt using the Foldseek structural tokens learns a structure-aware embedding.

A2 Thank you for suggesting additional comparisons with structure-aware models such as SaProt. We're now conducting experiments comparing MeToken with SaProt’s structure-aware embeddings. The results will be posted here once the experiment done.

Q3 The authors demonstrate one of the codebooks as an example but a more extensive analysis on this would be interesting. For example a very basic question would be: "Are the amino acid preferences (K for Sumoylation, S/T/Y for Phosphorylation, etc) correctly captured by the codebooks?"

A3 Thank you for this insightful suggestion. We have expanded our analysis of the codebooks to investigate whether they capture the well-documented amino acid preferences associated with specific PTM types. This analysis highlights how the tokenization process aligns with biological knowledge and how the model handles shared residues across different PTM types.

Codebook and Amino Acid Preferences Analysis

To analyze whether the codebook effectively captures these preferences, we examined the relationships between amino acids, PTM types, and their token indices. Below, we show the Top 5 groupings based on frequency:

From this table, we observe that the model effectively captures the known amino acid preferences for specific PTMs:

• Phosphorylation predominantly occurs on serine, threonine, and tyrosine residues, which are consistently associated with specific token indices.
• Acetylation primarily occurs on lysine residues, which are also captured and assigned to specific tokens.

This demonstrates that the tokenization process aligns well with established biochemical knowledge of PTM site preferences.

Multiple Token Assignments for the Same PTM and Residue

While the tokenization process captures the general preferences, we also observe that the same modification on the same amino acid can sometimes be mapped to multiple token indices. For example, acetylation on lysine is assigned to five distinct token indices, as shown below:

This result reflects the biological complexity of acetylation on lysine, which can occur in diverse micro-environments. Different structural and sequence contexts may result in different token assignments, indicating that the model captures subtleties in the local environments surrounding the modified lysine residues.

Case Study: Sumoylation on Lysine

Similarly, we analyzed sumoylation on lysine residues. Sumoylation, like acetylation, also occurs on lysine but involves distinct enzymatic processes. The token assignments for sumoylation on lysine are shown below:

Interestingly, some sumoylation sites and acetylation sites are mapped to the same token indices (e.g., Token 1332 and Token 1403). This suggests that the local micro-environments of these lysine residues share structural or sequence similarities. However, as we addressed in Q5, despite these overlaps in tokenization, the downstream predictor is still able to correctly distinguish between these two PTM types, leveraging additional features and learned relationships.

Q4 Information on the hyperparameters used to train MeToken and the baseline models are needed for reproducibility of the results.

A4 Thank you for raising this important point. We will include a comprehensive table detailing the specific hyperparameters used for training both MeToken and the baseline models. To summarize briefly, we trained MeToken using the AdamW optimizer with a learning rate of $0.0001$ for a total of 20 epochs. The batch size was fixed at 32. For the codebook, we used a size of $128 \times 26 = 3328$, meaning that each PTM type is assigned a sub-codebook with 128 discrete codes. The initial temperature for the vector quantization process was set to $\tau_v = 1$, and it decayed linearly to $\tau_v = 0.0001$ over epochs.

Q5 Distinguishing some PTMs from others is easier because of the distinct amino acid preferences in their modification sites. Hard evaluation setups could have been designed to understand the weakness and strength of the methodology. For example, does the model correctly differentiate a lysine residue being ubiquitinated or sumoylated?

A5 Thank you for pointing this out.

In this experiment, we visualized the final embeddings before prediction (referred to as "prediction embeddings") for lysine residues marked as either ubiquitinated or sumoylated. These embeddings were then analyzed using a SVM classifier to determine how separable these two PTM types are in the learned embedding space. The results showed that the model has a strong ability to distinguish between the two modifications, achieving an accuracy of 0.9881 and an F1-score of 0.99. These metrics indicate a clear separation in the learned embedding space, suggesting that the model has captured the relevant features needed to differentiate ubiquitination from sumoylation. A visualization of this separation is provided in the revised manuscript (Figure 22 in Appendix I), offering further clarity.

A similar analysis was conducted for O-glycosylation and phosphorylation, which occur on serine or threonine residues. Despite these modifications sharing similar sequence contexts, the model was again able to distinguish them effectively. The SVM trained on their prediction embeddings achieved an accuracy of 0.9934 and an F1-score of 0.99, further demonstrating the model's capacity to capture nuanced differences in the local micro-environments of residues. These results highlight the ability of the model to differentiate between closely related modification types, even when they occur on the same amino acid residues or share overlapping structural and sequence contexts. This indicates that the combination of sequence, structural features, and the tokenization process enables the model to learn distinct, generalizable representations.

Q6 The authors do not provide results on performance across different PTM classes. Are certain PTMs much easier to predict because of the available training data?

A6 Thank you for pointing this out. We agree that analyzing the performance of our model across different PTM classes is critical to understanding its strengths and weaknesses. To address this, we conducted additional experiments and evaluated the performance of our model across three major subgroups of PTMs: chemical groups (e.g., phosphorylation, acetylation), proteins/peptides (e.g., ubiquitylation, sumoylation), and complex molecules (e.g., glycosylation).

The results are summarized in the table below:

Certain PTM types are indeed easier to predict due to the availability of training data and clearer sequence motifs. For example, chemical group modifications like phosphorylation benefit from abundant data and well-characterized motifs, leading to higher predictive performance. PTMs with weaker sequence motifs or complex structural dependencies (e.g., protein/peptide and complex molecule modifications) are more challenging for the model. This highlights the need for additional structural context or specialized approaches to improve performance on these subgroups. Structural dependency plays a significant role in model performance. For instance, modifications that depend on accessibility or specific structural motifs (e.g., glycosylation) exhibit lower recall.

These findings and analyses will be added to the revised manuscript to clarify the model’s performance across different PTM categories and guide future work to address its limitations. Thank you again for this valuable suggestion!

Q7 How are the no-modification sites determined? Are these all sites in protein sequences where no modifications are reported or are they subsets based on amino acid types?

A7 The no-modification sites are determined based on residues for which no modifications have been reported in the dbPTM [1] database. This database contains experimentally verified data, with the majority of the reported modification sites being confirmed through MS-based proteomics techniques. Importantly, the no-modification sites in our dataset correspond specifically to those amino acid positions in protein sequences where no modification is listed, rather than being based on particular amino acid types. Therefore, these sites represent regions where modifications have not been experimentally observed in the database, providing a reliable set of non-modified residues for model training and evaluation.

[1] Li, Zhongyan, Shangfu Li, Mengqi Luo, Jhih-Hua Jhong, Wenshuo Li, Lantian Yao, Yuxuan Pang et al. "dbPTM in 2022: an updated database for exploring regulatory networks and functional associations of protein post-translational modifications." Nucleic acids research 50, no. D1 (2022): D471-D479.

Q8 Is the quantization technique and the codebook strategy different than the cited work FoldToken? The differences and similarities should be included in the relevant literature part as it is very closely related to the work introduced here.

A8 Thank you for this insightful question. We appreciate the opportunity to clarify the differences and similarities between our approach and FoldToken.

The main contribution of FoldToken [2] is its SoftCVQ (Soft Conditional Vector Quantization), which aligns the embedding induced by the binary code index with a smooth gradient using an adjustable temperature parameter. Our approach shares this similarity in that we also utilize an adjustable temperature to ensure smooth gradient optimization during training. However, there are key differences in the context and application of these techniques.

While FoldToken is designed for protein backbone generation and requires a highly expressive latent space to effectively reconstruct detailed 3D structures, our work focuses on protein PTM prediction, aiming to learn a compact latent space that captures relevant sequence-structure relationships specific to PTM sites. To achieve this, we introduce a uniform sub-codebook strategy, which helps to more effectively model PTM types by partitioning the latent space into smaller, specialized sub-codebooks for each PTM subtype. This is particularly useful in the context of PTM prediction, where we aim to capture the distinct modification patterns and functional relationships between amino acids at modification sites.

Furthermore, while FoldToken uses a large-scale binary codebook (codebook size = 65536) to allow for detailed protein structure reconstruction, we focus on designing a smaller (codebook size = 26$\times$128=3328), task-specific latent space tailored to PTM representation learning. The sub-codebook strategy in our model ensures that the embeddings focus on the most relevant features for PTM prediction, improving model efficiency and interpretability. Additionally, merely relying on smooth gradient optimization through adjustable temperature is not sufficient for effective PTM prediction, which is why we also incorporate the sub-codebook approach to explicitly guide the model toward the most important features for PTM classification.

We will update the relevant literature section to clearly discuss these similarities and differences, situating our approach within the context of FoldToken and other relevant works in the field, and highlighting how our use of sub-codebooks and compact latent spaces offers a distinct advantage for PTM prediction tasks. We hope this clarifies the differences between our method and FoldToken and we look forward to making these updates in the Appendix K of our manuscript. Thank you again for your valuable suggestion.

[2] Gao, Zhangyang, Cheng Tan, Jue Wang, Yufei Huang, Lirong Wu, and Stan Z. Li. "Foldtoken: Learning protein language via vector quantization and beyond." arXiv preprint arXiv:2403.09673 (2024).

Dear Reviewer nhjP,

We sincerely appreciate and highly regard your feedback, and we thank your kindness here again. Your contribution works a lot in refining our work.

As the author-reviewer discussion period passes by, we are eager to hear your feedback. Here, we kindly ask if you would consider increasing the score or confidence of our work if we successfully address your concerns. We would like to express our gratitude in advance.

Thanks again here, and your help assists us in improving our work.

Sincerely,

The Authors