Modeling All-Atom Glycan Structures via Hierarchical Message Passing and Multi-Scale Pre-training

Thanks for your valuable comments and constructive suggestions! We respond to your questions as below:

Q1: The technical novelty of the GlycanAA model could be considered incremental.

We appreciate the perspective on the novelty of the GlycanAA model and would like to emphasize the unique contributions our work bring to the field of glycan modeling. Glycans, unlike proteins or small molecules, possess a distinct hierarchical structure characterized by a diverse array of monosaccharides and intricate branching patterns. The GlycanAA model innovatively captures this complexity by (1) integrating atomic-level and monosaccharide-level interactions within a single heterogeneous graph framework and (2) performing hierarchical message passing to seamlessly bridge these two levels of structural information. The effectiveness of these two contributions is further verified in our ablation study (Section 5.3 of the paper).

Q2: The necessity of the custom pre-training approach should be better justified.

This suggestion is great. Indeed, the pre-training methods proposed in [a], i.e., attribute masking and context prediction algorithms, can be adapted to pre-train glycan representations. Therefore, we employ them to pre-train the GlycanAA model and name the pre-trained models as GlycanAA-Attribute and GlycanAA-Context, respectively. According to the results in the Table 1 of revised paper (updated on OpenReview), both pre-trained models show performance decay compared to the GlycanAA model without pre-training. We suggest that these two pre-training methods actually lead to trivial tasks during pre-training, which mainly causes the negative results. Specifically, the attribute masking method does not consider the correlation between atom and monosaccharide nodes during masking, and thus leads to the trivial prediction of a masked monosaccharide based on some of its characteristic atoms; similarly, the context prediction method could select highly correlated center and anchor nodes in an all-atom glycan graph, leading to a trivial prediction task. By comparison, the proposed PreGlycanAA model performs multi-scale masking carefully to ensure as little correlation left in the unmasked nodes as possible, leading to clearly better performance than the GlycanAA without pre-training.

[a] Hu, Weihua, et al. "Strategies for pre-training graph neural networks." ICLR, 2020.

Dear Reviewer 2kz8,

We thank again for your contributions to the reviewing process.

The responses to your concerns and the corresponding paper revision have been posted. We kindly remind that the author-reviewer discussion period will end in two days. We look forward to your reply and welcome any further questions.

Best regards,

Authors of Paper 1249

Thanks for your valuable feedbacks! We respond to your questions as below:

Q1: The novelty of the pre-training method is limited.

We would like to clarify that the novelty of our pre-training method lies in the adaptation and customization specifically tailored for the hierarchical and complex nature of glycan structures. On one hand, we design the multi-scale masking strategy where the masking processes of monosaccharides and their corresponding atoms are interactive, such that the model is facilitated to understand both the local atomic-level structures and the global monosaccharide-level structures. On other hand, such a pre-training method is well tailored to enhance the representation power of the proposed hierarchical encoder GlycanAA. Through the multi-scale masking-recovery process, the GlycanAA model learns to effectively perform hierarchical message passing so as to capture from local atomic-level to global monosaccharide-level dependencies. This claim about the coupling of GlycanAA and the pre-training method is supported by the results in Table 1 of revised paper (updated on OpenReview), where the competitive baseline RGCN benefits less from the pre-training method.

Therefore, our pre-training method is carefully designed for the all-atom structures of glycans and also the proposed hierarchical encoding approach, showing its novelty both in application and customization.

Q2: On some tasks, GlycanAA does not outperform the best baseline, which should be explained.

Thanks for pointing this out. Compared with the best baseline method RGCN, the performance of GlycanAA is equal or lower on 4 out of 11 tasks, i.e., phylum prediction, family prediction, immunogenicity prediction and glycosylation type prediction. The performance difference is not significant on phylum, family and immunogenicity prediction based on the one tailed t-test ($\alpha = 0.025$), while the performance difference is significant on the glycosylation type prediction task. Compared to RGCN, GlycanAA contains more parameters, i.e.,13.3M parameters (GlycanAA) v.s. 4.2M parameters (RGCN). Therefore, on a relatively small dataset like the dataset of glycosylation type prediction with 1,356 training, 163 validation and 164 test samples, GlycanAA is easier to overfit the training set, leading to inferior test performance. Such an overfitting phenomenon can be mitigated by the proposed pre-training method, where, after pre-training, the PreGlycanAA model outperforms RGCN on glycosylation type prediction. In the Section 5.2 of revised paper (updated on OpenReview), we supplement this analysis.

Q3: Why is it considered unpromising to directly apply small molecule encoders or monosaccharide-level glycan encoders to all-atom glycan modeling?

For applying small molecule encoders to all-atom glycan modeling, the main obstacle is the gap of molecular scale. These encoders are originally designed to model a small molecule with tens of atoms, while a glycan commonly contains hundreds of atoms, which greatly challenges the encoders by requiring them to model a much larger system. As shown in our experiments (Table 1 of the paper), performant small molecule encoders (e.g., Graphormer and GraphGPS) do not perform well when applied to model the all-atom structures of glycans.

For applying monosaccharide-level glycan encoders to all-atom glycan modeling, the main challenge is also the adaptation to a much larger system. Specifically, the encoders are originally designed to model a coarse-grained glycan structure with tens of elements (i.e., monosaccharides), and it is hard to directly apply them to a fine-grained all-atom glycan structure with hundreds of elements (i.e., atoms). In our experiments (Table 1 of the paper), we verify this point by applying a performant monosaccharide-level glycan encoder RGCN to all-atom glycan modeling (i.e., All-Atom RGCN in the table), where obvious performance decay is observed after such an adaptation from monosaccharide level to all-atom level.

Dear Reviewer JUXn,

We thank again for your contributions to the reviewing process.

The responses to your concerns and the corresponding paper revision have been posted. We kindly remind that the author-reviewer discussion period will end in two days. We look forward to your reply and welcome any further questions.

Best regards,

Authors of Paper 1249

Based on your insightful comments, we have posted the responses and paper revision. Please check them out.

We welcome any further questions and discussions that affect your rating.

Appreciate for your insightful comments and golden suggestions! We respond to your questions as below:

Q1: The novelty of the proposed method is limited.

We acknowledge the observation regarding the established nature of techniques such as message passing, masking-recovery, and pretraining-finetuning. However, we would like to emphasize that the novelty of our work lies in the innovative application and adaptation of these techniques to the complex and underexplored domain of glycan modeling.

Glycans present unique challenges due to their hierarchical structure, comprising both atomic and monosaccharide levels, which are crucial for accurately capturing their biological functions and interactions. The proposed GlycanAA model leverages a hierarchical approach that specifically addresses these challenges by integrating local atomic-level and global monosaccharide-level interactions within a heterogeneous graph framework. This hierarchical approach enables us to capture the intricate covalent and glycosidic bonds that are essential for understanding glycan functionality, which is not achievable with existing models.

Furthermore, our self-supervised pretraining strategy is tailored to harness the rich and unlabeled glycan data. The proposed multi-scale masking and multi-scale recovery methods well fit the hierarchical architecture of GlycanAA and guide it to capture the intricate dependencies within glycans, further facilitating the generation of highly informative representations.

Thus, while the underlying techniques are established, their novel application to glycan modeling greatly advances the field, offering new insights and capabilities that were previously unattainable.

Q2: The color presentation in Table 1 should be improved.

Thanks for the good advice. As you suggested, in the Table 1 of revised paper (updated on OpenReview), we use bold values, underlined values and italic values to represent the best, runner-up and third-place results, respectively.

Q3: Efficiency experiments are missing.

Thanks for pointing this out. In the Section 5.4 of revised paper (updated on OpenReview), we supplement the efficiency comparison between GlycanAA and RGCN, where our proposed GlycanAA performs hierarchical message passing on the all-atom level, while RGCN is a typical method with only monosaccharide-level message passing. Specifically, we evaluate their training and inference speed in terms of throughput (i.e., the number of samples processed in one second) and their training and inference memory cost in terms of Mebibyte (MiB). The evaluation is performed on the dataset of glycan taxonomy prediction for its good coverage of different kinds of glycans (#training/validation/test samples: 11,010/1,280/919, average #monosaccharides per glycan: 6.39, minimum #monosaccharides per glycan: 2, maximum #monosaccharides per glycan: 43). All experiments are conducted on a machine with 32 CPU cores and 1 NVIDIA GeForce RTX 4090 GPU (24GB), and the batch size is set as 256 for both models.

According to the results in the Table 2 of revised paper (updated on OpenReview), it is observed that, in terms of both speed and memory cost, GlycanAA does not introduce too much extra cost compared to RGCN during both training and inference. Specifically, for training/inference speed, GlycanAA is about 22% slower than RGCN, and, for training/inference memory cost, GlycanAA consumes about 19% more memory than RGCN. Such a moderate extra cost brings the superior performance of GlycanAA over RGCN in terms of weighted mean rank on the GlycanML benchmark, illustrating the worth of the hierarchical modeling approach of GlycanAA.

Dear Reviewer JYmY,

We thank again for your contributions to the reviewing process.

The responses to your concerns and the corresponding paper revision have been posted. We kindly remind that the author-reviewer discussion period will end in two days. We look forward to your reply and welcome any further questions.

Best regards,

Authors of Paper 1249

Thank you for the response. I've raised my score to a 5.

However, I still feel that these results don't demonstrate the importance of the hierarchical structure. The difference between PreRGCN and PreGlycanAA is quite marginal (and the same for the non pretrained versions). To me this doesn't definitively show that all-atom glycan encoders are truly necessary.

Thanks for your insightful review! We respond to your questions as below:

Q1: Could you compare the runtime of GlycanAA and RGCN?

This advice is great. In the Section 5.4 of revised paper (updated on OpenReview), we supplement the efficiency comparison between GlycanAA and RGCN. Specifically, we evaluate their training and inference speed in terms of throughput (i.e., the number of samples processed in one second) and their training and inference memory cost in terms of Mebibyte (MiB). The evaluation is performed on the dataset of glycan taxonomy prediction for its good coverage of different kinds of glycans (#training/validation/test samples: 11,010/1,280/919, average #monosaccharides per glycan: 6.39, minimum #monosaccharides per glycan: 2, maximum #monosaccharides per glycan: 43). All experiments are conducted on a machine with 32 CPU cores and 1 NVIDIA GeForce RTX 4090 GPU (24GB), and the batch size is set as 256 for both models.

According to the results in the Table 2 of revised paper (updated on OpenReview), it is observed that, in terms of both speed and memory cost, GlycanAA does not introduce too much extra cost compared to RGCN during both training and inference. Specifically, for training/inference speed, GlycanAA is about 22% slower than RGCN, and, for training/inference memory cost, GlycanAA consumes about 19% more memory than RGCN. Such a moderate extra cost brings the superior performance of GlycanAA over RGCN on 7 out of 11 benchmark tasks and also on the weighted mean rank, illustrating the “worth” of modeling glycans on the all-atom level.

Q2: How about the performance of the RGCN with the pre-training strategy.

Thanks for this good suggestion. To study the necessity of all-atom glycan modeling more in depth, we additionally pre-train the competitive RGCN model using a similar mask prediction algorithm and evaluate the pre-trained RGCN, denoted as PreRGCN, on the GlycanML benchmark. According to the results in the Table 1 of revised paper (updated on OpenReview), in terms of the weighted mean rank, PreRGCN outperforms RGCN, showing the benefit of pre-training. However, PreRGCN is just comparable to GlycanAA and clearly inferior to PreGlycanAA, and its improvement over RGCN is smaller than the improvement of PreGlycanAA over GlycanAA. These results verify the value of modeling glycans on the all-atom level and also illustrate the importance of hierarchical structures to our pre-training method.

Appreciate for your valuable comments and golden suggestions! We respond to your questions as below:

Q1: What structural or relational insights the proposed model captures at the glycan level?

We provide such an analysis through Visualization experiments (Section 5.5). According to the visualization of glycan-level representations on downstream task datasets, we have following observations and insights: (1) The GlycanAA model with randomly initialized weights can, to some extent, separate the glycans with different properties (e.g., immunogenic and non-immunogenic glycans). This result illustrates the ability of GlycanAA on capturing the structural differences between the glycans with different properties, resulting in discriminative glycan-level representations. (2) The pre-trained PreGlycanAA model can better distinguish the glycans with different properties, where the samples with the same property are gathered closer, and the samples with different properties are separated more far apart. This result demonstrates the effectiveness of the proposed pre-training method, during which the model well learns to cluster glycans based on their structural relevance.

Q2: The experiments regarding efficiency should be provided.

Thanks for pointing this out. To evaluate the extra computational cost brought by the hierarchical message passing process of GlycanAA, we compare it with RGCN, a typical method with only monosaccharide-level message passing, on their computational efficiency. This study is supplemented to the Section 5.4 of revised paper (updated on OpenReview). Specifically, we evaluate their training and inference speed in terms of throughput (i.e., the number of samples processed in one second) and their training and inference memory cost in terms of Mebibyte (MiB). The evaluation is performed on the dataset of glycan taxonomy prediction for its good coverage of different kinds of glycans (#training/validation/test samples: 11,010/1,280/919, average #monosaccharides per glycan: 6.39, minimum #monosaccharides per glycan: 2, maximum #monosaccharides per glycan: 43). All experiments are conducted on a machine with 32 CPU cores and 1 NVIDIA GeForce RTX 4090 GPU (24GB), and the batch size is set as 256 for both models.

According to the results in the Table 2 of revised paper (updated on OpenReview), it is observed that, in terms of both speed and memory cost, GlycanAA does not introduce too much extra cost compared to RGCN during both training and inference. Specifically, for training/inference speed, GlycanAA is about 22% slower than RGCN, and, for training/inference memory cost, GlycanAA consumes about 19% more memory than RGCN. Such a moderate extra cost brings the superior performance of GlycanAA over RGCN in terms of weighted mean rank on the GlycanML benchmark, illustrating the worth of the hierarchical modeling approach of GlycanAA.

Q3: Is there any more recent and competitive baselines?

In the Table 1 of revised paper (updated on OpenReview), we additionally compare with one recent competitive small molecule encoder, i.e., Uni-Mol+ [a], and three recent competitive protein encoders, i.e., GearNet [b], GearNet-Edge [b] and VabsNet [c]. According to the experimental results, GearNet-Edge performs best among these four models, while it is still clearly inferior to the proposed GlycanAA model in terms of weighted mean rank, i.e., 11.44 (GearNet-Edge) v.s. 4.66 (GlycanAA). This result demonstrates that it is unpromising to directly apply small molecule encoders and protein encoders to glycan modeling, where the unique structural complexities of glycans (e.g., a diverse array of monosaccharides and intricate branching patterns) greatly challenge these models that are originally designed for other biomolecules.

[a] Lu, Shuqi, et al. "Data-driven quantum chemical property prediction leveraging 3D conformations with Uni-Mol+." Nature Communications, 2024.

[b] Zhang, Zuobai, et al. "Protein representation learning by geometric structure pretraining." ICLR, 2023.

[c] Zhuang, Wanru, et al. "Pre-Training Protein Bi-level Representation Through Span Mask Strategy On 3D Protein Chains." ICML, 2024.

Dear Reviewer 44rZ,

We thank again for your contributions to the reviewing process.

The responses to your concerns and the corresponding paper revision have been posted. We kindly remind that the author-reviewer discussion period will end in two days. We look forward to your reply and welcome any further questions.  

Best regards,

Authors of Paper 1249

Table A: Results of one tailed t-test on performance improvements (significance level $\alpha = 0.025$). “-” indicates our model does not surpass the best baseline on this task.

 

In Table A, we conduct one tailed t-test with significance level $\alpha = 0.025$ to verify the significance of the proposed models’ improvements over the best baseline method on all benchmark tasks, where both our model and the baseline run for 3 times on each task. It is observed that, on 7 out of 11 tasks, GlycanAA outperforms the best baseline and at the same time shows statistical significance (i.e., with t-statistic surpassing the critical value 2.78 of this test), and, on 9 out of 11 tasks, PreGlycanAA outperforms the best baseline and at the same time shows statistical significance (i.e., with t-statistic surpassing the critical value 2.78 of this test). These results illustrate the significance of the performance improvements achieved by our models.

We hope this response can address your concern, and we welcome any further questions.