Modeling All-Atom Glycan Structures via Hierarchical Message Passing and Multi-Scale Pre-training

Thanks for your valuable comments! We respond to your concerns as below:

Q1: Why is the pre-training method less helpful to RGCN than to GlycanAA?

We deem that the less benefit of pre-training to RGCN mainly owes to its lower model capacity. Compared to GlycanAA that models both monosaccharide-level and atomic-level glycan structures, RGCN only models monosaccharide-level structures and is thus with lower capacity. As shown in the figure of this URL, the lower capacity of RGCN leads to its inferior pre-training performance (i.e., lower accuracy and higher cross entropy loss on masked monosaccharide prediction) against GlycanAA. These shortages of RGCN (i.e., lower model capacity and inferior pre-training performance) makes it benefit less on downstream tasks after pre-training, which is consistent with previous findings in other domains [a,b,c].

 

[a] A simple framework for contrastive learning of visual representations. ICML, 2020.

[b] Bert: Pre-training of deep bidirectional transformers for language understanding. NAACL, 2019.

[c] Strategies for pre-training graph neural networks. ICLR, 2020.

Q2: What is truly new beyond applying existing GNN methods to the glycan domain?

We argue that the proposed methods are clearly motivated and carefully designed to handle the complexity of modeling atomic-level glycan structures, instead of simply applying existing techniques to the glycan domain.

(1) For the model part, our model design is inspired by the fact that the backbone structure of a glycan mainly determines its biological properties, and the atomic-level structures of individual monosaccharides provide auxiliary information. Following this principle, we perform three steps of message passing to progressively enhance global backbone structural features with local atomic structural features, and the enhanced backbone features are finally readout for glycan representation.

(2) For the algorithm part, our pre-training algorithm is motivated by the fact that understanding the interactions between monosaccharides and their corresponding atoms is important to effective hierarchical learning, while only performing supervised learning on downstream tasks cannot guarantee acquiring such interactions. Therefore, in our pre-training algorithm, we facilitate the model to learn the interactions between monosaccharides and atoms. Specifically, we first perform an interactive masking process where each selected monosaccharide is masked along with its corresponding atoms, and, on such a masked glycan, the model learns to recover masked monosaccharides with the hints brought by the recovery of some of their atoms.

Q3: How the extra cost of GlycanAA affects its usage on large-scale glycan datasets?

Table A: Efficiency and effectiveness comparison between RGCN and GlycanAA on GlycanDomain-60K.

 

This question is great. To study the usability of GlycanAA on large-scale glycan datasets, we construct the GlycanDomain-60K dataset. We first collect all existing glycans deposited in the GlyTouCan database whose structures are complete (GlyTouCan is a regularly updated glycan database containing all discovered glycans), summing up to 60,152 samples. We then annotate each of them with a domain label (Eukarya, Virus, Bacteria or Archaea) based on their nearest neighbor in the domain classification dataset of the GlycanML benchmark, where the nearest neighbor is determined by a motif matching algorithm depicted in [d]. We name this large-scale glycan dataset with domain annotations as GlycanDomain-60K.

On this dataset, we compare the efficiency and effectiveness of RGCN (the most competitive monosaccharide-level baseline) and GlycanAA (our all-atom-level encoder). In specific, we respectively use the glycan domain classifier with RGCN and GlycanAA backbones to predict the domain labels of all glycans in GlycanDomain-60K, where both models are trained on the domain classification task of GlycanML. In Table A, we report the total processing time and the Macro-F1 score of predictions for these two models, where the test is done on a machine with 48 CPU cores and 1 NVIDIA GeForce RTX 4090 GPU under the batch size 256. According to the results, GlycanAA achieves a 50% higher Macro-F1 score than RGCN with less than 1 minute more processing time. Therefore, GlycanAA is applicable in processing large-scale glycan datasets, which achieves outstanding performance with little efficiency trade-off.

In the revised paper, we will add this analysis to better illustrate the usability and scalability of GlycanAA on large-scale glycan datasets.

 

[d] A motif-based analysis of glycan array data to determine the specificities of glycan-binding proteins. Glycobiology, 2010.

Thanks for your valuable comments and constructive suggestions! We respond to your questions as below:

Q1: Can the model handle more complex glycans with diverse glycosidic linkages?

We announce that the proposed GlycanAA model can handle any glycan no matter how complex its structure is. Basically, for a given glycan, GlycanAA extracts each of its monosaccharide as a node in the backbone-level graph, and, for each monosaccharide, its fine-grained atomic-level structure is further modeled by an atomic graph; GlycanAA constructs relational edges between different monosaccharides to capture glycosidic linkages, where it use 84 types of relations to model all possible glycosidic bonds that connect atoms at different sites with different stereochemical configurations.

The glycans with complex structures can be well modeled in this way. For example, starch is a complex kind of glycans composed of hundreds of glucoses (monosaccharide units) and thousands of $\alpha$1-4 and $\alpha$1-6 glycosidic bonds. By using the proposed backbone- and atomic-level graph modeling and multi-relational glycosidic bond modeling approaches, GlycanAA can well capture (1) the local structure within each glucose unit and (2) the global structures of starch formed by $\alpha$1-4 glycosidic bonds for its linear parts and $\alpha$1-6 glycosidic bonds for its branching parts.

Q2: What happens if structural features are included in the model?

Table A: Performance comparison between GlycanAA and GlycanAA-torsion on taxonomy prediction tasks. The Macro-F1 score for each task and the mean Macro-F1 score over all tasks are reported.

 

This question is great. First, we claim that GlycanAA owns a generic model framework which can easily incorporate various structural features. Here, we take the featurization of torsion angles of glycosidic bonds as an example. Specifically, by using the Carbohydrate Builder of GLYCAM, we get the 3D conformation of each glycan data in the dataset for taxonomy prediction. For each glycosidic bond in a glycan structure, it defines two torsion angles $\phi$ and $\psi$, as shown in the figure of this URL. To include these two torsion angles in GlycanAA, we respectively compute their sine and cosine values and map the resulting four values to the hidden space for message passing. We name this model variant as GlycanAA-torsion.

In Table A, we compare the performance of GlycanAA and GlycanAA-torsion on eight taxonomy prediction tasks. According to the results, GlycanAA-torsion outperforms GlycanAA on 5 tasks with at most 210 taxonomy categories, while GlycanAA-torsion is inferior on 3 tasks with at least 415 taxonomy categories. These results demonstrate that including torsion angle features can enhance the model's ability to fit the data, but it can also make the model more prone to overfitting, especially in complex tasks.

We will include this study in the revised paper to inspire more future work on glycan structure modeling, e.g., constructing distance-induced glycan graphs based on the vicinity of atoms and monosaccharides in 3D glycan conformations. We will also include the figure presenting glycosidic torsion angles in the revision for better understanding of glycan structures.

Q3: The selection of baseline for efficiency study.

We clarify that, in the efficiency study, we select the best-performing baseline (without pre-training), i.e., RGCN, for comparison with GlycanAA. Comparing to RGCN, GlycanAA performs better on all 11 benchmark tasks with a moderate 20% more computational cost, demonstrating that GlycanAA achieves remarkable performance gains against the most competitive baseline under acceptable efficiency trade-offs.

Appreciate your insightful comments and golden suggestions! We respond as below:

Q1: An ablation study focusing on the proposed hierarchical graph during the pre-training stage is recommended.

This suggestion is great. By removing the atom-level modeling part, the obtained model variant of GlycanAA essentially performs relational message passing among monosaccharides, which is basically RGCN. Therefore, we compare the pre-training performance of GlycanAA and RGCN by using the proposed mask prediction algorithm, where the monosaccharide mask ratio is set as 0.3 for both models. In the figure of this URL, we present the accuracy and cross entropy loss curves of pre-training for these two models. According to the results, compared to RGCN, GlycanAA performs clearly better in pre-training with higher accuracy and lower cross entropy loss, thanks to its higher model capacity (i.e., modeling both monosaccharide-level and atom-level glycan structures). By checking the benchmark results in the Table 1 of paper submission, we can observe that the pre-trained GlycanAA (i.e., PreGlycanAA in the table) achieves clearly more performance gains on downstream tasks after pre-training, compared to the pre-trained RGCN (i.e., PreRGCN in the table). This correlation between higher model capacity, higher pre-training performance and more performance gains on downstream tasks is also reported in other domains [a,b,c].

We will add this study to the revised paper version, so as to give more insights to pre-training glycan representations.

 

[a] A simple framework for contrastive learning of visual representations. ICML, 2020.

[b] Bert: Pre-training of deep bidirectional transformers for language understanding. NAACL, 2019.

[c] Strategies for pre-training graph neural networks. ICLR, 2020.

Q2: ProNet is a related work, which should be discussed and compared with.

Table A: Performance comparison between ProNet, GearNet, GearNet-Edge and GlycanAA on benchmark tasks. The best and second-best results are denoted by bold and italic, respectively.

 

Thanks for pointing this out. ProNet [d] is a representative model for protein structure modeling, which simultaneously captures the amino-acid-level, backbone-level and all-atom-level structures of a protein. In its implementation, ProNet passes the structural features at higher resolution (i.e., backbone-level and all-atom-level features) to the structural features at lower resolution (i.e., amino-acid-level features), and the graph convolution at lower resolution is biased by the features passed from higher resolution.

To investigate such a modeling approach in the glycan domain, we implement a ProNet for glycan modeling which follows the original architecture with four interaction blocks, and, in each interaction block, atom-level features are passed to monosaccharide-level features to bias the graph convolution operation. In Table A, we compare this ProNet with GearNet, GearNet-Edge and GlycanAA on benchmark tasks. According to the results, ProNet outperforms GearNet and GearNet-Edge in terms of weighted mean rank, while it is inferior to GlycanAA on all benchmark tasks. This result again demonstrates the effectiveness of the proposed hierarchical relational message passing scheme in GlycanAA, which well captures different kinds of dependencies within a glycan.

In the revised paper, we will supplement the above discussion and comparison for the interests of a broader audience.

 

[d] Learning Hierarchical Protein Representations via Complete 3D Graph Networks. ICLR, 2023.