GlycoNMR: A Carbohydrate-Specific NMR Chemical Shift Dataset for Machine Learning Research

We would like to thank reviewer dVzH for the comments and suggestions. Here we summarize and address reviewer dVzH’s concerns as follows.

1, Concerns on the significance of the dataset, insufficient description of protein difference.

Response:

The significance of the GlycoNMR dataset and benchmark can be summarized into two folds.

1, For attracting glycoscience domain expert interest in this ML direction, we believe GlycoNMR is a good starting point to apply ML to glycoscience where ML methods are relatively under-explored due to the lack of well-curated datasets and underdeveloped benchmarks. We do recognize that there exist more high-resolution tools to study protein structure than carbohydrate structure, with X-ray crystallography and cryogenic electron microscopy (cryo-EM) being the frontier [1], in addition to NMR. Carbohydrates cannot be easily crystallized or imaged in cryo-EM, so NMR is still the leading structure tool. For experimental glycoscience researchers, we would like to present our machine learning datasets and pipelines to showcase their effectiveness in aiding challenges of NMR prediction within the glycoscience field. The larger and more diverse (and publicly available) the experimentalists can help make datasets, the more powerful ML tools can become. On the other hand, collaboration with ML researchers is needed however to make the datasets ML-friendly, so we’ve tried to make substantial initial efforts, documentation, and demonstrations in this direction. We hope our study encourages greater future collaboration.

2, For machine learning researchers: we would like to introduce this emerging subfield, which holds the potential to lead to both fundamental research progress and industrial applications. For example, the current collection of an NMR spectrum may cost hundreds or thousands of dollars for a single data point, not counting all the preparation time and cost needed for the development of purification protocols for novel compounds, for example see (https://www.aiinmr.com/nmr-spectroscopy-q-a-blog/How-much-does-an-Eft-NMR-Cost and https://nmr.science.oregonstate.edu/industry-price-list). A well-curated machine learning tool for NMR chemical shift prediction may potentially lead to substantial cost savings in compound quality verification (e.g. predict which shifts should result if the correct compound was generated, a tough experimental problem for larger carbohydrates currently), or through enhanced theoretical understanding of the link between structure and NMR spectra generally.

For further detail on the significance of the GlycoNMR, please kindly take a look at the main GitHub page: (https://anonymous.4open.science/r/GlycoNMR-D381/README.md), including a brief tutorial on using our curated carbohydrate data to fit into a machine learning pipeline and a road map figure summarizing what we have covered in this current work, and what could be the potential research directions in follow-up work. Thus our study can hopefully help guide future ML researchers to contribute to the Glycoscience community, and vice versa. Data cleaning and preprocessing pages for GlycoNMR: (https://anonymous.4open.science/r/GODESS_preprocess-F9CD/README.md and https://anonymous.4open.science/r/GlycoscienceDB_preprocess-B678/README.md), these two repo records the general data preprocessing pipelines and tons of our work in aggregating the experimental results uploaded from various labs and preprocessing the simulational results. This can potentially aid future Glycoscience researchers or industrial professionals who are willing to share their data with the ML community. We believe that the following researchers using our dataset will achieve better performance than ours.

In addition, to address reviewer dVzH’s concerns in our manuscript. We have expanded the explanation and references related to the historical discussions about weakness in data standards and consistency in carbohydrate structure research relative to protein research, as well as new data complexities that are present in carbohydrate data, in the introduction and appendix:

See these references in the Introduction: “Particularly, existing structure-related carbohydrate NMR spectra databases are less extensive and less accessible to ML researchers than databases for other classes of biomolecules and proteins, leading to recent calls for improvement in standards and quality [2, 3, 4, 5].”

We are grateful to the reviewer sFB8 for the valuable suggestions to investigate the feature statistics on both datasets further. In summary, we have added three appendix subsections on pages 17, 18, and 19 to address reviewer sFB8’s concerns. Here, we reply to the concerns and questions in the following.

1, Concern: Feature Statistics and deeper validation of features.

We would like to thank reviewer sFB8’s suggestions, as the visualization could further illustrate the data statistics in Table 1. We provide a detailed feature analysis on both GlycoNMR.Exp and GlycoNMR.Sim in our Appendix A.3 and A.4. We kindly invite reviewer sfB8 to check our expanded version on page 17. In Appendix A.3, we visualize the feature statistics in six pie charts for atom-level and monosaccharide-level features.

For the atom level feature (first row), we included the dataset statistics of atom type, carbon atom position, and hydrogen atom position. According to the pie chart for atomic identity(top left), we observed that of all the 27267 atoms in GlycoNMR.Exp, 13010 are Hydrogen, 7663 are Carbon, 6257 are Oxygen, and 337 are other types of atoms, including Nitrogen, Phosphorus, and Sulfur.

To describe carbon atom position (top middle), ‘Other’ indicates the off-ring carbons. Similarly, for describing the hydrogen atom position (top right), ‘Other’ indicates off-ring hydrogens.

For the monosaccharide level feature, we included Amomer (bottom left, indicates the hydroxyl group), Configure (bottom middle, indicates Fischer project information), and Ring Size (bottom right, number of in-ring carbons). The ‘N/A’ of each figure indicates that the information is not contained in the PDB file.

In addition, we added Appendix C to analyze the feature contributions by calculating the Shapely values. Here we again present the Shapley value table:

We observe that the Shapley values across all features are positive. Notably, the ring position of carbon and hydrogen atoms is a critical feature in predicting NMR shifts. Moreover, integrating the stem type of monosaccharides into the 2D Graph Neural Network can marginally reduce the prediction error. Other features, like the modification group, anomer, configuration, and ring size, exert lesser influence.

2, Question: Ring Position vs. Shift Relationship

We provide the violin plot of the Ring position versus the NMR shift for both datasets in Appendix A.4. In each subplot, the x-axis indicates the ring position of the atom (Carbon / Hydrogen), and the y-axis indicates the NMR shift of the corresponding atoms.

We notice that the distribution of NMR shift values for the ring positions C1 and C6 significantly vary from those of C2, C3, C4, and C5, similar to hydrogens. In addition, the distribution of NMR shift values associated with the ring positions shows similarity when comparing the GlycoNMR.Exp and the GlycoNMR.Sim. This consistency indicates the value of using simulated NMR data to help develop ML approaches for studying carbohydrates.

For a general explanation of the ring position, a fundamental factor that determines the NMR shift value is the atom’s electronic environment, especially bonded or non-bonded interactions within 1-3 atom distances away from the atom of interest.

In glycoscience specifically, it is known that in the monosaccharide units, the index of carbon and hydrogen in the ring, is one of several major factors that define the electronic environment, which is reflected in our results [1]. These basic position tendencies are further modulated by a set of factors that can also alter the electrochemical environment, including the other features tested in our model, like anomer, configuration, temperature, and solvent, and more complicated modification or conjugation groups attached to glycans. This domain knowledge prompts us to integrate ring positions as one of the node features in our 2D-based and 3D-based Graph Neural Network models.

[1] Fontana, Carolina, and Goran Widmalm. "Primary structure of glycans by NMR Spectroscopy." Chemical Reviews 123.3 (2023): 1040-1102.

Thanks again for the valuable suggestions!

We would like to express our sincere gratitude to reviewer ozW2 for the detailed comments and suggestions we can use to improve the paper. Here, we address reviewer ozW2’s concerns and questions in the following.

1, Concerns on reproducibility and domain expertise involved:

We would like to thank ozW2 for pointing out that the domain expertise contribution is not introduced clearly enough in the manuscript, which may result in an insufficient demonstration of the reproducibility. Here, we address the reviewer’s ozW2’s concern as follows:

1. We summarize the data annotation and preprocessing steps of GlycoNMR.Exp and GlycoNMR.Sim. This summary includes an overview of and referral to the annotation GitHub repositories linked to our paper that provide extensive notes and scripts for preprocessing.

2. We present how domain expertise is involved in annotating the raw glycan data.

3. We present an example glycan for the data annotation that demonstrates the ambiguities.

Please see as follows, we also updated our Appendix L in response to ozW2’s suggestions.

For GlycoNMR.Exp:

We first carefully manually inspected the NMR chemical shift and structure files for every carbohydrate in the dataset in detail. We documented all inconsistencies between NMR and structure files (files that many different labs uploaded across decades) and used the list of various inconsistencies to make a formulaic and reproducible data curation/annotation protocol that could be extended to any experimental dataset of carbohydrates. Mismatches were mainly because of the inconsistent sequence ordering between monosaccharide IDs from the PDB structure files and NMR shift label files, as they are uploaded from various chemistry labs, structure files can be generated from separate software unrelated to NMR shifts, and format can change over the course of years. There were no automatic ways or well-specified routines for solving this matching problem, and significant domain expertise was required from the glycoscience domain for the initial data annotation protocol establishment. However, by the end of our process, we were able to reduce our curation protocols to understandable and reproducible annotation procedures. The complete protocol and corresponding preprocessing scripts are listed in this manuscript’s supplemental data preprocessing repository (https://anonymous.4open.science/r/GlycoscienceDB_preprocess-B678/README.md), and we’ve moved an abbreviated version of the steps into the Appendix L.2 and L.3.

Using the protocol, we reformulated all the PDB and label files we retrieved from Glycoscience.DB into interpretable consistent formatting and applied our initial data cleaning for the scraped carbohydrates. This step is mainly to unify the PDB (Protein Data Bank) format of the experimental carbohydrate data uploaded from various labs in different time periods. Each PDB file should include the information of atom type, atom 3D coordinates, and the monosaccharides that the atom belongs to. Carbohydrates like (https://anonymous.4open.science/r/GlycoscienceDB_preprocess-B678/experimental_data/FullyAnnotatedPDB_V2/DB26874.pdb) are dropped due to insufficient information (missing atom name that describes the ring position).