Contextual Molecule Representation Learning from Chemical Reaction Knowledge

On the Concern of code

We have restructured our code and shared it in an anonymous Git repository. Additionally, a comprehensive README file has been included, encompassing both pretraining and fine-tuning scripts, along with data processing details and the hyperparameters used during training. You can access the code repository through the following link: https://anonymous.4open.science/r/REMO-8E45/README.md.

Furthermore, we have made available a pretrained model, accessible through this link: https://drive.google.com/file/d/1t62Xo5Akco9Z04El_wtbcNR_9HN0wnnf/view?usp=sharing.

Should you have any questions regarding reproducibility or require further elaboration on specific details, please don't hesitate to reach out. We welcome inquiries and are ready to provide additional information as necessary.

Dear Reviewer,

We express our sincere gratitude for the invaluable suggestions you provided to enhance our manuscript. We kindly request your confirmation regarding any lingering issues that may require additional attention to meet your expectations and potentially elevate the overall assessment. Your time and feedback are highly valued, and we eagerly anticipate your response.

On the New Benchmark Reaction Type Classification

We conducted a comprehensive evaluation of REMO_IM on the Reaction Type Classification benchmark. The fine-tuning pipeline we employed aligns with the methodology used in MolR. Specifically, REMO_IM generates graph-level representations for both the reactants and products graphs, which are then concatenated to serve as input for a two-layer downstream classifier. The dataset utilized, USPTO-1k-TPL, comprises 400,000 reactions for training and 45,000 for testing, with 5% of the training set reserved for validation purposes. The results of this evaluation are as follows:

The superior performance of REMO_IM, as evidenced by these results, underscores its effectiveness compared MolR

We sincerely appreciate your insightful comments and suggestions. We have diligently revised our manuscript in accordance with your guidance. Below, we address your raised concerns and inquiries:

On the consensus of our motivations

We apologize for any inaccuracies in our initial explanation. Our intent is to illustrate that a diverse range of atoms can form stable molecules within the realm of organic chemistry, subject to chemical laws and principles, such as the Octet Rule. This diversity surpasses that seen in the dependencies between words in sentences, given the comparatively fewer constraints. For further reference, please see:

Chemistry LibreTexts. (n.d.). Combining Atoms to Make Molecules and Compounds. In Green Chemistry and the Ten Commandments of Sustainability (Manahan). Retrieved from https://chem.libretexts.org/Bookshelves/Environmental_Chemistry/Green_Chemistry_and_the_Ten_Commandments_of_Sustainability_(Manahan)/02%3A_The_Key_Role_of_Chemistry_and_Making_Chemistry_Green/2.12%3A_Combining_Atoms_to_Make_Molecules_and_Compounds

On the Concern of Weak Experiment Results

Additional experiments were conducted to **evaluate REMO_IM on the MoleculeACE benchmark **(Table 1). Our results demonstrate state-of-the-art performance, surpassing both ECFP+SVM and other deep-learning models. The average RMSE and RMSE_Cliff are 0.647(±0.003) and 0.747(±0.004), respectively. The full result is available in https://anonymous.4open.science/r/ICLR2024_5944-55E3/remoIM_moleculeace.csv

On the concern of lacking import baselines

We conducted a comparative analysis with MolR[1] using MoleculeNet for evaluation. Our findings indicate significant disparities between the claimed results of MolR and our own. After investigating MolR's methodology, we identified two key differences: their use of random data partitioning and a focus on single subtask prediction for multi-task datasets(Please be aware that the choice of split method, whether random or strictly scaffold-based, significantly influences the performance on MoleculeNet.). To address these differences, we concentrated on single-task datasets (HIV and BBBP) using the scaffold splitter method. Our re-evaluated results confirm REMO's superiority over MolR.

On More Baselines in the Benchmark ACNet

Thank you for your suggestion. We have added Mole-BERT to our set of baseline models for ACNet. Below is a detailed breakdown of Mole-BERT's performance across the 13 tasks within the ACNet Few Subset, using three different random seeds.

On More Baselines for the Benchmark Drug-Drug Interactions (DDI)

Upon applying GraphLoG and Mol-BERT to the DDI task, it became evident that REMO outperforms these methods. This superiority can be attributed to REMO's proficiency in capturing reaction information, as it learns from reaction data as opposed to single molecules. The results align with those observed in MolNet.

[1] CHEMICAL-REACTION-AWARE MOLECULE REPRESENTATION LEARNING, ICLR 2022.

Thank you for your positive feedback. We greatly appreciate your guidance. Based on your suggestions, we have diligently revised our paper. Addressing your concerns, we provide the following responses:

Q1

Our method constructs the graph through a systematic tokenization of nodes based on their types and adjacent bond types. This approach amalgamates these elements into a comprehensive dictionary of topology tokens for the dataset. For instance, a carbon atom linked with four single bonds constitutes a specific entry in this dictionary. Ultimately, our constructed dictionary encompasses approximately 2500 unique elements. The core objective of the reconstruction process is to accurately identify the corresponding label from this extensive dictionary.

Q2

Additional experiments have been conducted to assess REMO_IM's performance on MoleculeACE, with comprehensive results available at https://anonymous.4open.science/r/ICLR2024_5944-55E3/remoIM_moleculeace.csv. We observed an average RMSE of 0.647(±0.003) and RMSE_Cliff of 0.747(±0.004), affirming REMO_IM's state-of-the-art (SOTA) status when benchmarked against models like ECFP+SVM and other deep-learning frameworks. To clarify any misunderstandings, Table 2 in our paper features REMO_IM, which was concurrently pre-trained on both the reaction center reconstruction and identification tasks.

Q3

Table 3 highlights the distinct strengths of REMO-I and REMO-M across various tasks. REMO-I demonstrates superior performance in pharmacology-related tasks (e.g., BBBP, Tox21, Toxcast, SIDER), while REMO-M is more adept in biophysical tasks like HIV and Bace. Notably, REMO_IM, integrating both Identification and Masking pre-training, achieves the highest overall performance at 74.7, illustrating the synergistic benefits of these tasks.

Q4

To demonstrate the efficacy of our pretraining techniques, we applied REMO without pretraining to the DDI task. The results, as tabulated below, clearly show that our pretraining methods substantially improve the model's performance:

Q5

We apologize if there was any confusion. To clarify, we represent the presence of a specific atom as a reaction center using the variable p_i where p_i = 1/0. This representation is employed for a binary classification task, distinguishing whether a given atom serves as a reaction center or not. In cases where a molecule has multiple reaction centers, the vector [p_1, p_2, …, p_N] may contain multiple instances of the value 1. It's important to note that we do not extend the classification further to identify different types of reaction centers. If, however, there are multiple types of reaction centers (for example, 2 types), we can modify p_i to take values in the set {1, 2, 0). In this modified representation, p_i will denote whether a specific atom is a reaction center (when p_i > 0) and, if it is, indicate the type to which it belongs.

On Activity Cliff as a goal for pre-trained models.

Activity Cliff is defined as structural similar molecules may present significantly different binding affinities to targets. It is a good example to show the complexity for the property landscape of small molecules. Inferring from single molecules is challenging to predict activity cliff. Pre-trained model learns from single molecules, which learns a uniform space for molecular structures, though the structural-activity space is not uniform for the activity cliff problem, from a structural point of view, this discrepancy causes most pre-trained models could not yield optimised result, this problem is also studied in [3]. However, the intuition of our work aims at learning the chemical environment along with molecular structures, so the model would be aware of the molecules' position in chemical reaction space. Predicting activity cliff from structures is difficult, but activity cliff still obeys the rules in biochemistry. Intuitively, a model could be sensitive to activity cliff if it learns from the structural space and the chemical reaction space at the same time.

On baselines of MoleculeACE

The result of ECFP+MACC followed by SVM is attached at: https://anonymous.4open.science/r/ICLR2024_5944-55E3/svm_moleculeace.csv the average rmse and rmse_cliff is 0.658 and 0.744 respectively.

The result of D-MPNN (chemprop) is attached at: https://anonymous.4open.science/r/ICLR2024_5944-55E3/chemprop_moleculeace.csv the average rmse and rmse_cliff is 0.750(0.002) and 0.829(0.004), respectively.

it is worth noting that we evaluate our model REMO_IM on MoleculeACE after submission, the result is attached at: https://anonymous.4open.science/r/ICLR2024_5944-55E3/remoIM_moleculeace.csv the average rmse and rmse_cliff is 0.647(0.003) and 0.747(0.004), respectively. The result reaches SOTA, compares to either ECFP+SVM, or (ECFP+MACCs)+SVM, or other deep-learning based models.

On the difference in pretraining dataset

However, even if these are larger datasets, they do not necessarily have to be better. In fact, USPTO contains highly diverse, special data from patents. So it is not directly clear to me that this dataset is comparable to the pre-training data the other models use.

Undoubtedly, high-quality data can compensate for quantity limitations. However, the baseline models, which take only individual SMILES as input, may not fully exploit the wealth of chemical reaction information embedded in the USPTO data. To investigate this hypothesis, we disentangled chemical reactions into individual molecules, removing duplicates to create a dataset of approximately 2 million distinct SMILES, denoted as 'uspto-single.' Subsequently, we utilized this dataset to re-pretrain the GraphMAE for 100 epochs until the model's loss achieved convergence and stability.

Following the pretraining phase, we conducted a retest using MoleculeNet and present the results in the table below.

Notably, the performance experiences a decline when transitioning to the 'uspto-single' pretraining data. This observation suggests that the pretraining method, which is solely based on individual molecules, results in a loss of crucial chemical reaction information, leading to inferior performance. To address this limitation, there is a need to design a specialized pretraining strategy, such as REMO, to fully harness the rich information present in the USPTO data.

It is also not clear from the table if the REMO_x models are based on GIN or Graphormer.

We apologize for any confusion. Specifically, the REMO_IM-GraphMAE and REMO_IM-AttrMask configurations utilize GIN as the backbone. This choice is attributed to their continuous pretraining on the encoder provided by GraphMAE or AttrMask. In contrast, all other REMO settings, with the exception of these two, employ Graphormer as the backbone.

[3] Why Deep Models Often cannot Beat Non-deep Counterparts on Molecular Property Prediction?, arxiv 2023.

Dear Reviewer,

We express our sincere gratitude for the invaluable suggestions you provided to enhance our manuscript. We kindly request your confirmation regarding any lingering issues that may require additional attention to meet your expectations and potentially elevate the overall assessment. Your time and feedback are highly valued, and we eagerly anticipate your response.

Thank you for your insightful feedback. We greatly appreciate your guidance. Based on your suggestions, we have diligently revised our paper. Addressing your concerns, we provide the following responses:

On the missing references

We adhere the references to masking approaches for SSL here, and this would be in the paper of the final version:

masked language model (MLM) (Devlin et al., 2019; Lample and Conneau, 2019)

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In North American Association for Computational Linguistics (NAACL).

Guillaume Lample and Alexis Conneau. 2019. Crosslingual language model pretraining. arXiv preprint arXiv:1901.07291.

On the consensus of our motivations

"in molecule graphs the relationships between adjacent sub-units are mostly irrelevant." - We are sorry for the partially accurate information in the sentence when takes out along. We aim at pointing out that a wide variety of atoms can combine to form stable molecules under the context of organic chemistry, under the constraint of chemical laws and principles (e.g. Octet Rule). Especially, the variety is wider compares to the dependencies between words in sentences, as the constraints are less. A reference of this can be found in:

Chemistry LibreTexts. (n.d.). Combining Atoms to Make Molecules and Compounds. In Green Chemistry and the Ten Commandments of Sustainability (Manahan). Retrieved from https://chem.libretexts.org/Bookshelves/Environmental_Chemistry/Green_Chemistry_and_the_Ten_Commandments_of_Sustainability_(Manahan)/02%3A_The_Key_Role_of_Chemistry_and_Making_Chemistry_Green/2.12%3A_Combining_Atoms_to_Make_Molecules_and_Compounds

"In such cases, traditional masked reconstruction loss is far from being sufficient as a learning objective." - The Molformer paper shows that simple masking can recover structure quite well if enough pre-training data is available.

We agree on the fact that many molecular encoder trained on masking reconstruction from single molecules could recover structures quite well. However, it could be credited by a lot of factors. Firstly, most small molecule datasets are biased, the atom types are overwhelmingly biased to carbon, as seen in Supplementary F. This imbalance in dataset make the reconstruction easier when randomly mask an atom type. Moreover, as stated in [1], the masked node prediction task on single molecules might be too easy, for the limited size of vocabulary, and the valence constraints make the prediction even easier. Moreover, as stated in the Introduction part of our paper, the model may not learn useful knowledge that could transfer to downstream tasks from simple masked reconstruction, even though they may perform good in pre-training tasks.

"most biochemical or physiological properties of a molecule are determined and demonstrated by its reaction relations to other substances" - also needs references, esp. for ML readers

Textbook such as [2] explains it well, we will include it in the final version of our paper.

[1] Sun, R., Dai, H., & Yu, A. W. (2022). Does GNN Pretraining Help Molecular Representation? In Proceedings of the 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Retrieved from https://openreview.net/pdf?id=uytgM9N0vlR#:~:text=Different%20setups%20can%20lead%20to%20opposite%20conclusions.&text=In%20conclusion%2C%20different%20from%20the,is%20effective%20in%20molecular%20domain.

[2]LibreTexts. (n.d.). 13: Introduction to Biochemistry. Chemistry LibreTexts. Retrieved November 21, 2023, from https://chem.libretexts.org/

We are grateful for your valuable feedback and have meticulously revised our manuscript in light of your insightful recommendations. Below, we address your concerns and queries in a detailed manner:

On the Additional Information Gain of Distinct Pre-training Strategies

We acknowledge the similarity between the tasks of Reaction Centre Identification and Masked Reaction Centre Reconstruction, but emphasize their distinct characteristics. Reaction Centre Identification involves locating the reaction center on a molecule, whereas Masked Reaction Centre Reconstruction entails rebuilding the reaction center given its location, incorporating adjacent intra-molecular information and conditional molecules. As demonstrated in Table 3 of our paper, the REMO_I (Pretrained by Identification) and REMO_M (Pretrained by Masking) models exhibit specific advantages in different sub-tasks. Most notably, REMO_IM (Pretrained by both Identification and Masking) delivers the highest overall performance, highlighting the synergistic effect of these tasks.

On the Discrepancy Between Pre-training and Downstream Tasks

During pre-training, REMO does incorporate conditional molecules, despite some downstream tasks using only single molecules. REMO's training approach is designed to concurrently encode conditional and primary molecules using a singular encoder, with weight updates driven by gradients from both task branches. This method enables REMO to learn not just the atomic combinations within a molecule, but also its broader chemical context. Consequently, REMO's graph encoder, which is effective for single molecules, remains applicable to these tasks.

On REMO's Effectiveness in Addressing Activity Cliffs

We evaluated the REMO_IM model on the MoleculeACE benchmark (as seen in Table 1), with results accessible via https://anonymous.4open.science/r/ICLR2024_5944-55E3/remoIM_moleculeace.csv. The model's average RMSE and RMSE_cliff for the test set are 0.647(0.003) and 0.747(0.004), respectively, surpassing the ECFP+SVM method and achieving state-of-the-art performance. While REMO-M may not universally outperform fingerprint-based models, it demonstrates improvements in 16 specific tasks (RMSE metric) and 17 tasks (RMSE_cliff metric). REMO-I, on the other hand, outperforms fingerprint-based models in 16 and 15 tasks, according to RMSE and RMSE_cliff, respectively (detailed in Supplementary C.1, Figures 5 and 6).

Q1

Please refer to our explanation above regarding the additional information gain of distinct pre-training strategies.

Q2

We are unclear about the context of this question as REMO does not propose any method for conditional molecule generation. Nevertheless, we acknowledge the value of this direction for future research and appreciate the suggestion.

Q3

The 'Activity Cliff' concept highlights the challenge of predicting binding affinity differences in structurally similar molecules. This complexity mirrors the chemical reaction space of small molecules, as depicted in Figure 1A, where similar structures can react differently. REMO's training on chemical reaction information aims to capture this complexity, potentially aiding in understanding activity cliffs in small molecules. We believe that integrating chemical reaction knowledge into pre-training could be advantageous, and exploring this connection is a key direction for future research.

Q4

The observed differences are attributable to the choice of backbone models. REMO-IM utilizes the more advanced Graphformer, while REMO-IM Attrmask employs GIN, aligning with Attrmask for a balanced comparison.

Dear Reviewer,

We extend our heartfelt appreciation for the invaluable suggestions you offered to improve our manuscript. We kindly seek your confirmation on whether any outstanding issues remain that may need further attention to meet your expectations and potentially enhance the overall assessment. Your time and feedback are greatly valued, and we eagerly await your response.