FARM: Functional Group-Aware Representations for Small Molecules

References:

[1] Zaixi Zhang, Qi Liu, Hao Wang, Chengqiang Lu, and Chee-Kong Lee. Motif-based graph selfsupervised learning for molecular property prediction. Advances in Neural Information Processing Systems, 34:15870–15882, 2021.

[2] Shen Han, Haitao Fu, Yuyang Wu, Ganglan Zhao, Zhenyu Song, Feng Huang, Zhongfei Zhang, Shichao Liu, and Wen Zhang. Himgnn: a novel hierarchical molecular graph representation learning framework for property prediction. Briefings in Bioinformatics, 24(5):bbad305, 2023.

[3] Yongqiang Chen, Quanming Yao, Juzheng Zhang, James Cheng, and Yatao Bian. Hight: Hierarchical graph tokenization for graph-language alignment. arXiv preprint arXiv:2406.14021, 2024.

[4] Nianzu Yang, Kaipeng Zeng, Qitian Wu, Xiaosong Jia, and Junchi Yan. Learning substructure invariance for out-of-distribution molecular representations. Advances in Neural Information Processing Systems, 35:12964–12978, 2022.

[5] Biaoshun Li, Mujie Lin, Tiegen Chen, and Ling Wang. Fg-bert: a generalized and self-supervised functional group-based molecular representation learning framework for properties prediction. Briefings in Bioinformatics, 24(6):bbad398, 2023.

[6] Yongqiang Chen, Quanming Yao, Juzheng Zhang, James Cheng, and Yatao Bian. Hight: Hierarchical graph tokenization for graph-language alignment. arXiv preprint arXiv:2406.14021, 2024.

[7] Gabriel A Pinheiro, Juarez LF Da Silva, and Marcos G Quiles. Smiclr: contrastive learning on multiple molecular representations for semisupervised and unsupervised representation learning. Journal of Chemical Information and Modeling, 62(17):3948–3960, 2022.

[8] Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang. Pretraining molecular graph representation with 3d geometry. arXiv preprint arXiv:2110.07728, 2021.

[9] Zaixi Zhang, Qi Liu, Hao Wang, Chengqiang Lu, and Chee-Kong Lee. Motif-based graph selfsupervised learning for molecular property prediction. Advances in Neural Information Processing Systems, 34:15870–15882, 2021.

[10] Yin Fang, Qiang Zhang, Ningyu Zhang, Zhuo Chen, Xiang Zhuang, Xin Shao, Xiaohui Fan, and Huajun Chen. Knowledge graph-enhanced molecular contrastive learning with functional prompt. Nature Machine Intelligence, 5(5):542–553, 2023.

[11] Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. Uni-mol: A universal 3d molecular representation learning framework. The Eleventh International Conference on Learning Representations, ICLR 2023, 2023.

[12] Jun Xia, Chengshuai Zhao, Bozhen Hu, Zhangyang Gao, Cheng Tan, Yue Liu, Siyuan Li, and Stan Z Li. Mole-bert: Rethinking pre-training graph neural networks for molecules. The Eleventh International Conference on Learning Representations, ICLR 2023, 2023.

[13] Yuyang Wang, Jianren Wang, Zhonglin Cao, and Amir Barati Farimani. Molecular contrastive learning of representations via graph neural networks. Nature Machine Intelligence, 4(3):279– 287, 2022b.

[14] Alec Radford. Improving language understanding by generative pre-training. OpenAI, 2018. Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer on large-scale molecular data. Advances in neural information processing systems, 33:12559–12571, 2020.

Imprecise motivation on contrasive learning: The motivation of this work is vague. SMILES and graph of molecules contain the same information of molecule, i.e., a molecule can be reconstructed from both SMILES and graph. Why should we deal with them both instead of focusing on a single representation (SMILES or graph)? Since this paper strongly insist that they learn "chemically plausible" representation, this choice should be "chemically" justified (I know that the language model and the graph model may learn different features of molecules, but it is not a "chemical" motivation).

• While SMILES strings and molecular graphs indeed encode the same molecular information, they each bring distinct advantages for representation learning. Graph-based representations are highly effective in capturing the molecule’s topological structure, yet they often struggle to model long-range dependencies due to their inherently local connectivity. Conversely, SMILES representations are better suited to capturing these long-range dependencies, particularly when using attention mechanisms in language models.
• Our choice to use both representations is to capture both atom-level features and global molecular topology. We already make this point clear in the abstract: “FARM also represents molecules from two perspectives: by using masked language modeling to capture atom-level features and by employing graph neural networks to encode the whole molecule topology. By leveraging contrastive learning, FARM aligns these two views of representations into a unified molecular embedding.” Figure 1 also illustrates that the SMILES model is used for learning the atom-level representation (which is better at capturing long-distance dependencies between atoms), while the graph model is used for learning the overall structure of the molecule. Thus, we do not claim that the motivation for using contrastive learning is to learn a chemically plausible representation of molecules.
• The chemical information that FARM learns is gained by enhancing SMILES with functional group information (creating FG-enhanced SMILES) and incorporating functional groups directly into the graph representation (FG graphs). This approach introduces a chemically meaningful inductive bias into both representations, as opposed to relying on standard SMILES and molecular graphs. Consequently, the combined use of FG-enhanced SMILES and FG graphs enables a more comprehensive and chemically plausible molecular representation, capturing both local functional group characteristics and global molecular topology, as mentioned in line 398.
• Furthermore, our approach is consistent with recent research that uses contrastive learning across SMILES and molecular graph representations, demonstrating the complementary benefits of each representation for molecular representation learning [7].

Misclaim "L71: Our approach overcomes these limitation ...": The proposed approach cannot overcome "3D coordinates" (L66) and "long-range dependencies" (L68). This method does not deal with 3D coordinates nor long-range dependencies.

• Thank you to the reviewer for pointing out this oversight. We adjusted this statement to clarify our intention in the revised version of the manuscript. Our approach does not directly address 3D coordinates but is designed to effectively capture both topological information and certain long-range dependencies through the use of functional group graphs. By employing contrastive learning with these functional group graphs, which serve as a compact representation of molecular structures, we can capture essential relational patterns between distant functional groups more effectively.

Complexity of method: This work combines several objectives, e.g., link prediction, contrastive and MLM loss. However, the impact of each component is not thoroughly investigated. Furthermore, such a complex objective introduces several hyper parameters and makes the training unstable.

• Thank you for your insightful feedback. We have conducted an ablation study to evaluate the impact of each component in our method by training models with different configurations and testing their performance on MoleculeNet’s downstream tasks. As shown in Table 8 (Appendix E), we assessed several setups: molecular graph embeddings only, BERT with standard SMILES only, functional group knowledge graph only, functional group graph only, BERT with contrastive learning and functional group knowledge graph, and our full model (FARM) that combines contrastive learning with functional group knowledge graph and functional group initialization embeddings. The results demonstrate that FARM achieves the highest performance, confirming the positive contribution of each component to the overall performance.

Lack of novelty: Using functional groups in molecular representation learning has already been investigated by many works. Also each component of the framework, e.g., MLM, link prediction, and contrastive loss, has also been widely investigated. I could not find new (or novel) components in the overall framework.

We acknowledge that the use of MLM, link prediction, and contrastive learning are not novel concepts. However, our approach is distinct in its ability to both detect functional groups and effectively incorporate this functional group information to enhance molecular representations, leading to strong transfer learning capabilities, as demonstrated by performance in downstream tasks. Here’s how our work differentiates itself:

• Detecting functional groups: Existing motif-based tokenization methods do not strictly align with chemical functional groups. For instance, BRICS-based approaches (e.g., [1, 2, 3, 4]) generate substructures that do not map precisely to known functional groups. Methods using RDKit [5, 6] can only detect a limited set of functional groups (typically 85 traditional ones). Unsupervised strategies [7] often struggle with complex ring structures. In contrast, our approach implements algorithms to detect and incorporate a comprehensive set of chemical functional groups, including ring-containing groups, ensuring that the tokenization process aligns strictly with chemical features. We discuss this in more detail in the revised version of our manuscript (Related Works section, line 212).
• Incorporating functional group information: We are the first to directly inject functional group information into SMILES, adding richer chemical context to SMILES, thus we can effectively leverage language models (LMs) to learn molecular representations.
• Robustness in downstream tasks: Our model (FARM) demonstrates strong transfer learning capabilities - the core goal of pretrained models - outperforming other methods in 11 out of 13 tasks from the MoleculeNet benchmark. This suggests that FARM effectively learns molecular representations, leading to enhanced performance across various tasks.
• We also contribute by evaluating the diversity of large molecular datasets. Identifying functional groups and embedding this information into SMILES strings enables us to build a novel vocabulary, where each token reflects both an atom symbol and its associated functional group, serving as a measure of dataset diversity. Through this approach, we find that the ZINC dataset is significantly less diverse than ChEMBL, providing valuable insight for future work on dataset selection for foundation model training. Given the vast chemical space, it is essential for training data to represent a broad range of chemical structures. Our work is the first to conduct this type of evaluation, whereas previous studies have primarily focused on molecule length and atom types, which do not fully capture functional group diversity within datasets.

We have highlighted those points in the abstract and introduction of the revised manuscript.

In functional graph generation, what happens if the graph cannot be represented as a linear graph? Specifically, the node perturbation process in augmentation may be unclear. For example, if a functional group graph forms a triangle (a-b-c), the node perturbation could result in the same functional group graph, potentially generating incorrect negative samples.

• Thank you for your thoughtful comment. If the graph is not a linear graph, it doesn't affect the functionality of the node perturbation process. The key principle for perturbation is that as long as two nodes represent different functional groups (FGs), swapping them will still create a valid negative sample, even in non-linear graphs.
• It’s important to note that the graph we are referring to is a functional group graph, where each node represents a functional group, and edges represent the relationships between these groups. In this context, it is highly unlikely for a molecule to have exactly three functional groups where each functional group is bonded to the other two, forming a triangle. Chemical structures typically do not exhibit such close interconnections between functional groups. Instead, functional groups are more commonly connected in linear or branched configurations, rather than forming triangles.

In Figure 5, how do the removals of multiple functional groups and single functional groups yield parallel results? For instance, the orange molecule has three functional groups removed, while the green molecule has only one removed, yet both show similar results.

Thank you for your observation. The demonstration in Figure 5 aims to show that the effect of replacing one functional group with another on the molecular representation is consistent across different molecules (represented as parallel vectors in the molecular representation space). The results remain consistent regardless of the number of functional groups replaced. Specifically, the process involves replacing one type of functional group with another, such as substituting a -COOH group with a -OH group. Whether we replace a single -COOH group with a single -OH group, or multiple -COOH groups with an equivalent number of -OH groups, the overall effect on the molecular representation remains parallel in all cases. We will ensure that this explanation is made clearer in the manuscript, line 309.

In the knowledge graph construction, how are continuous values such as LogP discretized? The original continuous values may not correlate with other functional groups.

In our knowledge graph construction, continuous values such as LogP and water solubility are discretized by rounding them to the nearest integer. For example, a LogP value of 5.7 would be rounded to 6. This process results in 65 distinct categories for LogP and 14 categories for water solubility. We added this point to the revised version of the manuscript, line 870.

Will this approach be effective for generation tasks? A simpler generation task compared to SMILE LSTM could strengthen the contribution.

Thank you for your insightful comment. While we have not yet evaluated the effectiveness of our model for generation tasks, this is an area we plan to explore in future work. Our current focus is on building an effective model that provides robust molecular representations, which can be leveraged for transfer learning in downstream molecular property prediction tasks.

Why does Figure 5(b) depict the link prediction performance?

Figure 5(b) depicts the link prediction performance because the link prediction model is trained to learn the relationships between functional groups. In this case, we repeatedly replace one functional group (e.g., -COOH) with another (e.g., -OH) in the molecules. The consistent transitions (resulting in parallel vectors in the molecular representation space) observed across all test cases demonstrate that the model has effectively learned the connections and relationships between different functional groups.

The limitations of related works that address functional groups are unclear. The authors state that previous works “do not extend to detecting more complex functional groups, such as ring systems.” However, RDKit can identify ring systems, and simple IUPAC transformations could be applied to adapt this information for earlier works.

• Thank you for your feedback. Regarding functional group identification, RDKit can detect only 84 common functional groups. It primarily detects functional groups and substructures based on SMARTS (SMILES Arbitrary Target Specification) patterns, which limits its ability to identify more complex functional groups. For example, previous works using RDKit, such as FG-BERT, are also limited in the number of functional groups they can detect, with FG-BERT identifying only 47 groups. This is why we mention in our paper that RDKit’s functional group detection does not extend to more complex functional groups, such as ring systems—it is limited to detecting these 84 common functional groups.
• While RDKit can identify ring systems, it operates on molecular graphs, whereas its functional group identification typically works with sequence representations. As a result, RDKit cannot directly assign specific atoms in the graph to their corresponding functional group. This mismatch between traditional functional group detection using SMARTS patterns and ring system identification in molecular graphs makes RDKit incapable of identifying both traditional functional groups and ring systems simultaneously.

Which functional group detection algorithm is employed? The paper lacks a description of the algorithm that traverses the graph to identify functional groups, which is critical for the method. For instance, there should be clear priorities among functional groups to consistently identify intersected atoms across different functional groups.

Questions

• In this work, we develop a new functional group detection algorithm designed to strictly identify functional groups as defined in the Wikipedia page on functional groups, as well as other rings and fused ring systems. It is described starting at line 257 in the manuscript, works by traversing the molecular graph and assessing each atom based on various properties: atom type, bond types and neighbors, number of bonded neighbors, atom charge, and the presence of bonded hydrogen atoms. This structured approach ensures accurate identification of functional groups using specific chemical criteria. For example, the algorithm detects a ketone group (RCOR’) by identifying a carbon atom with no charge and three neighbors, one of which is a double-bonded oxygen atom. This mirrors how chemists recognize functional groups based on structural features, aligning the detected groups with established chemical conventions.
• Additionally, we address cases where a functional group may be a subset of another. The algorithm first checks for the presence of the larger functional group. If it’s not identified, the algorithm then checks for smaller functional groups, ensuring correct identification even in complex structures. We have clarified this point in the revised version of the manuscript, line 262.

We sincerely thank the reviewer for their thoughtful feedback and for highlighting key aspects of our approach. Below, we address each of the reviewers' comments in detail. The corresponding modifications in the revised manuscript are highlighted in blue for clarity.

The approach is quite similar to existing motif-based tokenization and fragmentation methods. While the authors define functional groups in terms of functional groups and fused ring systems, these could typically be identified through standard fragmentation tasks. An ablation study comparing FARM with other fragmentation methods would clarify its advantages. If applied under the same training conditions, does FARM demonstrate superiority?

• Our approach is not similar to existing motif-based tokenization and fragmentation methods. While our functional group detection algorithm may share similarities with motif-based methods in identifying ring and fused ring systems, we go beyond that by strictly detecting non-ring functional groups based on well-defined chemical rules. In contrast, some motif-based approaches may result in tokens that do not align with actual chemical functional groups. Functional groups such as alcohols, aldehydes, carboxylic acids, esters, amines, and others play a crucial role in determining molecular properties like hydrophilicity, lipophilicity (logP), reactivity, and solubility, which are important for tasks such as drug design or toxicity prediction.
• In terms of how we model functional group information, our work is also distinct from other methods. We are the first to directly incorporate functional group (FG) information into the SMILES representation, enabling us to leverage robust language models while also introducing chemical inductive biases.
• We have updated the related work section in the revised manuscript (line 212) to clearly highlight these key differences and emphasize the unique contributions of our approach. Thank you to the reviewer for pointing out that these differences were not clearly presented in the original version of the paper.

This work heavily relies on fused ring systems, which constitute over 99% of the identified functional groups. This raises concerns, as ring systems are generally not classified as functional groups. The paper emphasizes functional groups as the main contribution, as suggested by the method’s name. If ring systems are excluded, does the method still show superior performance?

Thank you for your valuable feedback. As mentioned in line 78 of our manuscript, we use the term functional groups to unify and expand upon related concepts such as "motifs," "fragments," "substructures," and "building blocks" because functional groups, in our approach, serve as fundamental structural elements that directly influence the chemical properties and reactivity of molecules. These terms often overlap in molecular chemistry, but we choose to adopt functional groups as a more general term to encompass various substructural features, including rings and fused ring systems. We also are not able to train the model excluding rings and fused ring systems, as it requires a long time for training the model from scratch.

The naming process for fused ring systems seems to overlook bond types, considering only ring indices and sizes. However, bond types, including single and aromatic bonds, are crucial for understanding the ring system. How are these bond types accounted for in the analysis?

Thank you for the question. While it's true that our naming process for fused ring systems primarily focuses on ring indices and sizes, we do not overlook the importance of bond types. In fact, the bond type information is implicitly captured within the SMILES notation, which we leverage during the naming process.

For example:

• Benzene (C6H6) is represented in SMILES as c1ccccc1. The aromaticity of the bonds is inherent in the SMILES string due to the lowercase "c". When we tokenize this structure in our framework, we generate the output c_6 1 c_6 c_6 c_6 c_6 c_6 1. The bond type (aromatic) is inherently captured within the SMILES string but is not explicitly represented in the tokenization output, as the bond type is already encoded within the SMILES structure itself.
• On the other hand, Cyclohexane (C6H12) is represented in SMILES as C1CCCCC1, and the tokenization output would be C_6 1 C_6 C_6 C_6 C_6 C_6 1, reflecting the structure of the cyclohexane ring without aromaticity, with bond types implicitly encoded within the SMILES string.
• Additionally, in the FG-enhanced SMILES, all special tokens, such as bond types (=, #), are preserved, ensuring that the bond information remains intact in the tokenized representation.

Thank you, Reviewer, for your valuable feedback. Once again, we emphasize that we use a scaffold split with a train-validation-test ratio of 8:1:1 and 3 random seeds for all MoleculeNet datasets in our benchmarking experiments. This approach aligns with the splitting method described in the link you provided for the BBBP dataset using the DeepChem library. It is also consistent with the splitting methods used in all papers listed in Tables 2 and 3 (for example [1, 2, 3, 4, 5, 6, 7]), as well as the UniCorn paper you mentioned in your comment. Our strong performance on the BBBP dataset stems from the robust transfer learning capability of our approach, rather than any unfair comparison.

Regarding the code release, we have already planned for it. The code and model will be made publicly available once our paper is accepted.

We apologize for the oversight in our initial response, where we forgot to upload the updated version of the manuscript. This is why the table detailing the performance on ADMET was missing. We have now corrected this and updated the manuscript accordingly.

References:

[1] Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang. Pretraining molecular graph representation with 3d geometry. arXiv preprint arXiv:2110.07728, 2021.

[2] Zaixi Zhang, Qi Liu, Hao Wang, Chengqiang Lu, and Chee-Kong Lee. Motif-based graph selfsupervised learning for molecular property prediction. Advances in Neural Information Processing Systems, 34:15870–15882, 2021.

[3] Yin Fang, Qiang Zhang, Ningyu Zhang, Zhuo Chen, Xiang Zhuang, Xin Shao, Xiaohui Fan, and Huajun Chen. Knowledge graph-enhanced molecular contrastive learning with functional prompt. Nature Machine Intelligence, 5(5):542–553, 2023.

[4] Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. Uni-mol: A universal 3d molecular representation learning framework. The Eleventh International Conference on Learning Representations, ICLR 2023, 2023.

[5] Jun Xia, Chengshuai Zhao, Bozhen Hu, Zhangyang Gao, Cheng Tan, Yue Liu, Siyuan Li, and Stan Z Li. Mole-bert: Rethinking pre-training graph neural networks for molecules. The Eleventh International Conference on Learning Representations, ICLR 2023, 2023.

[6] Yuyang Wang, Jianren Wang, Zhonglin Cao, and Amir Barati Farimani. Molecular contrastive learning of representations via graph neural networks. Nature Machine Intelligence, 4(3):279– 287, 2022b.

[7] Alec Radford. Improving language understanding by generative pre-training. OpenAI, 2018. Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer on large-scale molecular data. Advances in neural information processing systems, 33:12559–12571, 2020.

In the paper, the authors propose methods for FG-aware tokenization and fragmentation. Functional groups are identified based on known conventional groups or potentially using domain knowledge to define new groups. However, the set of rules for identifying new functional groups appears limited, raising questions about their generalizability. How do these rules compare to frequent subgraph mining, a widely-used technique in graph mining, where common subgraphs are often predictive features for small molecules?

• The functional group detection algorithm, detailed from line 262 in the manuscript, operates by traversing the molecular graph and evaluating each atom based on a range of properties: atom type, types and bonds of neighboring atoms, number of bonded neighbors, atom charge, and the presence of bonded hydrogen atoms. This systematic approach allows the algorithm to accurately identify functional groups within molecular structures by applying specific chemical criteria. For example, the algorithm can recognize a ketone group (RCOR’) by detecting a carbon atom that has no charge and three neighbors, one of which is a double-bonded oxygen atom. This approach mirrors how chemists manually recognize functional groups based on structural characteristics, ensuring that the detected groups align with known chemical conventions.
• Regarding generalizability, this rule-based method is robust for standard functional groups and can also be adapted to detect new groups. It may be complicated when dealing with very complex groups, but this is a minor limitation since most traditional functional groups in organic chemistry are relatively simple. In fact, our method has successfully identified over 100 well-known functional groups as documented in the Wikipedia page of functional groups.
• Compared to frequent subgraph mining (FSM), which identifies patterns that occur frequently in the data, our rule-based method is more targeted. While FSM can uncover recurring subgraphs, these subgraphs do not always correspond to functional groups as defined in chemistry. In contrast, our approach directly follows chemical principles, aligning closely with how chemists define functional groups. This ensures that our method reliably detects functional groups that are chemically meaningful, providing more accurate and chemistry-specific results. We highlighted this point in the modified version of the manuscript, line 222.

The use of a functional group knowledge graph to learn functional group embeddings is innovative, but some relationship types might provide an unfair advantage over other methods. For example, including properties like water solubility or lipophilicity (logP) could yield better results on downstream tasks related to those specific properties. It would be beneficial to assess the impact of removing such information from the knowledge graph to determine if the observed improvements are primarily due to these additional properties, which are not considered in other methods.

The use of properties like water solubility or lipophilicity (logP) in our functional group knowledge graph does not provide an unfair advantage. These properties are incorporated at the functional group level, not at the molecular level. These properties are only used to determine the similarity between functional groups, not to directly predict molecular properties. This ensures that the observed improvements in downstream tasks are driven by the functional group similarities rather than specific property-based enhancements.

The use of splits in the MoleculeNet datasets is inconsistent with the original MoleculeNet recommendations. Specifically, random splits are recommended for regression tasks such as ESOL, Lipophilicity, and FreeSolv. In this paper, the authors do not consistently clarify the splits used; for example, scaffold splits are mentioned in the appendix, but captions for Tables 7 and 8 indicate random splits.

Thank you for your comment. We would like to clarify that the random splits used in Tables 7 and 8 are part of our ablation study, where we evaluate the effect of adding different components to the model. These splits are not part of the benchmarking process (we presented it more clearly in the modified manuscript, line 1158). For the benchmarking tasks, we strictly adhere to the widely accepted scaffold split with an 8:1:1 ratio, which is consistent with prior works and essential for assessing the generalizability of the models. In fact, all of the benchmark models presented in Tables 2 and 3 use the same splitting method. We hope this clarifies the misunderstanding.

A significant challenge with MoleculeNet is the absence of a leaderboard with predefined splits, leading researchers to create custom splits, and sometimes even modify the original datasets, as seen in modifications to ESOL and FreeSolv in cases like this issue. This issue may reflect the use of dataset variants such as those described in this study.

To clarify, we did not make any modifications to the original datasets (such as the "modifications to ESOL and FreeSolv" mentioned by the reviewer). We used scaffold splitting with three different random seeds, ensuring robust evaluation without manipulating the dataset itself. Testing our model across multiple random splits within MoleculeNet datasets strengthens the reliability of our performance metrics and minimizes potential dataset-specific biases. Notably, this approach aligns with the consensus in the field for benchmarking models on MoleculeNet tasks.

The existence of multiple dataset versions and split schemes makes it difficult to accurately assess improvements toward state-of-the-art (SOTA) results, as subsequent studies often cite results without clarity on splits used. For instance, in "SELF-BART: A Transformer-based Molecular Representation Model using SELFIES" (NeurIPS 2024, AI4Mat, link), the reported MoleculeNet performance is challenging to compare with this paper due to inconsistent dataset versions and splits.

Our approach aligns with standard practices in the field, where most studies—including all models listed in Tables 2 and 3—use scaffold splitting with an 8:1:1 train-validation-test ratio and perform multiple runs with different splits to ensure robust evaluation. In contrast, the SELF-BART model uses a different split scheme, making direct comparison less reliable.

Conduct additional experiments on the TDC ADMET groups, which offer leaderboards and fingerprint-based baselines. TDC ADMET provides consistent splits, making future comparisons easier.

Thank you for your valuable suggestion. We have conducted additional experiments on the TDC ADMET groups as recommended and included the results in Table 9 of the revised manuscript. These results leverage the default splits, facilitating future comparisons. We believe this addition enhances the robustness of our evaluation.

Dear Authors,

First, I would like to thank you for conducting additional experiments on the ADMET dataset and addressing my previous concerns. I appreciate the effort you’ve put into this work.

Upon reviewing the results for the ADMET dataset, I observe that while the performance on 4 of the 22 datasets is promising, the outcomes for the remaining 18 datasets fall significantly short of the state-of-the-art benchmarks reported in the corresponding leaderboards. This raises concerns about the reliability of the experimental setup.

Furthermore, regarding the experiments with MoleculeNet, I am concerned about the lack of consistency in reported results across different works, which may stem from variations in dataset splits and evaluation protocols. This inconsistency makes it challenging to validate significant improvements and assess the true performance of the proposed method.

To ensure transparency and reproducibility, I strongly recommend that the authors open-source their code and carefully address any potential sources of data leakage in the experimental setup. Doing so will greatly enhance the credibility of the results and allow for independent verification of the findings.

Based on these concerns, I would like to lower my score and suggest the authors thoroughly revise the manuscript before resubmitting it for future consideration.

Best regards,

References

[1] Jun Xia, Chengshuai Zhao, Bozhen Hu, Zhangyang Gao, Cheng Tan, Yue Liu, Siyuan Li, and Stan Z Li. Mole-bert: Rethinking pre-training graph neural networks for molecules. The Eleventh International Conference on Learning Representations, ICLR 2023, 2023.

[2] Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. ELECTRA: Pretraining text encoders as discriminators rather than generators. In ICLR, 2020.

[3] Joshua David Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. In International Conference on Learning Representations, 2021.

[4] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. 2020. Graph contrastive learning with augmentations. Advances in neural information processing systems, 33:5812– 5823.

[5] Yuyang Wang, Jianren Wang, Zhonglin Cao, and Amir Barati Farimani. Molecular contrastive learning of representations via graph neural networks. Nature Machine Intelligence, 4(3):279– 287, 2022b.

[6] Sun M, Xing J, Wang H, et al. MoCL: data-driven molecular fingerprint via knowledge-aware contrastive learning from molecular graph[C]//Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining. 2021: 3585-3594.

[7] Trouillon, T., Welbl, J., Riedel, S., Gaussier, É., & Bouchard, G. (2016). Complex embeddings for simple link prediction. ICLR.

[8] Rossi, R. A., Sardiña, S. I., & Schein, E. S. (2020). A Survey of Network Embedding Methods for Graphs and Complex Networks. ACM Computing Surveys.

[9] Zaixi Zhang, Qi Liu, Hao Wang, Chengqiang Lu, and Chee-Kong Lee. Motif-based graph selfsupervised learning for molecular property prediction. Advances in Neural Information Processing Systems, 34:15870–15882, 2021.

[10] Shen Han, Haitao Fu, Yuyang Wu, Ganglan Zhao, Zhenyu Song, Feng Huang, Zhongfei Zhang, Shichao Liu, and Wen Zhang. Himgnn: a novel hierarchical molecular graph representation learning framework for property prediction. Briefings in Bioinformatics, 24(5):bbad305, 2023.

[11] Yongqiang Chen, Quanming Yao, Juzheng Zhang, James Cheng, and Yatao Bian. Hight: Hierarchical graph tokenization for graph-language alignment. arXiv preprint arXiv:2406.14021, 2024.

[12] Nianzu Yang, Kaipeng Zeng, Qitian Wu, Xiaosong Jia, and Junchi Yan. Learning substructure invariance for out-of-distribution molecular representations. Advances in Neural Information Processing Systems, 35:12964–12978, 2022.

[13] Biaoshun Li, Mujie Lin, Tiegen Chen, and Ling Wang. Fg-bert: a generalized and self-supervised functional group-based molecular representation learning framework for properties prediction. Briefings in Bioinformatics, 24(6):bbad398, 2023.

[14] Yongqiang Chen, Quanming Yao, Juzheng Zhang, James Cheng, and Yatao Bian. Hight: Hierarchical graph tokenization for graph-language alignment. arXiv preprint arXiv:2406.14021, 2024.

[15] Yifei Wang, Shiyang Chen, Guobin Chen, Ethan Shurberg, Hang Liu, and Pengyu Hong. Motifbased graph representation learning with application to chemical molecules. In Informatics, volume 10, pp. 8. MDPI, 2023.

The novelty is also a bit limited since similar methods based on functional groups have been well-explored in previous studies [1,2,3,4,5,6].

We acknowledge that using functional group (FG) information to improve molecular representation is not a novel concept. However, our approach is distinct in its ability to both detect functional groups and effectively incorporate functional group information to enhance molecular representations, leading to strong transfer learning capabilities, as demonstrated by performance in downstream tasks. Here’s how our work differentiates itself:

• Detecting functional groups: Existing motif-based tokenization methods do not strictly align with chemical functional groups. For instance, BRICS-based approaches (e.g., [9, 10, 11, 12]) generate substructures that do not map precisely to known functional groups. Methods using RDKit [13, 14] can only detect a limited set of functional groups (typically 85 traditional ones). Unsupervised strategies [15] often struggle with complex ring structures. In contrast, our approach implements algorithms to detect and incorporate a comprehensive set of chemical functional groups, including ring-containing groups, ensuring that the tokenization process aligns strictly with chemical features. We discuss this in more detail in the revised version of our manuscript (Related Works section, line 212).
• Incorporating functional group information: We are the first to directly inject functional group information into SMILES, adding richer chemical context to SMILES, then we can effectively leverage language models (LMs) to learn molecular representations.
• Robustness in downstream tasks: Our model (FARM) demonstrates strong transfer learning capabilities - the core goal of pretrained models - outperforming other methods in 10 out of 12 tasks from the MoleculeNet benchmark. This suggests that FARM effectively learns molecular representations, leading to enhanced performance across various tasks.
• We also contribute by evaluating the diversity of large molecular datasets. Identifying functional groups and embedding this information into SMILES strings enables us to build a novel vocabulary, where each token reflects both an atom symbol and its associated functional group, serving as a measure of dataset diversity. Through this approach, we find that the ZINC dataset is significantly less diverse than ChEMBL, providing valuable insight for future work on dataset selection for foundation model training. Given the vast chemical space, it is essential for training data to represent a broad range of chemical structures. Our work is the first to conduct this type of evaluation, whereas previous studies have primarily focused on molecule length and atom types, which do not fully capture functional group diversity within datasets.

We have highlighted those points in the abstract and introduction of the revised manuscript.

The paper does not sufficiently address the computational requirements of training FARM, given the added complexity from FG-aware tokenization and knowledge graph embeddings.

To address the reviewer’s concern about the computational requirements of training FARM with FG-aware tokenization and knowledge graph embeddings, we would like to clarify that the computational resources required for FARM are not significantly higher than those for training a standard language model (in this case is BERT model). The training process follows a multi-stage approach, where each stage is computationally manageable.

Multi-stage training:

• Step 1: The pre-training of the masked language model (MLM) is similar to the standard BERT pre-training procedure. The computational cost for this step is on par with that of training BERT.
• Step 2: We construct the FG knowledge graph and train a knowledge graph embedding model, such as ComplEx [7]. Training the knowledge graph embedding model does not require additional resources compared to training a language model. Knowledge graph embedding techniques have been well established and are scalable [8].
• Step 3: Construct the FG graphs and generate negative samples, then train the link prediction model to learn the high-level structure of the molecules, specifically how the functional groups (FGs) connect with each other. This step uses standard graph neural network techniques, such as the Graph Convolutional Network (GCN) model, and does not introduce significant computational overhead.
• Step 4: Fine-tune the MLM model by combining contrastive learning with structure representations. This step leverages pre-trained LMs and focuses on integrating the structural information into FG-enhanced SMILES representation, which only requires fine-tuning a BERT model and a linear layer to project the structural representation into a new space, facilitating better alignment with the MLM model’s learned embeddings.

Extracting molecular representations for downstream tasks: Convert the SMILES to FG-enhanced SMILES and use the pre-trained BERT model to extract molecular representations. This process does not require additional computational resources.

Regarding training time, there is always a trade-off between training duration and performance. However, with only a slight increase in training time, our method achieves a substantial improvement in performance. As shown in Figure 8 (Appendix D), the loss curves for the MLM during training on two datasets—standard SMILES and FG-enhanced SMILES—demonstrate that, after 500 training steps, both losses converge to approximately the same value.

We sincerely thank the reviewer for their thoughtful feedback and for highlighting key aspects of our approach. Below, we address each of the reviewers' comments in detail. The corresponding modifications in the revised manuscript are highlighted in blue for clarity.

The increase in tokenization granularity from 93 to 14,741 tokens could be seen as excessive, leading to training inefficiencies. While the authors acknowledge this, a deeper discussion on the trade-offs and potential mitigation strategies (e.g., pre-training optimizations) would enhance the paper.

• We appreciate the reviewer’s valuable insight regarding the trade-offs associated with increased tokenization granularity. While the jump from 93 to 14,741 tokens may seem large and potentially inefficient, our findings indicate that the benefits outweigh the potential downsides.
• In MoleBERT [1], the authors observed a negative transfer issue when using standard SMILES for masked language prediction. With only 118 tokens representing common chemical elements, the task becomes overly simplistic, failing to capture deeper, transferable knowledge. Other studies [2, 3] have shown that simpler pre-training tasks can limit a model’s ability to generalize effectively to new tasks.
• On the other hand, a typical BERT model for English operates with around 30K tokens, many of which are rare, yet BERT still learns effective word representations. This suggests that BERT can handle a larger vocabulary without sacrificing performance.
• In our work, as shown in Figure 8, our FG-enhanced SMILES model with ~15K tokens achieves a loss comparable to that of models using standard SMILES with only 93 tokens, suggesting that the model can handle masked language modeling effectively with this expanded vocabulary. Moreover, Table 8 illustrates that this increased tokenization granularity significantly boosts downstream performance. Specifically, our FG-BERT model, trained on functional group-enhanced SMILES, consistently outperforms the standard SMILES-based BERT, demonstrating clear improvements in representation quality.

The absence of 3D information limits the model’s capacity to handle stereochemistry and spatial effects, which are crucial in many chemical tasks. The authors mention this as a future direction, but its exclusion remains a significant limitation.

• We appreciate the reviewer’s insight into the importance of 3D information in molecular modeling. As noted in our paper, incorporating 3D spatial features is part of our planned future work. This study focuses on advancing functional group and molecular structure awareness within a 2D framework, which is highly effective across a range of tasks. It is worth noting that even without utilizing 3D information, our method demonstrates significant improvements on downstream tasks, as demonstrated in Table 2 and 3.
• Our goal is to establish a robust foundation in 2D molecular representation, setting the stage for future enhancements with 3D spatial features. This stepwise progression aligns with standard scientific practice, where model extensions build logically upon established foundations. We kindly suggest that limitations related to future work not be used to detract from the contributions made in the present study.

The paper briefly mentions augmentations for negative samples in contrastive learning, such as node deletion and swapping. A more detailed exploration of the impact of these strategies would provide clarity on their contribution to performance.

• Thank you for your comment on the augmentation methods used for negative samples in our contrastive learning approach. We note that node deletion and swapping are widely accepted augmentations for molecular graphs [4, 5, 6], serving to generate hard negative samples with high similarity to the original graph. These hard negative samples require the model to better distinguish between positive and negative examples, enhancing the learned features by making the model work harder to pull the FG-enhanced SMILES representation closer to the positive graph while pushing it away from the hard negative sample.
• In the context of molecular graphs, these augmentations are analogous to standard image augmentations, like cropping or flipping, which are used without exhaustive justification in image-based models. While an in-depth analysis of each augmentation’s impact is beyond this paper’s scope, we align with prior work in selecting these strategies, as they are known to effectively improve model robustness and representation quality in molecular graph tasks.