Towards Fine-grained Molecular Graph-Text Pre-training

We sincerely thank you for the precious time spent reading through the paper and giving constructive suggestions. We address your main concerns as follows:

W1: As compared with existing studies, e.g., MoleculeSTM, the main difference in this paper is to consider motif-level knowledge. Then the similar framework from MoleculeSTM seems can be extended in a straightforward way. Also, motif info is widely used in existing studies (though not in the molecule-text moltimodal scenario). Therefore, the authors need to better explain the novelties of the proposed approach.

Our main innovation boils down to the exploration of motif-level knowledge in molecular graph-text pre-training. Motifs determine the molecular properties, so it is crucial to incorporate motif-level knowledge during pre-training. We have utilized some existing techniques, but we stress that no previous solution can be directly applied to learning motif-level knowledge, because the lack of supervision signals for the alignment between motifs and words. To address the issue, we first fragment molecular graphs and descriptions into tokens of the same granularity within a unified semantic space, and then we design a masked multimodal modeling task, enabling FineMolTex to learn fine-grained alignment knowledge in a self-supervised manner.

W2: Modeling both molecule-level and motif-level knowledge may increase computational costs compared to models focusing solely on molecule-level representations, and the computational complexity should be discussed.

As indicated in our paper (line 491), we have provided the comparison of pre-training and inference times are provided in Appendix D. As shown in Table 8 in the Appendix, the pre-training and inference times for FineMolTex are comparable to those of the baselines.

As for the computational complexity, while the architecture of FineMolTex, MolCA, and MoleculeSTM are different, they all include a combination of transformer layers and a graph encoder. The total computational complexity can be expressed as $O(L_1N^2d_1+L_2Ed_2)$, where $d_1$ is the dimension of the input embeddings of the words, $d_2$ is the dimension of the input embeddings of the nodes, $L_2$ is the number of the transformer layers, $L_1$ is the number of the transformer layers, $E$ is the number of the edges of the graph, $N$ is the input sequence length (number of tokens). The actual computational costs differ based on the values of parameters $L_1$, $N$, $d_1$, $L_2$, $E$, $d_2$, which vary across models due to different design choices and input representations. Here we provide the overall parameter of FineMolTex. MolCA, and MoleculeSTM. We can see that the number of parameters in FineMolTex is significantly smaller than MolCA and is on the same order of magnitude as MoleculeSTM. Meanwhile, our model achieves the best performance across various downstream tasks.

W3: The effectiveness of FineMolTex will rely on accurate and meaningful motif extraction from molecular graphs. Can BRICS algorithm used in this paper meet this requirement? or is there other motif extraction algorithms that will show a performance difference?

Most of the motifs extracted by the BRICS algorithm are meaningful; however, some motifs, such as those composed of single atoms, may lack direct chemical significance. Despite this, the masked learning mechanism employed by FineMolTex demonstrates a strong ability to associate meaningful motifs with their corresponding textual descriptions. As shown in Figure 5b of our paper, our model successfully learns fine-grained alignment for the meaningful motifs.

To extract motifs from molecules, we could fragment molecules into motifs or use RDKit to extract specific motifs within the molecules. In our work, we utilize BRICS along with a post-processing procedure [1], which has been validated to be effective. RDKit can also directly identify specific functional groups in molecules. However, such methods are limited to detecting only a predefined set of functional groups (typically dozens) and lack generality. This restricts their application in broader motif-level representation learning. In contrast, our approach aims to generalize to various types of motifs, even those not explicitly predefined, by leveraging the masked multimodal modeling pre-training task.

[1] Zaixi Zhang, Qi Liu, Hao Wang, Chengqiang Lu, and Chee-Kong Lee. Motif-based graph self-supervised learning for molecular property prediction. NeurlPS 2021

Thanks sincerely for your careful reading and thought-provoking insights. We address your main concerns as follows:

W1: The figures require refinement; in Figures 1 and 2, some highlighted areas extend beyond the dashed boxes, and the <mask> tokens overlap with the text and motifs (e.g., in Figure 2(b), did you input "carboxylic"?).

Thank you for your suggestion. The input word is indeed "carboxylic." We have carefully refined the figure. The updated figure will be included in the revised version of the paper.

W2: Retrieve tasks need more metrics like recall.

Following the testing procedure of [1], we also utilize accuracy as the evaluation metric, as we believe it is sufficient to compare the performance of the models in our tasks. As you suggested, we have included the recall@20 metric in our evaluation. The recall@20 performance on the molecule-ATC, DrugBank-Description, and DrugBank-Pharmacodynamics datasets are presented in the tables below. As shown in the tables below, FineMolTex consistently outperforms baselines.

molecule-ATC

DrugBank-Description

DrugBank-Pharmacodynamics

W3: The experiments and datasets generally follow the MoleculeSTM framework but overlook more challenging and practical text-based molecule editing tasks, such as multiple-objective property-based editing, binding-affinity-based editing, and drug relevance editing. This omission significantly undermines the claim of "a notable improvement of up to 230% in the text-based molecule editing task.

• Multiple-objective editing:

We would like to clarify that we have conducted the multiple-objective property-based editing experiments, as shown in the second line of Figure 3 in the paper. These experiments demonstrate FineMolTex's capability to handle multiple objectives simultaneously, showcasing its effectiveness in achieving complex property-based editing tasks.

• Drug-relevance editing:

As you suggected, we have conducted experiments on drug relevance editing, and the results are summarized in the table below. As observed, FineMolTex consistently outperforms other baselines, highlighting its superior performance in this task.

• Binding-affinity-based editing:

The evaluation of the binding-affinity-based editing task conducted by [1] relies on pre-trained evaluation model. Unfortunately, this code has not yet been provided. To ensure accurate and fair comparisons, we are currently in the process of obtaining the code and training details for the evaluation model. Once the code and training details are available, we will include our results on the binding-affinity-based editing task in the revised version of the paper.

Q: Qualitative results that demonstrate the effectiveness of fine-grained alignment

Supervision Signal for Fine-Grained Alignment: To the best of our knowledge, there are no supervised signals for the motif-text alignment. So we use the masked multi-modal modeling (MMM) task to provide an implicit but effective supervision signal for fine-grained motif-text alignment. Specifically, during pre-training, motifs and text tokens are masked and their labels are predicted using tokens from both modalities. This design, coupled with the cross-attention mechanism, allows the model to align motifs and corresponding text descriptions. For instance, as shown in Figure 2 of the paper, the masked motif (e.g., SO₃⁻) can be reconstructed based on the relevant textual tokens such as "propanesulfonic," thus embedding motif-text relationships into the learned representations.

Qualitative Demonstration in Text-Based Molecule Editing Task: The results of the text-based molecule editing task provide strong evidence that FineMolTex has effectively learned motif-level knowledge. In this task, the model is instructed to generate molecules with specific motifs or properties. This task requires the model to recognize motif names and understand their properties. And FineMolTex significantly outperforms all baselines. Furthermore, we have included visual qualitative examples (Figure 4 in the paper and Figure 8 in the Appendix) that illustrate how the model replaces specific motifs or modifies molecules to exhibit desired motifs or properties. These results qualitatively confirm the model's ability to capture and utilize motif-level knowledge.

Qualitative Demonstration with LIME Interpretability: Our experiment, LIME-based interpretability analyses (Figure 5b and Figure 9 in the Appendix), have provided qualitative results that demonstrate the effectiveness of the alignment of our model. Specifically, we use our model to predict masked motifs based on text tokens. We can see that text tokens which are relevant to the masked motif indeed have more contribution to the predicting task.

We would greatly appreciate it if you could specify the type of qualitative demonstration that you would find most convincing. We are more than willing to provide additional results or experiments to meet your expectations and further enhance the presentation of our work.

Thanks again for your careful review. We will diligently revise the presentation of our paper and ensure that all experimental results discussed in this rebuttal are incorporated into the revised version.

[1] Liu S, et al. Multi-modal molecule structure–text model for text-based retrieval and editing[J]. Nature Machine Intelligence 2023

Thanks sincerely for your careful reading and review. We have addressed your concerns as follows:

W1: Comparison with HIGHT

We would like to clarify that HIGHT is currently under review for ICLR 2025 and has not yet been formally published. While HIGHT introduces the concept of motif-level knowledge, it detects only 38 predefined motifs for molecules and pre-train the model to learn only names of these motifs. This limitation restricts its generality and applicability in broader motif-level representation learning. In contrast, our approach focuses on an open vocabulary. By fragmenting molecules into motifs and texts into words, and employing the masked multi-modal modeling task, FineMolTex learns fine-grained alignment in a self-supervised manner. This design enables FineMolTex to generalize to a diverse range of motifs, including those not explicitly predefined, and to acquire comprehensive knowledge about motifs, including their names and properties. For instance, as demonstrated in Figure 5(b) of our paper:

• In Figure 5(b)(1), FineMolTex successfully learns the name of the masked token "triazole."
• In Figure 5(b)(2), it captures the aromatic property of the benzene ring.
• In Figure 5(b)(3), it not only identifies the name of the complex motif "organonitrogen" but also learns that this motif is "heterocyclic."

This ability to learn both the names and properties for diverse motifs demonstrates the broader scope and flexibility of FineMolTex.

Additionally, while HIGHT includes node-level information, we would like to emphasize that the information contained within a single node is rather limited [2]. In molecular graphs, critical insights are better represented at the motif or graph level, which is the focus of our work.

W2: Concern about the architecture

Our main innovation boils down to the exploration of motif-level knowledge in molecular graph-text pre-training. Motifs determine the molecular properties, so it is crucial to incorporate motif-level knowledge during pre-training. We have utilized some existing techniques, but we stress that no previous solution can be directly applied to learning motif-level knowledge, because the lack of supervision signals for the alignment between motifs and words. To address the issue, we first fragment molecular graphs and descriptions into tokens of the same granularity within a unified semantic space, and then we design a masked multimodal modeling task, enabling FineMolTex to learn fine-grained alignment knowledge in a self-supervised manner.

W3: Concern about not utilizing the architecture of BLIP-2

We would like to clarify that the architecture of BLIP-2 cannot be directly utilized for learning fine-grained alignment in our context. BLIP-2 relies on explicit supervision signals to establish alignment relationships, such as Image-Text Matching and Image-Grounded Text Generation. However, we lack explicit supervision signals for motif-level alignment. To address this, FineMolTex fragments molecules into motifs and texts into words and employs the masked multi-modal modeling task. This approach enables FineMolTex to learn a wide variety of motif-level knowledge in a self-supervised manner, capturing both the names and properties of motifs without requiring explicit supervision.

Additionally, our experimental results demonstrate that FineMolTex significantly outperforms MolCA [2], a model based on the BLIP-2 architecture, across multiple tasks. These results highlight the effectiveness of FineMolTex for learning fine-grained representation learning.

W4: The experiments could be expanded to include more tasks, such as molecule captioning and generation.

Following the testing procedure of [1], we have already evaluated the performance of our model on three downstream tasks: graph-text retrieval, molecule property prediction, and text-based molecule editing. Notably, text-based molecule editing is a form of molecule generation, which we believe is practical in real-world applications.

The experimental results across these tasks demonstrate the effectiveness of our model in capturing fine-grained knowledge. We appreciate your suggestion and will consider expanding our experiments to include additional tasks in future work.

W5: Comparison to 3D-MoLM

Thank you for your suggestion. We have conducted experiments on the graph-text retrieval task on DrugBank-Pharmacodynamics, molecule-ATC, and DrugBank-Description. The experimental results for 3D-MoLM and other models are provided in the tables below. As shown in the tables, FineMolTex consistently outperforms 3D-MoLM, demonstrating its effectiveness in capturing fine-grained knowledge. We will carefully include these results in the revised version of our paper.

DrugBank-Pharmacodynamics

molecule-ATC

DrugBank-Description

W6: Key hyperparameter details are not provided.

As indicated in our paper (line 265), details of the pre-training process can be found in Appendix C.4, where we provide the key hyperparameter details. We will revise the corresponding parts in the paper to ensure these details are more explicitly referenced and easier to locate.

Q1: Do you train the transformer and cross-attention layers from scratch? How many layers are used?

The transformer and cross-attention layers in our model are trained from scratch. The number of cross-attention layers is set to 2, resulting in 2 rounds of cross-modal interaction. For the molecule stream, the transformer layers are set to 2 in the first round and 3 in the second round. For the text stream, the transformer layers are set to 10 in the first round and 12 in the second round.

Q2: Why did you choose to train the transformer layers rather than using a pretrained LLM and then finetuning?

In our model, the transformer layers are not used to encode texts directly but to transform molecular graphs and texts into a joint embedding space and to learn fine-grained alignment representations. This specific purpose requires training these layers from scratch, as a pre-trained LLM cannot effectively perform this task.

We do utilize a pre-trained text encoder, SciBERT, to encode the text input. Our framework is flexible, and we could replace it with a pre-trained LLM.

Q3: Given that FineMolTex does not use a pretrained LLM, how does its computational efficiency (in terms of training and inference time) and memory cost compare with models that utilize LLMs?

As indicated in our paper (line 491) the comparison of pre-training and inference times are provided in Appendix D. As shown in Table 8 in our Appendix, the pre-training and inference times for FineMolTex are comparable to those of the baselines.

We provide the overall parameter of FineMolTex. MolCA, and MoleculeSTM. We can see that the number of parameters in FineMolTex is significantly smaller than MolCA and is on the same order of magnitude as MoleculeSTM. Meanwhile, our model achieves the best performance across various downstream tasks.

[1] Liu S, et al. Multi-modal molecule structure–text model for text-based retrieval and editing[J]. Nature Machine Intelligence 2023

[2] Liu Z, et al. MolCA: Molecular graph-language modeling with cross-modal projector and uni-modal adapter. EMNLP 2023

Thanks sincerely for your careful reading and thought-provoking insights. We address your main concerns as follows:

W1: Concern about innovation.

Our main innovation boils down to the exploration of motif-level knowledge in molecular graph-text pre-training. Motifs determine the molecular properties, so it is crucial to incorporate motif-level knowledge during pre-training. We have utilized some existing techniques, but we stress that no previous solution can be directly applied to learning motif-level knowledgebecause the lack of supervision signals for the alignment between motifs and words. To address the issue, we first fragment molecular graphs and descriptions into tokens of the same granularity within a unified semantic space, and then we design a masked multimodal modeling task, enabling FineMolTex to learn fine-grained alignment knowledge in a self-supervised manner.

W2: Concern about testing on PubChem dataset

Indeed, MoleculeSTM [2] was not tested on the PubChem dataset. Instead, [2] evaluated the model's performance on the DrugBank database, which provides valuable opportunities for exploring drug discovery tasks. And we follow their testing procedure. Specifically, three fields from DrugBank were utilized: the Description field, the Pharmacodynamics field, and the Anatomical Therapeutic Chemical (ATC) field. We believe that assessing model performance on datasets directly relevant to drug discovery offers more meaningful insights than testing solely on the PubChem dataset.

W3: Concern about testing on property prediction task.

Previous multimodal molecular graph learning methods [2, 3, 4] have consistently evaluated their models on molecular property prediction tasks. Following their work, we include this evaluation to ensure comparability and to demonstrate that FineMolTex learns better representations for molecules.

Q1: Combining a global representation with fine-grained representations.

Indeed, we use the same graph encoder, text encoder, and transformer layer for learn fine-grained representation and global representation. Thus the global representation inherently captures fine-grained knowledge as well. As you suggested, we test whether combining molecule representations with motif representations would improve performance. The results on the molecule-ATC dataset are shown in the table below. As observed, this combination leads to performance degradation. This may be because some motifs, such as single atoms, have little intrinsic meaning. These motifs are better suited to being part of the molecule representation, as considering them separately can introduce redundant information.

Q2: Does using fine-grained representation of motifs improve the explanability of your method?

Yes, using fine-grained representations of motifs improves the explainability of our method. In Section 4.4, "Predictions Based on Fine-Grained Knowledge," we predict the labels of masked motifs using text tokens and employ LIME [5] to quantify the significance of each text token in predicting these motifs. As shown in Figure 5b of our paper, text tokens with higher interpretive weights play a crucial role in the predictions. Importantly, we observe that these tokens are indeed relevant to the masked motifs, demonstrating the enhanced explainability provided by the fine-grained representations.

[1] Understanding the Limitations of Deep Models for Molecular property prediction: Insights and Solutions. In NeurIPS 2023.

[2] Liu S, et al. Multi-modal molecule structure–text model for text-based retrieval and editing[J]. Nature Machine Intelligence 2023

[3] Liu Z, et al. MolCA: Molecular graph-language modeling with cross-modal projector and uni-modal adapter. EMNLP 2023

[4] Su B, et al. A molecular multimodal foundation model associating molecule graphs with natural language, arXiv 2022

[5] Ribeiro M T, et al. " Why should I trust you?" Explaining the predictions of any classifier. KDD 2016