LLaMo: Large Language Model-based Molecular Graph Assistant

[W1] Evaluation of GPT-generated quality.

Thank you for your constructive feedback. During the generation, we implement a multi-step assessment process, detailed below:

Step 1: We prompt GPT-4 to generate multi-turn conversation instruction data about molecules using captions and IUPAC names from a well-established dataset, PubChem, without any demonstrations (zero-shot).

Assessment 1: We sampled over 100 subsets of data and observed that GPT-4 frequently generated incomplete conversations and refused generation.

Step 2: To address these issues, we first sample high-quality demonstrations from a small set of complete conversations generated by GPT-4 (zero-shot). Subsequently, we prompt GPT-4 to generate data with these demonstrations.

Assessment 2: We sampled 500 subsets generated via in-context learning. We found that conversations with a higher number of turns were more prone to generating incomplete and inaccurate outputs.

Step 3: We filter out incomplete conversations and those with many turns. Approximately 5% of the data was filtered out.

Assessment 3: We sampled 500 subsets from the filtered data and manually assessed their quality. We verified that the generated data contained accurate information about the given molecule, with no issues of incompleteness.

In addition, our ablation studies (Table 5) validate the effectiveness of GPT-4 generated instruction dataset. Instruction tuning with this data improves the performance of LLaMo, providing the model with more detailed and instruction-following guidance.

We believe that our data is high-quality and will make our instruction data publicly available for future research in the molecular domain.

[Q1] Why is the multi-level projector set as trainable during both stages?

We set the projector as trainable both stage 1 and stage 2 to follow existing Large Vision-Language Models such as LLaVA [1].

[Limitations1] Other tasks.

Thank you for the suggestion. We conduct additional experiments on the forward reaction prediction task to evaluate LLaMo’s ability in other spectrum of molecular tasks, as suggested. The forward reaction prediction is the task that targets the prediction of chemical reactions given the reactants. We utilize the forward reaction prediction dataset from Mol-Instructions. The experimental results are in the table below. From the table, our LLaMo shows the best performance compared to other models in all metrics.

[1] Liu, Haotian, et al. "Visual instruction tuning." NeurIPS 2023.

We sincerely appreciate the reviewer's comments and positive score.

As the reviewer suggested, we will include the discussion on the quality of GPT-generated data in the final version.

[W1] Details of functional group representations.

We appreciate your feedback and provide a comprehensive explanation regarding functional representations. We use simple hand-crafted features derived from the molecular graph as functional group information.

To construct functional group representations, we initially identify functional groups from the given molecular graph following [1]. We define three types of substructures (rings, non-cyclic FGs, and carbon-carbon single bonds) as functional groups.

Then, we vectorize the main characteristics of each functional group. Specifically, we one-hot encode the number of important atoms (e.g., carbon, oxygen, etc) and bonds (single, double, triple, aromatic bonds) contained in each functional group to represent the functional group. Each functional group is represented as $\mathbf{z}_{\text{FG},i}$, a vector encoding of its main characteristics.

Finally, the functional group representations $\mathbf{Z}\_{\text{FG}}$ are constructed by concatenating all individual functional group representations $\mathbf{z}\_{\text{FG}, i}$, which is formulated as $\mathbf{Z}\_{\text{FG}} = \left[\mathbf{z}\_{\text{FG},0}, \dots, \mathbf{z}\_{\text{FG},M}\right],$ where $M$ indicates the number of the functional groups in the given molecular graph.

We hope that this explanation clarifies the details of the functional groups. We will include these details in our camera-ready version if the paper gets accepted.

[W2] How is the graph modality treated in the instruction tuning phase?

As mentioned in Section 3.2, in the instruction tuning phase, the input molecular graph is first represented with a sequence of graph tokens and then the sequence of graph tokens is augmented into the text token inputs. In this phase, we train both multi-level graph projector and LLM, while keeping the graph encoder frozen.

[W3] Why are the motif and graph-level representations modeled separately?

Graph-level representations $\hat{\mathbf{P}}^{(0)}, \dots \hat{\mathbf{P}}^{(L)}$ are constructed based on outputs of each layer of GNNs, whereas motif-level representations $\hat{\mathbf{P}}^{(\text{motif})}$ are constructed based on functional group representations $\mathbf{Z}_{\text{FG}}$ as detailed in [W1] Details of functional groups. Due to these distinct construction methods, we model them separately.

[W4] Comparisons with vision-language model pipelines.

Compared to the Large Vision-Language Models (LVLMs) such as LLaVA [2], Instruct-BLIP [3], and mPLUG-owl [4], etc., our proposed LLaMo has several unique components tailored to the molecule domain, as summarized below:

• Multi-level graph projector: While most LVLMs usually construct visual tokens solely based on the outputs from the final layer of a visual encoder, our multi-level graph projector leverages node representations from all layers of a GNN to construct molecular graph tokens. This method enables the tokens to encapsulate richer information, reflecting the molecular graph structure at multiple levels of abstraction. This multi-level approach has been shown to improve the performance in various molecular graph-related tasks.
• Molecular graph specialized instruction data: We also present molecular graph specialized multi-turn instruction data automatically generated by GPT-4. This specialized multi-turn instruction data aims to improve the performance of the model in tasks related to molecular graphs by providing detailed and more contextually relevant examples. We believe that this data contributes to future research concerning the molecule domain.
• Functional groups (Motifs): LLaMo integrates molecule-specialized functional group information. The functional groups are statistically important subgraph patterns within a molecule exhibiting consistent chemical behaviors across different compounds. By incorporating the functional groups, our LLaMo has shown the best performance in Table 2 and 3. We have also demonstrated the effectiveness of the functional groups in Table 4.

We hope that this summarization clarifies the uniqueness of our work compared to the existing LVLM works.

[1] Ji, Zewei, et al. "ReLMole: Molecular representation learning based on two-level graph similarities." Journal of Chemical Information and Modeling 2022.

[2] Liu, Haotian, et al. "Visual instruction tuning." NeurIPS 2023.

[3] Dai, Wenliang, et al. "Instructblip: Towards general-purpose vision-language models with instruction tuning." NeurIPS 2023.

[4] Ye, Qinghao, et al. "mplug-owl: Modularization empowers large language models with multimodality." arXiv 2023.

[W1] Discussion of how multi-level graph projector differs from JKNet and Graph Transformers.

We appreciate your constructive comment. Our multi-level graph projector differs from JKNet and Graph Transformers (GTs) in two aspects: main purpose and architecture.

Main purpose

As shown in Figure 1, our multi-level graph projector functions as a projector (yellow box) that converts the graph encoder’s output representations into graph tokens. In contrast, both JKNet and GTs are designed as graph encoders (green box). Thus, our multi-level graph projector is orthogonal to these encoders and can be used in conjunction with them.

Architecture

• JKNet

Our projector encodes each graph level into separate tokens, while JKNet aggregates all GNN layers into a single representation. The separated level-aware tokens allow the attention module in the LLM to softly select the appropriate graph level, leading to better performance (Table 4, Multi-level graph projector (MGProj) v.s. JKNet with MLP projector (MLP w/ concat)) and interpretability (Figure 4).

• GTs

Our projector does not modify the GNN architecture, unlike GTs, which adopt self-attention operations instead of GNNs.

We will include this discussion in the camera-ready version if the paper gets accepted.

[W2] Would the use of functional groups as prior information cause information leakage?

We use simple hand-crafted features derived from the molecular graph as functional group information to minimize the potential risk of information leakage. They are not tailored to specific datasets or tasks. Specifically, we represent the functional group through one-hot vector encoding the number of key atoms (e.g., carbon) and bonds (e.g., single bond) in each substructure (e.g., rings). This representation relies solely on the inherent structure of the molecular graph.

Also, we check the performance improvement by the functional groups in our ablation study of the functional group (Table 4). The table indicates that our LLaMo without motif (functional group) still shows good performance (-0.6 BLEU, +0.2 METEOR). This suggests that the performance improvements are primarily attributed to the architecture and learning scheme of LLaMo, rather than the use of functional group information.

[W3] Quality of GPT-4 generated samples.

Thank you for your constructive feedback. During the generation, we implement a multi-step assessment process, detailed below:

Step 1: We prompt GPT-4 to generate multi-turn conversation instruction data about molecules using captions and IUPAC names from a well-established dataset, PubChem, without any demonstrations (zero-shot).

Assessment 1: We sampled over 100 subsets of data and observed that GPT-4 frequently generated incomplete conversations and refused generation.

Step 2: To address these issues, we first sample high-quality demonstrations from a small set of complete conversations generated by GPT-4 (zero-shot). Subsequently, we prompt GPT-4 to generate data with these demonstrations.

Assessment 2: We sampled 500 subsets generated via in-context learning. We found that conversations with a higher number of turns were more prone to generating incomplete and inaccurate outputs.

Step 3: We filter out incomplete conversations and those with many turns. Approximately 5% of the data was filtered out.

Assessment 3: We sampled 500 subsets from the filtered data and manually assessed their quality. We verified that the generated data contained accurate information about the given molecule, with no issues of incompleteness.

In addition, our ablation studies (Table 5) validate the effectiveness of GPT-4 generated instruction dataset. Instruction tuning with this data improves the performance of LLaMo, providing the model with more detailed and instruction-following guidance.

We believe that our data is high-quality and will make our instruction data publicly available.

[Q1] Performance of GPT-4 with prior information.

Good question. We test GPT-4's performance by adding the molecule's functional group information (FG info.) to the input prompts. We use the same FG info. used in our model. The results (below table) show that adding FG info. does not consistently improve GPT-4’s performance, which means that it is not always helpful on the molecular description task.

[Q2] Purpose of using different LLM on Table 2 and Table 3. Could better LLM result in better performance?

Table 2 and Table 3 report the performance of generalist and specialist models, respectively. The best-performing baseline for the generalist model, Mol-Instruction, uses LLaMa2 as its base model, while the best baseline for the specialist model, MolCA uses Galactica-1.3B. Thus, for a fair comparison with the established baselines, we employ different LLMs in each table.

To study whether a better LLM improves performance, we conduct additional experiments with LLaMo using LLaMa and LLaMa2 under the same setting as in Table 2. The result (below table) shows that LLaMa2 achieves a 6.9-point improvement on the BLEU metric compared to LLaMa, meaning that a better LLM results in better performance.

[Q3] SMILES edit tasks.

Thank you for the suggestion. We conduct additional experiments on the forward reaction prediction task to show LLaMo’s capability in editing SMILES as suggested. We use the forward reaction prediction dataset from Mol-Instructions. The results (below table) indicate that our LLaMo consistently outperforms other models in all metrics.

We sincerely appreciate the reviewer's positive comments and increased rating.

As the reviewer suggested, we will include a discussion on over-smoothing, the quality of GPT-generated data, and other tasks in the final version.

[W1] More performance comparison based on different GNN and LLM backbones.

Good question. We conduct additional experiments to evaluate performance based on different GNN and LLM backbones. Specifically, we compare pretrained Graph Convolutional Networks (GCNs) with our base GNN backbone (GIN) and use LLaMa-7B as an alternative LLM backbone, as suggested. We use the same experimental setting for reporting Table 2. The experimental results for molecule description generation are reported in the table below.

• Different base GNN (GCN v.s. GIN (ours))

  The table demonstrates that the pre-trained GIN shows better performance than the pre-trained GCN, with a 4.8 improvement in the BLEU metric for the molecule description generation task. This result shows the superior expressivity of the GIN model, which aligns with the result in Table 1 of [1].

• Different base LLM (LLaMa-7B v.s. LLaMa2-7B (ours))

  The table shows that the LLaMa2-7B achieves a 6.9 performance improvement in the BLEU metric compared to LLaMa-7B. This indicates that a more powerful LLM enhances the performance of our LLaMo.

[W2] Efficiency of multi-level graph projector.

Good point. Since we do not increase the number of graph tokens and GNN layers, our multi-level graph projector maintains efficiency comparable to other projectors. To showcase it, we measure the inference time (second per generated token) of the model with various projectors, including our multi-level graph projector in the table below. From the table, our MGProj has similar efficiency to other projectors. Interestingly, the inference time difference between the setups without graphs and with graphs is minimal. We think that this is because the number of input text tokens is 108.87 in average, which is significantly larger than the number of graph tokens (32).

[Q1] Details of learnable prompts (L138-139).

The learnable prompts means learnable tokens $\mathbf{P}^{(l)} = \left[\mathbf{p}_1^{(l)}, \dots, \mathbf{p}_b^{(l)} \right]$ in L138. We initialize them with values drawn from the normal distribution.

[Q2] Details of choosing the few-shot examples.

As mentioned in the Section F of the supplement (L885-890), we select the exemplars from the train split of each dataset, based on their similarity to the target molecule. To calculate the similarity, we use the Tanimoto similarity metric, a widely-used metric for comparing molecular structures. We select the four molecules with the highest similarity scores to the target molecule.

[1] Hu, Weihua, et al. "Strategies for pre-training graph neural networks." ICLR 2020.