Data-Efficient Molecular Generation with Hierarchical Textual Inversion

Dear Reviewer 9Hsy,

We sincerely appreciate your efforts in reviewing our manuscript. We respond to each comment in the following content. We carefully incorporated the discussions into the revised manuscript. We highlighted the revised contents in "$\text{\color{blue}blue}$" for your convenience to check.

[W1] It is worth mentioning some recent works on molecule generation in related works [1, 2, 3].

We appreciate your comment to improve our manuscript. Following your suggestion, we mentioned the related works [1, 2, 3] in Section 2 of our revised manuscript.

[W2] Could HI-Mol be adapted for 3D molecule generation by leveraging the structural information?

We strongly believe that our framework can be adapted for 3D molecular generation by leveraging the 3D structural information of molecules. Although we use the state-of-the-art text-to-molecule model (MolT5), which utilizes the SMILES string as the molecule representation, it is certainly possible to model the 'molecule' part of a text-to-molecule model as 3D molecule (with a recently proposed 3D molecular generative model [1] as a decoder architecture) without any methodological modifications of our method.

[W3] How to effectively interpret the learned tokens? Do they have chemical meanings?

Learned tokens have a chemical meaning (i.e., hierarchical features of molecules). For example, detail tokens represent features of a single molecule in the dataset. These learned tokens are effectively interpreted through visualization. For example, in Figure 2 of the manuscript, one can see that certain intermediate tokens represent molecules with chemical substructures such as long carbon chains or sulfonyl groups.

[1] Hoogeboom et al., Equivariant Diffusion for Molecule Generation in 3D, ICML 2022.
[2] Zhang et al., An Equivariant Generative Framework for Molecular Graph-structure Co-design, Chemical Science 2023.
[3] Flam-Shephered et al., Language Models Can Learn Complex Molecular Distribution, Nature Communications 2022.

Dear Reviewer 9Hsy,

Thank you very much again for your time and efforts in reviewing our paper.

We kindly remind that we have only three days or so in the discussion period.

We just wonder whether there is any further concern and hope to have a chance to respond before the discussion phase ends.

Many thanks,
Authors

Dear Reviewer Mfay,

We sincerely appreciate your efforts in reviewing our manuscript. We respond to each comment in the following content. We carefully incorporated the discussions into the revised manuscript. We highlighted the revised contents in "$\text{\color{blue}blue}$" for your convenience to check.

[W1] Clear motivation figure to demonstrate highly complicated and structured nature of molecules.

Thank you for the comment to improve the clarity of our manuscript. Following your suggestion, we added Figure 2 as a motivation figure in the introduction section of our revised manuscript.

[W2] The molecular language model part in the Section 3.2 should be moved to the Related Work section.

We appreciate your suggestion. Following your suggestion, we moved the Molecular language model part to the Related Work section in our revised manuscript.

[W3] Results of HI-Mol without grammar.

Thank you for the suggestion. We moved the QM9 results of HI-Mol without grammar to Table 3 (which is originally placed in Table 15 of Appendix G), in our revised manuscript.

[W4] In Table 6, why Valid decreases as the token hierarchical level increases? How to resolve it?

As the token hierarchy increases, Validity decreases because the model interpolates more levels of embeddings to generate more diverse molecules (measured by Unique. and Novelty), i.e., there is a trade-off between the diversity of generated molecules and their validity. However, we emphasize that an invalid molecule can be easily converted to a valid molecule via a simple modification algorithm (in Appendix H), which actually follows our desired distribution (see improvement in FCD score in the table below).

\begin{array}{l|c|ccccc} \hline \text{QM9} &\text{Grammar} & \text{FCD $\downarrow$} & \text{NSPDK $\downarrow$} & \text{Valid. $\uparrow$} & \text{Unique. $\uparrow$} & \text{Novelty $\uparrow$} \newline \hline \text{HI-Mol (Ours)} & \text{X} & 0.434 & 0.001 & 90.7 & 75.8 & 73.5 \newline \text{HI-Mol (Ours)} & \text{O} & 0.430 & 0.001 & 100 & 76.1 & 75.6 \newline \hline \end{array}

[W5] Rearrangement of the positions of the figures and tables.

Thank you for the comment to improve our manuscript. Following your suggestion, we carefully rearranged the positions of the figures and tables in our revised manuscript.

[Q1] What is ‘simple resampling strategy’ in Section 4.2?

Thank you for the opportunity to clarify this point. The ‘simple resampling strategy’ means that we ignore the generated sample when it is (1) invalid, (2) identical to an already generated sample, or (3) identical to a sample in the training dataset. By applying this strategy, we obtain a set of generated samples whose Valid., Unique., and Novelty scores are all 100 (note that sampling costs only 1.8 sec per molecule). We consider this scenario to align the Valid., Unique., and Novelty scores with some methods (e.g., GDSS) for a fair comparison in the FCD score. Even in this scenario, HI-Mol significantly outperforms those methods.

Dear Reviewer Mfay,

Thank you very much again for your time and efforts in reviewing our paper.

We kindly remind that we have only three days or so in the discussion period.

We just wonder whether there is any further concern and hope to have a chance to respond before the discussion phase ends.

Many thanks,
Authors

Dear Reviewer 89Qo,

We sincerely appreciate your efforts in reviewing our manuscript. We respond to each comment in the following content. We carefully incorporated the discussions into the revised manuscript. We highlighted the revised contents in "$\text{\color{blue}blue}$" for your convenience to check.

[W1] Novelty. Multi-level molecule representation and embedding interpolation are well-established.

We politely disagree with the reviewer’s concern; although the high-level concepts of multi-level representation and embedding interpolation have been used for various purposes and domains in the literature, their usage for our own purpose could be novel and not straightforward. Specifically, we strongly believe that our novelty lies in (1) introducing the textual inversion for molecular generation and (2) advancing the existing textual inversion technique (which originally uses a single shared token in the image domain) by introducing multi-level tokens. Here, (2) is motivated by our new finding that a naïve adoption of textual inversion using a single shared token does not perform well in molecular generation. To alleviate this issue, we introduce a new concept, namely multi-level tokens, into the textual inversion framework to capture the complex distribution of molecules. Furthermore, we carefully design the sampling strategy based on the interpolation, since it is not straightforward to sample molecules from our newly proposed multi-level token embeddings. Therefore, our method is not a straightforward integration of previous techniques, but a novel framework for successfully adapting the textual inversion to molecular generation by introducing multi-level tokens into the textual inversion framework.

[W2] Rationale for adopting textual inversion appears somewhat nebulous. Graph representations are generally more adept at modeling molecular structures.

For clarification, we use SMILES strings as the representation of molecules because our method is built upon the state-of-the-art text-to-molecule model, MolT5 [1], that utilizes SMILES strings, not graphs. On the other hand, if there exist graph-based text-to-molecule models, our method is also applicable to them (then, we would use graphs as the representation of molecules). Namely, our method is agnostic to the underlying molecule representation of the foundation models, and we do not argue for the superiority of the molecule representation between SMILES strings and molecular graphs.

[W3] Absence of baselines based on large-scale text-to-molecule model.

Although our framework is built upon large-scale text-to-molecule models, they are not meaningful baselines. This is because it is not feasible to prompt all the training molecules to the mentioned large-scale text-to-molecule models due to the maximum token length of the input prompt. Specifically, while the maximum token length of the text-to-molecule model is only 1024 and 512 for [1] and [2], respectively, we have thousands of molecules and each SMILES representation of a molecule can sometimes consist of more than 100 tokens. Nevertheless, one can utilize the mentioned text-to-molecule models [1, 2] by providing the task description only (without providing the training molecules). For your information, the table below shows that our method indeed significantly outperforms such a naïve prompting of the text-to-molecule models on the HIV dataset. This confirms that our improved performance is not just due to the utilization of the large-scale text-to-molecule model, but to our carefully designed textual inversion framework. We added these respects of discussion and the experimental details in Appendix F of our revised manuscript.

\begin{array}{l|cccccc} \hline \text{HIV} & \text{Active. $\uparrow$} & \text{FCD $\downarrow$} & \text{NSPDK $\downarrow$} & \text{Valid. $\uparrow$} & \text{Unique. $\uparrow$} & \text{Novelty $\uparrow$} \newline \hline \text{MolT5-Large-Caption2Smiles [1]} & 0.0 & 60.6 & 0.196 & 100 & 14.6 & 100 \newline \text{Text+Chem T5-augm-base [2]} & 0.0 & 62.2 & 0.188 & 100 & 90.2 & 100 \newline \hline HI-Mol (Ours) & 11.4 & 16.6 & 0.019 & 100 & 95.6 & 100 \newline \hline \end{array}

[1] Edwards et al., Translation between Molecules and Natural Language, EMNLP 2022
[2] Christofidellis et al., Unifying Molecular and Textual Representations via Multi-task Language Modeling, ICML 2023

Dear Reviewer 89Qo,

Thank you very much again for your time and efforts in reviewing our paper.

We kindly remind that we have only three days or so in the discussion period.

We just wonder whether there is any further concern and hope to have a chance to respond before the discussion phase ends.

Many thanks,
Authors

Dear Authors, I express my gratitude to the authors for their work. However, I still have some questions about certain aspects.

1. In the generation of the Hi-Mol method, are active molecules used as references? If so, have other generative models, including large text-to-molecule models, been fine-tuned based on active molecules?

2. I still think SMILES alone is not enough. Many graph-based generative models can also perform molecular optimization tasks. Examples of such models include Graph-GA, DST, RetMol, etc.

Dear Reviewer 5Up6,

We sincerely appreciate your efforts in reviewing our manuscript. We respond to each comment in the following content. We carefully incorporated the discussions into the revised manuscript. We highlighted the revised contents in "$\text{\color{blue}blue}$" for your convenience to check.

[W1] Clear description of method, e.g., “selected” in Eq. (1).

Thank you for the comment to improve the quality of our manuscript. For clarification, to learn the cluster-wise features of a molecular dataset, we propose a strategy to assign an appropriate cluster index to each molecule. Specifically, given a molecule, we assign (i.e., “select’’) the intermediate token $[I_k]$ where $k$ minimizes the reconstruction loss among the cluster indices $[1, K]$. This strategy allows us to cluster similar molecules (see Figure 2) in an unsupervised manner. We carefully revised the manuscript to provide a clear description of our method.

[W2] Figure 1 is not expressive enough to outline the method.

Thank you for the suggestion. Following your comment, we revised Figure 1 to better outline our method.

[Q1] Performance of the frozen Caption2Smiles model?

Following your suggestion, we provide the generation results of the frozen Caption2Smiles model [1] on the HIV dataset, where we utilize the task description of the HIV dataset as the text prompt. The table below shows that our method indeed significantly outperforms naïvely prompting frozen Caption2Smiles model, which implies that our proposed framework is essential in achieving the superior performance. We added these respects of discussion and the experimental details in Appendix F of our revised manuscript.

\begin{array}{l|cccccc} \hline \text{HIV} & \text{Active. $\uparrow$} & \text{FCD $\downarrow$} & \text{NSPDK $\downarrow$} & \text{Valid. $\uparrow$} & \text{Unique. $\uparrow$} & \text{Novelty $\uparrow$} \newline \hline \text{MolT5-Large-Caption2Smiles [1]} & 0.0 & 60.6 & 0.196 & 100 & 14.6 & 100 \newline \hline HI-Mol (Ours) & 11.4 & 16.6 & 0.019 & 100 & 95.6 & 100 \newline \hline \end{array}

[Q2] In Table 10, there is no benefit in increasing $K$ more than 10. Do the authors have a hypothesis?

Thank you for the insightful comment. We kindly remark that some previous works on molecular clustering have shown that a thousand of molecules can fall into $\sim$10 clusters [2], which could be a supporting hypothesis on our finding $K=$ 10. While the number of “clusters’’ of molecules can be defined in various ways, we also hypothesize that choosing a relatively small number of clusters (e.g., $K=$ 10) is sufficient in our framework. This is because the sharing of the coarse-grained common features (i.e., intermediate tokens) serves to regularize the fine-grained features (i.e., detail tokens) which are biased toward a single molecule in the embedding interpolation-based sampling. To verify our hypothesis, we conduct an additional experiment, where we replace the intermediate tokens with additional detail tokens for each molecule (i.e., $K=$ 2,113). The reported result below show that the overall performance is rather degraded with $K=$ 2,113 compared to $K=$ 10, 20, and 30. This verifies our hypothesis and justifies our choice of the number of clusters $K$ in our experiments. We added these respects of discussion in Appendix E of our revised manuscript.

\begin{array}{l|ccccc} \hline \text{$K$} & \text{FCD $\downarrow$} & \text{NSPDK $\downarrow$} & \text{Valid. $\uparrow$} & \text{Unique. $\uparrow$} & \text{Novelty $\uparrow$} \newline \hline 10 & 0.434 & 0.001 & 90.7 & 75.8 & 73.5 \newline 20 & 0.430 & 0.001 & 87.9 &77.3 & 73.8 \newline 30 & 0.436 & 0.001 & 88.9 & 77.2 & 73.9 \newline \text{2,113} & 0.443 & 0.001 & 86.2 & 75.4 & 72.6 \newline \hline \end{array}

[Q3] Could graph embeddings or vision embeddings of the molecules provide any benefit?

Thank you for your insightful comment for our future research direction. We strongly believe that graph embeddings or vision embeddings (such as the suggested GPT-4 vision examples) can be incorporated to improve the controllability of molecular generation of our framework, given some observations in the text-to-image domain that explore such a multi-modal approach to improve the controllability of image generation [3].

[Q4] In Table 15, what are the results at the 50% and 100% ratio?

Thank you for the suggestion to improve our manuscript. First, we would like to emphasize that our target problem is molecular generation in the low-data regime, and our method already outperforms state-of-the-art methods with only 2% of training data of the QM9 dataset. For this reason, we thought that the results using 50% and 100% training data are not that meaningful to show the superiority of our method. Nevertheless, for your information, we are currently running the experiments with 50% and 100% QM9 dataset. We will definitely include these results in our final draft (we hope your understanding that it is costly to train with such a large number of molecules in our local machine).

[1] Edwards et al., Translation between Molecules and Natural Language, EMNLP 2022
[2] Hariharan et al., MultiMCS: A Fast Algorithm for the Maximum Common Substructure Problem on Multiple Molecules, JCIM 2011
[3] Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models, CVPR 2022

Dear Reviewer 5Up6,

Thank you very much again for your time and efforts in reviewing our paper.

We kindly remind that we have only three days or so in the discussion period.

We just wonder whether there is any further concern and hope to have a chance to respond before the discussion phase ends.

Many thanks,
Authors