Can LLMs Generate Diverse Molecules? Towards Alignment with Structural Diversity

Q2. Have the authors considered the randomness of SMILES strings in measuring metrics? What effect does the randomness of SMILES have on the stability of method or evaluation?

To remove the randomness stemming from the non-uniqueness of SMILES, i.e., two distinct SMILES strings represent an identical molecule, we applied canonicalization [3]: converts SMILES strings representing an identical molecule into a unique SMILES string. We clarify this in Line 315 of our updated manuscript.

---

[1] Xie et al., How Much Space Has Been Explored? Measuring the Chemical Space Covered by Databases and Machine-Generated Molecules, ICLR 2023

[2] Hagberg et al., Exploring network structure, dynamics, and function using NetworkX, 2008

[3] Weininger et al., SMILES. 2. Algorithm for generation of unique SMILES notation, Journal of chemical information and computer sciences 1989

Dear reviewer xfWi,

We express our deep appreciation for your time and insightful comments. In our updated manuscript, we highlight the changes in $\color{blue}{\text{blue}}$.

In what follows, we address your comments one by one.

---

W1. The paper lacks discussion about relevant RL-based methods for diversity in molecular generation.

Thank you for the valuable suggestion! In Appendix A.1 of our updated manuscript (also in third footnote in Page 3), we incorporate discussions on prior RL methods for generating diverse molecules and describe how our work differs from them: they learn a model focused on a fixed target molecular property, whereas our work fine-tunes LLMs to generate diverse molecules given a prompt flexibly describing various target properties.

---

W2. Applying and evaluating diverse molecular generation on ChEBI-20 datasets may not suitable as a molecular description can be corresponded to a single molecule.

To address this comment, we would like to re-emphasize that the diverse molecular generation aims to increase the probability of the target molecule being included in the generated set of molecules by exploring a wide chemical space. Even for solving problems with one ground truth answer, generating diverse candidates increase probability of including the answer and being informative.

---

W3. It would be beneficial to introduce coefficient terms for description-matching and diversity rewards.

Thanks you for your suggestion! In our updated manuscript, We incorporate the coefficient terms $\lambda_{\text{match}}$ and $\lambda_{\text{div}}$ for description-matching and diversity rewards, respectively.

---

W4. In Figure 5, it seems that NCircles with $h=0.65$ and $h=0.75$ metrics do not follow the original metric definition where values at a higher $h$ should logically be lower.

First, we would like to clarify that our NCircles metrics follow the original definition of NCircles metric and are logically correct, but we found that the description of NCircles in our paper is insufficient. We have modified the description of NCircles in our updated manuscript with the following contents.

Given a set of molecules $\mathcal{M}$, the NCircles metric computes the size of the largest subset of molecules in which no two molecules are similar to each other. Specifically, it is defined with a Tanimoto similarity $T(\cdot,\cdot)$ and similarity threshold $h$ as follows:

$

\text{NCircles}_{h}=\max _{\mathcal{C}\subseteq\mathcal{M}} |\mathcal{C}| \quad \text{s.t. }T(x,y)<h, \forall x \neq y \in \mathcal{C},

$

where $\mathcal{C}$ is a subset of molecules. Every pair of molecules in $\mathcal{C}$ should have a similarity lower than $h$. Consequently, the value of NCircles metric is increased with respect to increasing $h$ as the condition becomes loose, e.g., $\mathcal{C}=\mathcal{M}$ when $h>1$.

Additionaly, we would like to clarify that this is identical to the definition of NCircles metric in the original paper [1], which is defined with a Tanimoto distance $d(x,y)=1-T(x,y)$:

$

\text{NCircles} ^{\text{dist}} _{t} = \max _{\mathcal{C}\subseteq\mathcal{M}} |\mathcal{C}| \quad \text{s.t. } d(x,y)>t, \forall x \neq y \in \mathcal{C},

$

where $t$ is a distance threshold and $\text{NCircles} ^{\text{dist}} _{t}=\text{NCircles} _{1-h}$.

---

Q1. Is it reasonable to represent each cluster as a circle in Figure 6? I wonder if this visualization accurately reflects clustering in chemical space.

We would like to clarify that the circle in Figure 6 is just a visualization purpose for explaining the NCircles in 2-D space. Therefore, this visualization does not accurately reflect clustering in chemical space. Specifically, we draw Figure 6 with the Fruchterman-Reingold force-directed algorithm used in the spring_layout of the NetworkX package [2], where the edge weight is defined with the Tanimoto similarity $T(x,y)$.

Thank you for the response! We would like to clarify the following points to address your comments.

---

W2-1. Diversity is not always an appropriate objective as similarity to a reference molecule can be significant rather than diversity, e.g., a molecular optimization task given a reference molecule.

First, we would like to clarify that we have not only considered the diversity but also considered the quality with $r_ {\text{match}}$ which can be defined with the similarity to the reference molecule in the molecular optimization tasks you mentioned. In these tasks, we would like to clarify that both similarity to a reference molecule and diversity are still important, as focusing only on similarity may generate multiple molecules similar to the reference but identical to other generated molecules, i.e., the set of generated molecules may not be informative.

Additionally, in the experiments on the ChEBI-20 dataset, we have already considered a setting similar to the settings mentioned above, where $r_ {\text{match}}$ is computed with the similarity to the target molecule satisfying the given description (as described in Equation 4 of Appendix B.2) to generate diverse molecules similar to the target molecule. We also have considered this in the evaluation (see response of W2-2).

Besides, there exist plenty of significant molecular optimization tasks where diversity is useful, e.g., drug diversification [1,2]. We firmly believe our algorithm will be useful in these scenarios.

---

W2-2. As MolT5 and BioT5$^{+}$ are designed to generate a single desirable, there are concerns about (1) mismatch between the baselines objectives and your proposed evaluation which prioritizes diversity and (2) experimental validity in comparing your method with MolT5 and BioT5$^{+}$ do not consider the diversity.

For (1), we would like to clarify that we have also considered the baseline objectives: evaluating how the generated molecules are similar to the target molecules, in defining the set of accepted molecules (as described in Line 348) which is also defined as $\mathcal{M}$ to measure $\text{NCircles}$ (as described in Line 350), and in defining $\text{Top }10$ scores (as described in Line 359), i.e., our comparisons do not only focus on diversity.

For (2), we would like to clarify that we compare our method with decoding schemes for diversified generation, e.g., beam search, rather than MolT5 and BioT5$^{+}$ which serve backbones. Our experiments are valid as they aim to show that applying existing decoding schemes is insufficient to obtain diverse and high-quality molecules similar to the target, while our method addresses these limitations.

---

[1] Krantz et al., Diversification of the drug discovery process, Nature Biotechnology 1998

[2] Nippa et al., Enabling late-stage drug diversification by high-throughput experimentation with geometric deep learning, Nature Chemistry 2024

Dear reviewer oKU9,

We express our deep appreciation for your time and insightful comments. In our updated manuscript, we highlight the changes in $\color{blue}{\text{blue}}$.

In what follows, we address your comments one by one.

---

W1. The authors should (1) discuss the limitation of using BLEU scores in comparing generated molecules and the target molecule satisfying a given description and (2) consider other RDKit-based scores.

We considered BLEU scores following prior studies [1] on text-to-molecule generation. However, we acknowledge their limitations in comparing the generated molecules with the target molecule: they simply measure textual similarity of SMILES strings that may fail to capture the structural or functional information.

To address your concerns, in Appendix D.2 of our updated manuscript (also in sixth footnote in Page 7), we (1) discuss the limitations of BLEU scores and (2) conduct additional experiments by replacing BLEU scores with Tanimoto and Dice scores [2] that are more concrete to capture the structural and functional information. One can observe that our fine-tuned model, even though it is trained with the BLEU score, consistently outperforms the baselines in these experiments. We also plan to fine-tune LLMs using Dice scores instead of BLEU scores and include the results in the Appendix.

---

W2. About minor errors or recommendation.

Thanks you for valuable suggestions! In our updated manuscript, we modify the symbols in Figure 7. We would like to defer naming until the end of the discussion period.

---

Q1. Can authors improve the explanation of NCircles metric?

To address your comment, we have modified explanation of NCircles metric in Section 4.1 of our updated manuscript and provided more detailed explanation in Appendix A.3 of our updated manuscript.

To measure the diversity of a given set of molecules $\mathcal{M}$, the NCircles metric computes the size of the largest subset of molecules in which no two molecules are similar to each other. Specifically, this metric is defined with a Tanimoto similarity $T(\cdot,\cdot)$ and a similarity threshold $h$ as follows [3]:

$

{{\text{NCircles}}_{h}} = \max _{\mathcal{C}\subseteq\mathcal{M}} |\mathcal{C}|\quad \text{s.t. }T(x,y)<h, \forall x \neq y \in \mathcal{C},

$

where $\mathcal{C}$ is a subset of molecules and every pair of molecules in $\mathcal{C}$ should have a similarity lower than $h$. The high NCircles metric implies that the given set of molecules covers a larger volume of molecular space, indicating diversity [3]. We measured this by defining $\mathcal{M}$ as the set of accepted molecules to capture quality and diversity.

What kinds of hyperparameters does it have? What are the conditions for two nodes to be close to each other? Do the overlap of the circles mean anything or are they a consequence of the force-directed algorithm?

Therefore, the NCircles metric does not have hyperparameters. Additionally, we guess that the latter two questions stem from Figure 6 which is just illustrated to explain NCircles in a 2-D space. Here, two nodes are close to each other when they have high similarity, the circle implies the boundary of threshold $h$, and the overlap of circles is just the consequence of the force-directed algorithm.

---

Q2. In Table 1, do the generated molecules correctly follow the given description?

Unfortunately, it is hard to directly evaluate whether the generated molecules follow the given description due to the complexity of qualitative description. However, it is worth noting that the results in Figure 7 are measured by evaluating whether the generated molecules exactly follow the molecular quantitative description, e.g., QED value or the number of hydrogen bond donors. This shows how generated molecules from our method correctly follow the given descriptions.

---

[1] Pei et al., BioT5+: Towards Generalized Biological Understanding with IUPAC Integration and Multi-task Tuning, ACL 2024

[2] Bajusz et al, Why is Tanimoto index an appropriate choice for fingerprint-based similarity calculations?, Journal of Cheminformatics 2015

[3] Xie et al., How Much Space Has Been Explored? Measuring the Chemical Space Covered by Databases and Machine-Generated Molecules, ICLR 2023

Dear reviewer FZz8,

We express our deep appreciation for your time and insightful comments. In our updated manuscript, we highlight the changes in $\color{blue}{\text{blue}}$.

In what follows, we address your comments one by one.

---

W1. The method directly adopts LLMs with reinforcement learning. The technical novelty of the method is limited.

We would like to clarify that the novelty of our work lies in introducing a new concept: leveraging LLM fine-tuning to generate diverse molecules, rather than simply proposing to adopt LLMs with reinforcement learning. We also note that our work is the first to explore the use of reinforcement learning for generating diverse output with LLMs, which has not been previously addressed in the LLM literature to the best of our knowledge.

---

W2. To help readers understand the impact of the approach, can authors provide additional explanations for the diversity metrics or introduce more similarity evaluation methods?

Thanks you for valuable suggestions! To address your comments, we provide additional explanations for the diversity metrics in Appendix A.3 of our updated manuscript. We also provide additional explanations of various similarity evaluation methods, e.g., Tanimoto, Dice, and cosine similarities, in Appendix A.2 of our updated manuscript. Additionally, we update first footnote in Page 1, Line 200 in Section 3, fourth footnote in Page 4, and Line 343 in Section 4.1 to provide high-level explanations of molecular diversity and similarity.

---

Q1. In Table 3, why is the comparison made when the number of generated molecules differs? Why did authors not include all relevant metrics?

The comparison with different numbers of generated molecules aims to show that our method consistently discovers more diverse molecules even when the baseline generates a larger number of molecules.

To address your comment, in our updated manuscript, we include all relevant metrics in Table 10 of Appendix D.1.

---

Q2-1. What LLM is used as the basis in Table 4?

We considered BioT5$^{+}$ in Table 4. In our updated manuscript, we clarify this in the captions of Table 4.

---

Q2-2. The performance improvement of Div-SFT seems to be minor.

The performance Div-SFT is not appropriate for evaluating the benefits of our algorithm, since it is a subroutine of our algorithm. Instead, one should focus on the result of reinforcement learning (Div-RL), which is our final model. The main purpose of Div-SFT is not to improve the performance, but to repurpose the base LLMs for generating multiple molecules before Div-RL.

Dear reviewer APHU,

We express our deep appreciation for your time and insightful comments. In our updated manuscript, we highlight the changes in $\color{blue}{\text{blue}}$.

In what follows, we address your comments one by one.

---

W1. The two-stage fine-tuning approach itself is not that novel.

This is not a weakness of our work, since using the two-stage fine-tuning approach is not our main idea. Our idea to leverage LLMs for diverse molecule generation is novel. We design new reward functions for maximizing the structural diversity of generated outputs. We newly construct experiments and metrics for comparing the diversity of our algorithm and existing LLM decoding algorithms.

---

W2. Although tested on standard datasets and metrics, this works lacks experimental or real-world validation of the practical utility of the generated molecules.

Evaluating on the standard datasets and metrics is sufficient, since the datasets and metrics were designed to be representative of the practical utility of the generated molecules [1,2,3]. Our belief aligns with the vast literature of computational methods for designing new molecules [4,5,6,7].

---

W3. The proposed approach may require extensive setup or hyperparameter tuning, which can hinder future adoption and reproducibility.

We clarify that our approach requires minimal setup and hyperparameter tuning due to following the widely used two-stage fine-tuning approach. Our work requires a similar number of hyperparameters compared to many of the two-stage fine-tuning approaches. We ensure reproducibility by providing the code in the Supplementary Materials, specifying hyperparameter values in Appendix B. We also plan to release the full parameters of the fine-tuned models once our work is accepted.

---

W4. The authors overstate that (1) "the first to explore the use of LLMs for generating diverse molecule" and (2) "the first propose a fine-tuning approach for LLMs to generate diverse solution".

We do not think this is an overstate, since we were unable to find any works that claim (1) and (2). We will be happy to tone down our claims if you could provide any reference that already claims (1) and (2).

---

[1] Edwards et al., Text2Mol: Cross-Modal Molecule Retrieval with Natural Language Queries, EMNLP 2021

[2] Polykovskiy et al., Molecular Sets (MOSES): A Benchmarking Platform for Molecular Generation Models, Frontiers in Pharmacology 2020

[3] Xie et al., How Much Space Has Been Explored? Measuring the Chemical Space Covered by Databases and Machine-Generated Molecules, ICLR 2023

[4] Hu et al. De novo drug design using reinforcement learning with multiple gpt agents, NeurIPS 2023

[5] Pei et al., BioT5+: Towards Generalized Biological Understanding with IUPAC Integration and Multi-task Tuning, ACL 2024

[6] Yu et al., LlaSMol: Advancing Large Language Models for Chemistry with a Large-Scale, Comprehensive, High-Quality Instruction Tuning Dataset, COLM 2024

[7] He et al. Evaluation of reinforcement learning in transformer-based molecular design, Journal of Cheminformatics 2024

Thank you for your detailed response! While it addressed some of my concerns, I still have the following questions:

1. "the first to explore the use of LLMs for generating diverse molecule" Your work focuses on using reinforcement learning (RL) to redesign the reward function and improve diversity, rather than leveraging Large Language Models (LLMs) themselves to generate diverse molecules. The pretraining stage didn't enhance diversity and the pertaining approach is not novel either. RL fine-tuning play the role of improve diversity.

2. "the first propose a fine-tuning approach for LLMs to generate diverse solution" Your proposal to fine-tune LLMs is not that novel, as similar approaches have been explored before (e.g., Reinvent [1]),(they didn't use your redesigned reward function). Your method uses the PPO algorithm, and the multiple stages can be regarded as multiple epochs in traditional RL. Therefore, novelty leads to diversity still relies on the redesigned reward function, but it's quite limited.

Therefore, I will keep my score.

[1]Reinvent 4: Modern AI–driven generative molecule design