LDMol: A Text-to-Molecule Diffusion Model with Structurally Informative Latent Space Surpasses AR Models

We greatly appreciate Reviewer mgY4 for the review and thoughtful feedback. Below, we provide detailed point-by-point responses to address your remaining concerns.

[mgY4] asked for additional metrics on text-to-molecule generation.

As the reviewer suggested, we measured the uniqueness, novelty, and prompt alignment score for the prompts we’ve tested using 1,000 samples. Validity is the proportion of generated SMILES that is valid. Uniqueness is the proportion of valid SMILES that are unique. The “align” score is the proportion of unique SMILES that match the given prompt. Novelty is the proportion of the unique SMILES that are not included in the training dataset. The alignment score was measured with pattern matching with the substructure described by the prompt.

We observed that even when stochastic sampling was enabled, AR models struggled to generate various samples from a single prompt. LDMol can generate molecules that align better with various hand-written prompts. Furthermore, its outputs were much more diverse than the previous AR models.

---

[mgY4] suggested the potential expansion toward different types of data and conditions.

We appreciate the reviewer’s expectation for the potential applicability of our approach towards the other types of data, and we also regard the expanding of carefully designed latent diffusion models as possible future works. As the reviewer suggested, this can be applicable to both different target chemical data domain itself or various biochemical conditions.

Thank you for your thoughtful feedback and for considering our work, and we appreciate your time and evaluation. Below we provide point-by-point responses on your questions and concerns.

[qAD8] asked for visualization of the latent space and its structural information.

To visualize the structural information encoded in the latent space of our encoder, we prepared 10 molecular clusters that contain 100 molecules, each sharing the common Murcko scaffold. Then, we obtained their latent vector from the LDMol encoder and visualized them in 2D via UMAP[1]. We also plotted the latent vector of 5,000 randomly sampled molecules. As shown in the following Figure 1 [LINK], the molecules with shared Murcko scaffold[2] have formed the clusters on the latent vector space.

---

[qAD8] expressed a concern on the text-to-molecule generation performance.

We kindly note that we had discussed that MolXPT and bioT5 had higher validity than ours in the manuscript(line 307~310, right column), yet they shorted in every metric for the similarity between the ground truth. We insist that the primary role of a text-to-molecule model is to generate a molecule that meets the user prompt, thus, similarity metrics should be prioritized when evaluating models.

We agree that character-wise metrics like BLEU and Levenshtein distance are not appropriate for SMILES generation tasks, and we included them as a convention for many studies in this benchmark. LDMol still showed the major performance improvement from the previous AR models on chemical similarities, including a 6.7~15.2%p increases in fingerprint similarities and the state-of-the-art exact match ratio of 0.530 in ChEBI-20 benchmark.

For the case study in Figure 4, we measured the uniqueness, novelty, and the prompt alignment score using 1,000 samples. The alignment score was measured with pattern matching with the structure described by the prompt. Please refer to the table in the response to the reviewer mgY4 below. It is shown that LDMol can generate molecules that match with various hand-written prompts, and its output is more diverse than the previous AR models.

---

[qAD8] questioned the usefulness of molecule-to-text retrieval and suggests text-to-molecule retrieval.

We want to emphasize that the primary contribution of LDMol is in the text-to-molecule generation, and the downstream tasks were done to demonstrate its applicability as a diffusion model compared to AR-based generative models. That said, we performed a paragraph-level text-to-molecule retrieval task on the PCdes test set and the MoMu dataset, measuring 64-way accuracy. Here, we had to estimate the ELBO of each query molecule with given text input using noise prediction error across different noise levels, and the query with the highest likelihood was selected. Although the exact calculation for ELBO would take 1,000 noise predictions per pair, LDMol already showed comparable performance with the baseline representation models with a rough estimation of NFE=25 while being a successful generative model at the same time.

---

[qAD8] questioned the usefulness of text-to-molecule generation.

As demonstrated by the applications of LLM on various data domains[3][4], natural text is a modality that enables the incorporation of various conditions like molecular properties, interactions, etc. This makes the generative model utilizing text conditions more applicable and expandable compared to the models trained with specific condition types. While its pratical utility still needs to be improved for real-world usage, text-conditioned molecule generation has received a growing interest as we noted in Related Works, and LDMol achieved the state-of-the-art performance over the baselines.

---

[1] UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction, arxiv 2018.

[2] The Properties of Known Drugs. 1. Molecular Frameworks, Journal of Medical Chemistry 1996.

[3] TableLLM: Enabling Tabular Data Manipulation by LLMs in Real Office Usage Scenarios, arxiv 2024.

[4] Can Language Models Solve Graph Problems in Natural Language?, NeurIPS 2023.

We appreciate the reviewer for your detailed comments and valuable suggestions. Below, we provide thorough point-by-point responses to address your concerns.

[bRG5] asked how the proposed contrastive learning can learn the structural differences.

We would like to remind the reviewer that in order to minimize the proposed contrastive loss, the feature cosine similarity from the different molecules in the batch needs to be minimized, which enables the encoder to reflect structural differences into their latent output. Please refer to the UMAP[1] visualization of LDMol’s latent space in the following Figure 1 [LINK], where the molecules with shared Murcko molecular scaffold[2] locate closer in the latent space.

---

[bRG5] asked how our latent space construction helps the alignment with the text conditions.

Compared to the raw SMILES tokens or structure-unaware latent from $\beta$-VAE(Figure 3-(b)), our encoder provides a latent where the proximity is more structurally meaningful, hence more “interpretable” for the later diffusion model that needs to link the text information. Indeed, fingerprint similarities and FCD on the ChEBI-20 dataset were improved by 7~15%p and 40% from the previous baselines, meaning that LDMol made a reasonable guess even when it was incorrect.

---

[bRG5] expressed concern about the claim on the model performance.

While there were many “text-conditioned” chemical diffusion models, we clarify that the term “textual data” generation refers to the text-like generation target of SMILES; most successful chemical diffusion models had targeted graph or point-cloud data, and diffusion models that generate SMILES[3] showed suboptimal performance compared to AR models. We insist that we’ve shown the potential of improving diffusion model performances on text-like data with carefully designed latent space.

---

Text-guided diffusion models as baselines.

Please note that TGM-DLM[4] is included as a text-guided molecule diffusion model baseline (line 340). TGM-DLM trained the diffusion model by naively treating the token index as a continuous real value, which led to a severe performance drop compared to LDMol.

---

Ablation study on contrastive learning.

Please refer to the revised ablation studies that we included in the response to the reviewer YcAV above. To replace our suggested latent space construction, we tested two common strategies of naive reconstruction and KL-regularized autoencoder. The latent space without any regularization was unable to learn by the diffusion model, and the latent space from $\beta$-VAE had suboptimal performance with notably low validity.

---

[bRG5] asked the prompt alignment score in the case studies.

Please refer to the table in the response to the reviewer mgY4 below, where we measured the prompt alignment score for the prompts we’ve tested using 1,000 samples. The result demonstrates that LDMol showed better and more consistent alignment along various hand-written prompts compared to the AR baselines.

---

[bRG5] asked for a clearer description of the molecule editing process.

Our text-guided molecule editing was done similarly to Delta Denoising Score(DDS), where the input data is optimized to match the target prompt by minimizing the difference between the model-predited noise with (source_prompt, source_data) pair and (target_prompt, target_data) pair. Please note that Supplementary material A.3 contains a detailed description and pseudocode of our DDS-based test-guided molecule editing.

---

The architecture of the encoder.

The encoder we’ve used consists of 12 bidirectional transformer layers of BERT$_{base}$, with a feature size of 1024 and 16 attention heads. We thank the reviewer for the suggestion, and we’ll include this information in Supplementary material A.2.

---

[bRG5] questions the statement of minimizing mutual information.

The mentioned statement describes our contrastive latent space construction with SMILES enumeration, compared to previously suggested molecular contrastive learning methods[4]. Since SMILES enumeration provides all possible variations under the SMILES grammar, the enumerated SMILES pair has minimal mutual information(i.e., the connectivity of atoms and bonds) compared to simple or local augmentations. As a result, while most contrastive learning focuses on “extracting” certain desired features, our contrastive learning can “fully preserve” the structural information as the augmentation invariant to become an autoencoder.

---

[1] UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction, arxiv 2018.

[2] The Properties of Known Drugs. 1. Molecular Frameworks, Journal of Medical Chemistry 1996.

[3] Text-Guided Molecule Generation with Diffusion Language Model, AAAI 2024.

[4] Graph contrastive learning with augmentations. NeurIPS 2020.

We thank Reviewer YcAV for the constructive feedback. Below, we provide point-by-point responses on your questions and concerns.

[YcAV] asked for the statistics of the enumerated SMILES.

Please note that the SMILES enumeration we’ve introduced is done by different SMILES constructions of the same molecule, and does not modify the given molecule. Therefore, the molecule size, atoms, bonds, etc., are the same as in the initial dataset PubChem, which contains a wide range of general molecules. We used 10 million randomly sampled molecules from PubChem to construct contrastive latent space, where the SMILES length over 500 were removed. No paired text descriptions were used in this training phase.

---

[YcAV] requested the analysis and ablation study on the effect of contrastive learning on the generation task.

Please refer to the revised ablation studies below. The first two rows (a) and (b) show the model performance by replacing contrastive learning-based latent space into two common latent space construction strategies of naive reconstruction and KL-regularized autoencoder. The latent space without any regularization was unable to learn by the diffusion model, and the latent space from beta-VAE had suboptimal performance with notably low validity.

Although we appreciate the reviewer’s suggestion, our latent construction of contrastive learning is currently inapplicable to AR models since they do not utilize any latent domain. It might be possible to apply contrastive loss into intermediate feature space similar to [1], but we believe this is beyond the scope of our work.

---

[YcAV] asked for further analysis on the compression layer

Since the latent space compression was done by a single linear layer, its role is to reduce the dimension for the diffusion modeling rather than the extra curation of the features. Please refer to the UMAP[1] latent visualization in the following Figure 1 [LINK], where the feature space is structurally informative before and after the compression layer.

Also, we consistently observed that the role of the compression layer should be minimized, as we had to leverage the smooth and regulated latent space from the contrastive learning. Please note that the model performance degraded as the diffusion model’s target domain deviates from the contrastive latent space, even when the compression layer capacity is increased(see row (e) of the table above).

---

[YcAV] suggested further clarification and examination of the SMILES enumeration.

Longer SMILES have more enumerated SMILES pairs as the reviewer noted, and we randomly selected one possible enumeration for each pre-training epoch. Therefore, the number of enumeration pairs for each training data that the model encountered are mostly equal, except for the molecules that are extremely small. Even fixing the single enumeration for each training data was enough for the encoder to learn the enumeration-invariant features, as we show the histogram of enumerated SMILES pair distances in the following Figure 2 [LINK].

---

[1] UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction, arxiv 2018.

[2] Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think, ICLR 2025