Chain-of-thoughts for molecular understanding

We sincerely appreciate your comments and efforts in reviewing our paper. We think our paper has improved significantly by adding a comprehensive ablation study. We address your question as follows. We also updated our manuscript by highlighting the updates in $\color{magenta}{\text{magenta}}$.

W1. The matching ratio-based rejection sampling needs further clarification including the weights between structural components, the impact of the number of samples $k$, and the efficiency or the running time of the rejection sampling.

Thank you for pointing out the unclarity. First, the weights between structural components are not adjusted and all components are treated equally.

Next, for the impact of the number of samples $k$, we conducted an additional ablation study on $k\in\{10,20\}$. It is notable that the results for $k=1$ (i.e., no matching ratio-based sampling) were already presented in Figure 6. We updated Figure 8 in our manuscript to include the additional results for $k\in\{10,20\}$. We observe that while applying matching ratio-based rejection sampling consistently improves performance, increasing $k$ beyond 5 does not result in significant further improvement. Therefore, our original choice of $k=5$ was sufficient to balance performance and efficiency.

Lastly, regarding the efficiency of the rejection sampling, the rejection sampling does not impact training efficiency, as it only influences the sampling process. During generation, the approach leverages normal beam search to generate multiple outputs. The time complexity of this process remains the same as the original beam search, which is O(d \times b \times \log(b)), where d is the number of successors per node, and b is the number of levels to explore. This ensures that the method is computationally efficient and scalable.

W2. An ablation study on each structural information component is needed.

Thank you for such an insightful comment! Following your suggestion, we conducted an ablation study on each structural information component. We report the new results in Figure 11 of Appendix B.3.

In this updated analysis, we evaluated the performance of molecule captioning on ChemT5-small using each structural information component individually. The results show that the molecular formula and chirality contribute the largest performance improvements. Additionally, incorporating each single component resulted in better performance compared to the baseline model without any Chain-of-Thought (CoT) integration. Notably, combining all the proposed structural elements yielded the best results, validating the effectiveness of our comprehensive approach.

W3. The molecular captioning performance of GPT-4o is too low compared to that of MolReGPT. Did the authors follow the prompt template used in MolReGPT?

We confirm that we followed the prompt template used in MolReGPT, as detailed in Table 8 and Table 10 of Appendix A.2 and A.3. It is notable that while we mostly adhered to the original prompt, we made slight modifications to incorporate the StructCoT. This adjustment ensures that the structural reasoning process is effectively introduced and leveraged within our framework. Despite adhering to MolReGPT’s prompt design, the reported molecular captioning performance of GPT-4o is lower than that of MolReGPT. This discrepancy arises because we did not apply any retrieval augmentation approach, which is a key component of MolReGPT’s methodology.

It is important to note that retrieval augmentation is orthogonal to our approach. Incorporating our StructCoT framework into MolReGPT could be an interesting direction for future work, potentially combining the strengths of both CoT and retrieval augmentation to further improve the molecular captioning performance.

W4. The baseline BioT5 is missing.

We regret that we are unable to conduct the experiments within the rebuttal phase due to computational limitations. In detail, fine-tuning BioT5 requires 100,000 steps with a batch size of 768, which could not be completed within the limited time and resources as it requires several hundred epochs. We plan to incorporate these experiments into our future manuscript. Although we are not able to incorporate StructCoT into BioT5, it is notable that we have already achieved competitive results with BioT5 in both tasks by incorporating ChemT5-base with our proposed StructCoT.

W5. The selected baselines are outdated and not strong enough.

To address your concern, we have conducted additional experiments using the recent MolInstructions [7] dataset for molecule captioning. The results detailed below and in Table 1, demonstrate that our approach also achieves improved performance on MolInstructions.

Moreover, we have also conducted additional experiments on retrosynthesis to verify the effectiveness of our method for other tasks. The results are detailed below and in Appendix B.4. We observe that incorporating our StructCoT not only improves molecular captioning and text-based molecule generation tasks but also more complicated retrosynthesis task.

[1] Bran, A. M., et al. "ChemCrow: Augmenting large-language models with chemistry tools." Nature Machine Intelligence 2024.

[2] Devlin, J., et al. "Bert: Pre-training of deep bidirectional transformers for language understanding." Proceedings of NAACL-HLT 2019.

[3] Bran, A. M., et al. "ChemCrow: Augmenting large-language models with chemistry tools." Nature Machine Intelligence 2024.

[4] Radford, A., et al. "Improving Language Understanding by Generative Pre-Training (GPT-1)." Arxiv 2018.

[5] Inaba, T., et al. "MultiTool-CoT: GPT-3 can use multiple external tools with chain of thought prompting." ACL 2023.

[6] Xenhu Chen, et al., "Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks." TMLR 2023.

[7] Fang, Yin, et al. "Mol-instructions: A large-scale biomolecular instruction dataset for large language models." ICLR 2024.

We sincerely appreciate your comments and efforts in reviewing our paper. We think our paper has improved significantly in terms of additional baseline, thanks to your comment. We address your question as follows. We also updated our manuscript by highlighting the updates in $\color{magenta}{\text{magenta}}$.

W1. The method is not novel as CoT is widely used in prompting LLMs for complex tasks.

We respectfully disagree with the assessment that it lacks novelty. We have addressed a key bottleneck in chemistry tasks. Even recent strong LLMs have shown limitations in chemistry tasks. Our work qualitatively identifies this bottleneck as the lack of capability to understand molecular structures and proposes StructCoT to integrate molecular structural information that directly relates to significant molecular properties. This innovation addresses a critical gap in applying LLMs to chemistry.

While we acknowledge that Chain-of-Thought (CoT) reasoning has been widely adopted in prompting LLMs for various complex tasks, our contributions specifically address unique challenges in the domain of molecular reasoning, which remain largely underexplored. Unlike natural language tasks, molecular representations (e.g., SMILES) lack inherent structural context, which directly impacts the properties of the molecules. StructCoT not only incorporates this missing structural information but also demonstrates its importance for enhancing the reasoning capabilities of LLMs in molecular tasks.

Moreover, a distinct contribution of StructCoT is its ability to assess the alignment between the generated structural rationale and the corresponding molecule. This enables a matching ratio-based rejection sampling mechanism, a novel approach to ensuring consistency of the rationale and the output. To the best of our knowledge, this is the first work to propose evaluating the coherence between a generated rationale and the final answer in molecular reasoning tasks.

W2-1. The meaning of molecular understanding is unclear as text-based molecule generation is not an understanding task.

We used the term "molecular understanding" as both molecular captioning and text-based molecule generation tasks require a deep understanding of the molecules. We clarified this in the first paragraph of the introduction that molecular understanding tasks in our context refer specifically to both molecule captioning and text-based molecule generation.

We used the term “molecular understanding” because both molecular captioning and text-based molecule generation tasks require a deep understanding of molecular structures and properties. These tasks necessitate a comprehensive understanding of the underlying molecular information.

W2-2. The meaning of "two fine-tuning frameworks" is unclear.

As mentioned in Figure 2, Sections 4.2 and 4.3, this means that we use two different frameworks for molecule captioning and text-based molecule generation.

W2-3. The word CoT is improper for molecular captioning task as it is generated by external tools.

First, we clarify that StructCoT is generated by the LLM during the text-based molecule generation task, which aligns with the general meaning of CoT as intermediate reasoning steps generated by the model.

In addition, our use of the term "CoT" follows prior works [3,5,6], where the CoT process has been extended to include reasoning enhanced by external tools. For instance,

• ChemCrow [3] employed external tools such as literature search and SMILES to price to generate proper reasoning for LLM.
• MultiTool-CoT [5] proposed to incorporate multiple external tools such as a calculator and knowledge retrieval during the reasoning process.
• PoT [6] introduced to incorporate a program interpreter into the CoT generation process.

These works demonstrate that incorporating external tools into the reasoning process does not conflict with the CoT framework. However, if you have alternative suggestions for terminology that better aligns with the context, we are open to adopting them.

W2-4. The x-axis label of Figure 6 is missing.

Thank you for pointing this out. The x-axis label represents epochs, consistent with Figure 8. We have added the x-axis label in the updated manuscript to ensure clarity.

We sincerely appreciate your comments and efforts in reviewing our paper. We think our paper has improved significantly in terms of additional tasks and advanced structural components, thanks to your comment. We address your question as follows. We also updated our manuscript by highlighting the updates in $\color{magenta}{\text{magenta}}$.

W1, Q1. Incorporating various tasks such as property prediction, reaction prediction, and synthesis planning may strengthen the paper.

Thank you for your insightful suggestions. To address your comments, we have conducted additional experiments on retrosynthesis to verify the effectiveness of our method for other tasks. The results are detailed below and in Appendix B.4. We observe that incorporating our StructCoT not only improves molecular captioning and text-based molecule generation tasks but also more complicated retrosynthesis task.

W2. While the proposed approach is practical, the method itself is too simple.

We acknowledge that our approach is straightforward; however, we respectfully disagree with the assessment that simplicity is our weakness. We have addressed a key bottleneck in chemistry tasks. Even recent strong LLMs have shown limitations in chemistry tasks. Our work qualitatively identifies this bottleneck as the lack of capability to understand molecular structures and proposes StructCoT to integrate molecular structural information that directly relates to significant molecular properties. This innovation addresses a critical gap in applying LLMs to chemistry.

W3, Q2. What is the reason for choosing the six proposed structural information components? Incorporating more advanced concepts (e.g., molecular fingerprints, electronic properties, 3D conformations, etc.) may improve the method.

Thank you for your insightful suggestions. In our work, we selected six "structural" features of molecules as they are highly relevant to molecular properties, as illustrated in Figure 3. Nevertheless, to address your comments, we conducted additional experiments incorporating three advanced components: fingerprint information (Morgan fingerprint) and electronic properties (topological polar surface area (TPSA) and molar refractivity (MR)). Specifically, the Morgan fingerprint encodes local substructures within a specified radius, TPSA represents the sum of the surface areas of all polar atoms and their attached hydrogen atoms, and MR quantifies the total polarizability of a molecule.

To verify the effectiveness of each additional descriptor, we evaluated the performance of molecule captioning using ChemT5-small. We provide the results in Figure 12 of Appendix B.3. We observed that incorporating all three additional descriptors together did not further improve the performance of StructCoT, although applying each additional descriptor individually improved performance. This validates the importance of structural information and the sufficiency of our proposed structural components.

We sincerely appreciate your comments and efforts in reviewing our paper. We think our paper has improved significantly in terms of advanced structural components, thanks to your comment. We address your question as follows. We also updated our manuscript by highlighting the updates in $\color{magenta}{\text{magenta}}$.

W1, Q1-1. Why do we need to employ general LLMs for molecular understanding tasks? Specialist models such as Chemprop may work better and external tools are more appropriate for acquiring structural information.

General large language models (LLMs), such as Llama and GPT-4o, excel in their ability to generalize across diverse domains, enabling them to adapt to a wide range of tasks that specialist models may not address effectively. Molecular understanding tasks, for instance, often require interpreting textual descriptions of molecules or integrating molecular reasoning with external knowledge. Generalist models, pre-trained on diverse text corpora, including scientific literature, are well-suited for these challenges. While the current performance of generalist models in some molecular tasks may lag behind specialist models, we anticipate that advancements in computational power and modeling capabilities will enable LLMs to surpass specialist models in the near future.

Moreover, we clarify that the integration of external tools to enrich structural information complements rather than replaces the utility of generalist models. StructCoT bridges this gap by enhancing generalist models with molecular structural information, leveraging their reasoning capabilities while addressing their limitations in understanding structural features. This integrated approach enables a broader application of LLMs in molecular understanding tasks, without being restricted by the narrower focus of specialist models.

Q1-2. What is the reason for choosing the six proposed structural information components? Incorporating more advanced concepts such as chirality and E/Z isomerism may improve the method.

Thank you for your insightful suggestions. In our work, we selected six "structural" features of molecules as they are highly relevant to molecular properties, as illustrated in Figure 3. Nevertheless, to address your comments, we conducted additional experiments incorporating three advanced components: fingerprint information (Morgan fingerprint) and electronic properties (topological polar surface area (TPSA) and molar refractivity (MR)). Specifically, the Morgan fingerprint encodes local substructures within a specified radius, TPSA represents the sum of the surface areas of all polar atoms and their attached hydrogen atoms, and MR quantifies the total polarizability of a molecule.

To verify the effectiveness of each additional descriptor, we evaluated the performance of molecule captioning using ChemT5-small. We provide the results in Figure 12 of Appendix B.3. We observed that incorporating all three additional descriptors together did not further improve the performance of StructCoT, although applying each additional descriptor individually improved performance. This validates the importance of structural information and the sufficiency of our proposed structural components.

[1] Edwards, C., et al. "Translation between molecules and natural language." ACL 2022.

[2] Christofidellis, D., et al. "Unifying molecular and textual representations via multi-task language modeling." ICML 2023.

W4. The technical novelty of the method is modest as the approach focuses on input augmentation rather than proposing a new model architecture.

We acknowledge the reviewer's observation but would like to clarify the scope and contributions of our work. While technical novelty is important in research, enhancing practical performance through straightforward approaches, such as input augmentation and constructing new datasets, is also a critical area of innovation. For instance, developing new datasets and benchmarks is widely recognized as a valuable research contribution [2,6]. Similarly, the general CoT method [9], which can be viewed as a simple input augmentation approach via few-shot learning rather than introducing new model architectures, has demonstrated substantial performance improvements. StructCoT aligns with this paradigm by employing structural information in the reasoning process, enriching the input to improve downstream performance.

While our primary focus is not on proposing a new model architecture, StructCoT introduces a significant contribution: the ability to assess the alignment between the generated structural rationale and the corresponding molecule. This enables a novel matching ratio-based rejection sampling mechanism, which ensures consistency between the rationale and the output. To the best of our knowledge, this is the first approach to evaluate the coherence between generated rationales and final outputs in molecular reasoning tasks.

W5. Incorporating more advanced concepts into the structural information may improve the method.

Thank you for your insightful suggestions. In our work, we selected six "structural" features of molecules as they are highly relevant to molecular properties, as illustrated in Figure 3. Nevertheless, to address your comments, we conducted additional experiments incorporating three advanced components: fingerprint information (Morgan fingerprint) and electronic properties (topological polar surface area (TPSA) and molar refractivity (MR)). Specifically, the Morgan fingerprint encodes local substructures within a specified radius, TPSA represents the sum of the surface areas of all polar atoms and their attached hydrogen atoms, and MR quantifies the total polarizability of a molecule.

To verify the effectiveness of each additional descriptor, we evaluated the performance of molecule captioning using ChemT5-small. We provide the results in Figure 12 of Appendix B.3. We observed that incorporating all three additional descriptors together did not further improve the performance of StructCoT, although applying each additional descriptor individually improved performance. This validates the importance of structural information and the sufficiency of our proposed structural components.

Minor. The word choice of CoT is misleading as the reasoning is generated by LLM.

First, we clarify that StructCoT is generated by the LLM during the text-based molecule generation task, which aligns with the general meaning of CoT as intermediate reasoning steps generated by the model.

In addition, our use of the term "CoT" follows prior works [3,5,6], where the CoT process has been extended to include reasoning enhanced by external tools. For instance,

• ChemCrow [3] employed external tools such as literature search and SMILES to price to generate proper reasoning for LLM.
• MultiTool-CoT [5] proposed to incorporate multiple external tools such as a calculator and knowledge retrieval during the reasoning process.
• PoT [6] introduced to incorporate a program interpreter into the CoT generation process.

These works demonstrate that incorporating external tools into the reasoning process does not conflict with the CoT framework. However, if you have alternative suggestions for terminology that better aligns with the context, we are open to adopting them.

Minor. Phrases like "We propose to" in the contributions section could imply a suggestion rather than a concrete result.

Thank you for the suggestion. We updated our manuscript with the sentence starting with "We design~".

We sincerely appreciate your comments and efforts in reviewing our paper. We think our paper has improved significantly in terms of additional baseline and advanced structural component, thanks to your comment. We address your question as follows. We also updated our manuscript by highlighting the updates in $\color{magenta}{\text{magenta}}$.

W1. Incorporating additional baselines [2] may strengthen the paper.

Thank you for your insightful suggestions. To address your comments, we have conducted additional experiments using the MolInstructions dataset for molecule captioning. The results detailed below and in Table 1, demonstrate that our approach also achieves improved performance on MolInstructions.

Moreover, we have also conducted additional experiments on retrosynthesis to verify the effectiveness of our method for other tasks. The results are detailed below and in Appendix B.4. We observe that incorporating our StructCoT not only improves molecular captioning and text-based molecule generation tasks but also more complicated retrosynthesis task.

W2, Q2. Comparison to BioT5 [4] could be additional baseline and Chemcrow [3] is not entirely appropriate as it is designed for different goals.

Regarding BioT5, we regret that we are unable to conduct the experiments within the rebuttal phase due to computational limitations. In detail, fine-tuning BioT5 requires 100,000 steps with a batch size of 768, which could not be completed within the limited time and resources as it requires several hundred epochs. We plan to incorporate these experiments into our future manuscript. Although we are not able to incorporate StructCoT into BioT5, it is notable that we have already achieved competitive results with BioT5 in both tasks by incorporating ChemT5-base with our proposed StructCoT.

For ChemCrow, we acknowledge that the comparison may not be entirely appropriate, as the reviewer mentioned, ChemCrow is primarily designed for practical synthesis tasks, but we included the comparisons to provide additional insights, as they share a similar motivation. To alleviate your concern, we added a discussion on the appropriateness of comparing to ChemCrow in Appendix B.3.

W3, Q1. There is no mechanism for handling missing structural information in captions during text-based molecule generation. How does the structural CoT affect captions beyond structural information?

In fact, StructCoT is specifically designed to address such scenarios. As detailed in Figure 2(b) and Section 4.3, we train the reasoning module to infer structural information from the given caption, regardless of whether explicit structural details are present in the caption. This approach ensures robustness in cases where structural information is incomplete or absent in the input caption.

Moreover, sketching the structural description before output generation enhances the model's understanding of molecular structures, effectively bridging gaps in the input. This improved structural awareness positively impacts the overall quality of the generated outputs, even in cases where the captions are less relevant to the structural description. As a result, by prioritizing structural reasoning, StructCoT enables more accurate outputs, demonstrating its effectiveness.

Q3. Although the method improves molecule captioning performance, the real-world applications of these gains are unclear. Are there specific downstream tasks that would benefit directly from enhanced captioning? This is a question regarding the general line of research, and not specifically pointed at this paper.

Molecule captioning is a widely used method to evaluate the models' understanding of molecules [1,2,8]. Enhanced molecule captioning complements precise and detailed descriptions of molecules, which facilitate the understanding of molecules.

For instance, improved captioning can enhance the interpretability of molecule generation models by providing human-readable rationales for generated structures. This can be particularly beneficial in tasks like virtual screening, where researchers need to quickly assess large libraries of molecules for potential candidates. Additionally, detailed molecular captions can assist in the training and evaluation of predictive models for bioactivity, toxicity, or other chemical properties, further extending the utility of the method. Thus, while molecule captioning might appear as a non-practical task, its implications are broad and impactful.

[1] Li, Sihang, et al. "Towards 3d molecule-text interpretation in language models." ICLR 2024.

[2] Fang, Yin, et al. "Mol-instructions: A large-scale biomolecular instruction dataset for large language models." ICLR 2024.

[3] Bran, A. M., et al. "ChemCrow: Augmenting large-language models with chemistry tools." Nature Machine Intelligence 2024.

[4] Pei, Q., et al. "BioT5: Enriching cross-modal integration in biology with chemical knowledge and natural language associations." EMNLP 2023.

[5] Inaba, T., et al. "MultiTool-CoT: GPT-3 can use multiple external tools with chain of thought prompting." ACL 2023.

[6] Xenhu Chen, et al., "Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks." TMLR 2023.

[7] Ye, F., et al. "ProteinBench: A Holistic Evaluation of Protein Foundation Models." arXiv preprint 2024. [8] Edwards, C., et al. "Translation between molecules and natural language." ACL 2022. [9] Wei, J., et al. "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models." NeurIPS 2022.

Thanks for your rebuttal. I have updated my score based on the additional experiments and clarifications.