Benchmarking Retrieval-Augmented Generation for Chemistry

Thank you very much for your constructive comments and support! Besides the general response, we specifically answer your questions below and the manuscript will be carefully revised accordingly.

Q1: Why use SPECTER? SPECTER2 is better and has adapters for various tasks.

A1: We selected four representative retrieval models to illustrate the performance across different retriever types: BM25 as a sparse retriever; Contriever and e5 as general-domain dense retrievers; and SPECTER as a scientific-domain dense retriever. To show that our conclusion is consistent for both SPECTER and SPECTER2, we will add the results of SPECTER2 in the revised draft.

Q2: What was the retrieval intention of the authors?

A2: Chemistry is a highly specialized and dynamic discipline. Thus, LLMs trained on general corpora often fail to generate grounded and accurate responses for chemistry, instead, they may produce hallucinated or outdated content. Retrieval presents a natural solution to these limitations, allowing models to retrieve and incorporate trusted chemical knowledge during inference. Therefore, it is a powerful way to mitigate hallucination.

Q3: There's no mention of code release or how to use the benchmark/toolkit.

A3: We will release the code soon. Please find our code at https://anonymous.4open.science/r/ChemRAG_anonymous-EB23/. We deleted some model/cache paths for anonymous purposes. It is very easy to run with different LLMs. For open-source LLMs, you can simply download a model from huggingface and provide the location of the model. For proprietary models, you can simply provide the model name. For adding a new retriever, you can simply run the index code that we provide. As for a new corpus, we have provided the code to index the corpus with retrievers.

Q4: The discussion of related work does not address the specific challenges

A4: Thanks for your suggestions! We have included the suggested section and the mentioned paper in our revised draft.

Q5: No statistical significance testing has been applied to the experimental results

A5: For benchmark paper, it is very rare to include the statistical significance test. To make sure our result is reproducible, we set the temperature to 0 for all models except Deepseek and o1. We run the experiments three times and the mean is reported. There are 1,932 questions in total, and we believe this is substantial that the improvement from using RAG does not rely on some individual data points.

Q6: Given the datasets have been published previously (one of them several years ago) it is reasonable to assume that LLMs will have seen them during training.

A6: We think this is a common issue for all LLMs. Even LLMs have seen the test data during training, they may not be able to leverage it well (e.g. baseline in Table 3), and using RAG may be helpful for providing related knowledge for the task (also shown in Table 3).

Q7: There is no “limitations” section which should critically assess how generalisable the contribution is.

A7: Thanks for your suggestion! The limitation section is not required by COLM. We will add a limitation section in the appendix in our revised draft.

Reference

[1] Hendrycks, D., et al. Measuring Massive Multitask Language Understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.

[2] Papers with Code: MMLU Leaderboard

[3] Zhang, D., et al. ChemLLM: A Chemical Large Language Model. arXiv:2402.06852 [cs.AI] (2024). https://doi.org/10.48550/arXiv.2402.06852.

[4] Wang, X., et al. SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models. Proceedings of the 41st International Conference on Machine Learning, PMLR 235:50622-50649, 2024.

[5] Fang, Y., et al. Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Models. Proceedings of the International Conference on Learning Representations (ICLR), 2024.

Q3: Another shortcoming I see is the experimental work which does not make use of suitable baselines A3: We have already considered SOTA models. For MMLU [1], GPT-4o is the strongest model based on [2]. For ChemBench [3], in their original paper, the strongest models are ChemLLM and GPT-4. For SciBench [4], in their original paper, GPT-4 performs the best. For Mol-Instruct [5], the best models are MolT5 and ChemT5, and we have included the results below.

Comparison with MolT5 and Text-Chem T5

Chain-of-thought Experiment

We have also included the chain-of-thought (CoT) experiments demonstrated below.

We notice that CoT does better than baseline and solely using RAG in multiple-choice settings (MMLU and ChemBench4K), but performs worse than baseline and solely using RAG in open QA (SciBench and Mol-Instruct.).

Comparing with other methods may not be necessary. Because we focus on building a benchmark that makes using and researching RAG for chemistry more convenient and focus on understanding the performance of current RAG systems, rather than claiming current RAG systems can beat many other systems. Please refer to our general response to all reviewers for more information.

Thank you very much for your constructive comments! Besides the general response, we specifically answer your questions below and the manuscript will be carefully revised accordingly.

Q1: The work appears a bit incremental as the “comprehensive benchmark” collection

A1: Although RAG demonstrated its power in the general domain, its effectiveness in the chemistry domain is not well explored and this is because of lack of benchmark data and chemistry knowledge source. This paper is not aimed at developing a new retriever, but to propose a benchmark on RAG for chemistry and systematically study RAG in the challenging Chemistry domain by constructing corpus, incorporating existing retrieval methods, and selecting representative chemistry-related tasks. It is non-trivial to construct the corpus, incorporate retrieval algorithms, select the representative tasks, and implement the RAG pipeline. We solve these challenges by working with researchers in chemistry and biochemistry. They expect LLMs to work in their domain-specific tasks in molecule description generation, molecule generation, property and reaction predictions. We carefully collect data from sources commonly used by domain experts:

• PubChem is the largest database of freely accessible chemical information in the world. For each chemical compound in PubChem, we collect its names, properties, and description and transform them into JSON format.
• Semantic Scholar composes a large amount of full-text scientific papers and includes cutting-edge chemistry research information. It also contains chemical reactivity and bioactivity information. We collect 1,849,956 full-text chemistry papers from Semantic Scholar, then we chunk them into chunks of 512 tokens with a 50-token overlap.
• USPTO contains a large amount of information about reactions: reactants, reagents, products, and yields. We transform these attributes into JSON format.
• Open-source textbooks: We parse the PDF textbooks to texts with Mathpix (https://mathpix.com/). Then we split the parsed textbook into chunks of 512 tokens with a 50-token overlap. The curated corpus is large-scale and comprehensive, containing millions of molecules and 1,849,956 chemistry papers, covering chemical names, properties, reactivity, bioactivity, and reactions. We will add the corpus curation section in the revised draft.

The tasks in our constructed benchmark are carefully selected to represent a wide range of chemistry tasks by collaborating with the aforementioned domain experts. The proposed benchmark makes using RAG for chemistry and research in this domain much more convenient. Please refer to our general message for more information.

As for the copyright issue, all the data we collect from is open-source, and we do our best to make sure it is allowed to re-distribute.

Our Contribution:

To the best of our knowledge, this is the first work of benchmarking RAG for chemistry. It is also the first work that covers a wide range of aspects and systematically studies the effect of current RAG systems on chemistry tasks. Both the dataset and the benchmark will serve as valuable references and contributions to both the RAG and chemistry communities.

Q2: While each dataset looks like a valuable resource it remains unclear how representative the chosen data actually is for a specific practical setting.

A2: The domain experts of researchers in chemistry and biochemistry guide us through selecting the corpora and task datasets. For the corpora, we have PubChem which contains names, properties, and description for each chemical compound; Semantic Scholar which contains full-text chemistry literature and is valuable for analyzing the usage, property, and chemical activity; USPTO which contains critical information for reactions; and PubMed which contains important information for biochemistry. Our domain experts confirm these data sources are quite representative and are the data sources they use in their work.

For the task datasets, our domain experts confirm that the datasets represent all the settings that a LLM may be used for, since they mainly leverage LLMs for understanding the molecules, getting information of molecules, predict reactions, and generate molecules.

Q5: You describe your benchmark as “expert-curated.” Could you clarify who the experts were and what their role was in curating the data?

A5: The experts are researchers in chemistry and biochemistry. They mainly involve in 1) selecting data when curating the corpora, including the data source, specific information, data processing; and 2) choosing the chemistry-specific tasks. Please refer to our general response for more information.

Q6: Consider including the names of the LLMs and retrievers used directly in the table and figure captions to improve clarity and readability.

A6: Thanks! We have updated our draft accordingly.

Reference

[1] Friel, R., et al. RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems. arXiv:2407.11005 [cs.CL] (2025). https://doi.org/10.48550/arXiv.2407.11005.

[2] Xiong, G., et al. Benchmarking Retrieval-Augmented Generation for Medicine. In Findings of the Association for Computational Linguistics: ACL 2024, pages 6233–6251, Bangkok, Thailand. Association for Computational Linguistics.

[3] Hendrycks, D., et al. Measuring Massive Multitask Language Understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.

[4] Papers with Code: MMLU Leaderboard

[5] Zhang, D., et al. ChemLLM: A Chemical Large Language Model. arXiv:2402.06852 [cs.AI] (2024). https://doi.org/10.48550/arXiv.2402.06852.

[6] Wang, X., et al. SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models. Proceedings of the 41st International Conference on Machine Learning, PMLR 235:50622-50649, 2024.

[7] Fang, Y., et al. Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Models. Proceedings of the International Conference on Learning Representations (ICLR), 2024.

Q4: The comparative evaluation is not very comprehensive.

A4: Our evaluation settings are the widely used ones in the community [1,2]. We have already considered SOTA models. For MMLU [3], GPT-4o is the strongest model based on [4]. For ChemBench [5], in their original paper, the strongest models are ChemLLM and GPT-4. For SciBench [6], in their original paper, GPT-4 performs the best. For Mol-Instruct [7], the best models are MolT5 and Text-ChemT5, and we have included the results below.

Thank you for mentioning using chain-of-thought prompting, we have included the experiments demonstrated below. We notice that CoT does better than baseline and solely using RAG in multiple-choice settings (MMLU and ChemBench4K), but performs worse than baseline and solely using RAG in open QA (SciBench and Mol-Instruct.).

Comparing with chemistry agents may not be necessary. First, we focus on building a benchmark that makes using and researching RAG for chemistry more convenient and focus on understanding the performance of current RAG systems, rather than claiming current RAG systems can beat many other systems. Second, these chemistry agents are in different settings, for instance, ChemAgent requires a few ground truths for reasoning while we don’t provide any demonstration in our setting.

Thank you very much for your constructive comments! Besides the general response, we specifically answer your questions below and the manuscript will be carefully revised accordingly.

Q1: No new retrieval algorithms or corpora are proposed

A1: Although RAG demonstrated its power in the general domain, its effectiveness in the chemistry domain is not well explored and this is because of lack of benchmark data and chemistry knowledge source. This paper is not aimed at developing a new retriever, but to propose a benchmark on RAG for chemistry and systematically study RAG in the challenging Chemistry domain by constructing corpus, incorporating existing retrieval methods, and selecting representative chemistry-related tasks. It is non-trivial to construct the corpus, incorporate retrieval algorithms, select the representative tasks, and implement the RAG pipeline. We solve these challenges by working with researchers in chemistry and biochemistry. They expect LLMs to work in their domain-specific tasks in molecule description generation, molecule generation, property and reaction predictions. We carefully collect data from sources commonly used by domain experts:

• PubChem is the largest database of freely accessible chemical information in the world. For each chemical compound in PubChem, we collect its names, properties, and description and transform them into JSON format.
• Semantic Scholar composes a large amount of full-text scientific papers and includes cutting-edge chemistry research information. It also contains chemical reactivity and bioactivity information. We collect 1,849,956 full-text chemistry papers from Semantic Scholar, then we chunk them into chunks of 512 tokens with a 50-token overlap.
• USPTO contains a large amount of information about reactions: reactants, reagents, products, and yields. We transform these attributes into JSON format.
• Open-source textbooks: We parse the PDF textbooks to texts with Mathpix (https://mathpix.com/). Then we split the parsed textbook into chunks of 512 tokens with a 50-token overlap. The curated corpus is large-scale and comprehensive, containing millions of molecules and 1,849,956 chemistry papers, covering chemical names, properties, reactivity, bioactivity, and reactions. We will add the corpus curation section in the revised draft.

The tasks in our constructed benchmark are carefully selected to represent a wide range of chemistry tasks by collaborating with the aforementioned domain experts. The proposed benchmark makes using RAG for chemistry and research in this domain much more convenient. Please refer to our general message for more information.

Our Contribution:

To the best of our knowledge, this is the first work of benchmarking RAG for chemistry. It is also the first work that covers a wide range of aspects and systematically studies the effect of current RAG systems on chemistry tasks. Both the dataset and the benchmark will serve as valuable references and contributions to both the RAG and chemistry communities.

Q2: Shallow findings and analysis.

A2: Thanks for the suggestions! We will update more detailed analysis in the revised draft.

Q3: It is not chemistry-specific enough, other than the used datasets.

A3: The corpus constructed by us is created for chemistry and it integrates diverse sources of chemical knowledge, capturing a broad spectrum of information on chemical compounds, reactions, mechanistic insights, and scientific discourse. The retriever and the method of LLMs utilizing the retrieved knowledge are unique in a chemistry setting and should be presented in a separate paper. The goal of our paper is to provide all the components for users and researchers: chemistry-focused corpora (created by us), baseline retrievers, and chemistry tasks. In the future, researchers do not need to construct another database, run baseline retrievers, or find chemistry-related tasks, but simply utilize our proposed benchmark.

Thank you very much for your constructive comments! Besides the general response, we specifically answer your questions below and the manuscript will be carefully revised accordingly.

Q1: The contribution seems a bit limited

A1: Our work is beyond combining previous work. The corpora is curated by us. Curating the corpus for ChemRAG benchmarking is non-trivial. We solve this issue by discussing with researchers in chemistry and biochemistry. They expect LLMs to work in their domain-specific tasks in molecule description generation, molecule generation, property and reaction predictions. We carefully collect data from sources commonly used by domain experts:

• PubChem is the largest database of freely accessible chemical information in the world. For each chemical compound in PubChem, we collect its names, properties, and description and transform them into JSON format.
• Semantic Scholar composes a large amount of full-text scientific papers and includes cutting-edge chemistry research information. It also contains chemical reactivity and bioactivity information. We collect 1,849,956 full-text chemistry papers from Semantic Scholar, then we chunk them into chunks of 512 tokens with a 50-token overlap.
• USPTO contains a large amount of information about reactions: reactants, reagents, products, and yields. We transform these attributes into JSON format.
• Open-source textbooks: We parse the PDF textbooks to texts with Mathpix (https://mathpix.com/). Then we split the parsed textbook into chunks of 512 tokens with a 50-token overlap.

The curated corpus is large-scale and comprehensive, containing millions of molecules and 1,849,956 chemistry papers, covering chemical names, properties, reactivity, bioactivity, and reactions.

The tasks in our constructed benchmark are carefully selected to represent a wide range of chemistry tasks by collaborating with the aforementioned domain experts.

The proposed benchmark makes using RAG for chemistry and research in this domain much more convenient. Please refer to our general message for more information.

Q2: It might lack some insights on how to build a better retriever from the analysis from this paper.

A2: Current sparse and dense retrieval methods still fall short in the chemistry domain. We could first construct a synonym dictionary and use the synonyms of a molecule to retrieve positive documents which are used to train a retriever. In this way, texts about the same molecule but with different names are likely to be retrieved which resulting in a better retriever. We could also use an LLM to do the retrieval by using its reasoning capability to write search queries. The focus of this paper is to benchmark RAG systems for chemistry. In addition, building a better retriever itself is a lot of contribution and may be a new paper, which is why we refrain from constructing a new retriever in this paper.

Here is an example of the analysis from this paper:

Query: What is the molecular weight of aspirin?

Document1: The molecular weight of CH3COOC6H4COOH is 180.16 g/mol.

Document2: Aspirin can cause developmental toxicity.

In this example, CH3COOC6H4COOH is the formula of aspirin, but current retrievers don't give Document1 high scores because it doesn’t know CH3COOC6H4COOH is aspirin. By training with synonyms, the texts with CH3COOC6H4COOH and aspirin will be closer in the embedding space. Alternatively, when using an LLM to generate queries, the LLM may first find the formula of aspirin, and then use the formula to search for the molecular weight.

Dear Reviewer vCXR,

As we approach the end of the discussion period, could you please check the authors' responses and see if they have addressed your concerns? Thank you very much for your efforts.

Best, AC