Generative Hierarchical Materials Search

Thank you for the positive feedback of our work! Please find our response to your questions below.

LLM finetunine baseline

The reason that we only included few-shot prompting baselines in the end-to-end evaluation of Table 1 is because, to our knowledge, there is no natural-language-to-crystal-structure dataset available for finetuning LLMs to generate crystals from loose natural language descriptions, as this work is among the first to study this problem. Hence, few-shot prompting is the most natural baseline. However, we do evaluate the subcomponents of GenMS to previous baselines, and show that the formula-conditioned generation in GenMS performs favourably compared to diffusion model [1] and finetuned language model baselines [2] in Table 5 and Table 6 of the paper.

[1] Yang, M., Cho, K., Merchant, A., Abbeel, P., Schuurmans, D., Mordatch, I., & Cubuk, E. D. (2023). Scalable diffusion for materials generation. arXiv preprint arXiv:2311.09235.

[2] Antunes, Luis M., Keith T. Butler, and Ricardo Grau-Crespo. "Crystal structure generation with autoregressive large language modeling." arXiv preprint arXiv:2307.04340 (2023).

Why GenMS uses zero-shot prompting but baselines use few-shot prompting

We consider GenMS in a zero-shot prompting setting because of the practical motivation of materials discovery, i.e., structures to be discovered may not have similar existing structures to use as few-shot prompt. However, when we tried zero-shot prompting of cif files using LLMs directly, we observed close to zero percent valid cif files being generated. Therefore we had to include a few cif file examples as input to the LLMs to specify the format of the cif files.

Details of the LLM used and potential ways to improve the LLM module

Per the reviewer’s suggestion, in addition to Gemini long context, we have also used GPT4 as the LLM module, and observe that GPT4 performs similarly (and occasionally better) than Gemini long context. We did not further tune the prompts, which may lead to different results.

We have also conducted two additional studies for RAG. First, we eliminate half of the retrieved Wiki text, and observe the match rate in Table 4 drops from 0.89 to 0.85. Second, we replace the text retrieved from Wikipedia by text retrieved from Materials Project [1] (entries that contain the keyword Perovskite), and observe that the match rate in Table 4 drops to 0.86, which is still better than no retrieval at all. We will include these additional results in the final version of the paper. We think there are many other interesting aspects of the system that can be improved, such as using research articles to further enhance RAG. We see our work as the first that calls for further contribution for language-to-structure generation.

Thank you for the detailed review. Please see our response below.

The "end-to-end" terminology.

We agree that our usage of "end-to-end" is not conventional and can be misleading. "End-to-end" in this paper’s context refers to end-to-end automation from formulae proposal to structure generation and verification, as opposed to end-to-end modeling. We have clarified the usage of "end-to-end" in the paper and reduced its frequency for clarity.

The multi-objective optimization formulation

Thank you for recognizing this work’s contribution despite its simplicity. We believe that formally formulating the materials search problem from natural language input is important, so that many problems in materials discovery can be considered under the same framework. Under the formulation of the multi-objective optimization, rule-based pruning is only one proposed approach to solve this objective. One can also perform constrained optimization, planning, or reinforcement learning with external feedback to solve this multi-objective optimization problem, which we consider to be interesting future work.

Heuristic rules and their effectiveness

We would like to clarify that the heuristics considered in this paper are only examples of many constraints and heuristics that practitioners in materials science may want to enforce, as opposed to a set of hand-designed heuristics that work the best for materials discovery. The current heuristics, especially those in $R_\text{hi}$, are only basic checks on instruction following. As practitioners use this framework, we anticipate additional constraints and heuristics to be enforced depending on the specific types of materials being discovered and the additional domain knowledge of the respective users. We have rewritten line 116-120 to highlight that $R_\text{hi}$ considered in this paper are only examples of high-level heuristics that can be optimized.

Comparison to finetuned LLM and diffusion models.

The reason that we only included few-shot prompting baselines in the end-to-end evaluation of Table 1 is because, to our knowledge, there is no natural-language-to-crystal-structure dataset available for finetuning LLMs to generate crystals from loose natural language descriptions, as this work is among the first to study this problem. Hence, few-shot prompting is the most natural baseline. However, we do evaluate the subcomponents of GenMS to previous baselines, and show that the formula-conditioned generation in GenMS performs favourably compared to diffusion model [1] and finetuned language model baselines [2] in Table 5 and Table 6 of the paper. [1] and additional work on diffusion models has demonstrated better performance than GNNs ([3]), hence we did not include comparison to GNN generation.

[1] Yang, et al. Scalable diffusion for materials generation. arXiv preprint arXiv:2311.09235 (2023).

[2] Antunes, et al. Crystal structure generation with autoregressive large language modeling. arXiv preprint arXiv:2307.04340 (2023).

[3] Zeni, Claudio, et al. "Mattergen: a generative model for inorganic materials design." arXiv preprint arXiv:2312.03687 (2023).

Detailed explanation of LLM and property GNN used

The LLM and GNN used are described in detail in the previous work [1] and [2]. We have included these detailed descriptions in the appendix for the final version of this paper.

[1] Reid et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024

[2] Amil Merchant, Simon Batzner, Samuel S Schoenholz, Muratahan Aykol, Gowoon Cheon, and Ekin Dogus Cubuk. Scaling deep learning for materials discovery. Nature, 624(7990):80–85, 363 2023.

Why chemical formula

Chemical formulas are both connected to low-level properties of structures (e.g., conductivity) and high-level semantics that a user might specify using natural language (i.e., a crystal with carbon and silicon). Hence, chemical formulas are natural intermediate information. Moreover, chemical formula exists both in high-level datasets such as textbooks, wiki articles, and research papers, as well as in low-level crystal structure databases such as Materials Project [1] and ICSD [2], making them a natural interface as LLMs and diffusion models are trained on high and low level datasets, respectively.

[1] Jain, Anubhav, et al. "Commentary: The Materials Project: A materials genome approach to accelerating materials innovation." APL materials 1.1 (2013).

[2] Hellenbrandt, Mariette. "The inorganic crystal structure database (ICSD)—present and future." Crystallography Reviews 10.1 (2004): 17-22.

Why UNet

We have experimented with different architectures including only using transformers with local residual connections as opposed to a UNet-like architecture, but these alternatives did not work as well in practice. We did not experiment with GNNs for generation, because GNNs with graphs represented by adjacency matrices scale quadratically in the number of atoms. On the other hand, the modeling complexity of GenMS only scales linearly with the number of atoms, as the compact crystal representation of GenMS does not consider edges.

Limitations of only generating chemical formula

Generating formula is in fact nontrivial, especially for complex structures such as metal-organic frameworks (MOFs), which often have long formulae. Formulae implicitly capture rich human knowledge such as charge balancing, conductivity, etc. This knowledge is implicitly captured by LLMs and elicited during the formula generation process. That being said, we believe that formulae do not have to be a hard interface between high-level semantic and low-level structure generation. Exploring alternatives like including additional textual information such as space groups are interesting future works to explore, as well as directly generating structures from loose natural language, once such a dataset becomes available for finetuning LLMs.

Thank you for the thoughtful feedback. Please see our response below.

More comprehensive baselines

The reason that we only included few-shot prompting baselines in the end-to-end evaluation of Table 1 is because, to our knowledge, there is no natural-language-to-crystal-structure dataset available for finetuning LLMs to generate crystals from loose natural language descriptions, as this work is among the first to study this problem. Hence, few-shot prompting is the most natural baseline. However, we do evaluate the subcomponents of GenMS to previous baselines, and show that the formula-conditioned generation in GenMS performs favourably compared to diffusion model [1] and finetuned language model baselines [2] in Table 5 and Table 6 of the paper.

[1] Yang, M., Cho, K., Merchant, A., Abbeel, P., Schuurmans, D., Mordatch, I., & Cubuk, E. D. (2023). Scalable diffusion for materials generation. arXiv preprint arXiv:2311.09235.

[2] Antunes, Luis M., Keith T. Butler, and Ricardo Grau-Crespo. "Crystal structure generation with autoregressive large language modeling." arXiv preprint arXiv:2307.04340 (2023).

Concerns around contribution for a research paper.

We would like to highlight the nontrivial aspects of this work. Even though the solution of combining LLMs with diffusion models might seem trivial, the reason behind such a design and the process of how GenMS is the first to arrive at this design is nontrivial. Our contributions include recognizing the knowledge gap in high-level textbooks and low-level materials structure datasets, recognizing the gaps between models of different abstraction levels (LLMs for high-level semantics and diffusion models for low-level details), and properly addressing these gaps using both models at the abstraction level with abundant data. We hope that these insights, as well as our novel hierarchical, multimodal search algorithm may inspire solutions to problems in other domains that involves discovery using natural language.

Thank you for recognizing the significance of this work! Please see our response to your questions below.

Ad-hoc decisions in the framework.

We would like to clarify that the choice of $R_\text{hi}$ and $R_\text{lo}$ are not ad-hoc decisions made by the framework, but highlighting what practitioners can enforce in the materials search process. Checks currently implemented in the current framework are rather basic, e.g., formulae containing the desired elements, formulae consist of valid chemical elements, don’t contain duplicates, and have desired element ratios. There are certainly other checks that can be incorporated into the current framework, such as "must contain element Si" and "have less than 5 chemical elements", which gives a domain expert additional control in the generation process. We have rewritten lines 116-120 to reflect that $R_\text{hi}$ and $R_\text{lo}$ are only examples of what can be optimized in GenMS, as opposed to arbitrary choices we made.

Additional ablations.

As discussed in the previous paragraph, the goal of GenMS is not to develop the best heuristics but to allow diverse checks and constraints to be incorporated. That being said, the choice of what information to include in RAG is indeed an interesting ablation. We conducted two additional studies. First, we eliminate half of the retrieved Wiki text, and observe the match rate in Table 4 drops from 0.89 to 0.85. Second, we replace the text retrieved from Wikipedia by text retrieved from Materials Project [1] (entries that contain the keyword Perovskite), and observe that the match rate in Table 4 drops to 0.86, which is still better than no retrieval at all. We will include these additional results in the final version of the paper. However, we were unable to conduct more ablations, such as using research articles containing keywords as retrieval text in RAG due to the lack of a dataset with easily retrievable research articles in materials science. We leave this investigation for future work.

[1] Jain, Anubhav, et al. "Commentary: The Materials Project: A materials genome approach to accelerating materials innovation." APL materials 1.1 (2013).

Incorporating LLMs for formula checking.

We agree that using LLMs for formula checking is more general than using regular expression checkers. In fact, we are already using LLMs to rank the top W formulas as described on line 122. To investigate the reviewer’s suggestion of LLM feedback, we use Gemini long context [1] to judge whether structures are nonmetals. For 25 known nonmetals, LLM determins 23 are nonmetals, achieving a success rate of 92%. However, using LLM feedback for novel structures is more challenging, as we do not have ground truth for whether the LLM feedback is correct.

We believe that similar to RL with feedback for LLMs, where diverse heuristic and reward functions such as human feedback, AI feedback, and machine feedback have been jointly used to improve system performance, GenMS can also benefit from diverse heuristics and sources of feedback, including feedback from LLMs.

[1] Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean365 baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 366 Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv 367 preprint arXiv:2403.05530, 2024

Latency of the system

Thank you for bringing up this question. We have measured the latency of the system on TPU-V4 with batching enabled (using batch size 64 for the structure branch and energy GNN), and report the results below.

The latency of the language branch is measured on Gemini long context model and latency of structure branch is measured on the compact diffusion model. The latency of the energy prediction GNN is high due to the paddings applied to the structures to pad structures to the same number of nodes, and due to the breadth of the search (i.e. predicting energies for 2500 structures). We will include this result in the final version of the paper.

Concerns around the requirement of domain knowledge

While it is compelling to eliminate the need for any domain knowledge, we think that some level of domain knowledge is hard to avoid, especially given the end goal is to have materials scientists use the framework to discover and even synthesize novel materials. Therefore, the framework should provide a way for material scientists to specify and verify the generation. Instead of getting rid of domain knowledge, GenMS leverages domain knowledge both through in-context learning and through heuristic/reward design, which gives practitioners the flexibility to generate and verify crystals based on their need.

That being said, we are not leveraging more domain knowledge to evaluate GenMS than previous work such as [1] and [2]; the validity metrics in Table 1 are standard in previous generative models for materials work. The results of GenMS are easy to reproduce, with the language prompt provided in Appendix A.2 and validity metrics following previous work ([1], [2]).

[1] Xie, Tian, et al. "Crystal diffusion variational autoencoder for periodic material generation." arXiv preprint arXiv:2110.06197 (2021).

[2] Yang, M., Cho, K., Merchant, A., Abbeel, P., Schuurmans, D., Mordatch, I., & Cubuk, E. D. (2023). Scalable diffusion for materials generation. arXiv preprint arXiv:2311.09235.