MatExpert: Decomposing Materials Discovery By Mimicking Human Experts

We appreciate the valuable feedback and have carefully considered the points raised. Below, we address your concerns. We hope our revisions clarify all the questions and strengthen the quality of our work.

W1 Unconditional Generation Task Clarification

AW1 We apologize for any confusion regarding the definition of “unconditional” in our comparison.

To clarify:

Misunderstanding. It seems there might be a misunderstanding regarding our use of the term "unconditional." In our context, "unconditional" refers to generating materials without any predefined constraints on properties such as bandgap, space group, or composition. The goal is to create stable materials that have not been seen in the database.

No contradictory. The retrieval stage in MatExpert operates as a retrieval-augmented-generation (RAG) technique, where we retrieve materials from the training set. This approach ensures that no new information or knowledge beyond what is available in the training data is introduced. The seed structure is selected randomly from the training set and does not impose any constraints, ensuring a fair comparison with other baseline methods. We will clarify this distinction in the revised manuscript to address any potential misunderstandings.

W2 Performance Improvement and Stability Measurements

AW2 We understand the importance of demonstrating the stability of out-of-distribution (OOD) generated samples. Here is how we consider this issue:

We define anything new in the generated materials as out-of-distribution (OOD). For the "unconditional generation" task, we have already calculated stabilities for all materials generated by MatExpert and the baseline models, including stability and metastability metrics. This calculation covers all generated materials that passed the validity tests.

There are two phases in our stability calculations:

Pre-relaxation with M3GNet: We initially estimate stability using M3GNet predictions.

Correction by DFT Relaxation: We further refine these estimates through relaxation with DFT.

This two-step approach ensures that our stability assessments are comprehensive and align with standard practices, providing a fair comparison with other baselines.

For the definitions of stability and metastability, please refer to Appendix C.6. For meta-stability, the calculation was applied to the generated materials that passed the validity test, which indicates the rate of compounds with relaxed hull energy $\hat{E}\_{\text{hull}} < 0.1$ by M3GNet. For stability, those identified as metastable by M3GNet undergo further DFT relaxation, and we report the percentage with $\hat{E}_{\text{hull}} < 0.0$.

Regarding "conditional generation," the convex hull of NOMAD is not publicly available, so we lack reference energy data for NOMAD materials. Due to computational resource limitations, it is not feasible to calculate the convex hull by ourselves for over 2 million materials in NOMAD. Therefore, we only focus on the performance of the generated materials that meet user-defined property constraints for "conditional generation”.

I appreciate the reviewer for the detailed responses.

My concerns have been partially addressed. Regarding W3, the authors provided experimental evidence showing that the proposed method is more stable than diffusion-based models, and it seems it is because of the RAG mentioned in W1.

However, it is still questionable for me how the authors conducted unconditional generation. The provided prompts are unclear if they are conditional or unconditional. Other reviewers have commented similar aspects, but I am not fully persuaded by the rebuttal.

For this reason, I'd like to raise the score to 4, but the system does not offer the rating of 4, so I am just leaving this comment here.

We appreciate the valuable feedback and have carefully considered the points raised. Below, we address your concerns. We hope our revisions clarify all the questions and strengthen the quality of our work.

W1 Novelty of Multi-Stage Generation

AW1 We appreciate your references to other works [1, 2, 3]. However, our contribution focuses on novel material discovery, which is distinctly different from the tasks addressed in those papers. Specifically, [1] is a positioning framework rather than a specific method; [2, 3] propose retrieval-augmented generation (RAG) methods for textual documents in material science. our approach introduces a new paradigm of retrieval, refinement, and generation for conditional material generation, leveraging material structures rather than NLP corpora.

[1] Miret, Santiago, et al. “Are LLMs Ready for Real-World Materials Discovery?”

[2] Zhang, Huan, et al. “HoneyComb: A Flexible LLM-Based Agent System for Materials Science”

[3] Chiang, Yuan, et al. “LLaMP: Large Language Model Made Powerful for High-fidelity Materials Knowledge Retrieval and Distillation”

The novelty of our work can be divided into the following points:

Novel Paradigm: Our framework introduces a new paradigm for material discovery by integrating retrieval, refinement, and generation stages. Specifically in the context of conditional material generation, MatExpert demonstrates remarkable flexibility and ability to generate a novel material conditioning on the user-predefined constraints on properties with this new paradigm and finetuned LLMs. To the best of our knowledge, it is the first to work with this paradigm and demonstrate effectiveness on large-scale material benchmark datasets.

Novel Retrieval Approach: Our retrieval approach, the first step of MatExpert, is different from traditional retrieval-augmented generation (RAG) approaches commonly used in natural language processing (NLP). Traditional RAG methods typically focus on retrieving relevant text snippets or knowledge pieces from large NLP corpora to enhance the generation process. In contrast, our approach is tailored specifically for material generation, where we retrieve material structures from databases instead of textual information. This structural retrieval serves as a seed for subsequent modifications, providing a tangible and scientifically grounded starting point for material generation.

Novel Cain-of-Thought Approach: In our framework, we use a Chain-of-Thought (CoT) reasoning approach to enhance the transition stage. GPT-4 takes both the true material structure (answer) and the user query as inputs to generate detailed modification steps (thought). These steps serve as ground-truth pathways for fine-tuning LLaMA. LLaMA, in turn, takes the query alone as input and is trained to generate both the modification pathways (thought) and the resulting material structure (answer). This approach enables LLaMA to apply expert-like reasoning, ensuring accurate and effective material modifications. With the inclusion of contrastive learning and CoT reasoning, our framework provides interpretable pathways for material modifications, allowing users to evaluate the reliability and reasonableness of generated outcomes.

Dataset Innovation: Our work collects and leverages the NOMAD database, comprising over 2.8 million materials, to set a new standard in material generation models. This extensive dataset, larger and more diverse than those typically used, like MP-20 or Carbon-24, reduces the risk of overfitting and enhances model generalization. Our approach not only improves material generation performance but also encourages the use of large-scale, high-quality datasets for more robust discoveries in materials science.

3. What’s the Novelty of Our CoT

We acknowledge that CoT reasoning has been widely studied and demonstrated to improve reasoning tasks in prior works. However, our method introduces distinct advancements tailored to the material generation context, which we elaborate on below:

1) Traditional CoT with Prompts:

• In traditional CoT methods, CoT reasoning is often achieved through prompt engineering, where step-by-step reasoning is embedded directly in the input prompts. These prompts guide the LLM to simulate a logical reasoning process for problem-solving tasks, such as multi-step mathematical or scientific reasoning.

• While effective, these CoT prompts typically rely on implicit reasoning paths derived from a single query-response interaction. They do not benefit from external, validated reference steps or explicit intermediate feedback loops.

2) Our CoT with GPT-4-Generated Ground-Truth Pathways:

In contrast, our approach leverages GPT-4 to generate detailed, step-by-step ground-truth transition pathways during the training phase. These pathways are not limited to a single static prompt but are derived from:

• A carefully designed training dataset of <source material, target material, transition pathway> triples.

• GPT-4’s generation capabilities, which provide a comprehensive breakdown of how the material properties and structures should be modified to meet the target specifications.

By using these ground-truth pathways, we explicitly integrate validated intermediate steps into the CoT reasoning process, enabling our system to replicate expert-like reasoning for the iterative modification of materials.

We hope this clarification addresses your concern regarding the novelty of MatExpert. Please let us know if this response resolves your questions or if further clarification is needed.

Inputs for MatExpert and HoneyComb

Below is a description a bulk material: The formation energy per atom is -1.828. The band gap is 1.1081. The energy above the convex hull is 0.0491. The elements are Co, Li, Mn, Ni, O. The spacegroup number is 5.

Based on the information, could you generate a description of the lengths and angles of the lattice vectors and then the element type and coordinates for each atom within the lattice?

1. HoneyComb

RAG outputs from HoneyComb

['"The answer to the question by running pre-defined tools, is: The information extracted from the academic papers will be used to generate a description of the lengths and angles of the lattice vectors and the element type and coordinates for each atom within the lattice.. The following are the tools used and the responses: [(AgentAction(tool=\'Arxiv Search\', tool_input=\'crystal structure lattice vectors elements coordinates\', log=\'This question requires understanding of crystallography and material properties. I should search for academic papers or articles related to crystal structures and lattice vectors.\\\\nAction: Arxiv Search\\\\nAction Input: \\"crystal structure lattice vectors elements coordinates\\"\'), \'Published: 2015-03-25\\\\nTitle: Crystal Structure Representations for Machine Learning Models of Formation Energies\\\\nAuthors: Felix Faber, Alexander Lindmaa, O. Anatole von Lilienfeld, Rickard Armiento\\\\nSummary: We introduce and evaluate a set of feature vector representations of crystal\\\\nstructures for machine learning (ML) models of formation energies of solids.
...
Here, is the fraction of the grain boundary monolayer available for segregated atoms at saturation, is the actual fraction covered with segregant, is the bulk solute molar fraction, and is the free energy of segregation per mole of solute. Values of were estimated by McLean using the elastic strain energy , , released by the segregation of solute atoms. The solute atom is represented by an elastic sphere fitted into a spherical hole in an elastic matrix continuum. The elastic energy associated with the solute atom is given by: where is the solute bulk modulus , is the matrix shear modulus , and and are the atomic radii of the matrix and impurity atoms, respectively. This method gives values correct to within a factor of two (as compared with experimental data for grain boundary segregation), but a greater accuracy is obtained using the method of Seah and Hondros, described in the following section. Free energy of grain boundary segregation in binary systems Using truncated BET theory (the gas adsorption theory developed by Brunauer, Emmett, and Teller), Seah and Hondros write the solid-state analogue as: where is the solid solubility , which is known for many elements (and can be found in metallurgical handbooks). In the dilute limit, a slightly soluble substance has , so the above equation reduces to that found with the Langmuir-McLean theory. This equation is only valid for . If there is an excess of solute such that a second ph\\""']

Final outputs from HoneyComb

To generate a description of the lengths and angles of the lattice vectors and the element type and coordinates for each atom within the lattice, specific crystallographic data is required. This includes detailed information about the unit cell dimensions, angles between lattice vectors, and atomic positions within the unit cell. The provided information about the bulk material, such as formation energy, band gap, energy above the convex hull, elements involved, and spacegroup number, does not include these specific crystallographic details.

1. **Lattice Vectors and Angles**: The spacegroup number (5) indicates a specific symmetry, but without additional data such as unit cell parameters (a, b, c) and angles (, , ), the exact lengths and angles of the lattice vectors cannot be determined.

2. **Atomic Positions**: Similarly, while the elements present (Co, Li, Mn, Ni, O) are known, the exact coordinates of these atoms within the lattice require detailed structural data, which is not provided.

In conclusion, the necessary information to describe the lattice vectors and atomic coordinates is not available in the provided text. Therefore, it is not possible to generate a description of the lattice structure based solely on the given information.

2. MatExpert

(Please refer to Figure 3 in our manuscript to view the results of the structure retrieval and the final outputs.)

This comparison highlights how structure retrieval delivers precise and property-aligned outputs, whereas RAG-based retrieval may struggle to achieve the same level of specificity and may fail to adequately respond to the query.

We hope this clarification addresses your concern regarding the design choice of structure retrieval. Please let us know if this response resolves your questions or if further clarification is needed.

We appreciate the valuable feedback and have carefully considered the points raised. Below, we address your concerns. We hope our revisions clarify all the questions and strengthen the quality of our work.

W1 Data Usage Clarification

AW1 We apologize for any confusion regarding the data used for training and testing. To clarify:

1. For unconditional generation, MatExpert was trained exclusively on the MP-20 dataset, aligning with the datasets used by other baseline models.

2. For conditional generation, MatExpert was trained exclusively on the NOMAD dataset.

It is important to note that MatExpert was not trained on both datasets simultaneously. We ensure that our comparisons remain fair and consistent across different models and tasks. We originally stated this in lines 315–317 and lines 389–392. The clarification is now further included in line 316, line 321, and line 370 in the updated manuscript.

W2 Figure 5 Clarification

AW2 Thank you for pointing out the issues with Figure 5. The last two bars represent MatExpert-80B ($\tau$=1.0) and MatExpert-80B ($\tau$=0.7). We also added numerical values on the top of each bar for clarity and included missing legends. Please check the figure in the updated manuscript.

From the figure, we can observe that:

• MatExpert consistently achieves high scores in both structure and composition diversity compared to CDVAE and various Crystal-LLM configurations, indicating its superior ability to explore a broad chemical space.
• MatExpert demonstrates the capacity to generate novel structures and compositions aligned with the distribution of the test sets, as shown by its consistent novelty scores close to the test set.
• Crystal-LLM faces challenges with larger models exhibiting lower novelty. In contrast, MatExpert consistently maintains high novelty across different model sizes.

W3 Retrieval Stage Limitations

AW3 Our method uses the retrieved material as a template, which inherently supports both the benefits and challenges of this strategy:

Role of the Template: The retrieved material serves as a foundational template, allowing the model to leverage existing structures that align closely with desired properties. This can enhance the accuracy of generated materials by providing a realistic starting point.

Trade-offs and Limitations: There is a trade-off between the extent of modifications possible and the quality of the predicted properties. While larger modifications may be more challenging and could impact accuracy, small modifications often maintain desirable characteristics of the original material and allow for incremental improvements.

Observations and Empirical Evidence: Empirically, we have observed that the limitations of this approach are not significant in our current framework. The use of templates generally supports successful material generation without substantial drawbacks.

Avoiding Cumulative Errors: Since the process of MatExpert involves only two rounds of conversation with one step of refinement, minimizing the potential for error accumulation. Our empirical results show that despite taking the risk of error propagation, the overall performance improvements justify the multi-stage approach.

Future Improvement: With additional computational resources, we can have the following improvement directions: 1) implement feedback loops to allow for iterative refinements, thus addressing any initial inaccuracies in the retrieval stage. 2) With tools like pymatgen and m3gnet, we can check whether the retrieved material at the initial step is similar to the requested property constraints to avoid error propagation. Additionally, we only use a very trivial retrieval approach with only one step of refinement to demonstrate the effectiveness of the multi-stage process, we believe with the most advanced RAG technique, it will be further improved.

We appreciate the valuable feedback and have carefully considered the points raised. Below, we address your concerns. We hope our revisions clarify all the questions and strengthen the quality of our work.

W1 Concern about computational complexity

AW1 We acknowledge that the multi-stage design may increase computational complexity compared to single-step models. Here’s how we consider this issue:

Comparative Complexity: We have two important baselines to consider: CDVAE and CrystaLLM. Our approach is more efficient than diffusion-based methods like CDVAE, as it avoids the complexity inherent in diffusion processes. While our method does have higher computational complexity than the one-step LLM approach used by CrystaLLM, it only requires two rounds of inference, resulting in approximately twice the time cost. Below is a summary of the inference speed for these methods:

Please note that due to differences in hardware, inference library implementations, and hyperparameters such as the number of iterative steps in CDVAE, the computational costs are provided only for reference.

Better performance: As our framework operates offline rather than in real-time, it is important to consider the trade-off between efficiency and performance. While our method involves a multi-stage process, it does not incur exponentially higher time costs compared to other state-of-the-art methods. Instead, it achieves significantly better performance, demonstrating the superiority of our approach in balancing computational efficiency with enhanced material generation quality.

Efficient Measures and Further Improvement: We have implemented efficient algorithms, such as FlashAttention for the 70B model, to streamline the process and reduce computational overhead without compromising performance. We are open to further improving computational efficiency, especially as more open-source resources like efficient LLM inference libraries become available.

W2 Concern about cumulative errors of multi-stage process.

AW2 We acknowledge the concern regarding cumulative errors in the multi-stage process. Here’s how we consider this issue:

Only limited rounds: Since the process of MatExpert involves only two rounds of conversation with one step of refinement, the potential for error accumulation is minimized. Our empirical results show that despite taking the risk of error propagation, the overall performance improvements justify the multi-stage approach.

Further Improvement: With additional computational resources, we can have the following improvement directions: 1) implement feedback loops to allow for iterative refinements, thus addressing any initial inaccuracies in the retrieval stage. 2) With tools like pymatgen and m3gnet, we can check whether the retrieved material at the initial step is similar to the requested property constraints to avoid error propagation. Additionally, we only use a very trivial retrieval approach with only one step of refinement to demonstrate the effectiveness of multi-stage process, we believe with the most advanced RAG technique, it will be further improved.