Scalable Diffusion for Materials Generation
We scale up diffusion models on a novel unified representation of crystal structures and generate orders of magnitude more novel stable materials verified by Density Function Theory calculations compared to previous generative modeling approaches.
摘要
评审与讨论
This paper introduces a diffusion-based method for material generation. To adapt to the typical diffusion framework such as DDPM, the authors propose a new crystal representation, claiming to represent any crystal structures, for compatibility with UNet inputs. This representation integrates the locations of atoms within the crystal as additional dimensions of the element table, resulting in an image-like tensor analogous to those used in image-based diffusion models. In addition to applying existing metrics for evaluating material generation, the authors also propose a new evaluation approach using DFT to evaluate the physical validity of the generated materials. Experiments comparing a set of existing methods together with conditional generation are conducted to demonstrate the effectiveness of the proposed method.
优点
-
The paper presents a solid and novel technical contribution. The methodology distinctively diverges from existing diffusion-based material generative models by introducing a novel representation, more attuned to image-based diffusion models, which has been extensively explored in existing literature.
-
A notable strength of this paper lies in the introduction of a new evaluative metric designed to assess the physical validity of generated materials. The emphasis on the synthesizability of material generation is a refreshing approach, addressing an area that has seen limited exploration within the community. Using the DFT relaxation method from material science enhances the practical applicability and utility of the material generation method, providing deeper insights into its effectiveness and practicality.
-
The experimental outcomes presented in the paper are quite promising. The proposed method outperforms existing models by significant margins in most evaluated cases.
缺点
-
The authors claim that the proposed representation is capable of capturing any crystal structure, as well as being "scalable and flexible." While the completeness of the representation is clear and well-articulated, there appears to be a lack of detailed discussion regarding its redundancy or compactness. It would be beneficial if the authors could provide further explanations or experimental insights that illuminate how compact or redundant this representation might be in practice.
-
In the section discussing conditional generation, the use of conditioning variables seems somewhat unclear. Specifically, on Page 4 under "Conditioned Diffusion with UniMat," the conditioning variables are directly concatenated with the noisy material along the last dimension. However, this approach raises questions as there appears to be a disparity in the feature spaces of the conditioning variables and the noisy material. For reference, in image conditional generation, cross-attention modules are commonly used to align the input condition (like text) with the image space effectively. Without the incorporation of a similar module in this work, the mechanism by which the conditioning variables guide the generation process remains unclear.
-
Regarding the method's performance, it is noted that the proposed method does not attain 100% validity on larger datasets like MP20, in contrast to simpler approaches such as CDVAE. It would enhance the paper if the authors could delve deeper into this issue, offering more insights or explanations. Including potential solutions or future directions in addressing this limitation in the limitation section would also be quite valuable.
问题
-
In Figure 1, it might be beneficial to improve clarity by adding more descriptions or labels to the element table. It actually takes me some time to decipher that it represents an element table. Providing a more explicit explanation regarding the motivation and benefits of utilizing the element table for representation would also be advantageous. It seems that one evident benefit is the shared similarities in properties among neighboring elements in the table, potentially providing a useful prior for generation. Incorporating such observations and expanding on the motivations in the Introduction section would also be helpful.
-
In Section 2.2, it could be helpful to include references to DDPM and extend the discussion slightly to incorporate considerations of other diffusion models, elaborating on why DDPM was the chosen approach. The discussion doesn’t need to be overly complicated: a straightforward explanation, such as the effective performance of DDPM in the authors’ use case, accompanied by some contextual background, would enhance the readers' understanding.
-
In the section "Conditioned Diffusion with UniMat" on Page 4, the statement "While the unconditional ... training distribution" could be refined for precision and accuracy. It might be more accurate to state that DDPM primarily learns the score function rather than directly learning the training distribution, making it challenging to quantify the extent of overlap between the learned and training distributions.
-
The section "Drawbacks of Learning Based Evaluations" in Section 2.3 is quite motivating. However, it might be more seamlessly integrated by briefly mentioning its main points in the Introduction. This could help prepare the reader for the detailed discussion that follows in Section 2.3.
-
On Page 5 the reference format is wrong at the end of the second paragraph.
Thank you for the positive feedback. Please let us know if our response below addresses your questions and concerns.
How compact or redundant is the UniMat representation.
The answer to this question depends on the types of materials we are modeling. For small crystals with only a few atoms in a unit cell, the UniMat representation is not compact and in fact quite redundant (e.g., 99% atoms are null atoms). However, this representation allows handling of complex structures with up to thousands of atoms. We have included this discussion in the conclusion section.
Align input conditions and generations effectively.
UniMat in fact uses self-attention layers to attend over the concatenated conditioning variable and noising materials. The skip connections in a UNet can been seen as a form of cross-attention (conditioning variables are passed to attention layer through the previous skip connection). This approach to conditioning is common in spatial and temporal super-resolution diffusion models [1] [2].
[1] Video Diffusion Models. Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, David J. Fleet. [2] Imagen Video: High Definition Video Generation with Diffusion Models. Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, Tim Salimans.
Low validity on MP20.
We have found MP20 to be a more difficult dataset where scaling up the model helps with validity (Figure 4). The validity metric from CDVAE is only a proxy metric that does not fully capture the generation quality. For instance, 100% validity could be due to overfitting to the training set and mostly generating training compositions. Hence, converge metrics and property statistics should be take into account, for which UniMat has better performance. We have included this limitation in the conclusion and limitation section.
Clarity of Figure 1 and motivation of periodic table representation.
We have updated Figure 1 to indicate each cells are entries in the periodic table. We have added additional motivations for the periodic table representation in the Introduction (second paragraph on Page 2).
The choice of DDPM.
Since the denoising process of a DDPM nicely corresponds to the process of gradually moving atoms in space until they reach their target location, we choose DDPM over other diffusion models (e.g., denoising score matching). We included this intuition on Page 4.
In the section "Conditioned Diffusion with UniMat" on Page 4, the statement "While the unconditional ... training distribution" could be refined for precision and accuracy.
Thank you for catching this detail. We have updated the text on Page 4.
Mentioning drawbacks in learning based evaluations in the Introduction.
Thank you for the suggestion. We have updated the Intro section (second to last paragraph on Page 2).
Dear Reviewer, we would like to ask if your concerns around the technical details of this work have been addressed, or if there were any other issues that would prevent you from increasing your score. Please let us know, and thank you for your time.
Thanks for the authors' rebuttal. I will keep my positive rating.
This paper proposes to use the diffusion model to generate novel crystal structures so as to discover novel materials. One challenge of crystal generation is the representation of a crystal structure. In this paper, the authors tackle this problem by using the atom locations in the the periodic table, the 3D coordinates of the atom in the crystal as well as maximum number of atoms per chemical element to represent a crystal. The authors proposed methods for evaluating the generated material.
优点
-
The proposed method is shown to be better than previous methods quantitatively in most cases (Table 1).
-
The proposed method generates crystal structures closer to those in the test set than the baseline method CDVAE.
缺点
-
There is no innovation in the diffusion model and the AI part. This paper just uses the standard diffusion model, and the conditional diffusion model to generate crystal structures.
-
I understand this paper may be a good paper for material science. Another venue related to material science, physics or chemistry may be a good venue to maximize the impact of this work. This paper presented at ICLR may have a small number of audience. In addition, Sec. 2.3, evaluating the generated materials using energy, is purely material science and has nothing related to AI. AI Researchers probably cannot evaluate the correctness and novelty of Sec. 2.3. Also, for the AI community, we do not learn any novel AI knowledge from this paper.
问题
I would suggest the authors submit this work to a more related venue to maximize the impact of this work.
Thank you for the feedback. Please let us know if our response below addresses your concerns.
Lack of innovation.
We disagree that this work has “no innovation in the diffusion model and the AI part”. Despite the advances in generative models, their adoption to material science surfaces many unanswered questions. For instance, whether crystals should be represented using voxel images, pointclouds, phase fields, or graphs is unclear. How to jointly model discrete atoms and continuous locations has not been previously addressed. How to properly evaluate generated materials has been largely unattended. Whether generative models can scale to large material datasets has not been considered. Our work provides the first set of answers to many of these questions.
Irrelevance to the AI community.
We disagree that “evaluating generated materials using energy is purely material science and has nothing related to AI.” How to properly evaluate generations lies at the core of generative models, where many metrics such as the Fréchet inception distance has been studied by the core AI community. Instead of leaving evaluation to material scientists, we believe it is more constructive for AI researchers to also understand and engage in application-specific evaluations of generative models. We also disagree that “we do not learn any novel AI knowledge from this paper”. In addition to the answers this work provides to crystal representations, joint modeling of discrete atoms and continuous locations, scaling up, and proper evaluations, this work illustrates diffusion models’ capability in modeling relationships other than spatial and temporal in images and videos (for which diffusion models were invented), opening up wider applications of diffusion models.
Furthermore, “applications to physical sciences (physics, chemistry, biology, etc.)" is listed as a subject area for ICLR 2024, and our submission significantly improves on the methodology and evaluation dimensions. These two dimensions cover two of the most important challenges of applying ML to any field.
Dear Reviewer, we would like to ask if your concerns regarding the novelty and relevance of this work have been addressed. Please let us know, and thank you for your time.
The paper presents an new approach to materials generation using diffusion models and a novel materials representation. The authors employ diffusion models, originally designed for image generation, to generate complex material structures.
This approach is to find broad applications in materials science and chemical engineering, addressing the long-standing challenge of efficiently generating diverse materials, especially in larger and more complex systems. The models jointly handle continuous atom locations and discrete atom types, overcoming challenges associated with large and complex systems. The models are trained and tested on several datasets and are compared with previous methods. The results show that the models provide better superior generation quality compared to previous state-of-the-art methods.
优点
The paper's approach is innovative, offering a fresh perspective on materials generation.
UniMat is the standout contribution of this work. It offers an elegant solution to the representation of materials, particularly in the context of the periodic table. The concept of sparsity in representation, with adaptability to chemical system size, is novel. The utilization of diffusion models together with UniMat represents a clever combination of ideas.
The generated materials of diffusion models are validated through DFT calculations. This rigorous approach ensures the stability and reliability of the generated structures.
The paper also provides a detailed background on related work in materials generation, diffusion models, and evaluation methods. This context helps readers understand the significance of their contributions. The training hyperparameters and computational resources provided in the appendix are clear and understandable.
缺点
The quality improvement of the paper is significant, especially for scaling up to large materials datasets. However, it would be helpful to provide a more in-depth analysis of the quantitative metrics and benchmarks used to make these comparisons.
问题
The focus of the paper is primarily on crystalline materials. Expanding the applicability of UniMat and diffusion models to non-crystalline or amorphous materials is an area that has not been explored but could be of interest to researchers in diverse fields.
The UniMat representation is a powerful concept, but its complexity might deter some researchers. Some examples in the appendix could be helpful.
It would be good to have more explanation about UniMat’s advantages. E.g. Will it save some memory or is it efficient in computing? These are also important when generating new structures.
Thank you for recognizing the significance of this work! Please see our response to your questions below.
More in-depth analysis of the quantitative metrics and benchmarks.
We totally agree that rigorous metrics and benchmarks are essential for further attesting scalability. Since we are the first to scale generative models for materials to large material datasets with millions of stable materials, we proposed standardized evaluation metrics and in-depth discussion on why these metrics make sense (Section 2.3). We hope these metrics will be utilized by future work for evaluating generated materials at scale.
Expanding UniMat to other materials.
This is indeed an exciting direction that we are actively exploring. In addition to non-crystalline or amorphous materials, the UniMat representation can also capture other chemical structures such as catalysts and proteins. We believe scaling to wide scientific data may eventually enable foundation models for science with broad knowledge in biology, chemistry, and quantum physics. We have included this in the conclusion section of the paper.
Additional examples of UniMat representation.
We have made animations visualizing the diffusion process to aid reader understanding. We will make this animation available in the final submission. We have also updated Figure 1 and its caption to make the UniMat representation easier to understand.
UniMat’s advantage.
The advantage of UniMat lies in modeling flexibility which enables scalability. UniMat also has benefit in computational efficiency compared to traditional search and substitution in materials discovery. UniMat has limited memory efficiency, as the representation is sparse. We have updated the conclusion to reflect these.
This work proposes a diffusion model for the task of material generation. Their model takes the material with atom locations as input and performs the denoising process by moving atoms from random locations to their original locations. The output results in crystals. The method is evaluated on three material generation datasets and compared against previous work in the topic.
优点
-
The paper is well-written and easy to follow. The theoretical background is well explained and clear.
-
The idea of modeling the atom movement for material generation using diffusion models and the denoising process is interesting and novel to the best of my knowledge. I am not an expert in materials science, so I am not sure about the method novelty here.
缺点
-
The utilized benchmarks seem to be saturated with values close to 100% performance. The performance gain is marginal and therefore could be a random improvement. Also, in some of the cases, the previous work has already achieved 100%, so there is no room for improvement.
-
There is another work that uses diffusion models for the same task on the same datasets [a]. Although [a] uses diffusion models in a different way compared to this work, it has similar or better performance in some cases.
[a] Pakornchote, Teerachote, et al. "Diffusion probabilistic models enhance variational autoencoder for crystal structure generative modeling." arXiv preprint arXiv:2308.02165 (2023).
问题
-
Since ICLR is an ML conference, the paper would benefit from explaining the different evaluation criteria and their importance in the material generation task. E.g. what are the property statistics exactly and do they have higher importance compared to validity?
-
The paper could be contrasted and compared against [a].
Thank you for the positive feedback. Please let us know if our response below addresses your concerns.
Existing benchmark saturated.
This is indeed the case. We have updated the text to reflect this (the last sentence of the first paragraph on Page 7). Saturation of existing benchmarks and unreliability of learning based evaluations (elaborated in Section 2.3) were indeed what motivated us to propose three metrics based on DFT calculations in Section 2.3. We found that despite UniMat and CDVAE have close performance in learning based evaluations, UniMat performs drastically better than CDVAE under DFT calculations (i.e., what considered to be near ground truth evaluation in material science).
Related work on diffusion model for materials.
Thank you for pointing out the related work. We have updated Table 1 with comparison to this work and the related work section. We note that out of 19 metrics being reported, UniMat performs better than DP-CDVAE on 16 metrics. We hypothesize that DP-CDVAE uses a separate VAE to predict lattice parameters and number of atoms, which limits its flexible in jointly modeling of lattice parameters, number of atoms, and atom locations.
Relative importance of evaluation metrics.
From our experience working with material scientists, the metrics in Table 1 can only serve as a proxy as opposed to reliable evaluation under the context of materials discovery, due to reasons listed in Section 2.3 (first paragraph). The property statistics are distributional distances (e.g., earth mover's distance) between property distributions (e.g., distribution of number of elements) in generated materials versus test materials. We have elaborated this more in the first paragraph of Section 3.1. If the property is predicted by another model (e.g., formation energy), the reliability of such evaluation is dependent on the quality of the other model, making the metric less reliable (compared to properties that don’t require prediction from another model, such as the number of elements). Not that property statistics are also proxy metrics, and the most reliable metrics should be DFT based.
Dear Reviewer, we would like to ask if your concerns around the saturated benchmark and related work have been addressed, or if there were any other issues that would prevent you from increasing your score. Please let us know, and thank you for your time.
I thank the authors for providing the feedback. The rebuttal addresses most of my concerns. The only concern that keeps me from increasing the rating is that I think the proposed method is novel in terms of the application in material science, but not much novel in terms of technical contribution in representation learning. Therefore, I keep my original rating.
We thank all reviewers for their feedback. We have updated our manuscript using blue colored text for any changes we made to the manuscript in addressing the feedback. Please see our detailed responses under each reviews.
This paper applies diffusion models to remote sensing. It trains a large generative model on high-resolution remote sensing data. It considers both unconditional and conditional cases.
This is a pure application paper and reviewers did not reach an agreement after discussion. In particular, Reviewer Xv1e was not convinced by the authors' response that this paper has sufficient innovation in the AI/ML part. This paper may have a significant contribution to the material science part, but I cannot evaluate it confidently as I have little experience in material science.
I think this this a borderline paper. I am recommending an acceptance but I wouldn't mind if the paper gets rejected.
为何不给更高分
Reviewer Xv1e was not convinced by the authors' response that this paper has sufficient innovation in the AI/ML part.
为何不给更低分
I think this this a borderline paper. I am recommending an acceptance but I wouldn't mind if the paper gets rejected.
Accept (poster)