Compositional Search of Stable Crystalline Structures in Multi-Component Alloys Using Generative Diffusion Models

We thank the referee for the feedback provided to the project.

We understand the fact that the presentation aspect of the work is not thoroughly designed for the machine learning community. To solve this, we are planning to do a major revision on the text. Many aspects of the presented framework are rooted deeply in materials science - for example, when utilizing Molecular Dynamics simulations, the stability of structures is a major concern. We will be more careful about the extent to which we describe such matters - whether they should constitute the main text of the article, or serve as confirmation within the appendix - in order to better suit the area of interest with our next submission. In the following, we are answering to the referee’s questions:

1. One data point is a crystal graph: apart from the node features (atom types), it is equipped with lattice vectors - those are three vectors along which the nodes repeat periodically. Lattice vectors can be seen as the features of the entire structure (graph), similarly to other input parameters: the value of the property (bulk modulus), and - for P-CDVAE - also a label that describes the phase (whether it’s an FCC or BCC structure). The output of the model has the same format and while theoretically, the sample space is not constrained to graphs of a certain size, in practice, all training examples were composed of 54 atoms, just like the training examples.

2. The input to the model is a crystal structure. The model is trained on the task of data denoising/reconstruction - after the structure is encoded, Gaussian noise is applied to the representation. The model learns to reconstruct the input graph and a loss function is applied based on the errors in reconstructed, which can be seen as a self-supervised task: in absence of external labels, a reconstruction task is performed based solely on the input data.

3. The crystal phase of the training examples (FCC or BCC) was straightforward to assess - the results of MD simulations aligned perfectly with the composition-to-phase relation visualized in FIG. 2, based on (Wu et al., 2017). Therefore, the structures in the dataset were annotated with corresponding crystal phase labels based purely on the composition. To assess the phase of the output structures, OVITO software was used. A loss function was applied to the model’s fully connected subnetwork to enforce correct prior phase prediction (against the phase labels). The training didn’t involve the loss function that explicitly punishes the mismatch between the phase of the emergent output structure and the label (or uses the phase of the output structure in any other way), but it did involve the coordinate loss. In that sense, the observation of the phase of output structure was not a part of the machine learning pipeline and was only conducted to analyze the results (fig. 4)

4. The goal of the operation was indeed to enrich the dataset with optimized structures that cannot be simply seen as sampled from uniform distribution (in terms of initial conditions, that then converge to locally stable spatial distribution). In hindsight, we admit that “augmentation” strikes as an inadequate name for this procedure - given that the goal of augmentation is usually to add examples to the dataset through simpler operations in order to account for simple feats (e.g. symmetries) of already existing cases.

5. We’re sorry to admit that this is one of the very unfortunate phrases that make the ideas behind the methodology obscure. A paraphrase that better conveys what was meant here is: “In order to cover more arbitrary orderings of atoms that are possible for a given atomic composition”.

We express our gratitude to the referee for acknowledging the novelty of the work in the high entropy alloy discovery field.

We also understand the referee’s concerns regarding the presentation aspect of the work. When correcting the paper, our most important focus will be to largely clarify the description of our methodology, mostly the model architecture and the improvements we’ve included in the source model. We have indeed used too much materials science and physics jargon. To address this, we will focus on explaining these concepts, such as section 3 of the paper as the referee mentioned, in a more comprehensive manner for a machine learning scientist reader. In addition, we will provide detailed information, together with clear examples, in the supplementary materials section. This includes defining the concept of “Phase” in this context. The other concepts that we recognize need further explanation and clarification are for example: BCC, FCC, CALPHAD, MD, DFT, MEAM and etc. All the mentioned presentation issues will be carefully taken care of in the new version of the paper.

We also agree with the referee on the issues regarding the figure captions and the “local-search” algorithm caption and description. In the new version, we will revise all the figure captions by adding more details about that specific figure. Also, we will add a detailed explanation of the algorithm with all the variables explained in the text. The LaTeX mathematical mode will be used for the equation as well. We will also work on the architecture of the paper, in a way that the readers have an easy time going through the paper.

The “related work” section will be indeed revised thoroughly, explaining how this work is connected with the aforementioned literature.

Regarding the generalization of the experiment, we need to mention that the choice of experimental data was mainly dictated by the availability of high-quality interatomic potentials, necessary for embedding the computationally efficient MD simulations within the workflow. This raised concerns about the generalization capability of the approach. While MD poses some limitations to employing the workflow to arbitrary kinds of structures, we will put more focus on highlighting the possibility of applying the neural model itself to other data. We will contrast the results with the baseline CDVAE model on existing datasets, such as the Carbon dataset that posed a challenge to the original architecture.

Finally, we are planning to make the dataset creation workflow available when the work is revised thoroughly. This will certainly make sense when the changes to the CDVAE architecture is reliable enough so that “any'' kind of dataset gives valuable results for high entropy alloy discovery.

We Thank the referee for the valuable feedback.

The G-vector based metrics are not used during the training of the model and are only utilized for final evaluation. The choice of scoring the local structure was made based on our primary assumptions regarding the task itself, as well as the way we expect CDVAE to act in our setting: Firstly, we assume that the training step of the diffusion model is indeed aimed at the exact reconstruction of the training example, not just the local patterns. That being said, we expect the local patterns to have a large influence on the bulk modulus: they introduce distortions in the crystal lattice, influencing the structure’s mechanical properties. To provide sufficient freedom for the model to form diverse patterns, we provide a relatively large supercell of 54 atoms. When vaguely saying “patterns”, we mean atomic clusters of various sizes: this could be, for example, a bunch of atoms with a bigger radius surrounding one with a smaller radius. Within the supercell, those patterns can appear shifted by symmetric transformations (translation/rotation), which doesn’t concern us at all, or separate clusters of atoms could be reconstructed correctly, but have different relative positions to each other, which we consider an imperfection, but not the most critical one.

To justify why we don't expect the perfect match overall, let's assume the following scenario: the model's fully-connected atomic composition submodule, which is queried before Langevin dynamics start, predicts that the structure consists of 52 atoms of one kind and 2 atoms of another kind. For training, noise is applied to the positions of those atoms - therefore, we ideally expect them to end up in the same place as before applying the noise. In sampling, the atoms will start in randomly chosen positions, to then form a regular lattice. We don't control for the exact placement of the two atoms of a certain kind, but we want their relative positions to be the same. One of the ways to check if this goal was achieved is to utilize tools to match two crystal structures in order to account for rotational or translational symmetries before evaluating the pattern. In the experimental setting of our choice, due to imperfect reconstructions by the model, as well as overall size of the matched structures, the usage of those tools proved to be very problematic. We opted for the alternative approach - producing the embeddings of each atom, based on the positions (distances and angles) and atom types of their neighbours. This descriptor-based approach also gives a measure of fulfilling our goal - in order to assume the perfect reconstruction, the relative positions/atom types of their neighbours have to be the same - only then, the atom’s embedding will be an exact match to the ground truth.

The model in question (one used for the final generation of structures) did use the local search data for training. While it found structures with highest bulk modulus than the best examples seen during training, the magnitude of improvement upon this baseline is indeed small. The inclusion of the examples produced with local search had significant impact on the accuracy of reconstruction of the structures - one can hypothesize that the initially produces dataset, which consisted only of randomly-initialized structures, was not the best domain for a diffusion model, which is expected to iteratively distinguish between the random noise and the underlying signal. Local search certainly helped in the regard of including information that is certainly “non-random”. As the degree of feature improvement may be seen as insufficient, we acknowledge that further tests should be done to better distinguish between what was seen during the training and what was not.

Both training and testing datasets consist only of 54-atom structures. Based on the training data provided, the model generalizes to different (and varying) numbers of atoms. 1150 examples were created for the database. We use the output of our crystal phase classifier both during the training and sampling. In practice, for the studied case, the distinction between different crystal phases is a trivial matter: it can be reliably predicted based solely on the summary formula (nevertheless, perhaps not as easy for the denoising GNN, which, for this application, is not equipped with any pooling layer or mechanism that could correctly view the structure “as a whole”; apart from the raw structure encoding passed to each atom). One can assume the difference between the two approaches mentioned in the question would be marginal on NiFeCr dataset, however, for the cases where this prediction is more intricate, the distinction between them could be much more relevant.