We sincerely appreciate your valuable feedback and insightful questions regarding our manuscript. Your comments have provided significant guidance to enhance our research, and we are grateful for your thoughtful review.

The latexdiff with the updated paper is available in the Supplementary Material at the following URL: https://openreview.net/attachment?id=NVKwjCIAAX&name=supplementary_material

Below, we address the weaknesses and questions you have raised.

—---------

Weakness1 :The novelty of the method To optimize the property of materials with the gradient of property prediction methods is not new. It was proposed in section 5.3 of [1]

Thank you for your comment regarding the novelty of our method. While it is true that both our approach and the model proposed in Section 5.3 of the CDVAE paper (referred to as CDVAE-5.3) utilize gradients for optimization, there are three significant advantages.

Firstly, CDVAE-5.3, may experience a decrease in predictive accuracy due to its inability to utilize the most accurate models. It is important to note that CDVAE-5.3 uses a latent space for exploring new materials, a technique very similar to FTCP, which we used as a benchmark in SMOACS. A crucial difference lies in the use of latent variables for property prediction. As discussed in Section 4.1, CDVAE-5.3, similar to FTCP, may experience a decrease in predictive accuracy due to its inability to utilize the most accurate models. SMOACS circumvents this limitation, as demonstrated in Table 1.

Secondly, the entanglement of different crystal components within the latent space can complicate the preservation of specific crystal structures. FTCP struggles with this issue, as shown in Table 3. CDVAE-5.3 may also struggle with this issue. In contrast, SMOACS directly handles crystal structures, allowing for the optimization while maintaining their specific configurations, as outlined in Section 3.3.

Thirdly, CDVAE-5.3 is unable to ensure electrical neutrality and preserve specific crystal structures because generative models are probabilistic models. While FTCP also fails (and CDVAE-5.3 may fail) to maintain these aspects, as shown in Table 3, SMOACS successfully performs conditioning-based optimization, as demonstrated in Sections 4.3 and 4.4. This capability is crucial in materials design outlined in our Introduction.

We appreciate your feedback and have realized the necessity of clarifying these distinctions. We have now included a new paragraph to show the novelty in Introduction as follows: "SMOACS is the first method that directly optimizes the space of crystal structures using a gradient-based approach. We achieve this by making the entire crystal structure differentiable, which involves decomposing it into various components and representing atomic species as atomic distributions. Unlike traditional methods that convert crystal structures into latent variables—thereby entangling their elements—our approach maintains the independence of each component. This independence facilitates the preservation of crystal structures and ensures electrical neutrality by precisely specifying the atoms at each site. Furthermore, unlike generative models that probabilistically generate materials satisfying certain conditions, our method can inherently guarantee electrical neutrality and the preservation of crystal structures. Moreover we can add additional constraints as long as they are differentiable."

Weakness2: The evaluation. only two properties are optimized, how about more properties like the mechanic properties? How to handle the conflict between different property prediction models.

Thank you for your valuable suggestion regarding the evaluation of additional properties. In response to your advice, we have expanded our experiments to include new properties, as summarized in Section A.13. To evaluate properties distinct from the electronic property (band gap), we conducted experiments using models trained on two datasets, which facilitate the optimization of mechanical properties (bulk modulus) and superconducting transition temperatures. For each of these properties, we conducted experiments using ALIGNN models trained accordingly. Although the success rate appears lower, we do not regard this as a significant concern. SMOACS demonstrates high efficiency, capable of optimizing thousands of samples within a few ~ 30 minutes using a single A100 GPU, as detailed in Section 4.2.

Our study now optimizes six properties in total: bulk modulus, the energy difference from the convex hull, critical temperature of superconductors, band gap, formation energy, and tolerance factor. This expansion allows us to address a broader range of properties, including mechanical properties, thereby providing a more comprehensive evaluation of our approach. Additionally, we add the following sentence to the first paragraph of Section 4 to ensure readers are aware:

“For experiments utilizing models trained on datasets other than MEGNet, please refer to Section A.13.”

Thank you very much for your insightful comments. Regarding the success rate, we would greatly appreciate it if you could specify the target success rate you consider appropriate for this problem, along with your reasoning. Additionally, if you are aware of any better methods than our initialization approach (please note that our method is not completely random but is based on physical principles), we would greatly appreciate it if you could share the details, as it would be valuable for our future research. We will address your other concerns in a subsequent response.

We appreciate your important comments.

1. Why CDVAE and FTCP can not utilize the most accurate models? The key idea of all these models including your SMAOCS is utilizing the gradients.

We would like to first clarify that FTCP does not use gradients. It employs the sampling mechanism of the VAE to sample around data close to the target properties, as described in Section A.6. As far as we know, there is no method applying gradient-based optimization to FTCP.

[Why cannot use most accurate models]: Firstly, authors of CDVAE and FTCP propose to predict physical properties from latent variables; On the other hand, the most accurate models such as ALIGNN and Crystalformer require crystal structures as inputs, which is distinctly different from latent variables. Therefore, CDVAE and FTCP cannot use such models.

[On the use of gradients]:Secondly, while both our approach and CDVAE use gradients, our approach does not require the training of a generative model such as CDVAE and therefore goes far beyond the contributions of CDVAE. We acknowledge that, in principle, it might be possible to connect models such as Crystalformer following the decoder of CDVAE (it was not originally proposed in CDVAE paper[1]). However, our method only requires an off-the-shelf, web-available, pre-trained ALIGNN, which differs from CDVAE that requires the training of a generative model. CDVAE is trained on the perovskite structure dataset, Peov-5, for generating perovskite structures, and this is necessary regardless of whether ALIGNN is used or not. In contrast, SMOACS does not require such retraining and can generate perovskite structures even using only the web-available ALIGNN trained on various crystal structures.

[Difference between FTCP and SMOCAS]: Thirdly, as detailed in “Response to your concerns and questions (1/3) Weakness 1,” the entanglement of different crystal components within the latent space complicates the preservation of specific crystal structures. FTCP (and CDVAE-5.3 may also) struggles to maintain electrical neutrality and specific configurations, as evidenced in Table 3. In contrast, SMOACS directly manages crystal structures rather than latent space, enabling optimization while preserving specific crystal structures, as outlined in Section 3.3, and successfully performs conditioning-based optimization. This capability is crucial for materials design, as discussed in our Introduction.

In conclusion, FTCP and CDVAE utilize latent variables for prediction, thereby they can not adopt the most accurate models such as ALIGNN as a predictive model. A significant contribution of SMOACS lies in its ability to incorporate off-the-shelf models without the necessity for additional training. By directly managing crystal structures, enabling optimization while preserving specific crystal structures and electrical neutrality. Thus, although both SMOACS and CDVAE employ gradient-based optimization, there are critical differences.

2. I still disagree with the claim that “multiple targeted properties can be optimized with SMAOCS” for two reasons. The experiments only show the optimization of the tolerance factor, band gap, and formation energy. However, the success rate is quite poor.

Thank you very much for your comment.

[Difficulty in new benchmarking] Firstly, the problem we are tackling involves exploring unknown territories and may not have solutions, unlike machine learning benchmark tasks with known correct answers; therefore, a low success rate is inevitable. For example, developing perovskite solar cells (Sections 4.3 and 4.4) present challenges, including the incorporation of lead and relatively low power conversion efficiencies (strongly related to the bandgap optimization we are pursuing), and may not have ideal materials. Consequently, their widespread practical implementation has remained limited.

[Comparison with other methods] Secondly, although we are tackling more challenging problems than other studies, our success rate is not significantly different. MatterGen[3], in Figure 4, performs energy minimization searches under the condition of a given crystal structure (i.e., solving two conditional problems), addressing a total of 14 conditional problems related to crystal structures. However, out of these 14 cases, seven have success rates below 20%, with some even falling below 10%. In our experiments involving crystals having a similar number of atoms (Section 4.3), we achieved a success rate of approximately 10 ~ 20% under five conditions: three optimization targets, electrical neutrality, and preservation of the crystal structure. In Section 5.3 of CDVAE, they calculate scores after selecting the best out of 10 samples; thus, when aligning their evaluation method with ours, their success rate would be roughly 1/10, falling below 10%. Therefore, compared to other studies, our success rate is by no means low.

[Practical point of view] Thirdly, we believe practical usefulness in materials design is more important than the absolute success rate. The ultimate goal is to synthesize promising materials, and if machine learning can propose promising candidates faster than synthesis, the method is valuable. As we showed in the “Additional experiment regarding Weakness 2,” we are able to obtain approximately 500 successful candidates per hour using a single GPU. Thus, the success rate of SMAOCS is sufficient for practical purposes.

[Ensuring electrical neutrality and preserving specific crystal structures] Fourthly, we have solved problems where existing methods such as FTCP and TPE failed, achieving accuracy sufficient for practical materials design. This demonstrates that we have, for the first time, solved a problem that was previously unsolved, which holds great significance. Particularly, the difficulty of calculating electrical neutrality discussed in Section 4.4 is noteworthy. This indicates that while materials proposed by methods other than SMOACS may achieve the desired properties, they cannot verify the electrical neutrality essential for realistic materials. In contrast, SMOACS exhibits a significant advantage by being able to bypass this limitation.

As discussed above, although the success rate may appear low compared to benchmarks such as ImageNet, we conclude that SMAOCS is sufficiently effective.

3. Randomly initializing the crystal structure is not a good way to handle the conflict between different property prediction models.

We apologize for the insufficient description given in section 3.2. We would like to emphasize that while our initialization is not done in a completely random way.

[physics based initialization] Firstly, our initialization method is not entirely random; it is based on physics domain knowledge. In Section 4.2, we select initial structures from the MEGNet dataset, mimicking common techniques such as element substitution, where new materials are derived from existing crystal structures. In Section 4.3, we use randomly distorted perovskite structures with specified atomic coordinates and charges to ensure they are likely to exist. This approach effectively leverages physics knowledge and is reasonable.

[local minima are natural from the physical point of view] Secondly, from a condensed matter physics perspective, conflicts between different property predictions are natural and inherently difficult to overcome. Although the crystal structure search space appears vast, practical materials occupy limited regions. Only a limited number of crystal structure types exist, and achieving specific properties can be challenging depending on the structure (e.g., the face-centered cubic structure family is typically metallic with a bandgap of 0 eV). Consequently, the search space is sparse with many regions where property gradients are zero (e.g., in the region of face-centered cubic structure family, gradient toward >0 eV might not exist), leading to conflicts and local minima. Overcoming these local minima is difficult and may result in unstable structures outside the data distribution. We believe this characteristic contributes to the low success rates observed in FTCP, TPE and SMAOCS.

[physically appropriate strategy] Thirdly, given the sparsity and gradient issues, starting from known crystal structures and optimizing locally is the most efficient strategy to overcome these conflicts. Our approach aligns with this by initiating optimization based on real crystal structures.

As discussed, our initialization method is efficient and utilizes physics knowledge. Your feedback has highlighted the need to improve our writing style. We have revised the opening sentence of Section 3.2 and included references to the Appendix for further details: “In SMOACS, the initial structures for the optimization process are obtained from two sources: an existing dataset and self-generated structures based on typical perovskites with modified lattice constants (for details, see Section A.6 and A.7).”

4. However, I still take this idea as incremental work. I encourage the author to explore a better way to handle the conflict of different property prediction models and improve the success rate.

As discussed above, our methodology efficiently addresses the conflicts using a physics-based perspective approach. Furthermore, we tackle challenging problems for which solutions may not exist, making a lower success rate inevitable; indeed, other approaches often fail under these conditions. The success rate of our method is sufficient for materials design and remains adequately practical in its current state.

Since overcoming local minima is challenging from the physical point of view, improving the success rate can be achieved by further exploring the vicinity of promising initial values rather than adopting better optimization techniques. For example, by using the results from the first SMOACS run to perturb promising initial values, we can further explore the surrounding area of promising solutions[4].

I hope that all of your concerns are resolved. Thank you very much for your time and consideration!

[4] Fujii, Akihiro, et al. “Enhancing Inverse Problem Solutions with Accurate Surrogate Simulators and Promising Candidates.” arXiv preprint arXiv:2304.13860 (2023).

—---------

Weakness 2: Compared with the generative approach (FTCP), SMOACS seems to struggle to find (meta-)stable, and thus synthesizeable, materials as measured by formation energy and inter-atomic distances. Furthermore, stability of a material is a relative property not directly computable from formation energy—for a material to be stable it needs to be in a more favorable energy state than competing phases---making it even harder to say that the materials found by SMOACS are potentially synthesizable. I would say that this is a limitation of the purely optimization-based approach (which only cares about optimizing the target properties of interest) compared with the generative approach (which also tries to generate structures that are “in distribution”, i.e. similar in some sense to the training data samples). I would have liked to see a more in-depth discussion about this limitation (??) of the proposed method.

Thank you for your comments and insights regarding the stability of materials. We acknowledge that SMOACS appears to struggle more than generative models in finding (meta-)stable materials. However, our primary objective is not solely focused on stability. For instance, FTCP often fails to satisfy the specified structure or bandgap requirements. We also agree that considering the possibility of other phases with different composition ratios is crucial in examining practical stability of materials. However, calculating all possible competing phases and determining which ones are the most stable is a very challenging task for both SMOACS and generative models. In fact, Fig. 2(b) of MatterGen's paper[1] reveals that a relatively small proportion of those that are generated by the generative model was actually the most stable. From this perspective, taking account of the practical stability of materials remains as a future challenge for both SMOACS and generative models.

We appreciate your feedback and will include a following discussion about the stability in the Conclusion section in the revised manuscript. “However, simply minimizing the formation energy does not guarantee that the material is synthesizable; it is necessary to carefully investigate the stability of competing phases.”

—---------

Weakness 3: The authors deliberately use worse property prediction models for the generative approach (FTCP). … I would have liked to see comparison with a generative model based on SOTA prediction networks.

As you have pointed out, in our optimization experiments using FTCP (e.g., Table 2), we employed a property prediction model with lower accuracy. However, the accuracy of the property prediction model does not influence the optimization results presented in Tables 2, 3, and 4, ensuring that our experiments remain fair. In these experiments, we optimized the crystals so that the predicted property values from the property prediction model fall within specified ranges (e.g., 2.00 ± 0.04 eV), rather than relying on the ground truth property values. In essence, we treated the predicted values from the property prediction model as virtual true values for the purpose of optimization. This facilitates a comparison between the optimization algorithms of generative models and SMOACS, under the assumption that each employs an ideal prediction model. Therefore, whether we use a state-of-the-art (SOTA) model or a less accurate one, the prediction accuracy does not impact the experimental outcomes. This approach ensures fairness irrespective of the prediction model employed. Nonetheless, we understand that this aspect might not have been clear in our paper. To address this, we have revised sentence in the introductory paragraphs of Sections 4.2 and add sentence on second to last paragraph Section 4.3, such as "We evaluated the structures based on the band gap and formation energy values predicted by their respective predictors."

We sincerely appreciate your valuable feedback and insightful questions regarding our manuscript. Your comments have provided significant guidance to enhance our research, and we are grateful for your thoughtful review.

The latexdiff with the updated paper is available in the Supplementary Material at the following URL: https://openreview.net/attachment?id=NVKwjCIAAX&name=supplementary_material

Below, we address the comments, weaknesses and questions you have raised.

—---------

Strengths : The proposed approach to directly optimize the representation of the material to maximize certain (ML-predicted) properties is interesting. … by adding explicit constraints to the optimizer (not illustrated in the paper)

Thank you for your valuable comments. Upon reading the final comment in the Strengths section, which states "by adding explicit constraints to the optimizer (not illustrated in the paper)," we realized that this part may be unclear. We have included the following paragraph in the Introduction to describe strengths beyond those you have pointed out.

“SMOACS is the first method that directly optimizes the space of crystal structures using a gradient-based approach. We achieve this by making the entire crystal structure differentiable, which involves decomposing it into various components and representing atomic species as atomic distributions. Unlike traditional methods that convert crystal structures into latent variables~\citep{REN2022314}—thereby entangling their elements—our approach maintains the independence of each component. This independence facilitates the preservation of crystal structures and ensures electrical neutrality by precisely specifying the atoms at each site. Furthermore, unlike generative models that probabilistically generate materials satisfying certain conditions, our method can fundamentally guarantee electrical neutrality and the preservation of crystal structures. Moreover we can add additional constraints as long as they are differentiable.”

—---------

Weakness1 : In the numerical evaluation the authors show a high “success rate” for SMOACS, …..the numerical evaluation leaves me wondering if this is really a fair comparison.

Thank you for your comments and insights regarding the numerical evaluation and comparison of the methods presented in our manuscript. We value your attention to the detailed implementation and the fairness of our comparisons. In response to your concerns, we'd like to first clarify that the implementation details for both the FTCP and TPE methods are fully documented in Sections A.6 and A.7 of the supplementary material. Concerning the fairness of the comparison, we have made deliberate efforts to ensure that both SMOACS and the alternative methods are evaluated under comparable conditions. As indicated in the last paragraph of Section A.4, various parameters were adjusted for the FTCP. With respect to TPE, we designed an objective function analogous to that used in SMOACS and optimize all objectives equally. To ensure consistency and fairness in comparison, we set 𝜆 = 1.0 in Equation 14 for SMOACS, preventing any advantage that could arise from adjusting weights of the objective functions. Additionally, the bandgap margin of 0.04 eV mentioned is not arbitrarily designed to favor SMOACS but is rather based on the typical discrepancies between DFT calculations and experimental values. Note that the superiority of SMOACS holds across various margin values as shown in Table A.10.

Thank you very much for your positive feedback and for raising our score. We sincerely appreciate your time and effort in reviewing our work.

To further address your concern, we kindly ask for more detailed guidance.

First, regarding the tuning of FTCP and TPE, as well as the comparison method with SMOACS, we have explained in "Response to your concerns and questions (1/4)" under "Weakness 1" that this constitutes a fair comparison.

Second, concerning your apprehensions about the band gap width, we have clarified in "Response to your concerns and questions (3/4)" under "Weakness 3" that using a fixed band gap width was necessary for a fair comparison. Additionally, in "Add experiments using the prediction errors of the predictors as margins for band gap target," we have conducted experiments using the prediction errors of the models as margins for the band gap target, as per your suggestion, and included the results in Section A.14.

if there are any specific concerns that remain unresolved, we would appreciate it if you could kindly point them out.

Thank you for your important comments. We will address your concerns by dividing our response into two parts: (1) the evaluation method of FTCP, and (2) the appropriateness of including FTCP as a comparative method.

(1) I still believe that the numerical evaluation is too tailored to the proposed method. I've read through the supplementary material, but it's still not clear to me how FTCP is set up to target the specified bandgap range (which is what results in a low success rate for this method).

[Reproduction of the methodology of the authors of FTCP] First, we have followed the methods proposed by the authors of FTCP in our experiments, so your concern about the evaluation being "too tailored to the proposed method" does not apply. For example, the experiment in Section 4.2 is conducted exactly as described in Section A.6: “FTCP selects initial data from the training dataset where the band gap is close to the target and the formation energy is below -0.5 eV, and then uses an encoder to convert this into latent variables. Next, we add noise to these latent variables using a normal distribution with a mean of 0 and a standard deviation of 0.6, then decode them back into crystal structures for evaluation.” This procedure is consistent with the method described in the Workflow section of the FTCP paper. And as described Section 4.2 :“We optimized and evaluated the structures based on the band gap and formation energy values predicted by their respective predictors ” from latent variables of FTCP because “It employs prediction branches to predict properties from these variables.” as described in Section 2. Moreover, as stated in Section A.4, we used the official implementation, thereby fully adhering to the material design method with specified bandgap proposed by the FTCP authors.

[Simultaneous optimization of more targets] Second, our paper proposes a novel problem of simultaneous multiple properties optimization. The number of property evaluation metrics in each paper is as follows: CDVAE (Section 5.3) evaluates one property, MatterGen evaluates two (e.g., crystal structure and energy (S.U.N)), FTCP evaluates four (bandgap, energy, electrical neutrality, and minimum interatomic distance), while our research evaluates six properties (bandgap, energy, electrical neutrality, preservation of crystal structure, tolerance factor, minimum interatomic distance). Therefore, conventional methods not designed to handle this complexity naturally achieve lower scores.

In summary, we have strictly followed the original FTCP method without intentionally distorting the evaluation process. Although the scores for other methods may appear low, this is expected because we are tackling a new and more difficult problem.

(2) And if it's difficult to guide FTCP towards the desired properties, then perhaps it's not the most suitable generative method for the comparison? (see e.g. https://www.cell.com/matter/fulltext/S2590-2385(24)00242-X for a review of generative models that can be used for property -> structure generation)

[FTCP is a common benchmark] Firstly, FTCP is a widely used benchmark in this field and is optimal for comparison. According to Google Scholar, FTCP has been cited 128 times, indicating its broad acceptance as a benchmark. It has also been used as a benchmark method in recent studies like MatterGen (Dec. 2023). Since we aim to compare our work with well-known methods in the field, FTCP is one of the most suitable choices.

[The number of evaluation metrics of FTCP is the most close to ours] Secondly, FTCP is appropriate from the perspective of the number of evaluation metrics. As previously mentioned, we are solving a problem involving six property evaluations. FTCP evaluates four properties while CDVAE and MatterGen do one and three properties. So it is suitable to compare our method with others that assess a similar number of properties.

[All generative models share common limitations] Thirdly, generative models are probabilistic models and share common limitations. Generative models such as FTCP generate materials probabilistically, and even when conditioned on specific crystal structures and electrical neutrality, they cannot guarantee these properties entirely. Additionary, they cannot apply adaptive constraints without retraining. Therefore, while using state-of-the-art generative models such UniMat or MatterGen might offer some performance improvement, they cannot fully address the challenges (also, UniMat and MatterGen's datasets and codes are not publicly available, making direct comparison difficult). In contrast, our method can guarantee these properties by restricting the optimization range and atomic distribution.It can also apply adaptive constraints without retraining. Consequently, similar results are expected regardless of which generative model is used, with some variations in scores.

[Success rate is competitive with the latest model] In fact, although it is not a direct comparison, our success rate is not significantly different from MatterGen[3] even though we are tackling more challenging problems than they are. In Figure 4 of MatterGen, they perform energy minimization searches (S.U.N) under the condition of a given crystal structure (i.e., solving two conditional problems), addressing a total of 14 conditional problems related to crystal structures. However, out of these 14 cases, seven have success rates below 20%, with some even falling below 10% even though they use larger dataset and fine-tuning. In our experiments involving crystals having a similar number of atoms (Section 4.3), we achieved a success rate of approximately 10 ~ 20% under five conditions: three optimization targets, electrical neutrality, and preservation of the crystal structure. Therefore, compared to MatterGen, our success rate is by no means low. In conclusion, FTCP is a widely recognized benchmark method with a number of property evaluations similar to ours. Additionally, there is no fundamental reason to expect significant differences even if other methods are used. Therefore, FTCP is an optimal choice for comparison.

We sincerely appreciate your valuable feedback regarding our manuscript. Your comments have provided significant guidance to enhance our research, and we are grateful for your thoughtful review.

The latexdiff with the updated paper is available in the Supplementary Material at the following URL: https://openreview.net/attachment?id=NVKwjCIAAX&name=supplementary_material

Below, we address the weaknesses you have raised.

—---------

Weakness 1 : Novelty: The method uses property prediction model and use backpropagation to optimize the properties. This kind of method has been employed in many prior works such as [1][2][3] and don't seem to be novel.

As you mentioned, employing backpropagation to optimize properties is indeed well-established, with applications across dynamics, image manipulation[4], and nanoscale material design[5]. However, to the best of our knowledge, our approach, which involves directly optimizing crystal structures via backpropagation, appears to be unprecedented. A significant aspect of our work is transforming non-differentiable features of crystal structures—such as crystal types and elemental species—into a differentiable format, which is a contribution in this area. Although Konno et al. [6] developed the transformation of compositions, we apply this method to incorporate site-specific constraints on atomic probabilities for gradient-based optimizations, which constitutes our novel contribution. Your comments have highlighted the necessity of situating our work within the broader context of existing literature. To address this, we have introduced a comprehensive discussion in the related works section under "Gradient-Based Optimization." Additionally, we have now included a new paragraph to show the novelty in Introduction as follows: "SMOACS is the first method that directly optimizes the space of crystal structures using a gradient-based approach. We achieve this by making the entire crystal structure differentiable, which involves decomposing it into various components and representing atomic species as atomic distributions. Unlike traditional methods that convert crystal structures into latent variables—thereby entangling their elements—our approach maintains the independence of each component. This independence facilitates the preservation of crystal structures and ensures electrical neutrality by precisely specifying the atoms at each site. Furthermore, unlike generative models that probabilistically generate materials satisfying certain conditions, our method can fundamentally guarantee electrical neutrality and the preservation of crystal structures. Moreover we can add additional constraints as long as they are differentiable."

—---------

Weakness 2 : One limitation of the method is that it heavily relies on good property prediction models. In the case of not enough data, the result may not be good.

Thank you for highlighting the dependency on the accuracy of property prediction models. Indeed, if a model lacks predictive accuracy, it becomes challenging to propose high-quality materials. Nevertheless, a key advantage of our approach is its capacity to leverage various high-performance models. This allows us to propose materials for crystal structures with sparse data—such as perovskites—by utilizing models that have been trained on larger datasets. We demonstrate this ability in Section 4.3, where we effectively optimize perovskite structures using the widely accessible the ALIGNN and Crystalformer models, which has been trained on the MEGNet dataset including various crystal structures. This example underscores our method's adaptability and effectiveness even with limited data scenarios.

—---------

Weakness 3 : Clarity: The method needs to clearly state how the method differs from the prior methods (not just listing the prior methods). Also, it needs to be more clear when introducing the method (e.g., in the section 3.1).

Thank you for your critical feedback. To address your concerns, we have clearly outlined this by adding a new paragraph in the Introduction section as described in Weakness 1.

—------

I hope that all of your concerns are resolved. Thank you very much for your attention!

[4] Xia, Weihao, et al. "Gan inversion: A survey." IEEE transactions on pattern analysis and machine intelligence 45.3 (2022): 3121-3138.

[5] Ren, Simiao, Willie Padilla, and Jordan Malof. "Benchmarking deep inverse models over time, and the neural-adjoint method." Advances in Neural Information Processing Systems 33 (2020): 38-48.

[6] Konno, Tomohiko, et al. "Deep learning model for finding new superconductors." Physical Review B 103.1 (2021): 014509.

Thank you very much for your valuable feedback and for raising our score.

[The latest diffusion-based models are not publicly available] Firstly, we would like to point out that comparing our method with the latest diffusion-based models is challenging due to their unavailability. As mentioned in the "Deep Generative Models" section of Related Works, there are limited studies that address both desired material properties and structural stability simultaneously, which is the focus of our research. Among these, UniMat[1] and MatterGen[2] are the most recent efforts utilizing diffusion models for this purpose. However, neither their datasets nor codes are publicly accessible, making it impossible to conduct direct comparisons. (Although UniMat has a GitHub page, the code is not provided.)

[The number of evaluation metrics of FTCP is the most close to ours] Secondly, FTCP is more appropriate from the perspective of the number of evaluation metrics than MatterGen and UniMat. The number of property evaluation metrics in each paper is as follows: CDVAE (Section 5.3) evaluates one property, MatterGen and UniMat evaluate two (e.g., crystal structure and energy (S.U.N)), FTCP evaluates three (bandgap, energy, electrical neutrality, and minimum interatomic distance), while our research evaluates six properties (bandgap, energy, electrical neutrality, preservation of crystal structure, tolerance factor, minimum interatomic distance). So it is suitable to compare our method with FTCP that assesses the most similar number of properties.

[Success rate is competitive with the latest models] Thirdly, although it is not a direct comparison, our success rate is not significantly different from MatterGen[3] even though we are tackling more challenging problems than they are. In Figure 4 of MatterGen, they perform energy minimization searches (S.U.N) under the condition of a given crystal structure (i.e., solving two conditional problems), addressing a total of 14 conditional problems related to crystal structures. However, out of these 14 cases, seven have success rates below 20%, with some even falling below 10% even though they use larger dataset and fine-tuning. In our experiments involving crystals having a similar number of atoms (Section 4.3), we achieved a success rate of approximately 10 ~ 20% under five conditions: three optimization targets, electrical neutrality, and preservation of the crystal structure. Therefore, compared to MatterGen, our success rate is by no means low.

[All generative models share the same limitations] Fourthly, we believe that FTCP serves as a suitable substitute for the latest diffusion models in our comparisons. Generative models, including FTCP (VAEs) and the latest diffusion models, generate crystals probabilistically and cannot guarantee electrical neutrality or specific crystal structures and cannot apply adaptive constraint without retraining. This fundamental limitation remains unchanged even with the latest diffusion models. In contrast, our method can ensure those. Our objective is to demonstrate that, unlike these generative models, our method can maintain electrical neutrality and specific crystal structures without retraining, as shown in Table 3. To summarize, FTCP is more appropriate from the perspective of the number of evaluation metrics. Additionally, FTCP is sufficient as a representative of generative models—including the latest diffusion models—and the latest diffusion models cannot be utilized because their data is not publicly available. Additionally, although it is not a direct comparison, our success rate is not significantly different from MatterGen.

We hope this addresses your concerns and will consider incorporating these comparisons into our revised manuscript.

[1] Yang, Sherry, et al. "Scalable diffusion for materials generation." arXiv preprint arXiv:2311.09235 (2023).

[2] Zeni, Claudio, et al. "Mattergen: a generative model for inorganic materials design." arXiv preprint arXiv:2312.03687 (2023).

Question 1 : How are you obtaining gradient signals for properties that are not part of the property prediction model, such as oxidation state?

Regarding the oxidation state, we do not utilize gradient signals to achieve electrical neutrality. Instead, we first determine D patterns of oxidation states that achieve electrical neutrality, as explained in Section 3.1. (e.g. [+2,−1,−1] and [+4,−2,−2] in Section 3.1.) We then employ the method illustrated in Figure 2 to create masks for each atomic site, thereby restricting the possible elements that can occupy each site. This ensures that the optimized materials are electrically neutral. Regardless of the optimization process's success or failure, electrical neutrality is inherently achieved through the predefined oxidation state patterns and the masking technique. Additionally, we realized that the procedure for obtaining the D patterns of oxidation states that achieve electrical neutrality might not have been sufficiently clear. To address this, we have enhanced the following sentence in Section 3.2 to clarify the 'D': In SMOACS, optimizations begin with crystal structures from a dataset or those randomly generated. These crystal structures must satisfy electrical neutrality and generate D of oxidation number patterns based on the compositions of initial crystal structures (see Section A.11 for details).

In our framework, we rely on gradient signals for optimization unless we implement specific strategies, such as the approach we used for oxidation states. For example, in Section 4.3, we optimize the tolerance factor because it is expressed through differentiable computations, enabling the use of gradient-based methods. However, certain properties pose challenges. The number of atoms in the initial structure cannot be described in a differentiable form, which means it cannot be optimized using gradient signals. As a result, the initial structure and the number of atoms it contains must be set as hyperparameters rather than optimized variables.

—---------

Question 2 : Can you add more details on the compute cost and infrastructure used to train SMOACS and how it compares to the baselines studied in the paper?

Thank you for your insightful question. We have compiled the software information and included it in the implementation details of Section A.4. We recognize the importance of providing details about the computational infrastructure used to train SMOACS, including information about GPUs and other resources. We have included this information in section A.4.

We did not include a comparison of the computational costs between SMOACS, TPE, and FTCP in the original manuscript because making a fair comparison is challenging due to constraints in our current computational setup. Specifically, SMOACS is implemented using PyTorch, whereas the official implementation of FTCP utilizes TensorFlow 1.x. TPE primarily runs on CPUs, but the component responsible for obtaining predicted values employs GPUs, and implementing parallelization for TPE is not straightforward. Due to these differences and the complexities involved in optimizing each method under varying hardware conditions, we have not conducted measurement experiments for a direct comparison.

Nevertheless, although we did not perform a formal comparison, we might say that SMOACS operates entirely on GPUs, which may be easy to facilitate more efficient generation of a larger number of samples compared to TPE. For example, using CrystalFormer, we can simultaneously optimize 2,048 crystal structures containing five atoms each, and the computation completes in a matter of minutes.

—-------

Question 3 : While SMOACS performs better than the bayesian optimization baselines, the success rate is still low in many cases. Can you provide some intuition as to why and how future work could improve on this?

Thank you for your thoughtful question. The reason the success rate is not higher could be primarily due to the difficulty in overcoming local minima. Gradient descent algorithms can be trapped in local optima, which limits the overall success rate in certain cases. However, we do not consider the low success rate to be a significant issue. SMOACS is highly efficient; using a single A100 GPU, we can optimize 2,048 samples in a matter of minutes. This allows us to repeat the optimization process multiple times, enabling us to obtain a large number of successful optimization samples. Your comments have underscored the importance of incorporating this information into the main body of our text. Consequently, we have added the following sentences:

“SMAOCS can easily scale this computation and can optimize 2,048 samples simultaneously in just a few minutes using a single A100 GPU. This allows us to repeat the optimization process multiple times, enabling us to obtain a large number of successful optimization samples. “ in Section 4.1

—-------

I hope that all of your concerns are resolved. Thank you.

We sincerely appreciate your valuable feedback and insightful questions regarding our manuscript. Your comments have provided significant guidance to enhance our research, and we are grateful for your thoughtful review.

The latexdiff with the updated paper is available in the Supplementary Material at the following URL: https://openreview.net/attachment?id=NVKwjCIAAX&name=supplementary_material

Below, we address the weaknesses and questions you have raised.

weakness1 : The experiments to just one dataset, which means the scope is a bit limited. The authors should consider running further experiments based on crystal structure datasets [1], as well as Perov-5 for perovskites [2].

One of the main strengths of our approach is its ability to maintain specific crystal structures during optimization using any pretrained model. Consequently, the dataset models trained are less crucial in our method. What is more significant is verifying that our technique can be applied effectively across diverse types of models. To validate this, we have conducted experiments with two web-available pretraineds: a GNN model (ALIGNN) and a Transformer model (Crystalformer). We demonstrated that our method can successfully optimize using both models. On the other hand, we understand your concerns. We are currently investigating whether we can conduct experiments using models trained on other datasets.

weakness 2 : The paper could be strengthened by adding more experiments with additional property prediction models, such as commonly used machine learning potentials [3] which should also be covered in the related work section. In the case that some models cannot be used (e.g., because they cannot predict relevant properties) this should also be explained.

Thank you for your valuable suggestions. After careful consideration, we have concluded that machine learning potentials serve different purposes from our research objectives. Machine learning potentials are primarily designed to predict energies and forces but typically do not extend to electronic properties like the band gap. Therefore, incorporating these models into our study may not align with our research objectives.

The number of factors being explored, our research is on par with other studies. MatterGen explores five factors: stability (S.U.N.), magnetism, limitation of atomic species (composition formula and supply chain risks), and 14 types of crystal structures (space symmetry). In our paper, we also explore five factors: in addition to stability, we consider eight types of band gaps (Table A.3, A.4, and A.5), crystal structures, tolerance factor, and electrical neutrality. Although the tolerance factor is not a value through a prediction model, please note that it is an optimization target as shown in Equation (17). Therefore, in Sections 4.3 and 4.4, we optimize three properties. Based on the above discussions, we believe that the extensiveness of our experiments is sufficient.

—----

weakness 3 : In terms of related work, the authors could also benefit from discussing the application of reinforcement learning [4] and GFlowNets [5] for crystal structure design. The same should also be applied to diffusion models [6], flow matching models [7] and language models [8] [9]. It would be good to add details about how those methods compare.

Thank you for your critical insights. Each of the suggested papers seems very important in the field of material design. We appreciate this opportunity to expand our discussion. We have updated the introduction and related work sections to include these papers.

However, we believe that Reference [6], which addresses CSP (Crystal Structure Prediction) to search for appropriate crystal structures from compositions, has a somewhat different objective from the material design we are conducting in the main text. Therefore, we have added a discussion in Section A.3 where we have done a similar experiment.

Thank you for your questions. We are pleased to provide clarifications below.

Question 1: Could you clarify if you need a model trained exactly on the property to apply SMOACS? If so, would it be possible to fine-tune a machine learning potential model toward that property?

Yes. Optimizing crystal structures with SMOACS to achieve a specific property requires a predictive model for that property. We acknowledge that fine-tuning machine learning potential (MLP) models toward the property of interest (e.g., band gap) should, in principle, allow applying SMOACS. (Of course, in this case, differentiable connections from input to output are also necessary.)

However, we are afraid that such fine-tuning for applying to SMOACS may not provide expected good performance because of their architecture optimized for prediction of energy and forces. For example, in all state-of-the-art MLP (e.g. [11], [12]) models, the total energy is expressed as a sum of atomic energy to attain transferability to systems having different number of atoms from those in the training dataset. Obviously, when predicting the band gap or Tc, such a form, that is, decomposition into individual atomic contributions, is unreasonable. Models like ALIGNN and Crystalformer, which do not decompose properties into atomic-level contributions, might be more appropriate for these tasks.

For this reason, we limited the models used in our experiments to those with proven performance in predicting properties such as band gap, formation energy, and bulk modulus, such as ALIGNN and Crystalformer.

Question 2: Why is the low success rate not a significant issue? SMOACS can optimize thousands of samples in a few minutes to about 30 minutes on a single A100 GPU. By repeating the optimization or running it in parallel on multiple GPUs, we can obtain dozens to hundreds of successful samples within an hour, even with a success rate around 10%.

SMOACS can optimize thousands of samples in a few minutes to about 30 minutes on a single A100 GPU. By repeating the optimization or running it in parallel on multiple GPUs, we can obtain dozens to hundreds of successful samples within an hour, even with a success rate around 10%. In materials design, candidates usually undergo time-consuming processes like DFT calculations or experimental synthesis. Since SMOACS generates promising candidates much faster than these steps, the low success rate doesn’t hinder the overall efficiency of material discovery. Therefore, the success rate isn’t a significant concern in practical applications.

While similar issues might apply to FTCP and TPE, their situations differ from SMOACS. In TPE, we must check electrical neutrality after optimization, even if all specified properties are met. This becomes serious when proposing large systems, as discussed in Section 4.4. FTCP requires extra verification to ensure both electrical neutrality and the specified crystal structure, adding complexity. In contrast, SMOACS inherently ensures electrical neutrality and the specified crystal structure, eliminating these concerns.Note that for the sample evaluation speed description, it is detailed in Section 4.2, not Section 4.1, as previously mentioned.

We hope the above addresses your questions and provides a clearer understanding of our methodology. Thank you again for your valuable feedback.

[11] Musaelian, A., Batzner, S., Johansson, A. et al. Learning local equivariant representations for large-scale atomistic dynamics. Nat Commun 14, 579 (2023). https://doi.org/10.1038/s41467-023-36329-y

[12] Park, Yutack, et al. “Scalable Parallel Algorithm for Graph Neural Network Interatomic Potentials in Molecular Dynamics Simulations.” Journal of Chemical Theory and Computation (2024)

Related Works and Writing

• v3Du presented several studies closely related to our research. We strengthened our paper by incorporating them into the Introduction and Related Works sections.

• p9py pointed out the lack of a concise formulation of the problem and description of the proposed method early in the paper. We revised our manuscript by adding a new first paragraph to the Introduction to clearly state these points.

• p9py highlighted the potential application of training-free conditional generative models to crystal structure generation. We have stated that no such training-free conditional generative models exist for crystal structure generation and have added the presented research to the related works section.

Limitations of the Proposed Method

• uNXA indicated that our method heavily relies on the accuracy of pretrained models, and if the accuracy of these models is low, good material proposals cannot be made. However, as p9py pointed out that it is "straightforward to use any available predictive model," we stated that this can be resolved by using high-quality off-the-shelf models. Considering that new and more accurate property prediction models are published at ICLR every year, it is reasonable to expect that our method's will be able to use more accurate models.

• stg3 expressed concern that the success rate is "poor." However, detailed justifications for this assessment were not provided. We pointed out that SMOACS is solving problems that previous studies could not address, as p9py pointed out. Furthermore, we provided evidence that MatterGen [2] and CDVAE [1], which tackle problems with fewer optimization targets than ours, have achieved similar scores to our results. Additionally, we demonstrated that obtaining 500 successful samples in 1 GPU-hour is sufficiently practical for material design. Moreover, as stg3 himself (or herself) pointed out, since we are considering a new problem setting in an unknown domain where solutions may not even exist, we argued that it is natural for the success rate to be lower than standard machine learning benchmark scores. Thus, the success rate is sufficient rather than ‘poor’.

• stg3 also expressed concern that randomly initializing the crystal structure is not a good way to overcome local minima. However, detailed justifications for this assessment were not provided. We answered that our initialization method is not entirely random; it is based on physics domain knowledge. In SMOACS, the initial structures for the optimization process are obtained from two sources: an existing dataset and self-generated structures based on typical perovskites. We mimic common techniques in physics such as element substitution. From a condensed matter physics perspective, although the crystal structure search space appears vast, practical materials occupy limited regions. Given this, our strategy of starting from known crystal structures and optimizing locally is the most efficient way to optimize crystals rather than using a good optimization technique to overcome local minima. We have included the details of the initialization in the main text of the manuscript and added a reference to the Appendix.

• p9py noted that, compared to generative models, there is a limitation in our method's ability to generate stable materials (being in a more favorable energy state than competing phases) and suggested that this should be discussed in the paper. However, MatterGen's paper [2] reveals that only a relatively small proportion of materials generated by the generative model were actually the most stable. From this perspective, accounting for the practical stability of materials remains a future challenge for both SMOACS and generative models.

Comparison with other methods

• p9py expressed concern that the evaluation method of FTCP used in our comparison might be "too tailored to the proposed." As we have thoroughly described in the main text and Appendix, our evaluation method strictly follows the methods of the FTCP authors, including the use of their code, and thus is not "too tailored to the proposed." Furthermore, as stg3 pointed out, our paper "proposes a novel problem of simultaneous multiple properties optimization" and we need to recognize that we are addressing a problem that is challenging for conventional material design methods. The scores for FTCP may appear low, but this is expected because we are tackling a more difficult problem.

• uNXA suggested comparing with the latest diffusion models. Additionally, p9py is concerned about whether there are other suitable generative models besides FTCP. We explain as follows: FTCP is the most appropriate from the perspective of the number of property evaluation (optimization) metrics. Additionally, FTCP is sufficient as a representative of generative models, including the latest diffusion models, because they share the same inherent limitations of generative models. Moreover, the latest diffusion models such as MatterGen cannot be utilized because their data and code are not publicly available. Furthermore, although it is not a direct comparison, our success rate is not significantly different from that of the latest diffusion model, MatterGen.

Others

All the points raised here are important, but they were explicitly stated in the Appendix of the initial manuscript.

• p9py pointed out that the implementation is unclear and that it is not evident whether a fair comparison is being made. We explained that we have already detailed this in the Appendix and that it constitutes a fair comparison.

• p9py noted that updating the graph data during optimization is necessary. We indicated that this is described in the Appendix A.4.

[1] Tian Xie, ICLR2022, “Crystal Diffusion Variational Autoencoder for Periodic Material Generation”

[2] Zeni, Claudio, et al. "Mattergen: a generative model for inorganic materials design." arXiv preprint arXiv:2312.03687 (2023).