Invariant Tokenization of Crystalline Materials for Language Model Enabled Generation

Dear Reviewer vCYe,

Thank you for your recognition that our approach addresses the challenge of representing crystal structures in a unique and invariant way under different mathematical descriptions. For your concerns and questions, we provide point-to-point responses below.

1. The paper should discuss the computational complexity differences between language models and specialized crystal models, as language models generally have higher parameter counts and computational demands.

Thank you very much for your suggestion, and we do think this is important to point out. Thus, we include the computational complexity and efficiency in generation comparisons with specialized crystal models like CDVAE and DiffCSP in Table 11 attached in the above general responses. As we can see in the table, LLMs based Mat2Seq indeed has more parameters, but with similar amount of GPU resources following DiffCSP running efficiency comparisons, Mat2Seq is significantly faster in generation.

We additionally show that decreasing the model size by 8 times will not influence the RMSE a lot and can further increase the running efficiency. For the smaller model, it achieved RMSE of 0.039 which is significantly better than DiffCSP with RMSE of 0.049, and more than 3 times faster in generation process.

2. The comparison of language models is limited. The experiments only compare with CrystalLLM, despite mentioning other methods.

Thank you for your question. We'd like to mention that we use Table 1 to directly show the failures of other two LLMs for crystal generation, and crystalLLM from Meta team that are finetuned instead of training from scratch is not directly comparable due to unfair training logic (they are using pre-trained large language models). We tend to believe the crystal structure prediction task established by DiffCSP and CrystaLLM is a reasonable and powerful benchmark to compare with, but other LLMs based crystal generation methods are either not directly comparable, or provide no evaluations for this task.

3. The rationale for comparing only with CrystalLLM in Section 4.2 and not with other methods from Section 4.1 should be clarified.

Thank you very much for this question. The reason for not including CDVAE and DiffCSP in Section 4.2 is also simple: we just want to compare with baselines in a fair way. For the tasks defined in Section 4.2, it requires the model to be trained on CrystaLLM dataset with 2.3 million crystal structures, while CDVAE and DiffCSP have only been trained on much smaller dataset like MP20 and MPTS52 with less than 50k training samples. It is unfair to compare with them, so we do not include them in the comparison in Section 4.2.

Additionally, the major contribution of this work, just as you mentioned before, is addressing the challenge of representing crystal structures in a unique and invariant way under different mathematical descriptions. To show this, we comprehensively compare our Mat2Seq with CrystaLLM that does not address this.

Furthermore, we tend to believe the performance gains beyond CDVAE and DiffCSP are already clearly shown by comparisons in Section 4.1.

4. For crystalline materials, isn't the original data already the most primitive unit? If we first apply Niggli cell reduction before using CIF files to describe unit cells, can we ensure a consistent representation?

Unfortunately no. Although usually the original data is the most primitive unit cells, there are a lot of different most primitive unit cells for the same crystal structure, as shown in Figure 1, like shifting periodic boundaries.

For your second question, actually the baseline method CrystaLLM uses Niggli cell reduction before using CIF files, but as we can see in Table 1, it fails to maintain a consistent representation for the same crystal structure before and after periodic transformations that will not change the crystal structure at all.

5. While generating continuous crystal structures in 3D space seems natural, I am curious whether this token generation method can ensure that the reconstructed 3D crystal structure is continuous. Specifically, can the generated sequences always be converted back into a meaningful crystal representation?

To show this, it is better to demonstrate the validity and stability of generated crystals from Mat2Seq. We have provided additional experiments for validity evaluations and stability evaluations in Table 8 attached above in the general responses. It can be seen that Mat2Seq achieves competitive validity ratio and remarkable stability ratios (around 50% of the generated structures can fall into the energy hull region of E_{hull} < 0.1 eV following FlowMM pipeline).

For your second question, as we can see from Table 8, CDVAE, DiffCSP, and the most recent FlowMM published after the NeurIPS deadline, none of them can guarantee a 100% validity rate, which means the generated crystals by these SOTA methods are not always meaningful.

6. When performing conditional generation, it appears that generating structures for each specific property requires fine-tuning under specific conditions. This could result in high complexity in practical applications.

Yes, currently we tend to utilize the pretrained model on that 2.3 M dataset because this pretrained model is available and potentially powerful beyond training from scratch. This kind of finetuning will potentially introduce less complexity compared with training from scratch. We will release the pretrained model for others to use.

Additionally, a foundation model covering various properties require a huge amount of crystal structures with corresponding properties. This kind of dataset is very expensive to establish and currently out of our scope. And this is a promising direction to move forward.

Thank you again for your questions. If you have any other questions, we are more than willing to answer.

Yours sincerely, Authors.

Dear Reviewer Sr8M,

Thank you very much for your time invested in reviewing our work. For your concerns and questions, we provide point-to-point responses below.

1. About authors' validation whether Mat2Seq helps LM learns from SE(3) equivariance or if the improved performance is due to the new token representation.

Thank you for this question. Actually, we'd like to point out that the only major difference between Mat2Seq and CrystaLLM is the SE(3) invariance 1D crystal sequence used, while the tokenization process is similar. To be specific, we used the exactly same model settings and very similar tokenization process following CrystaLLM, with the aim of showing the importance of SE(3) invariance and periodic invariance sequence representations. With CrystaLLM naturally served as the ablation of the inclusion of SE(3) invariance and periodic invariance sequence representations implemented by Mat2Seq, we show by comprehensive experimental results in the original manuscript (Tables 2 and 3) and in the attachment of the global rebuttal (Table 12) that Mat2Seq sequences will yield better performances.

Thus, by using CrystaLLM as a solid baseline and ablation, we believe the importance of SE(3) invariance and periodic invariance sequence representations for materials is clearly demonstrated.

2. We slightly organized your questions and concerns for the evaluation of Section 4.2 (Weakness 2, Question 4) and 4.3 (Weakness 2, Question 5), and provided responses below.

First of all, we agree with your point that for section 4.2 it is better to further include evaluation of the atomic distance differences between the real and generated structures using RMSE to better assess the generation accuracy. We provide detailed hit rate and RMSE in the new Table 12 attached above in general responses. It can be seen in the table that Mat2Seq significantly performs better than CrystaLLM in terms of Hit Rate (whether the generated structure matches with the literature structure), and RMSE.

Additionally, your suggestion for section 4.3 to include metrics like uniqueness, validity, and novelty is indeed very important for aiding researchers in discovering new structures with desired properties. Thus, we follow your suggestion and conduct evaluations for the generated crystals when conditioned towards lower or higher band gap values, with results shown in Table 9 attached above in general responses. It can be seen in Table 9 that Mat2Seq achieves remarkable validity with 88% for lower band gap condition and 90% for higher band gap condition, with good uniqueness of 98% and 92%, and good novelty of 86% and 99%.

3. About figure 1 and "Change Lattice" operation.

Thank you for your suggestion. Currently we show two types of periodic transformations that will alter unit cell structures significantly to demonstrate the failures of previous methods. For sure, we can update Figure 1 to include more demonstrative examples of all transformations that can result in different unit cell structures once we can update the paper.

For your second question, the "Change Lattice" operation does not correspond to cell expansion. Let's use a simple example to better demonstrate this. For example, you have a 2D material with atom at the origin of a cell, with cell lattices (0, 1) and (1, 0). A change lattice operation means you can actually change the lattice vectors to (1, 0) and (1, 1) without changing the area (or volume for 3D structures) of the cell at all. It is still a minimum unit cell, but with different lattice structures. We will also add these texts in the appendix once we have the chance, to enhance the clarity.

4. How were the values obtained, especially the 30% proportion?

Thank you for your question. The evaluation is conducted as follows: (1) for all crystal structure in MP 20 dataset, we transform it to a different unit cell representation without changing the crystal structure at all, e.g., by shifting the periodic boundaries as shown in Figure 1, to obtain a second representation of the same crystal. (2) We feed the original and the second crystal structure into 1D sequence representations of different methods, and then compare the differences between the output of these two different inputs for the same crystal structure. If the sequence mismatches, e.g., the resultant coordinates or type for the first atom is different, then it is a failure. (3) We go through the whole dataset and calculate the success rate. 30% means only 30% of the structures before and after the transformation will have the same sequence representation when using CrystaLLM.

5. The band gap zero is a classification task distinguishing metals from non-metals. This oversimplification could lead to misleading results.

Thank you for your question. However, we kindly disagree with you for this point. We want to mention that the band gap of a material is the energy difference between the top of the valence band and the bottom of the conduction band. It is calculated by subtracting the energy of the valence band maximum (E_v) from the energy of the conduction band minimum (E_c). We treat materials with Eg = E_c - E_v < 0.5 eV as a separate group, which will indeed include both true metals/semimetals with (Eg = 0) and small gap materials 0< Eg <0.5 eV), but we do not regard this group as pure "metal" group. Similarly, we separate materials with 0.5 <= Eg < 1.0 eV, 1.0 <= Eg < 1.5 eV, ... etc. as individual groups. This is just our current grouping protocol. For your interested applications, you can easily divide materials into different groups, e.g. separating Eg=0 as an individual group for metal electrode applications, and 0<Eg<0.5 as another group for mid-infared and thermoelectric applications.

With extensive additional experimental results and further clarifications, we hope we addressed your concerns and questions. If you have any other questions, we are more than willing to answer.

Yours sincerely, Authors.

Dear Reviewer 5Kst,

Thank you for your time invested in reviewing this work. We provide point-to-point responses to your questions and concerns.

1. Weakness 1

Thank you very much for raising this question. However, we kindly disagree with this and we feel there might be a potential misunderstanding of our proposed method.

Let's begin from a question. Could Niggli cell reduction algorithms or other widely used 1D crystal representations including CIF files achieve uniqueness or SE(3) invariance? The answer to this question is No. More specifically, Niggli cell reduction can only give you a set of lattice vectors and cannot determine a unique crystal unit cell. Additionally, there is a previous work CrystaLLM that has been using Niggli cell reduction when converting the crystal structures into 1D sequences. However, as we show in Table 1, CrystaLLM cannot achieve unique crystal 1D sequence representations.

Furthermore, let's look into our proposed method Mat2Seq and demonstrate how it achieves the desired properties including uniqueness, completeness, and SE(3) invariance. The niggli cell reduction is only used in the first step when we want to uniquely determine a set of lattice vectors. After that, we need to uniquely determine a corresponding unit cell. And then, we need to determine a unique ordering of atoms in the cell, and features used to completely represent the crystal 3D unit cell structures. The Niggli cell reduction is only an initial step of our proposed method, and to the best of our knowledge, Mat2Seq is the first work that can achieve uniqueness, completeness, and SE(3) invariance when converting 3D crystal structures into 1D sequences in the field of materials science. Thus, we kindly disagree that this problem is solved by previous Niggli cell reduction algorithms or any other widely used methods in materials science.

Last but not least, we want to point out that all previous LLM based crystal generation methods fail to achieve uniqueness and SE(3) invariance. It is valuable to propose a method to address this limitation of the current process in this direction.

2. Weakness 2

We appreciate your question. To clarify, as we mentioned in the beginning of section 3, we begin with Requirements for ideal crystal sequence representations in Section 3.1, and then move forward to how to determine a SO(3) equivariant unit cell in Section 3.2, then, Section 3.3 is used to demonstrate how to convert a SO(3) equivariant unit cell into SE(3) invariant 1D sequence.

Specifically, in Section 3.3, we show that for given determined SO(3) equivariant and periodic invariant unit cells $\mathbf{M}=(\mathbf{A}_u, \mathbf{P}_u, \mathbf{L}_u)$, we represent them by SE(3) and periodic invariant sequences that are complete to guarantee the full reconstruction of crystal structures.

More specifically, the SO(3) equivariant unit cell obtained in Section 3.2 cannot be directly used as input for LLMs, because a given crystal structure can have infinite number of SO(3) equivariant unit cells that differ by a rotation transformation.

Additionally, to show that the converted 1D sequence satisfies all the requirements, we provide detailed proofs in Section 3.4.

With these being said, we will appreciate any additional specific suggestions you have to improve this section, and we will revise the main paper accordingly once we have the chance.

3. Weakness 3

Thank you very much for your valuable question.

Following your suggestion, we further followed a very recent work FlowMM in ICML 24 published after the NeurIPS submission deadline to establish a fair comparison in terms of Validity, Stability, and S.U.N. It is worth noting that the calculation of stability and S.U.N. is very expensive (requires extensive DFT calculations) and only has been used by very recent works even published after NeurIPS deadline. We show the results in Table 8 attached above in general responses. It can be seen in the table that Mat2Seq achieves competitive results in terms of validity, stability, and S.U.N., even 28% better beyond FlowMM in terms of S.U.N. (stable, unique, and novel) that published in ICML 24 after NeurIPS deadline.

4. Question 1

Sure, we'd like to mention that a concrete example of the Mat2Seq input for a crystal structure is shown in Figure 2 in the main paper. The whole Mat2Seq sequence for that crystal structure is "formula Ag 4 Hg 2 I 8 \n space_group_symbol I-4 \n lattice_parameters \n a 6.5361 b 6.5361 c 13.1629 \n alpha 90.0000 beta 90.0000 gamma 90.0000 \n Ag 2 0.0000 0.0000 0.0000 \n Ag 2 0.0000 0.5000 0.7500 \n Hg 2 0.0000 0.5000 0.2500 \n I 8 0.2400 0.7596 0.6200 \n\n", where all these symbols and numbers are mapped to int values by a mapping dictionary.

5. Question 2

Thank you for your valuable question. The nature of Autoregressive language models is the conditional probabilities of the next token given its predecessors, $p(C_i|\theta)=\prod_{j=1}^{n_i}p(c_j | c_1:c_{j-1}; \theta)$. Thus, if you want to establish a conditional distribution $p_\theta (\cdot|s)$ to generate 3D crystal structures possessing the property $s$, it is natural to place the target property values at the beginning of the text other than anywhere else.

6. Question 3

No, the irreducible atom sets refer to the subset of the atoms in the minimum unit cell. For example, as you can see in the structure of Ag4Hg2I8 in Figure 2 in the main paper, there are 14 atoms in the minimum unit cell, however, the irreducible atom set only contains 2 Ag, 1 Hg, and 1 I. This is because the positions of other seven I atoms can be fully recovered by using I 0.2400 0.7596 0.6200 and the space group transformations for I-4 group, and that's exactly why there is a number 8 in "I 8 0.2400 0.7596 0.6200" after symbol I.

With these clarifications provided, we hope we addressed your questions and concerns. And if you have any other questions, we are more than willing to answer.

Yours sincerely, Authors.

Dear Reviewer JJwt,

Thank you very much for your recognition of our work in terms of insights and contributions. For your questions and concerns, we provide point-to-point responses as follows.

1. Lack of experimental data validation for the generated data. Need to verify the generated data with more experimental results as currently it have multiple limitations. Need to compare the accuracy with already existing CIF data of crystals.

Thank you for your insightful comments. We totally agree that the comparison with experimental data (or in other words, crystal structures that have been experimentally observed) is important to demonstrate the ability of the proposed method for real world applications, beyond synthetic crystal structures obtained from pure DFT calculations.

Actually, to demonstrate this, we conduct experiments by using Mat2Seq to discover recently experimentally discovered crystal structures from literature in Section 4.2. The visual comparison between the Mat2Seq generated structure with the experimentally observed crystal structure is shown in Figure 4, which shows our proposed method's ability for experimentally observed crystal structures. We also add one experiments to show the match rate and RMSE in Table 12 attached above in the general response.

Additionally, the datasets used in Section 4.1 also contain a large number of crystal structures that are experimentally observed. For example, in MP20 dataset test set, there are 3819 crystal structures that are experimentally observed. To show the accuracy of Mat2Seq for experimental data, we additionally conduct evaluation experiments on these 3819 crystal structures with results shown in Table 10 attached above in general response. It is worth noting that Mat2Seq achieves 65.2% match rate with 0.042 RMSE on these 3819 crystal structures.

2. For current limitations listed in the paper.

Thank you for your valuable question.

(1) Designing a LLM-based generative method for all atomic systems while satisfying uniqueness and completeness is highly nontrivial and a open question. We tend to address this problem in future works.

(2) Different from other generative methods like diffusion based or flow based methods, LLM based methods generate crystal structures in an autoregressive manner. This difference makes a lot of techniques used by previous diffusion or flow based methods invalid for LLM-based methods. And this is an open frontier to be explored as also discussed by previous LLM-based methods. In this paper, we proposed a potential solution for this problem, and we tend to believe there are rooms for future works to improve.

(3) As discussed by the ML community, usually more data will result in more powerful ML models. However, generating large scale crystal structure dataset is very expensive and currently out of our scope, so we list this as a potential direction to further improve the power of ML methods for crystal structure generation.

We hope we addressed your concerns. If you have additional questions or concerns, we are more than willing to answer.

Yours sincerely, Authors.