LLM Meets Diffusion: A Hybrid Framework for Crystal Material Generation

We thank the reviewer for the valuable feedback. Below, we provide detailed point-by-point responses to each of the comments and questions:

Questions:

Q1. Limited LLM Baselines Comparison in Table-1

To the best of our knowledge, Crystal-Text-LLM is the most widely used LLM-based model for material generation in the literature. However, we have also conducted experiments on the de novo generation task using LLaMA-2 13B and the latest LLaMA model, LLaMA 3.1 8B model. Results for MP-20 are as follows:

Models	Struct Validity	Comp Validity	COV-Precision	COV-Recall	F1-Score	Density	#Elements
LLaMa-2 7B	96.40	93.30	94.90	91.1	92.96	3.61	1.06
LLaMa-2 13B	95.51	92.41	97.91	88.91	93.19	2.13	0.1
LLaMa-3.1 8B	93.90	87.90	99.40	90.58	94.74	0.8309	0.1713
CrysLLMGen	99.94	93.78	99.84	98.52	99.17	0.664	0.459

We observe a similar trend for all the LLaMa baseline models as well.

Q2. Including Bayesian Flow Networks (e.g. CrysBFN) Baseline

We thank the reviewer for the suggestion. The results of CrysBFN for the de novo generation task are reported below.

Models	Struct Validity	Comp Validity	COV-Precision	COV-Recall	Density	#Elements
DiffCSP	100	83.25	99.76	99.71	0.3502	0.3398
CrysBFN	100	87.51	99.79	99.09	0.2070	0.1628

Overall, the results follow similar trends as observed with other generative models reported in Table 3. Similar to other diffusion-based generative models (like DiffCSP), these models generally perform well in structural validity but struggle with discrete components, such as accurately identifying atomic types, leading to lower compositional validity. \We will include these results in the revised manuscript and appropriately cite CrysBFN.

Q3. Use of Llama-2 instead of Llama-3 — Justification and Practical Considerations

We have conducted experiments with LLaMA 3 (LLaMa-3.1 8B) as well, and the results are reported in the table above (See Q1). We observe a similar trend for both LLaMA-2 and LLaMA-3 models. Therefore, we adopted the LLaMA 2 model, as it is more computationally efficient and better suited for our limited fine-tuning data.

Q4. Details on conditional and unconditional fine tuning of the LLMs

For fine-tuning LLMs, we adopted the strategy proposed in [1],[2]. We fine-tune the pre-trained LLaMA-2 model on a dataset of crystal structures represented as strings, accompanied by prompts instructing the model to generate bulk materials by providing lattice parameters (lengths and angles), along with atom types and atomic coordinates. Specifically, we use the following prompt:

Below is a description of a bulk material. [The chemical formula is Pm₂ZnRh, ... <Other Conditions>]. Generate a description of the lengths and angles of the lattice vectors and then the element type and coordinates for each atom within the lattice: [Crystal string]

The text in [Conditions] enables conditional generation; omitting it allows for unconditional generation. [Crystal string] represents the string encoding of atoms within the lattice.

Q5. Distribution Alignment Between LLM Outputs and Diffusion Training Samples

In our design, both components, LLM and Diffusion, are trained independently on the true data distribution using the available training dataset. However, we acknowledge the reviewer’s concern regarding the potential distribution shift between the real training data and the data generated by the LLM.

To address this, during sampling, we do not directly feed the LLM-generated intermediate representation into the diffusion model for the full T-step denoising process. Instead, we inject this intermediate representation at an intermediate step, denoted as $\tau$, where T > $\tau$ > 0. The value of $\tau$ is treated as a tunable hyperparameter.

We determine the optimal $\tau$ using a validation dataset. Specifically, for each material in the validation set, we generate prompts and obtain outputs from CrysLLMgen, where we try different values of $\tau$ (900, 800, 700, 600, 500, 400, 300, 200, and 100) by injecting the LLM output into the diffusion model at those respective steps. For each $\tau$, we evaluate the final generated materials using metrics for both CSP and Gen tasks. The $\tau$ value that yields the best validation performance on these metrics is selected for our final model for final evaluation.

Q6. CSP Task results on MPTS-52 dataset

We thank the reviewer for the suggestion. We ran both LLaMA-2 (7B) and our proposed CrysLLMGen on the MPTS-52 dataset for the CSP task and compared the results with existing baseline generative models. The results are as follows:

Models	Match Rate	RMSE
CDVAE	5.34	0.2106
DiffCSP	12.19	0.1786
FlowMM	17.54	0.1726
CrysBFN	20.52	0.1038
LLaMA-2 7B	39.96	0.1092
CrysLLMGen	42.16	0.0977

Results for CDVAE, DiffCSP, FlowM, and CrysBFN are taken from the respective papers. We observe that CrysLLMGen outperforms all baseline models in both Match Rate and RMSE. We will include these results in the revised manuscript.

Q7. Source of LLM Results in Table-2 for CSP Task

We selected LLMs as one of the baselines for our study and trained them for the CSP task. Our goal was to include both generative models and LLMs as baselines for both tasks, allowing for a comprehensive comparison with our proposed CrysLLMGen model.

Q8. Typo regarding MP-20 RMSE in Table-2

We thank the reviewer for pointing this out. It was an oversight on our part, and we will rectify it in the revised manuscript.

Q9. Evaluation of SUN Rate for DiffCSP++ in Table 4

Since SUN and Stability Rate were not reported in the original DiffCSP++ paper, we generated 10K materials using DiffCSP++ and computed the Stability and SUN scores using CHGNet (as described in Section 5.4). The results are reported accordingly.

To further verify the reviewer’s comment, we examined the FlowMM[2] and FlowLLM[3] papers, where Stability Rate and SUN Rate for other baselines such as CDVAE, DiffCSP, and FlowMM, measured using CHGNet, are reported. We verified these values and observed a similar trend in our results. While the absolute values we obtained are slightly higher, the rank order among baselines (CDVAE, DiffCSP, FlowMM) remains consistent with our observations.

[1] Gruver, Nate, et al. "Fine-tuned language models generate stable inorganic materials as text." ICLR 2024

[2] Miller, Benjamin Kurt, et al. "Flowmm: Generating materials with riemannian flow matching, ICML-2024

[3] Sriram, Anuroop, et al. Flowllm: Flow matching for material generation with large language models as base distributions. NeurIPS - 2024

We thank the reviewer for the valuable feedback and are pleased to know that you appreciated our work and the thoughtful insights provided. Below, we offer detailed point-by-point responses to each of reviewer's comments.

Question:

Q1. Regarding making τ learnable or adaptive for better generalization

The suggestion given by the reviewer is appreciated and indeed a good one. However, in this work, we do not treat τ as a learnable or adaptive parameter. Instead, τ is considered a hyperparameter, which we tune using the validation dataset. Making τ learnable or adaptive is an interesting direction and remains a potential avenue for future work. This has been noted in Appendix B: Limitations and Future Work.

Q2. Regarding prompt flexibility and robustness

Again, a good suggestion by the reviewer regarding the robustness and flexibility of the prompt. However, we have followed the existing literature in this regard. Most LLM/text-based material generation works to date (e.g., Crystal-Text-LLM [1], FlowLLM [2], TGDMat [3]) rely on template-based prompts while adapting pretrained LLM/BERT models for text-conditional generation. Accordingly, we adopted a similar framework to maintain consistency with prior work. That said, introducing greater flexibility and robustness into prompt design is an interesting direction that warrants further exploration, and we consider this a potential avenue for future work.

Q3. Regarding property-guided generation capability

Again, a good suggestion by the reviewer regarding the property-guided generation capability of our proposed framework, enabling conditional generation based on target physical or chemical properties. However, implementing this would require a pretrained property predictor for each target property to assess whether the generated material aligns with the desired property values. The main challenge here lies in selecting an appropriate property predictor, as the literature offers many architectural choices, and currently, there is no standard benchmark for this task. Unfortunately, none of the existing works have addressed this issue too.

In our work, we followed the existing literature in this regard. Specifically, we adopted the evaluation methodology used in [1], where the number of atoms and space group are evaluated for text-conditional generation. These are intrinsic material properties that can be validated without requiring any pretrained property predictor. Nonetheless, property-guided generation is an interesting direction that warrants further exploration, and we consider it a promising avenue for future work.

Q4. Regarding joint training between LLM and diffusion model

In our current proposed framework, there is no interaction between the LLM and the diffusion model; both components are trained independently as standalone modules. While jointly training the LLM and diffusion model is an interesting direction, it presents several challenges—such as designing an appropriate optimization framework and establishing an effective feedback mechanism between the two modules. There are many nuances in this design that require further exploration and significant engineering effort. Therefore, we consider this as a potential avenue for future work, which has been noted in Appendix B: Limitations and Future Work.

Q5. Regarding invalid or unrealistic atom types

We agree with the reviewer’s observation that LLMs can generate invalid or unrealistic atom types. In our experiments, we observed a small percentage of such materials generated by the LLM models, which we filtered out during post-processing using simple validation scripts.

Q6. Regarding LLaMa's default tokenize

We have followed the existing literature in this regard. As LLM backbone, LLaMA-2 have demonstrated strong performance in material generation tasks across several recent studies[1][2]. Additionally, LLaMA-2 tokenizes numbers as individual digits by default, a feature shown to significantly enhance performance on arithmetic-related tasks. However, we have also conducted experiments on the de novo generation task using latest LLaMA model, LLaMA 3.1 8B model. Results for MP-20 are as follows:

Models	Struct Validity	Comp Validity	COV-Precision	COV-Recall	F1-Score	Density	#Elements
LLaMa-2 7B	96.40	93.30	94.90	91.1	92.96	3.61	1.06
LLaMa-3.1 8B	93.90	87.90	99.40	90.58	94.74	0.8309	0.1713

We observe a similar trend for all the LLaMa baseline models as well.

Overall, the suggestions provided by the reviewer are very insightful, and we agree that there is significant potential to enhance our proposed hybrid framework. However, CrysLLMGen, as presented in this paper, represents one of the initial attempts at designing hybrid models for material generation. While there is considerable scope for further development, these directions require deeper exploration and will be pursued in future work. We will include these in "Appendix B: Limitations and Future Work" section.

[1] Gruver, Nate, et al. Fine-tuned language models generate stable inorganic materials as text. ICLR 2024

[2] Sriram, Anuroop, et al. Flowllm: Flow matching for material generation with large language models as base distributions. NeurIPS - 2024

[3] Das, Kishalay, et al. Periodic materials generation using text-guided joint diffusion model. ICLR 2025

One point I would still appreciate clarification on is the supervision signal used for LLM fine-tuning. While the paper discusses using CIF-like sequences as input, it would help to explicitly confirm that the LLM is trained via next-token prediction on tokenized ground-truth structures from datasets like MP-20. Additionally, reporting the training loss or validation perplexity would help assess the convergence and quality of the LLM component.

We thank the reviewer for the valuable feedback. Below, we provide detailed point-by-point responses to each of the comments.

Regarding loss functions of LLM:

For fine-tuning, we adopted the strategy proposed in Crystal-text-LLM [1], wherein the pretrained LLaMA-2 model was fine-tuned using a next-token prediction objective to generate string representations of materials. We will report training loss or validation perplexity in the revised manuscript.

Regarding Chemistry-Specific Tokenization:

We appreciate the thoughtful feedback regarding the limitations of general-purpose tokenizers in handling domain-specific scientific data.

For fine-tuning, we followed the strategy proposed in Crystal-text-LLM [1], where the authors reported (Appendix A.1) that instead of creating a domain-specific vocabulary and training models from scratch, it is effective to use LLaMA-2’s existing tokenizer and fine-tune the pretrained model directly. They conducted an ablation study comparing this approach with one that involved fine-tuning LLaMA-2 models using crystal-specific tokens. However, they found that the latter was more challenging to train and did not yield any performance gains over using the shared tokenizer. Based on these findings, we adopted their approach. The potential benefit of developing a domain-adapted tokenizer remains an open question and is left for future exploration.

Regarding Invalid LLM Outputs

We thank the reviewer for the question and apologize for any confusion. Invalid compositions are filtered out immediately during the sampling process and are discarded on the spot. The following pseudocode illustrates our sampling approach:

num_samples = 10000  // total number of valid samples required
generated_structures = []
while len(generated_structures) < num_samples:
    new_structure = CrysLLMGen.sample()
    if new_structure.atom_type is INVALID:
        continue
    else:
        generated_structures.append(new_structure)

Hence, invalid compositions are filtered immediately during sampling, and only valid structures are retained. Once 10,000 valid structures are collected in the generated_structures list, the reported evaluation metrics are computed on this set. All reported metrics for gen task and stability are computed only on valid 10K materials, and Invalid samples are excluded from metric evaluations.

Discard Rate: On average, we observe approximately 2-5% of the sampled structures are discarded due to invalid atom types or compositions.

[1] Gruver, Nate, et al. Fine-tuned language models generate stable inorganic materials as text. ICLR 2024

Thank you very much for your valuable feedback. We are pleased to know that you appreciated our work and the thoughtful insights you provided. Below, we offer detailed, point-by-point responses to each of your comments.

Weaknesses :

• W1. Novlety of our proposed framework and difference with FlowLLM

By design, our proposed CrysLLMGen bears some resemblance to FlowLLM; however, it is not merely an adaptation of FlowLLM’s pipeline with the flow-matching module replaced by a diffusion model. The key innovations and novel aspects of CrysLLMGen are as follows:

• Generative Module: FlowLLM employs a flow-matching model as its generative component, whereas CrysLLMGen uses a diffusion model. We analyzed the performance of both the flow matching and diffusion models in the context of unconditional material generation. Our observations indicate that DiffCSP, the diffusion-based model for material generation, outperforms FlowMM, the flow matching model, across most benchmark metrics for this task. Referencing from the FlowMM paper [1], the summary of results from de novo generation on the MP-20 dataset is as follows:

Models	Struct Validity	Comp Validity	COV-Precision	COV-Recall	Density	#Elements	Stability Rate	SUN Rate
DiffCSP(1000 Steps)	100	83.25	99.76	99.71	0.35	0.125	5.06	3.34
FLowMM(1000 Steps)	96.85	83.19	99.58	99.49	0.239	0.083	4.65	2.34

These observations indicate that the diffusion model is a more natural choice for material generation compared to flow matching, as it consistently produces more valid and stable materials.

• Integration Strategy: In FlowLLM, the material structures generated by the LLM are directly passed into the flow-matching module for refinement. In contrast, since LLM generated material structure are intermediate representations, we inject these representations into our diffusion model at an intermediate timestep $\tau$ , where 0 ≤ $\tau$ ≤ T, and initiate the denoising process from that point. This design allows for more effective refinement and generation, leveraging the strengths of both LLM and diffusion models.

• Parallel Training: We did not adopt the sequential training paradigm of FlowLLM, where a large language model (LLaMA-2) is first fine-tuned on the training dataset, and the samples generated from this LLM are then used to train the flow matching framework. Instead, in our approach, both the LLM and the diffusion module are trained in parallel using the same training dataset.

• Experimental Evaluation: Finally, our experimental results demonstrate that CrysLLMGen outperforms FlowLLM across most standard benchmark metrics for the de novo material generation task (as shown in Table 3 of the paper) across different datasets. These results highlight the overall effectiveness of our proposed approach.

[1] Miller, Benjamin Kurt, et al. "Flowmm: Generating materials with riemannian flow matching, ICML-2024

• W2. Clarification on Architectural Differences and Performance Gains in CrysLLMGen

We thank the reviewer for this insightful question. First, we clarify that both CrysLLMGen and FlowLLM use the same underlying LLM architecture (LLaMA-2) in the first stage of their respective hybrid pipelines. The key differences, and our primary innovations, lie in the second stage of the pipeline. Specifically, there are two main distinctions:

• Generative Module: FlowLLM employs a flow-matching model as its generative component, whereas CrysLLMGen uses a diffusion model. We evaluated the performance of both flow-matching and diffusion models in the context of unconditional material generation and found that diffusion models consistently outperform flow-matching models across the majority of the evaluation metrics.

• Integration Strategy: In FlowLLM, the material structures generated by the LLM are directly passed into the flow-matching module for refinement. In contrast, since LLM generated material structure are intermediate representations, we inject these representations into our diffusion model at an intermediate timestep τ, where 0 ≤ τ ≤ T, and initiate the denoising process from that point. This design allows for more effective refinement and generation, leveraging the strengths of both LLM and diffusion models.

These two innovations collectively contribute to the superior performance of CrysLLMGen compared to FlowLLM, as shown in Table 3.

• W3. Ablation Study

As suggested by the reviewer, we conducted an ablation study in which we replaced the FlowMatching module of FlowLLM with a diffusion model (following the DiffCSP framework). In specific, we first used data sampled from LLaMA-2 to train the diffusion model. During sampling, given a prompt, we first generate A, X, L from the LLM, which are then passed to the diffusion model (injecting at the final step T) for further refinement. The results we observed from this ablation are as follows:

Models	Struct Validity	Comp Validity	COV-Precision	COV-Recall	Density	#Elements
FlowLLM w\ Diffusion	99.88	82.50	98.54	97.55	0.724	0.5388
CrysLLMGen	99.94	93.78	99.84	98.52	0.664	0.459

We observe performance degradation. In specific, 12.03% performance degradation in compositional validity. These results suggest that training the diffusion model on true ground-truth data (rather than on LLM-sampled data), along with employing intermediate denoising to refine LLM-generated structures, enhances the effectiveness of our proposed hybrid framework.

Questions:

• Q1. Clarification on #Element Metric Discrepancy

We thank the reviewer for pointing this out. For consistency, we have reported benchmark results as presented in the respective original papers. The results for LLaMA-2 (7B) are taken from the Crystal-Text-LLM paper [2]. However, we also conducted our own sampling using the fine-tuned LLaMA-2 (7B) models, and the results we obtained for the de novo generation task are as follows:

Models	Struct Validity	Comp Validity	#Elements
LLaMA-2 7B	96.40	93.78	0.462
CrysLLMGen	99.94	93.78	0.459

We observe both compositional validity and # Element metrics are similar the two models. We will update it in the revised manuscript.

[2] Gruver, Nate, et al. "Fine-tuned language models generate stable inorganic materials as text." ICLR 2024

• Q2. Visualizations of intermediate and final samples

We thank the reviewer for this valuable suggestion. We will include visualizations of a few generated material samples in the revised manuscript

Paper Formatting Concerns

We thank the reviewer for these thoughtful suggestions and appreciate the attention to detail regarding formatting and presentation.

• We will include representative examples of sampled structures in the revised manuscript, enabling experts to visually assess the quality of the generated materials.
• Regarding the Denoising Network section (Section 4.4), we agree that the technical depth may be better suited for the Supplementary Material. We will move this section accordingly to maintain the flow and readability of the main paper.
• We acknowledge the issue with Figure 1 and will replot it at higher resolution to ensure clarity and better visual quality.
• Additionally, we appreciate the recommendation to include examples of intermediate textual representations and details on coordinate tokenization. We will incorporate these elements in the revised manuscript or the camera-ready version to enhance reproducibility of our approach.

We thank the reviewer for the valuable feedback. Below, we provide detailed point-by-point responses to each of the comments and questions:

Weaknesses :

• W1. Difference with FlowLLM

By design, our proposed CrysLLMGen bears some resemblance to FlowLLM; however, it is not merely an adaptation of FlowLLM’s pipeline with the flow-matching module replaced by a diffusion model. The key innovations and novel aspects of CrysLLMGen are as follows:

• Generative Module: FlowLLM employs a flow-matching model as its generative component, whereas CrysLLMGen uses a diffusion model. We analyzed the performance of both the flow matching and diffusion models in the context of unconditional material generation. Our observations indicate that DiffCSP, the diffusion-based model for material generation, outperforms FlowMM, the flow matching model, across most of the evaluation metrics for this task. Referencing the FlowMM paper [1], the summary of results from de novo generation on the MP-20 dataset is as follows:

Models	Struct Validity	Comp Validity	COV-Precision	COV-Recall	Density	#Elements	Stability Rate	SUN Rate
DiffCSP(1000 Steps)	100	83.25	99.76	99.71	0.35	0.125	5.06	3.34
FLowMM(1000 Steps)	96.85	83.19	99.58	99.49	0.239	0.083	4.65	2.34

These observations indicate that the diffusion model is a more natural choice for material generation compared to flow matching, as it consistently produces more valid and stable materials.

• Integration Strategy: In FlowLLM, the material structures generated by the LLM are directly passed into the flow-matching module for refinement. In contrast, since LLM generated material structure are intermediate representations, we inject these representations into our diffusion model at an intermediate timestep $\tau$ , where 0 ≤ $\tau$ ≤ T, and initiate the denoising process from that point. This design allows for more effective refinement and generation, leveraging the strengths of both LLM and diffusion models.

• Parallel Training: We did not adopt the sequential training paradigm of FlowLLM, where a large language model (LLaMA-2) is first fine-tuned on the training dataset, and the samples generated from this LLM are then used to train the flow matching framework. Instead, in our approach, both the LLM and the diffusion module are trained in parallel using the same training dataset.

• Experimental Evaluation: Finally, our experimental results demonstrate that CrysLLMGen outperforms FlowLLM across most standard benchmark metrics for the de novo material generation task (as shown in Table 3 of the paper) across different datasets. These results highlight the overall effectiveness of our proposed approach.

[1] Miller, Benjamin Kurt, et al. Flowmm: Generating materials with Riemannian flow matching, ICML-2024

• W2. Improvement of CrysLLMGen over LLMs

The reviewer’s observation regarding the incremental improvement of CrysLLMGen over the base 7B LLM model may have come from the results of the de novo generation task on the Perov dataset (a simpler benchmark) reported in Table 3. However, if one examine the results on the more challenging MP-20 benchmark dataset, there are significant gains. Here is the summary of the results for the de novo generation task and stability metrics (detailed results are provided in Tables 3 and 4 of the paper):

Models	Struct Validity	Comp Validity	COV-Precision	COV-Recall	Density	#Elements	Meta Stability Rate	MSUN Rate	Stability Rate	SUN Rate
LLaMA-2 7B	96.40	93.30	94.90	91.10	3.610	1.060	36.27	23.92	6.910	5.29
CrysLLMGen	99.94	93.78	99.84	98.52	0.664	0.459	45.20	35.94	11.98	9.21
% Improvements	3.672	0.514	5.205	8.144	81.60	56.69	24.62	50.25	73.37	74.10

We clearly observe that CrysLLMGen outperforms the LLaMA baseline across most benchmark metrics by a significant margin. Notably, the overall average improvement across all metrics is 37.81%.

We observe that CrysLLMGen significantly outperforms the LLaMA-2 7B model on most of the metrics for material generation.

• W3. Comparing with additional generative models: Mattergen, SymmCD, CrysBFN

We thank the reviewer for highlighting the baseline models. All the mentioned baselines have been evaluated on the De Novo Generation Task using the MP-20 dataset, and the results are as follows:

Models	Struct Validity	Comp Validity	COV-Precision	COV-Recall	Density	#Elements
Already reported
DiffCSP	100	83.25	99.76	99.71	0.3502	0.3398
FlowMM	96.85	83.19	99.58	99.49	0.2390	0.083
New experiments
Mattergen	100	86.34	99.45	99.59	0.4597	0.2538
SymmCD	88.10	85.32	96.82	99.53	0.201	0.3639
CrysBFN	100	87.51	99.79	99.09	0.206	0.1628

Overall, the results follow similar trends as observed with other generative models reported in Table 3. Similar to other diffusion- and flow-based generative models, these models generally perform well in structural validity but struggle with discrete components, such as accurately identifying atomic types, leading to lower compositional validity. While there may be some quantitative differences, the overall observations remain consistent across most of these models. We will include these results in the revised manuscript and appropriately cite these works.

Questions:

• Q1. In-depth discussion of the performance gap between Llama-2 (7B) and CrysLLMGen (7B)

Check W2.

• Q2. Motivation behind Choosing Diffusion Model

In an unconditional setup, DiffCSP performs better than FlowMM. We analyzed the performance of both the flow matching and diffusion models in the context of unconditional material generation. Our observations indicate that DiffCSP, the diffusion-based model for material generation, outperforms FlowMM, the flow matching model, across most benchmark metrics (6 out of 8) for this task. Referencing from the FlowMM paper, the summary of results from de novo generation on the MP-20 dataset is as follows:

Models	Struct Validity	Comp Validity	COV-Precision	COV-Recall	Density	#Elements	Stability Rate	SUN Rate
DiffCSP(1000 Steps)	100	83.25	99.76	99.71	0.350	0.125	5.06	3.34
FLowMM(1000 Steps)	96.85	83.19	99.58	99.49	0.239	0.083	4.65	2.34

These observations indicate that the diffusion model is a more natural choice for material generation compared to flow matching, as it consistently produces more valid and stable materials.