SimXRD-4M: Big Simulated X-ray Diffraction Data and Crystal Symmetry Classification Benchmark

Thank you for recognizing our work. We have carefully addressed your comments and made corresponding improvements to our manuscript. As you acknowledge the significance of our work, we kindly request your endorsement and consideration of a higher score, should our revisions meet your satisfaction.

Weaknesses-Baseline for Dataset: In particular, if you simulated PXRD patterns with Pymatgen and GSAS-II [6] and trained models on these, how would it perform on experimental data, as compared to training on SimXRD?

Thank you for your comment. As you suggested, the generative ability for testing on experimental data is crucial. However, due to time constraints, regenerating a large database based on MP using tools like Pymatgen, GSAS-II, and recalculating benchmarks is highly challenging. We have added further clarification in lines 153-160 to emphasize this point, as summarized below:

"Earlier studies primarily relied on widely-used software tools, such as Pymatgen (Ong et al., 2013), FullProf (Rodr´ıguez-Carvajal, 2001), and GSAS-II (Toby & Von Dreele, 2013), to simulate patterns for training. However, these approaches often showed varying degrees of performance drop when applied to experimental test sets. SimXRD addresses these limitations by employing the newly developed PysimXRD, which accounts for a wider range of real-world conditions."

Weaknesses-Presentation of Figures: Displaying 4-6 examples of these in a large figure (preferably in the main text).

Thank you for your suggestion. Figure 2 demonstrates that, with a carefully selected set of parameters (optimized according to the experimental pattern), the simulated pattern aligns well with the experimental one.

We taken your have advice and added a new section in Appendix B.9 SIM2REAL GAP (lines 1181-1232), which provides additional cases for PysimXRD and GSAS-II. The parameter searching process requires a refinement function, so we have only compared our approach with GSAS-II. (Due to the page limitations of the ICLR conference, we did not present this in the main text.)

Weaknesses-Simulation Fidelity Evaluation: Can you take maybe 10 experimental patterns with known ground truth structures, and simulate patterns with SimXRD, Pymatgen, and GSAS-II? Then see the R_w goodness of fit value as compared to the experimentally observed.

Thank you for your comment. We have taken your advice and added an additional section in Appendix B.9 SIM2REAL GAP (lines 1181-1232), providing a comparison between PysimXRD and GSAS-II. We present the experimental patterns, refined crystal constants, and fitting Profile factors.

Weaknesses-Address Structure Solution: You seem not to cite the relevant works in the field: please look at them and cite them [1, 2, 3, 4, 5].

Thank you for your suggestions. We have added the latest research to our paper, as shown below:

"More recent studies (Guo et al., 2024a;b; Li et al., 2024; Riesel et al., 2024; Lai et al., 2024) on powder XRD for crystal prediction and generation have employed advanced ML architectures, achieving significant progress and noteworthy results."

Weaknesses-Writing: I think you can emphasize more how this dataset itself is valuable.

Thank you for the comment.

• We have emphasized the challenge of sim2real in the revised version as follows: (lines 073–079) "The limited scale of experimental data presents two significant limitations in this field : (1) insufficient data to comprehensively test generative capabilities and (2) the difficulty of effectively utilizing small-scale experimental datasets for model tuning and transfer learning.

• We have also enhanced the "Experimental Data Generalization" section (lines 153-160), summarized as follows: "Earlier studies primarily relied on widely-used software tools, such as Pymatgen (Ong et al., 2013), FullProf (Rodr´ıguez-Carvajal, 2001), and GSAS-II (Toby & Von Dreele, 2013), to simulate patterns for training. However, these approaches often showed varying degrees of performance drop when applied to experimental test sets. SimXRD addresses these limitations by employing the newly developed PysimXRD, which accounts for a wider range of real-world conditions."

Questions: Are you able to generate PXRD patterns of varying Q-ranges and/or x-ray wavelengths?

Yes, we can. SimXRD is simulated within specific Q-ranges, as it is designed to function as a standard database. In contrast, PysimXRD is open-source, allowing for high-fidelity simulations across arbitrary diffraction angles or Q-ranges.

I appreciate your revisions, especially Figures 7 and 9, that show that you can achieve realistic PXRD patterns that adhere to experimental observations. As such, I will raise the score. Looking forward to seeing this work in print.

Thank you for reviewing our paper and providing feedback. We have responded to all of your comments comprehensively and welcome further discussion to clarify and resolve these matters.

Weaknesses: The main contribution of this work is unclear. The analyses and benchmarks do not provide much insight into ML research.

Thank you for your comment. We would like to emphasize that our insight is highly related to ML research:

• We provide comprehensive, high-fidelity datasets for fundamental data-driven model developments.
• We benchmarked 21 different ML models, including those developed specifically for PXRD data and others for sequence data, offering a broad and systematic evaluation.
• We highlight the long-tailed distribution in symmetry classification, which is an ML problem. We evaluate the model performance under different objective functions to show that addressing it improves the model performance.
• We study the model generalization ability to different experimental factors and crystals. Identifying such problems is vital for the ML community.

As recognized by other reviewers, "enabling advancement in crystallographic machine learning research; can be a useful resource for the community."（Z7kM）, "a strong motivation and need for this paper in the research community" (Qfc3), "a reproducible, adaptable contribution to the community (UoDH), the contribution of our work is clear and highly relevant to the ML community.

Questions 1: The simulated dataset uses lattice plane distance as the x-axis. How is it related to 2 theta, which is more commonly used in XRD data, and what’s the reason for this choice?

Thank you for your comment. In spectroscopy, the d-spacing (d) representation is preferred over 2-theta. The d-I pattern is independent of the wavelength of the incident X-rays, making it universally applicable across all experimental X-ray source. In contrast, 2-theta patterns are comparable only when the incident X-rays share the same wavelength. This inherent limitation does not apply to d-I patterns, which offer greater versatility.

If a 2-theta pattern is required, a d-I pattern can be easily converted using the wavelength of the incident X-rays and the fundamental Bragg equation.

Questions 2: The generalizability of models to experimental data is assessed by training on SimXRD and testing on RRUFF. Is RRUFF representative of experimental data influenced by various factors (materials type, physical condition, equipment, etc.)? How do the authors ensure this is a rigorous assessment?

RRUFF is a representative experimental database for testing model performance, as it is one of the largest experimental databases currently available. To ensure a rigorous assessment, we train models on simulated XRD patterns without any observation of the experimental patterns, thus their performance on RRUFF reflects their generalization ability to corresponding experimental patterns.

Questions 3: Line 157, “almost all related research…due to the lack of easily accessible datasets”, what does this mean?

Thanks for the question. What we mean is that the study of powder XRD currently lacks a broad community. Our work seeks to engage diverse research communities by offering open-source data and ML workflows, thereby fostering model innovation and advancing technological development in this area.

Questions 4: Line 345, is “being biased by classes of high frequency” a problem of CNN? It should be more related to the objective/loss function.

Thanks for the question. Both model architectures and training strategies are crucial for modeling imbalanced class learning. As shown in Figure 5 of the revised pdf, although CNN11 is trained with the CNN models, it performs better on low-frequency crystals.

Questions 5: Line 48, structure analysis being challenging for domain experts is irrelevant to the search-match method.

Thank you for your question. We emphasize that symmetry identification is the first and most critical step in structural analysis.

Questions 5: Line 59, what challenges do introducing large-scale databases present?

Thanks for your question. Introducing large-scale databases presents challenges related to data quality, consistency, and accessibility. Our work provides a standardized ML workflow for powder XRD databases, with a comprehensive benchmark for model architecture comparison, model training, and model innovation to facilitate the development of this field.

Questions 5: Line 79, repeated words.

Thanks for the comment. We modified the expression.

Questions 5: Line 216, the details on the Materials Project are irrelevant to this work.

Thanks for the comment. Any XRD simulation database is based on crystal databases. The end-to-end inference process involves deriving crystal information from XRD patterns. The details of crystal databases provide a thorough understanding of the quality of the database and the task settings.

Thanks for the response. I am still doubtful about the main contribution, but I acknowledge that the work is solid and comprehensive. Some of my concerns are not addressed:

• Q2: how is the claim "RRUFF is a representative experimental database for testing model performance" supported? One possible way is to assess the data distribution of RRUFF. Otherwise, this statement is not rigorous (not that it's wrong, but it's an overclaim).
• Q3: "almost all related research…due to the lack of easily accessible datasets" is logically strange. The research in this field is driven by a need in materials science to decode XRD data efficiently; dataset is a tool, not the goal.

Nonetheless, I appreciate the time and effort the Authors put into improving the paper. I will raise my rating incrementally.

Dear reviewer xPRn,

We sincerely appreciate your reply and are glad to hear that we have addressed your remaining concerns well.

Therefore we kindly ask you to consider lifting overall ratings.

Thanks for your feedback again.

Best regards,

Authors.

We sincerely thank you for recognizing the significance of our work. We have carefully addressed your concerns one by one to improve the quality of this paper. If you are satisfied with the revisions, we kindly request your approval by considering an improved score.

Weaknesses A: There is no validation of SimXRD to show that the patterns it creates match reality.

Thank you for your comment. We provide evidence of the fidelity of SimXRD in two aspects:

1. The experimental generative ability of SimXRD. Our results show that the performance of ML models trained on SimXRD and tested on RRUFF yields very consistent results.
2. In the revised version, we provide a more comprehensive analysis of the Sim2Real gap.

• In Appendix B.4 XRD Pattern Simulation (lines 1033–1070), we present a detailed theoretical framework for data generation.
• Figure 6 illustrates how the multiphysical coupling observed in experimental XRD can be incorporated into the simulation parameter space by introducing more realistic conditions.
• In Appendix B.8 Sim2Real Gap (lines 1182–1232), we include case studies that compare the fidelity of pattern generation with GSAS-II. The results demonstrate that PysimXRD has the capability to generate high-fidelity simulated patterns.

Weaknesses B: The poor performance of the models on the long-tailed distribution.

Thank you for your comment. Several baselines have shown promising results in our testing. The benchmark results indicate that CNN11 demonstrated strong performance in both in-library and out-of-library crystal system classification. Additionally, our findings suggest that PathTST and bidirectional time-series models (e.g., Bidirectional-GRU) show potential for studying XRD data, achieving over 90% accuracy for crystal system classification and 80% for space group classification. Furthermore, the results highlight that label smoothing and focal loss can address the long-tailed distribution of crystal systems to a certain extent, which may warrant further investigation.

Weaknesses C: I do not understand Figure 2.

We apologize for the confusion. We have added a detailed caption to provide further clarification for Figure 2. Specifically, we add (lines 235-243):

• Figure A: The transformation from a crystal structure to an XRD pattern involves considerations such as finite grain size, specific orientation, thermal vibrations, zero-shift corrections, and the scattering background. The recorded patterns are represented in Q-space, with lattice distances sorted along different diffraction directions. Using the crystal database, PysimXRD generates simulated XRD patterns for ML training, enabling intelligent symmetry identification tasks.
• Figure B: We compared experimental measurement patterns with simulated XRD patterns generated using PysimXRD for a Li-rich layered oxide cathode structure, primarily composed of Li2MnO3. The simulation incorporates multiphysical coupling, resulting in generated patterns that align closely with experimental measurements, exhibiting small residual error.

Weaknesses D: Overall writing clarity (outside of the appendix) could use some improvement.

Thank you for your suggestions. We have revised the manuscript to enhance its overall readability.

Questions 1: In line 36: Transitional methods for what? I assume symmetry identification, though extra clarity would be appreciated.

Thanks for the suggestion, we have revised it as follows:

"Traditional methods of symmetry identification involve a search-and-match process (Altomare et al., 2008)."

Questions 2: Line 156 claims that existing symmetry methods rely on 1D CNNs? As it is currently written, it is saying that every technique that has been introduced uses 1D CNNs, which contradicts lines 296-308.

Thank you for your comment. Most models developed for PXRD are based on 1D CNNs. In lines 296-308 (Original version), other benchmarks discussed in this paper include various models developed for sequence data, rather than specifically for PXRD data. We have revised this sentence to be more rigorous:

"Most existing methods for symmetry identification rely on one-dimensional convolutional neural networks to build classification models."

Questions 3: Lines 296-308: is this supposed to be a comprehensive list of techniques? As it is written, that is what it seems to be saying.

Thanks for the question. It consists of all the existing symmetry classification models (i.e., CNN-based models). Moreover, we also evaluate advanced machine learning models, including recurrent models and transformers, that have not been adopted for such tasks. To the best of our knowledge, our benchmark considers a comprehensive list of techniques in the domain of symmetry classification.

Regarding updates to Figure 2 (A, B): Thank you for the update and caption. What confuses me still is the flow of “information” through the system. I see labeled images like: “crystal”, “reciprocal space”, etc, but I do not understand where the process starts, and how it precedes until completion. There are three red arrows, along with 3 green arrows, but they do not create a narrative. Something in the form: “image 1” -> “image 2” -> “image 3” -> … is the way I see similar diagrams made in literature. Since this diagram is one of the primary explanations for the contribution of the paper, further refinement and clarity of it is vital to the paper.

——— Satisfied Concerns ———

Regarding Sim2Real gap: Thank you of the updates. This completely satisfies my concerns about this area.

Regarding long-tailed distribution performance: This explanation satisfies my concerns about this area of the paper, so long as it is noted that this is an area requiring future exploration.

Regarding the answers to the two questions: I am satisfied with the response to each, and I have no follow up questions.

——— Score Update ———

I have updated the score incrementally for now. I have changed it from a 5 to a 6.

Thank you for your insightful feedback, which has significantly improved the quality of our manuscript. We have carefully addressed each of your comments, as detailed below. We greatly appreciate your recognition of the importance of this work in advancing phase identification. If you find our revisions satisfactory, we kindly request your favorable consideration for an improved score.

Weaknesses 1: Limit technical novelty from an architectural perspective.

Thanks for your comment. We would like to clarify that our goal is to propose novel datasets and benchmarks (datasets and benchmarks track), which are also crucial besides architectural contributions. In this work, we provide a comprehensive and accessible powder XRD database, along with a benchmark designed to standardize symmetry identification tasks. We emphasize our technical contributions as follows:

• We identify and highlight the heavy long-tailed distribution in symmetry classification, which has not been explored before and is vital for developing accurate models.
• Through evaluating 21 model architectures, we find that: (1) Most existing CNN models are unsuitable for symmetry identification within the large scale of the SimXRD database; (2) Convolutional neural networks without pooling have achieved the best performance in most tasks; (3) The raw transformer encounters difficulties in learning XRD patterns while PatchTST achieves significant performance improvement, demonstrating subsequence-level patches are beneficial to identify peaks.
• Through model interpretability analysis(lines:810-821), we discovered that: CNN11 places greater emphasis on characteristic peaks, aligning closely with the traditional search-match method. In contrast, BiGRU selectively focuses on relatively strong peaks, progressively reducing the emphasis on weaker peaks based on their intensity.

Weaknesses 2: The validation against real-world data is limited. The RRUFF dataset represents only 0.07% of the simulated dataset size.

Thanks for your comment. We would like to clarify that labeled experimental data are indeed highly valuable, which is also the motivation for constructing a simulated XRD dataset. At present, RRUFF is the largest open-sourced experimental dataset as far as we know. Models trained on SimXRD can generalize to RRUFF, demonstrating the significance of our datasets.

Weaknesses 2: The comparison between simulated and real-world patterns is demonstrated only for Li2MnO3. A more comprehensive analysis of sim and real-world gaps would be helpful.

Thanks for the constructive comment. In the revision, we emphasize the challenges posed by limitations in experimental patterns (lines 073–079): (1) insufficient data to comprehensively test generative capabilities, as you highlighted, and (2) the difficulty of effectively utilizing small-scale experimental datasets for model tuning and transfer learning. Additionally, we have conducted a more comprehensive analysis of the gaps between simulated and real-world data. This includes a supplement in Appendix B.4 XRD Simulation(lines 1033–1070), which examines the influence of simulation parameters, and a new section in Appendix B.8 Sim2Real Gap (lines 1182–1232), which compares the fidelity of pattern generation with GSAS-II.

Weaknesses 3: The paper lacks investigation of model failure modes and interpretability analysis. For example, analyzing what patterns CNNs learn and how they make decisions could reveal important insights and potentially guide future architectural improvements.

Thank you for your valuable suggestions. Based on your advice, we have added a new section, Appendix A.3 Feature Analysis (lines 792–821), in the revised manuscript. The key observations are summarized as follows:

• Machine learning models primarily rely on high-intensity peaks (characteristic peaks) to infer symmetry categories.
• Models with better performance tend to extract more detailed information from relatively weak-intensity peaks, rather than relying solely on the "three strongest diffraction peaks" as in the search-match approach.
• CNN11 places greater emphasis on characteristic peaks, aligning closely with the traditional search-match method. In contrast, BiGRU selectively focuses on relatively strong peaks, progressively reducing the emphasis on weaker peaks based on their intensity.