Code-Generated Graph Representations Using Multiple LLM Agents for Material Properties Prediction

We sincerely appreciate your time and thoughtful evaluation, as well as your recognition of our methodological design, theoretical foundations, and experimental framework. We are particularly grateful for your recognition of our motivation, ”Creating new representations of materials structures that satisfy necessary physical constraints could help ML-based materials modeling and design”. Below, we will address your concerns point by point.

Q1: Materials representations need to trade off conciseness and expressiveness. Are six types of constraints all necessary?

R1: We agree that representations should balance simplicity and expressiveness. At the same time, the importance of the six types of constraints proposed in this paper has been substantiated in corresponding studies (ComFormer, PerCNet, and PDD e.t.). These constraints can help capture the structure information of crystals and enhance the accuracy of property prediction. Improving the prediction accuracy of ML-based methods can benefit material science in various aspects.

Most importantly, our framework offers fully customizable constraints for crystal representation. Domain experts can freely modify these constraints, such as physical constraints, representational simplicity constraints, and other specific constraints, based on application needs, enabling the creation of tailored crystal characterization codes for various scenarios. We selected six representative constraints to demonstrate our framework's capability in generating crystal representations that simultaneously satisfy multiple constraints that existing algorithms fail to meet collectively.

Q2: How to combine merits from different code pieces? Is the ‘base’ code provided by user?

R2: In the initial iteration, we manually provide the 'base' code to Agent_A. This code is designed to meet fundamental properties, including permutation, rotation, and translation invariance. It is worth noting that the base code can alternatively be generated by LLMs.

The merits of the generated codes are integrated throughout our framework's process. We not only retain the codes (produced by Agent_B) that meet specific constraints but also document the reasoning (merits and demerits) by Agent_C. Then, the parent codes with merits can be effectively selected through Formula 1 and 2. Subsequently, the codes and their corresponding reasoning are sent back to Agent_A. Agent_A evaluates the codes and their rationale, devising an improved plan accordingly.

Q3: Why choose to generate code instead of natural language description? (Using LLMs decreases trustworthiness.)

R3: This question helps shed light on our motivation. The primary advantage of code lies in its superior clarity and interpretability. Its structured nature allows humans to easily understand the reasoning behind the LLM's outputs, thereby boosting trust in its results. In contrast, natural language descriptions of crystal structures lack the same level of transparency, making it harder to discern the rationale behind the LLM's conclusions and potentially undermining confidence.

Q4: “Given these, I don't see an advantage of using LLM here.”

R4: It is worth noting that our goal is to assist human experts in accelerating material research instead of completely replacing them. This acceleration manifests in two aspects.

First, while human experts excel in their specialized fields, they often invest significant effort in acquiring cross-disciplinary skills, such as coding or understanding graph learning. Rep-CodeGen enables materials science professionals to concentrate on their core expertise by reducing the need to master complex computer science concepts, thereby saving both time and resources.

Second, the generated representations can be viewed as a novel source of knowledge that can inspire the researchers. Rep-CodeGen shows the capability to rapidly generate a diverse array of solutions. For instance, to ensure that representations satisfy periodic equivariance (i.e., changes in lattice coordinates should be reflected in the representation), current graph representation methods typically rely on obtaining sufficiently large cutoffs. By contrast, RepCode-Gen introduces innovative strategies beyond the traditional cutoff method, such as converting periodic data into length and angle metrics, incorporating angles between lattice and edge vectors, and proposing supercell configurations extending unit cells along periodic dimensions.

Thank you very much for your time and for recognizing the originality of our work, the contributions to the problem, and the experimental design. Below, we will address your suggestions and questions point by point.

Q1: It is recommended to provide the initial code to help readers better understand the extent of changes during the evolution process.

R1: Due to space limitations, we are unable to present the complete code here.The initial code will be provided in the appendix. In terms of design philosophy, the initial code determines neighbors of atoms within the unit cell based on a distance cutoff, and the features are limited to the distances between neighbors. However, the representation code that satisfies all conditions after the framework evolution differs in both the neighbor determination method and the features.

Q2: It is recommended to provide the complete prompts, particularly the descriptions of the constraints, to enhance reproducibility and clarity.

R2: We will include the complete prompts in the appendix. Below is our full description of the constraints:

Permutation_invariance: Changing the atomic indices should not alter the graph representation.

Rotation_invariance: Rotating the atomic coordinates should not change the graph representation.

Reflection_equivariance: Performing a mirror symmetry on the lattice and atomic coordinates should result in a change in the graph representation.

Lipschitz_continuity: If the coordinates of the crystal undergo continuous changes, the corresponding graph representation should also change continuously. This means the neighbor relationships of each atom (i.e., which atom is a neighbor) should not change, but the attributes of the corresponding neighbors (such as distance or angle) should vary accordingly.

Periodicity_equivariance: The graph representation should implicitly incorporate lattice information, meaning that any modification to the lattice coordinates (e.g., scaling the lattice by 1.5 times) should result in a change in the graph representation.

Translation_invariance: Translating the atomic coordinates should not affect the graph representation.

Q3: Could there be situations where constraints conflict with each other, meaning that satisfying some constraints might make it impossible to satisfy others?

R3: Theoretically, this situation is highly unlikely to occur because these physical constraints are grounded in real-world conditions, making conflicts almost improbable. Moreover, even if conflicts were to arise, the framework would evolve in a direction that satisfies the majority of the constraints.

Q4: In the conclusion, the authors mention that the proposed representation method could also be applied to crystal generation. Could the authors provide more details on how this representation would be utilized in crystal generation tasks?

R4: First, taking state-of-the-art algorithms in crystal generation tasks, such as CDVAE[1], DiffCSP[2] and DiffCSP++[3], as examples, the graph representation we propose can serve as input to denoising models and be directly applied to crystal generation tasks.

Second, the “constraints” that Rep-CodeGen can solve, are not limited to physical constraints but can also include arbitrary requirements from human experts. Therefore, our framework can generate new crystal representation codes that meet the needs of various scenarios, such as sequence-based crystal representations required by LLMs.

[1]T. Xie, X. Fu, O.-E. Ganea, R. Barzilay, T. Jaakkola, Crystal diffusion variational autoencoder for periodic material generation, arXiv preprint arXiv:2110.06197 (2021).

[2]R. Jiao, W. Huang, P. Lin, J. Han, P. Chen, Y. Lu, Y. Liu, Crystal structure prediction by joint equivariant diffusion, Advances in Neural Information Processing Systems 36 (2024).

[3]Jiao, Rui, et al. "Space Group Constrained Crystal Generation." The Twelfth International Conference on Learning Representations.

Thank you very much to the reviewers for your time and for recognizing the innovation of our method and its contributions to the field. We will revise the paper carefully. Below, we will summarize your reviews and provide a response to each point separately.

Q1: The experiments are stopped at five due to resource limitations. Why the results of GPT+Rep-CodeGen and Seek+Rep-CodeGen are 0 for five constraints in Table 2? Does this mean that the framework is only effective for QianWen?

R1: Although the choice of five constraints is employed due to the resource limits, it can show a clearer comparison between LLMs with and without our framework. The zero results for GPT+Rep-CodeGen and Seek+Rep-CodeGen occur because we capped the experiment at 1,000 programs to ensure a fair comparison. We should also pay attention to the results other than three constraints which also reflect the evolutionary capability of our framework. Thus, the framework is not limited to QWen and can also be applied to other LLMs.

Q2: Although the generated codes are interpretable, the paper does not provide detailed insights into understanding the logic and structure of these codes. There is also no discussion on how to further optimize these codes to improve performance.

R2: In this paper, the framework goal is to identify crystal representations that satisfy all constraints, and thus we treat the entire evolutionary process as a black-box procedure. Our framework enables better utilization of the interdisciplinary knowledge of LLMs while guiding them to optimize and generate code effectively. Since the entire process is automated by the LLM, not interpreting the intermediate codes does not affect the final outcome.

Q3: The experiments used specific network architectures of PerCNet. Can the impact of network choices on the results be further discussed?

R3: We use PerCNet because it is currently the only network architecture in this field capable of handling dihedral angles. We employ this network to ensure a fair comparison between the representations obtained by our framework and those of PerCNet. In the future, we plan to use LLMs to simultaneously generate both the representations and the corresponding network architectures, rather than solely relying on network architectures from other algorithms.

Q4: The test set is constructed manually, will it be better to employ the LLMs to generate the test set?

R4: Using LLMs to generate the test set is possible, but in terms of effectiveness, a manually designed test set would be better. Due to the hallucination issue of LLMs, they might generate answers that appear correct but are actually wrong. Since the test set determines the direction of evolution, a manually designed test set would be more reliable and effective.

Q5: Will the framework get stuck in local optima when facing more complex constraints? If so, how can mechanisms be designed to avoid this?

R5: It is unlikely to fall into local optima because our program selection method (i.e., Equation 1 in the paper) ensures that the two parent programs have a higher probability of satisfying different constraints, thereby maintaining the diversity of the parent programs.

Thank you for your valuable time and your recognition of our work's motivation, novel contributions, and experimental design. We provide point-to-point responses to your questions as follows.

Q1: The neighborhood selection is the main difference between the 'new' representation and the existing methods. Why does this change affect the property predictions in Table 1?

R1: The neighbor selection method in the generated representation enhances the graph's capability to capture long-range atomic interactions. As a result, the energy-related properties are improved. Specifically, the interactions between atoms are not limited to adjacent atoms, and long-range interactions can extend to distant atoms, which play a key role in the stability of crystals. Taking ionic crystals as an example, the Coulomb force between ions is a type of long-range interaction with a relatively large range of action. Cations and anions are attracted to each other through the long-range Coulomb force, which maintains the structural stability of the crystal. In our representation, neighbors are determined based on periodic regions instead of a cutoff value, which can cover the atoms over a longer distance. Besides, the long-range interactions also affect the electrical and thermal properties of crystal materials.

Q2: The LLMs may train on methods and their codes in Table 3. Thus, the 'new' representation may be a combination of the existing codes. Then, how can the framework satisfy the unseen constraints?

R2: LLMs can be trained not only on materials science research papers but also on works from other fields, such as protein design and drug discovery. With all these training resources, LLMs have a better understanding of atom interactions inside the materials. Therefore, we aim to guide the LLMs in generating the desired representations. To the best of our knowledge, the neighborhood selection for the new representation, which is based on periodic regions, has not been observed. As shown in Table 2, the representations satisfying four and five can be viewed as unseen constraints. Among these representations, we also observed 'new' representations that are different from existing methods.

Q3. This framework costs a lot of tokens to generate a target code. How to compare the cost from both time and money to DFT calculations? Besides, what is the size of LLMs in Table 2? As the LLMs become more and more powerful, a new version of GPT or Deepseek may accomplish this task well.

R3: The cost of Rep-CodeGen is minimal compared to DFT calculations. We utilize the Qwen2.5-Coder-7B model to develop three agents, opting for the 7B version over the Qwen2.5-Coder-32B primarily due to cost considerations. While the Qwen2.5-Coder models are open source, the operational expense of the 32B model is significantly higher than that of the 7B model. Additionally, the 7B model offers considerably faster inference times. Our aim is to create a framework accessible to a broad spectrum of researchers. Specifically, we only utilize one NVIDIA RTX 24G 3090 GPU for all experiments.

A larger LLM can benefit the framework, but a single one may not accomplish the task well, as shown in Table 3. GPT-3.5 (gpt-3.5-turbo used in our experiments) and DeepSeek (DeepSeek-Coder-v2 used in our experiments) represent a larger general-purpose LLM and a larger code-generation LLM, respectively. Nonetheless, their performance remains unsatisfactory.