LLM-Prop: Predicting Physical And Electronic Properties of Crystalline Solids From Their Text Descriptions

Some details are not clear. How large is the pre-trained T5 model you used? The T5-small, T5-base, or T5-large? Specify this is important for the comparison of your efficiency.

We did add the number of parameters for the T5 model and the MatBERT baseline on table 2 and 3, and table 5. We will specify that this corresponds to T5 small in the revision.

You have mentioned there’s some related work that also used finetuned LLMs for crystal representation. They collect crystal text descriptions based on Robocrystallographer too. Could you compare your dataset with theirs about the text content and text quality? Showing the advantage of your dataset is important for your contribution to data collection.

Unfortunately, they did not release their data sets which makes it impossible to compare our benchmark with the data they used. Since we did not have access to their train-test splits, we finetuned MatBERT (the model that they finetuned on their data) on our benchmark so as to make a fair comparison with LLM-Prop.

Section 5.1 “The possible reason for this improvement might be that LLMProp can easily access the most important information for volume prediction, e.g. space group information, from the text descriptions compared to GNNs.” Could you please make some more ablation about your text information to prove your statement? Also, this is important to measure the importance of different types of information.

This is indeed a great point, we will include this ablation in the revision.

Could you please make a task description to show the importance of the two tasks (Band gap and volume) in the experiments? Many other properties are predicted in related works such as energy per atom and bulk modulus. Why did you choose these two tasks?

Our motivation to focus on the physical properties (Band gap and Volume) was based on the fact that while these properties are very important they are understudied in many prior works probably because prior methods seem to not perform well on these two tasks.

Although the task of crystal property prediction is an interesting problem for AI for science, the innovation of methodology is not surprising. This paper only uses the language model to make the prediction which is less novelty.

Although the idea of using LMs to make predictions is not novel, finetuning LLMs to achieve SOTA results on crystal property prediction using their descriptions compared to domain-specific models requires innovative approaches and to the best of our knowledge, LLM-Prop is the first method that is based on finetuning a non-domain-specific LLM to achieve that. Different techniques (see Table 6) that we combine together to finetune T5 encoder enable LLM-Prop to outperform a naively finetuned MatBERT despite having a three times fewer parameters than MatBERT which has been pretrained on domain-specific data. As the results show, we also improved over the GNN-based methods, achieving a significant difference in performance (see Table 3-5).

There are several recent works focusing on the text-rich graph where each node has text descriptions by jointly leveraging the GNN and LLM. These works are also related to the problem studied in this paper and should be discussed and compared.

Although there are works that use GNN and LLM, for example Zhao et al. 2023, we would like to remind the reviewer that these works are specifically implemented for tasks that are related to text-attributed graphs (for e.g. knowledge graphs and paper citation graphs) such as node classification. In those works each node is a text/document and nodes are interconnected in a graph format. Our work specifically deals with the task of crystal property prediction where each crystal is represented by a text description and does not require building a knowledge graph between different crystals, it directly predicts crystal properties using a finetuned LLM on a text description of that crystal. Although simple, our approach achieves state-of-the-art performance, outperforming even state-of-the-art GNN-based methods.

The specific challenge in this problem is not clarified. What is the specific challenge in the problem of crystal property prediction?

Crystals, unlike molecules, are rigid bodies. They have strong symmetries that are challenging to capture by even the current state-of-the-art GNNs (Choudhary et al. 2021). Our work shows that working in the description space of these crystal structures is a promising way to predict the properties of crystals. The text descriptions that are the input to our proposed method already have information about symmetries such as space groups and by having a powerful pretrained language model like T5 that can easily understand text and extract useful information from the crystal descriptions, we get the crystal representation that is used to accurately predict its properties.

My top concern is the technical depth of the paper. For the method, this paper is an implementation for predicting crystal properties tasks with the decoder part of T5. The discussion of choice of T5 is weak and Input Processing is also trivial.

We did not use the decoder part of T5 but rather built on the encoder of T5 with a carefully designed finetuning approach. We refer the reviewer to the last paragraph of the Introduction section and section 3.2.1 and 3.2.2 for our detailed discussion of why we chose T5. We also refer the reviewer to section 5.2 and Table 6 that discuss the detailed contribution of the input processing to the overall performance of LLM-Prop. Please, note that the input processing is very crucial when handling scientific data. For instance this very recent work (Golkar et al. 2023) shows that encoding numbers in scientific text differently improves the LLMs performance substantially.

For the Data collection in Sec 3.1, I did not see the challenging part and discussion about the process of data collection including quality control. I am not sure about the quality of data at all.

This is a good point. We will indeed add more details for this part in the revision. We will add details on how Robocrystallographer works, how long it took us to collect the structure data from the Materials Project database, and how to generate the descriptions with Robocrystallographer. We also add details on quality control where for instance we discarded crystals with shorter descriptions (less than 5 tokens), and statistics of the final datasets such as the average length of the descriptions, total number of tokens, average number of numerical values in each description, etc.