ELoRA: Low-Rank Adaptation for Equivariant GNNs

Thank you for the valuable comments and suggestions.

Q1: The meaning of $K$.

A1: $K$ means the number of channels. $K_0^1$ means the number of channels of the first tensor when $l=0$.

Q2: The meaning of the superscript of $K$.

A2: The superscript of $K$ does not represent a power. The $K$ in Figure 3 corresponds to the $K$ in the equations of Section 4.2. Here, the superscript of $K$ is used to identify different tensors, while in Equation (7), the subscript of $k$ is used for the same purpose. We will address this in our revised paper.

Q3: The ambiguity of the multiplication symbol.

A3: In line 254, it is used to denote the dimension of a matrix. In lines 256–257, it means $R \ll \min(K^3_{l_3}, K^2_{l_2} \cdot K^1_{l_1})$. The use of the multiplication symbol here might cause misunderstanding, and we will revise it.

Q4: The choice of $K_3$.

A4: $K_3$ is a hyperparameter of the neural network and can be specified arbitrarily.

Q5: The connection between $K$ and datasets.

A5: Here, $K_1$, $K_2$, and $K_3$ are not uniquely determined by the dataset; rather, they are hyperparameters of the network and can be set arbitrarily. This is similar to the hidden dimensions in an MLP, which can be larger than the input dimension. The number of channels used in our model is provided in Section E.4, which is typically set to 128.

Q6: Code availability.

A6: ELoRA is implemented based on e3nn. We have provided the code in the anonymous repository https://anonymous.4open.science/r/ELoRA/README.md.

Thank you for your feedback and we will correct all the typos, improve the writing and enhance the readability according to your comments. We will polish the describtion of Figure1 and add reference link for the proved propositions.

Q1: The novelty of ELoRA.

A1: We propose a PEFT method for SO(3)-equivariant MPNNs, which is technically novel in its path-dependent low-rank adaptation. To the best of our knowledge, no existing PEFT methods preserve equivariance.

Q2: The comparative experiment of different finetuning methods.

A2: We will compare two other popular fine-tuning methods: Readout fine-tuning (freezing the previous layers and only fine-tuning the Readout layers) and Adapter fine-tuning. Table 1 depicts the RMSE results of these fine-tuning methods on Cu and Sn datasets. Adapter fine-tuning performs worse because the equivariance is destroyed under the fine-tuning process. Readout fine-tuning is not as accurate as full-parameter fine-tuning because only the last few layers are retuned, which lacks some flexibility compared to full-parameter fine-tuning.

Table 1 The comparasion of four different finetuning methods.

Q3: Number of parameters after using ELoRA.

A3: In ELoRA, the path-dependent weights can be merged into the model weights after training, just like in the original LoRA. Thus, it will not increase the number of the model's parameters. The last row of Table 1 records the number of trainable parameters of the four different fine-tuning methods.

Q4：The applicability on other SO(3)-equivarant MPNNs.

A4: We add experiments with ELoRA on SevenNet [1] (another representative SO(3)-equivariant MPNN), as shown in Table 2. The conclusions drawn from Table 2 are consistent with those from Table 1. Our proposed ELoRA can be adapted to other SO(3)-equivariant MPNNs in a user-friendly manner.

Table 2 The finetuning methods on SevenNet.

[1] Park, Yutack, et al. "Scalable parallel algorithm for graph neural network interatomic potentials in molecular dynamics simulations."

Q5: Results analysis on inorganic dataset.

A5: The prediction errors of energy and forces should be considered jointly, as they are typically optimized together during training. We cannot evaluate based solely on either energy or forces. For a comprehensive analysis, we can equally combine energy and force RMSEs as a joint metric. Under this metric, ELoRA achieves the best accuracy on 9 out of 10 datasets, as Table 3 shows.

Table 3 Summed RMSE of Energy and Force.

Q6: Clarify of preseved equivariance in ELoRA.

A6: ELoRA is a parameter-efficient fine-tuning method. Its parameters can be merged into the base model after tuned, the network structure remains the same. Thus, the equivariance is preserved.

ELoRA projects the equivariant features into a lower-dimensional space through equivariant operations. These projected features remain SO(3)-equivariant, as proved in Proposition 4.4.

Q7: The explanation of the pretraining dataset's sparsity and the materials' diversity.

A7: The diversity refers to the combinatorial diversity of material structures. For example, even small organic molecules composed of carbon（C）, hydrogen（H）, oxygen (O), and nitrogen (N) atoms can theoretically form up to $10^{60}$ possible structures.

The sparsity refers to the fact that labeled data with ab initio accuracy is in a small number. Computing structures' quantum properties (e.g., energy and atomic forces) require expensive Density Functional Theory (DFT) calculations. Currently, only a small fraction structures has high-quality DFT labels. The available labeled structures represent sparsity in the large structure space.

Q8: The organization of the paper.

A8: For a detailed explanation of the Section 3, please refer to Reviewer iY6i, Q1.

Many thanks to your questions and suggestions.

Q1: The fairness in using only 50 samples in rMD17 experiments.

A1: In practical downstream applications, fine-tuning methods are often expected to perform well with limited first-principles training data, as DFT calculations are computationally expensive. To simulate such realistic requirements, we construct low-data scenarios (e.g., extract only 50 training samples from each small dataset in rMD17) to demonstrate the accuracy performance of dedicated models (ACE, NequIP) and fine-tuned pretrained model such as MACE.

Q2: Full-parameter finetuning results of 3BPA and AcAc.

A2: Our experiments show that ELoRA based MACE is better than full-parameter fine-tuned MACE, especially in the out of domain test set. Table1&2 show the from-scratch-training and finetuning reults of 3BPA and AcAc.

Table1 3BPA results

Table2 AcAc results

Q3: The comparison with SOTA results PACE.

A3: We will cite the PACE paper and include the results of PACE in the organic experiments. PACE represents SOTA results among dedicated models that are individually trained for each small dataset. However, our work focuses on developing novel fine-tuning techniques for pretrained models.

In the case of rMD17, PACE achieves high accuracy when 1000 samples are used in training, as the PACE paper shows. In low-data scenarios, when training the PACE model with 50 samples per dataset, the accuracy cannot be reached, which is achieved with 1000 training samples. While, ELoRA makes it possible to obtain high-precision downstream models when a small amount of training data is provided.

Table3 lists PACE and MACE results. The second and third columns show the MAE for PACE with 1000 and 50 training samples respectively. The fourth column reports the MAE results of MACE ELoRA. Table 3 reveals that PACE requires abundant training data to maintain its high accuracy. Its performance will drop at 50 training samples. ELoRA maintains prediction precision even with this minimal training set.

Table3 rMD17 results

Q4: The analysis on SVD decomposition of weight matrices.

A4: Please refer to Reviewer iY6i, Q1.

Q5: Reduction in the number of parameters.

A5: The pretrained model covers multiple elements spanning the periodic table. But for downstream tasks, we retain only weights relevant to the task's elements and prune all others. This causes the reduction of model parameters. We will clarify this issue in the revised paper.

We thank the reviewer for the professional and valuable comments.

Q1: The analysis on SVD decomposition of weight matrices.

A1: We apologize for the analysis of the SVD decomposition, which could be misleading. Section 3 serves as a transitional part, aiming to convey the necessity of fine-tuning on pre-trained models rather than training models from scratch. Then, we naturally introduce our innovative fine-tuning method, ELoRA.

In Section 3, Figure 1 provides a qualitative interpretation from the chemical space perspective, comparing the coverage function space among pre-trained models, fine-tuned models, and models trained from scratch. The SVD decomposition analysis intends to offer quantitative information, demonstrating that the weights of fine-tuned models exhibit higher similarity to the pre-trained models than those trained from scratch models.

The function space represented by deep neural networks is complex due to its high dimension and nonlinear nature. To our knowledge, we have not yet found an alternative theoretical explanation to better characterize the learned function spaces. As a result, we use spectral data in each layer to figure out the knowledge embedded in model weights. It may not be a rigorous analysis. If the reviewers consider the SVD analysis insufficiently compelling, we can move the SVD experiments to the Appendix and move the AcAc and 3BPA experiments to the main text.

As for spectral analysis on ELoRA-fine-tuned weights ($W_{\text{ELoRA}}$), we add the comparison on the pre-trained weights $W_0$ and $W_{\text{ELoRA}}$ (see Link: https://anonymous.4open.science/r/ELoRA/picture/spectra.png). The results show that $W_0$ and $W_{\text{ELoRA}}$ maintain high similarity. The distribution of cosine similarity is close to that of $W_0$ and full fine-tuned weights ($W_{\text{Full-parameter}}$). The added figure indicates that $W_{\text{ELoRA}}$ and $W_{\text{Full-parameter}}$ have high similarity to $W_0$. We will add this figure in the revised manuscript.

Q2: The claim of "equivariant MPNNs"

A2: Our proposed fine-tuning method applies specifically to SO(3)-equivariant MPNNs. We will clarify the scope by specifying SO(3)-equivariant MPNNs in our revised paper.

Q3: Choice of decomposition.

A3: In Equation (8) $W^0_{l_3 l_2 l_1} + \Delta W^0_{l_3 l_2 l_1} = W^0_{l_3 l_2 l_1} + B_{l_3 l_2 l_1} A_{l_3 l_2 l_1}$, the weight matrix $W_{l_3 l_2 l_1}$ has dimensions $K^3_{l_3}$, $K^2_{l_2}$, and $K^1_{l_1}$. We merge $K^2_{l_2}$ and $K^1_{l_1}$ because the computation involves a transformation from an intermediate tensor of dimension $K^2_{l_2} \cdot K^1_{l_1}$ to an output tensor of dimension $K^3_{l_3}$, making this merging a natural design choice, as illustrated in Figure 3. There exist various weight decomposition method, identifying other more effective weight decomposition strategies (such as Tucker decomposition) will be one of our future works.