Efficient and Scalable Density Functional Theory Hamiltonian Prediction through Adaptive Sparsity

We thank all your valuable comments very much, and would like to discuss more on the 4th question of ‘Other Strengths And Weaknesses’:

We had conducted a series of ablation study to examine the contribution of each module, and the results had been included in the Appendix B.2. As shown in Appendix Table 6 (we also listed the results below), we removed the Sparse Pair gate, the Sparse TP gate, and the Vectorial Node Interaction block from the SPHNet model, respectively, to see their impact on the model efficiency. We found that the two kinds of sparse gates can improve the speedup ratio from 1.73x to 7.09x, demonstrating the effect of sparse gates on the model acceleration. Also, each of the Sparse pair gate and Sparse TP gate alone can provide significant acceleration, improving the speedup ratio from 3.98x to 7.09x and 2.73x to 7.09x. You can find the detailed analysis in Appendix B.2.

Additionally, we add a similar study by adding the Sparse pair gate and Sparse TP gate to the QHNet model here. Specifically, we applied the Sparse Pair gate on the second Non-diagonal pair block and applied the Sparse TP gate on the Node-wise interaction blocks and both Diagonal and Non-diagonal pair block. As shown in the table below, the sparse gate can effectively accelerate the training speed of the QHNet model and save computational consumption. We will also add this part of the experiment to the Appendix in our future version also.

Table: The effect of sparse gates on SPHNet/QHNet model on PubChemQH dataset.

Besides, we conduct an extra ablation study to evaluate the effect of different modules in the SPHNet architecture. Specifically, the standard SPHNet model has 4 Vectorial Node Interaction blocks, 2 Spherical Node Interaction blocks, and 2 Pair Construction blocks. We removed all the sparse gates and reduced the number of these three kinds of modules to 1, respectively, and observed the model performance. As shown in the table below, we found that both Vectorial Node Interaction block and Spherical Node Interaction block significantly affect the model performance, indicating the design of architectures with progressively increased irreps orders has an important positive impact on the models. Interestingly, we found that remove one Pair Construction block would not strongly affect the model accuracy, suggesting that there is actually room to further speed up the model. We will explore this further in our future work.

Table: The effect of different modules on SPHNet model on PubChemQH dataset.

Thanks for your valuable comments and suggestions. We address each of the comments individually below.

Weakness：

Your assumption regarding the effect of the sparse gate is correct. As demonstrated in the ablation study presented in Table 6 of Appendix B.2, both types of sparse gates significantly improve speed with only a slight loss in accuracy. Therefore, compared to QHNet, the performance improvement in accuracy of our method primarily stems from SPHNet’s architectural design. Given that SPHNet has already achieved strong performance, the minor accuracy loss due to sparsity is entirely acceptable. For more details on the ablation study of different sparse gate, please refer to Appendix B.2 of the paper. Additionally, further experiments on applying the sparse gate to QHNet can be found in our response to the 4th Reviewer EBLX.

Besides, to further address your concerns, we conducted an ablation study to evaluate the effect of different modules in the SPHNet architecture. Specifically, the standard SPHNet model has 4 Vectorial Node Interaction blocks, 2 Spherical Node Interaction blocks, and 2 Pair Construction blocks. We removed all the sparse gates and reduced the number of these three kinds of modules to 1, respectively, and observed the model performance. As shown in the table below, we found that both Vectorial Node Interaction block and Spherical Node Interaction block significantly affect the model performance, indicating the design of architectures with progressively increased irreps orders has an important positive impact on the models. Interestingly, we found that remove one Pair Construction block would not strongly affect the model accuracy, suggesting that there is actually room to further speed up the model. We will explore this further in our future work.

Table: The effect of different modules on SPHNet model on PubChemQH dataset.

However, since our model has very different architecture and components from other models, and the specific module in the model is not a one-to-one correspondence, it's hard to make a substitution of the specific module and carry on the ablation study with the other model like QHNet. We are sorry that we are not able to carry on the ablation study with a different model and explain why the SPHNet has such performance improvements compared to others. We'd like to give a try in the future to explore more about this.

Questions For Authors 1:

Thank you very much for your interest! We are very glad to share our data with the whole community. But our organization require a review process for open sourcing data, which might take some time. We are already actively promoting this data open source process and hope to be able to share our data soon.

Questions For Authors 2:

Thank you for your question. We have actually evaluated the relationship between model performance and sparsity ratio in Section 5.4. As shown in Figure 3, for all three datasets, the Hamiltonian MAE remained stable within a certain sparsity range. However, when the sparsity rate reached a particular threshold, we observed a significant increase in the Hamiltonian MAE, which we interpret as the upper limit of sparsity. This suggests that a suitable range of sparsity has little impact on model accuracy while significantly improving computational efficiency. We provide a detailed analysis of this in Section 5.4.

Thank you very much for your valuable comments and suggestions. Below are our detailed responses.

Claims And Evidence

1. Thank you for your question. As you noted, SPHNet’s lightweight design allows it to run 1.73× faster than QHNet on the PubChemQH dataset. However, the primary acceleration comes from sparse gates. To isolate their effect, we included ablation study in Appendix B.2. Beside, to further address your concerns, we test these gates on QHNet. Please refer to our answer to the 4th reviewer EBLX for the details due to the length limitation of the response.

2. The computational cost of key sparsity has already been included in Figure 3, where the number above each “$\times$” symbol represents the training speed at that specific sparsity level. In Appendix Figure 8, we provide a more detailed visualization of how time scales with sparsity. Additionally, we have listed the complete results for training speed and GPU memory usage at each sparsity level on the PubChemQH dataset in the table below. The results indicate that as sparsity increases, both time and memory costs decrease in an approximately linear manner.

Experimental Designs Or Analyses:

1. We agree with your opinion that the sparse gates cause suboptimal results. As we had analysis in Section 5.3, since the water molecule is very small (only 3 atoms) compared to other molecules, there is not much room for the atom pairs and TP combination reductions. Therefore, spare gates may remove the necessary interaction combinations within the system and cause poor results.

   However, we would like to declare that the sparse gates are not designed for these extremely small molecules, and we are more concerned about their performance on larger molecules, which turn out to be very efficient on the large molecule dataset PubChemQH.

Other Strengths And Weaknesses

1. Sorry for the confusion caused by the word "probability." A clearer term would be "percentage." Specifically, in the second phase, TOP(·) selects elements whose weights are within the top $( 1 - k )$ percent, with no randomness involved—selection is purely based on learnable weights. We have revised the manuscript for clarity.

   Regarding path selection, the learnable weights continue to update in phase two, as they participate in subsequent operations (Equations 4 and 5). Only selected paths' weights are updated, while TOP(·) consistently selects the highest-weighted elements. As weights evolve, the selected paths adjust accordingly, allowing the sparse gate to gradually learn the optimal path set.

2. Thank you for the question. The $w_{ij}$ is a $R^{k}$ vector, where $k$ is the number of elements in the complete set $U_c= \{(\ell_1, \ell_2, \ell_3) \mid \ell_3 \in [|\ell_1 - \ell_2|, \ell_1 + \ell_2] \}$ in the tensor product. And the $w_{ij}^{\ell_1, \ell_2, \ell_3}$ is a single number from the $w_{ij}$ vector. Note that in Equation 12 we selected the $w^{\ell_1, \ell_2, \ell_3}_{ij}$ that within the $U^{TSS}_p$ set.

3. We used a fixed scheduler step $( t = 3 )$ across all datasets and experiments, as results remained stable, indicating minimal impact on performance. We recommend setting $( t = 3 )$, but users can adjust it with minimal tuning cost.

   For sparsity rate, Section 5.4 shows performance remains stable within a reasonable range, with degradation occurring only beyond a certain upper limit, which depends on molecular size. Users can select this parameter based on our experimental results for their specific molecular sizes.

Other Comments Or Suggestions:

1. Thank you for pointing that out—the more precise shape should be $n \times n_0$, where $n_0$ corresponds to the number of occupied orbitals. The reason we sometimes write $C$ as $n \times n$ is that the KS equation is typically solved for all eigenstates, including virtual orbitals. We will revised this and running title in the revised version.

Questions For Authors

1. Please see previous responses.

2. Thank you for raising this question. We addressed this in Section 4.1. To ensure that the combination selection is as unbiased as possible, we initialized the learnable matrix $W$ (the sparsity weights) as an all-one vector. This means that, at the beginning, all combinations are considered to have the same importance.

3. Thank you very much for pointing this out. Here, the result of inner product is still a irrep with different order of feature, and the superscript stands for the order of irrep feature we used (actually, all the superscripts in our paper stand for the order of irreps feature). But we just found that there is a typo in the writing of the formula. The correct formula should be $<x_i,x_j>^{1:}$, and the superscript ${1:}$ stands for the irrep feature larger than order zero. We are very sorry about the mistake and have fixed it in the manuscript.

Thank you very much for your valuable comments and suggestions. Below are our detailed responses.

Other Strengths And Weaknesses

Thank you for pointing that out—your notation is the more precise way to express it. $C$ should indeed be $n \times n_0$ in Equation 1, where $n_0$ corresponds to the number of occupied orbitals. We sometimes write $C$ as $n \times n$ because the KS equation is typically solved for all eigenstates, including virtual orbitals. We appreciate your keen attention to notation and will make sure to clarify this in the revised version.

Other Comments Or Suggestions

1. Thank you for pointing this out. We are carefully considering this, and while we may not be able to provide a definitive new name in the short term, we will update this in the subsequent versions.

2. The higher coefficient rates imply that as the molecular system grows larger, we can apply a higher sparsity rate to further accelerate the model without sacrificing accuracy. We apologize for the confusion and have revised this to 'higher sparsity rates' for better clarity.

3. As you mentioned, compared to the tensor product operation, all these operations only introduce minimal computational overhead and have little impact on the overall speed. However, this discussion is still meaningful, and we will include it in the subsequent version of the manuscript.

   In the three-phase sparsity scheduler, for a given unsparsified set $U$, the additional computational overhead in the first phase has a complexity of $\mathcal{O}(|U|)$, contributed by the RANDOM(·) operation. The second phase has a computational overhead of $\mathcal{O}(|U| \log{|U|})$, arising from the TOP(·) operation. Since we fix the learnable weight matrix and the selected elements, there is no additional computational overhead in the third phase. For detailed information, please refer to Equation 3.

   For the sparse TP gate, the computational overhead comes from the element-wise multiplication of two weight vectors (Equation 5), so its complexity is $\mathcal{O}(|U_c|)=\mathcal{O}(L^3)$.

   For the sparse pair gate, the additional computational overhead mainly comes from the linear layer $F_p(\cdot)$ in Equation 7, with its complexity being $\mathcal{O}(d_{hidden}|U_p|)$, where $|U_p|$ is always the square of the number of atoms and $d_{hidden}$ is the hidden feature dimension. Other operations, including the inner product (Equation 6) and the weight calculation (Equation 9), are necessary operations in our framework, even without the sparse pairwise gate.

4. -7. Thank you for the comments. We have revised the manuscripts accordingly.

Questions For Authors

1. Thank you for the question. We agree with your point. In fact, in our experiments, the number of selected short-distance pairs still constitutes the majority of all selected pairs, which aligns with the nearsightedness of electronic matter, as shown in Figure A. Each bar represents the fraction of selected pairs within the range from k to k+1 Å relative to the total number of selected pairs. Therefore, when we say “as the pair length increases, the probability of a pair being selected also rises,” we mean that pairs at longer distances are indeed retained at a higher proportion in Figure B—though their absolute count remains significantly lower compared to short-distance pairs. For example, among 10 pairs at 25–26 Å, 6 may be selected, whereas among 300 pairs at 4 Å, 100 may be selected.

   There are two possible reasons for the observed tendency to retain long-distance pairs. First, long-range interactions, including electrostatic interactions and weak van der Waals forces, are crucial for accurately describing large molecules. Consequently, incorporating such interactions could be beneficial for overall accuracy, a property that has been studied in previous research (Knörzer J, et al. https://arxiv.org/pdf/2202.06756; Li Y, et al. https://arxiv.org/pdf/2304.13542.). Second, since long-distance pairs constitute only a small fraction of all pairs, once sufficient data has been collected to characterize short-range interactions, selecting more long-distance "hard samples" could contribute to improving final accuracy.

   However, due to the lack of large molecular system datasets, we conducted a similar experiment on the QH9 dataset (Figure A-B), where we found that the selected ratio was similar to the proportion of short-distance pairs in the PubChemQH dataset. However, since the maximum atomic distance in this dataset is less than 8 Å, we did not observe a preference for retaining long-distance pairs. This is reasonable, as long-range atomic interactions typically become significant only when the interatomic distance exceeds 12 Å. As larger molecular datasets become available in the future, we hope to conduct further experiments to validate this hypothesis.

2. We have discussed that in the previous responses.