Infusing Self-Consistency into Density Functional Theory Hamiltonian Prediction via Deep Equilibrium Models

Thanks for your review. We will address each point in our response accordingly.

Weakness 1:

Thank you for your feedback. DEQH is a general method specifically devised to instill self-consistency into pre-existing models, while QHNet is a distinct network that predicts the Hamiltonian directly. Specifically, DEQH functions as a Hamiltonian solver, learning the iterative solving process $H^* = f(H^*, Z, R)$, while QHNet operates as a predictor $H = f(Z, R)$, where $H, Z, R$ denote Hamiltonian matrix, atomic number, and atomic coordinates resepctively. f represents a neural network and $H^*$ denotes the fixed point of the equation. In our research, we utilized QHNet as the fundamental backbone and incorporated DEQH into it, thereby creating DEQHNet. As mentioned in the Supplementary Material A.11, we opted for QHNet over PhiSNet due to the current implementation of PhiSNet being restricted to single-molecule support. This limitation stems from PhiSNet’s matrix prediction module, which is tailored to predict matrices of fixed size for identical molecules.

DEQH introduces self-consistency by approaching the problem of Hamiltonian solution from the vantage point of a fixed-point iteration, which necessitates the inclusion of the Hamiltonian in the network's input. To ensure the broad applicability of the Hamiltonian across various graph neural networks, it must be transformed into signals on the graph. This is demonstrated in Section 4.1, where we transposed the Hamiltonian, overlap matrix, and other equivalent tensors into invariant node features.

Furthermore, we provide a PDF document in the global rebuttal section, which includes a figure delineating the distinction between the off-the-shelf model used for Hamiltonian prediction and the DEQH model. We understand the importance of clearly communicating these differences and will further elaborate on these points in the revised version of our paper.

Weakness 2:

Thank you for your question. In Appendix A.7, we provide detailed definitions of the metrics. H refers to the Mean Absolute Error (MAE) of the Hamiltonian matrix, while epsilon and Psi are the eigenvalues (orbital energy) and eigenvectors (orbital coefficients) obtained by solving generalized eigenvalue problems. The metrics for epsilon and Psi are the MAE of the orbital energies and the cosine similarity of the orbital coefficients, respectively.

We explain in Appendix A.9 why epsilon is higher than the baseline in many cases. We selected a molecule randomly, added a Hermitian-Gaussian noise matrix to its Hamiltonian, and solved the corresponding generalized eigenvalue equation. We observed that as the noise on the Hamiltonian increased, the range of errors in the orbital energy also increased. The orbital energy errors reported for both QHNet and DEQHNet on the MD17 and QH9 datasets fall within this range, suggesting the experimental results are reasonable. This implies that only when the MAE of the Hamiltonian is sufficiently small can we ensure the corresponding orbital energy deviation is also small. In repeated experiments on the MD17 dataset, we found that when the Hamiltonian MAE of the model is at the same level, it is possible to obtain both a lower and a higher orbital energy MAE than the baseline. We believe this contributes to the performance difference observed between DEQHNet and the baseline model. This somewhat suggests that using existing Hamiltonian error measurements cannot definitively reflect the errors in the eigenvalues. This is a nontrivial, separate research question that we plan to investigate in the future.

We will elaborate on this point more clearly in the revised version of our paper to provide a comprehensive understanding of the model's performance.

Limitations:

Thank you for your insightful comments. Indeed, the DEQH model introduces self-consistency using DEQ, but it remains a supervised learning task that requires DFT data as labels. The statement "DEQH model is a Hamiltonian solver" refers to the DEQH model's ability to learn the iterative process of solving the Hamiltonian through DFT data, rather than it being a Partial Differential Equation (Schr"odinger equation) solver. With this mechanism, DEQH is a better way for supervised learning of Hamiltonian in that it exploits more information in a model of limited size by passing through the model multiple times until convergence, and that the model learns the iteration map which is a physical mechanism that generalizes better.

Your idea of bypassing DFT computations is intriguing and aligns with our thoughts. As we mentioned in Appendix A.11, some existing methods introduce self-consistency using unlabeled data. Our approach is entirely orthogonal to these methods. Therefore, combining these methods to achieve a trade-off between DFT computational cost and accuracy is a promising direction worth exploring.

Thank the authors for response. I still think the ablation study is hard to interpret, but I appreciate the additional comparison with other benchmarks. I have increased score accordingly

We are pleased that our response has satisfied you and are very grateful for the increased score.

As outlined in our rebuttal, the DEQH functions as a Hamiltonian solver, learning the iterative solving process $H^* = f(H^*, Z, R)$, while the QHNet acts as a predictor with the formula $H = f(Z, R)$. In practice, if the dataset includes overlap matrices, the network input can also incorporate this information, treating it similarly to the Hamiltonian by transforming it into equivariant node features (even if it's absent from the dataset, the calculation of the overlap matrix is quite straightforward and can be generated during data preprocessing). The overlap matrix offers a wealth of detailed information and can be computed easily with $Z$ and $R$. Accordingly, the equations for the DEQH and the modified Hamiltonian predictor become $H^* = f(H^*, Z, R, S)$ and $H = f(Z, R, S)$, respectively, where $S$ is the overlap matrix. In line with this, we conducted several experiments in our ablation study, testing the QHNet ($H = f(Z, R)$), QHNet w/ S ($H = f(Z, R, S)$), DEQHNet ($H^* = f(H^*, Z, R, S)$), and DEQHNet w/o S ($H^* = f(H^*, Z, R)$) on the Uracil dataset. The results depicted in Figure 4 indicate that:

• Incorporating the overlap matrix as an input leads to a lower Mean Absolute Error (MAE) in predicting the Hamiltonian (DEQHNet's Hamiltonian MAE is lower than DEQHNet w/o S, and QHNet's Hamiltonian MAE is higher than QHNet w/ S). Furthermore, the inclusion of the overlap matrix results in higher similarity in the orbital coefficients, suggesting that the overlap matrix is beneficial for the network's learning of the Hamiltonian and also supports our experimental hypothesis that the overlap matrix provides orbital information, which in turn improves the results for the orbital coefficients.
• Regardless of the presence of the overlap matrix in the network's input, the DEQH's Hamiltonian MAE is consistently lower than that of the QHNet-based Hamiltonian MAE, indicating that the DEQH model benefits from the introduction of self-consistency.

We will ensure that all new experimental results are included in the manuscript and will promptly refine our paper as soon as we are permitted a polish for the final version. We deeply appreciate your valuable feedback.

We sincerely appreciate the reviewer's recognition of the innovation and significance of our work and will address each point in our response accordingly.

Weakness:

Thank you for your insightful question. The Hamiltonian exhibits self-consistent iterative properties. Specifically, in the DFT solution process, the information from the previously obtained Hamiltonian is used to construct the current Hamiltonian for the current DFT solution. This process is repeated until certain criteria meet a set threshold. We hypothesize that introducing DEQ enables the network to learn the iterative mechanism of solving the Hamiltonian, rather than directly learning the solution to the Hamiltonian. This shift could allow the network to better capture inherent physical laws, thereby improving generalizability. Compared to models without self-consistency, DEQ introduces additional orbital information to the network input. This, coupled with the introduction of scientific priors, can expedite DFT convergence even when the model's output is simply used as the initial value for DFT (as shown in Appendix A.8). In contrast with methods that introduce self-consistency (which often include DFT computations in the loss function during model training), our method offers an architectural enhancement. The DEQH model does not directly include DFT computations, leading to a more efficient computational process. Empirically, if the dataset and molecular size are relatively small and the cost of DFT computations during training is acceptable, previous methods introducing self-consistency might outperform ours (as suggested by DEQHNet's results on water, where DEQHNet requires more data). However, when the dataset is large enough, or when it includes large molecules where DFT computation cost is prohibitive, the benefits of the DEQH model become more significant. As mentioned in Appendix A.11, since these methods are orthogonal, combining them could be a promising direction for future exploration.

We will make sure to discuss these points more clearly in the revised manuscript.

Questions:

Thank you for your insightful question. Compared to works introducing self-consistency, our method enhances the model architecture. As mentioned in the related works section of our paper, several methods introduce this attribute during the training process at the loss level, requiring direct DFT computation. In contrast, the DEQH model does not directly involve DFT computation, leading to more efficient computational performance. Empirically, if the data volume and molecular size are relatively small, and the DFT computational cost during training is acceptable, previous works introducing self-consistency might perform better (as suggested by DEQHNet results on water, where DEQHNet requires a larger data volume). However, when the data volume is sufficient, or the data contains larger molecules making the DFT computation cost too high, the improvements offered by the DEQH model become more substantial. As we mentioned in Appendix A.11, since these methods are orthogonal, combining them is a direction worth exploring.

Thanks for your review. We will address each point in our response accordingly.

Weakness 1:

Thank you for your feedback. The DEQH model is designed to predict the Hamiltonian. Given the self-consistent iterative properties of the Hamiltonian, we regard these properties as a fixed-point iteration problem, which inspires us to use DEQ to introduce self-consistency into existing models for Hamiltonian prediction.

We provide a PDF document in the global rebuttal section, which includes a figure delineating the distinction between the off-the-shelf model for Hamiltonian prediction and the DEQH model. Furthermore, we will revise our manuscript to simplify the technical language and include more background information about DEQs and Hamiltonian prediction. We will also add further explanations about the integration of these two components and how this combination contributes to the overall model. We hope that these revisions will make our paper more comprehensible to readers with various backgrounds.

We appreciate your feedback and will make sure to address this concern in the revised version of our paper.

Weakness 2:

Thanks for your insightful suggestion. Although DEQH model could be extended to periodic systems, DEQHNet, being built on the foundation of QHNet, is tailored for non-periodic systems. Hence, a direct comparison with DeepH-E3 and DeepH-2 may not be suitable.

In response to your feedback, we include additional benchmark results for SchNOrb [1], PhiSNet [2], and DeepH [3] on the MD17 dataset as follows.

However, it should be noted that the data used in DeepH is re-labeled by OpenMX[4]. Furthermore, considering the high training cost of the original PhiSNet, we provide PhiSNet results from the QHNet paper for a more balanced comparison.

We appreciate your suggestion and will make sure to incorporate these benchmark results in the revised version of our paper.

[1] Schütt K T, Gastegger M, Tkatchenko A, et al. Unifying machine learning and quantum chemistry with a deep neural network for molecular wavefunctions[J]. Nature communications, 2019, 10(1): 5024.

[2] Unke O, Bogojeski M, Gastegger M, et al. SE (3)-equivariant prediction of molecular wavefunctions and electronic densities[J]. Advances in Neural Information Processing Systems, 2021, 34: 14434-14447.

[3] Li H, Wang Z, Zou N, et al. Deep-learning density functional theory Hamiltonian for efficient ab initio electronic-structure calculation[J]. Nature Computational Science, 2022, 2(6): 367-377.

[4] Ozaki T, Kino H. Numerical atomic basis orbitals from H to Kr[J]. Physical Review B, 2004, 69(19): 195113.

Weakness 3:

Thanks for your feedback. We understand the importance of making source code available for reproducibility and transparency in research. We have prepared an anonymous link (https://anonymous.4open.science/r/nips-rebuttal-80C4/) for the review.

Question 1:

Thanks for your question. In Section A.9 of Supplementary Material, we discussed this difference in detail. We randomly selected a molecule, added a symmetrized (Hamiltonian matrices need to be symmetric) Gaussian noise matrix, and solved the corresponding generalized eigenvalue equation. As shown in Fig. 5, when the noise on the Hamiltonian gradually increased, the error range of orbital energy also increased. The orbital energy errors we reported for both QHNet and DEQHNet on the MD17 and QH9 datasets fall within the range shown in the figure, which suggests that the experimental results are reasonable. This observation implies that only when the Mean Absolute Error (MAE) of the Hamiltonian is sufficiently small can we ensure that the corresponding orbital energy deviation is also small. Additionally, we repeated the experiments on the MD17 dataset and found that when the Hamiltonian MAE of the model is at the same level, it is possible to obtain both a lower and a higher orbital energy MAE than the baseline. In the comparison between QHNet and PhiSNet, we observed similar situations. This somewhat suggests that using existing Hamiltonian error measurements cannot definitively reflect the errors in the eigenvalues. This is a nontrivial, separate research question that we plan to investigate in the future.

We will elaborate on this point more clearly in the revised version of our paper to provide a comprehensive understanding of the model's performance.

Question 2:

Thanks for your question. Indeed, our approach is designed to be highly adaptable and can be applied to off-the-shelf machine learning models. As stated in the introduction, our method introduces self-consistency to off-the-shelf models used for Hamiltonian prediction through the use of Deep Equilibrium (DEQ) models. To incorporate this into existing models, one simply needs to add a block that processes node features constructed from the Hamiltonian and overlap matrix inputs. This makes it relatively straightforward to combine any existing model with DEQ.

We believe this adaptability is one of the key strengths of our method and will further highlight this in our revised manuscript.

Thank you for sharing your further comment. Nevertheless, we are afraid that your comment is based on a factual misunderstanding. The fact is, DEQHNet outperforms QHNet in all cases, including ethanol and malonaldehyde, with the only exception of water due to limited available labels (500; other molecules have 25,000 labels) as we have explained in the paper. Such results indicate that DEQHNet improves upon existing end-to-end Hamiltonian prediction methods universally.

We are not sure about the specific reason why you have the misunderstanding, but we would like to stress that the primary comparing metric should be Hamiltonian MAE, which directly measures the prediction accuracy of the models. Other metrics, i.e., orbital energy and orbital coefficients, are derived quantities from the Hamiltonian. As we mentioned in the rebuttal, the prevailing MAE metric for Hamiltonian has been found not perfectly aligning with metrics of these derived quantities, which has also been observed in the comparison between QHNet and PhiSNet. In the appendix of our paper and in the rebuttal, we have explained why our method performs well in terms of Hamiltonian error and orbital coefficients. We also pointed out that the reported orbital energy falls within the error range of the current Hamiltonian. Specifically, in Section A.9 of the Supplementary Material and the Question 1 section of the rebuttal, we discussed this discrepancy in detail. We randomly selected a molecule, added a symmetrized Gaussian noise matrix (Hamiltonian matrices need to be symmetric), and solved the corresponding generalized eigenvalue equation. As shown in Fig. 5, as the noise on the Hamiltonian increased, the error range of the orbital energy also increased. The orbital energy errors we reported for both QHNet and DEQHNet on the MD17 and QH9 datasets fall within the range shown in the figure, suggesting that the experimental results are reasonable.

Additionally, we can provide further evidence from the QHNet paper, which observed a similar phenomenon. In their study, the PhiSNet achieved a Hamiltonian error of only 15.67x10$^{-6}$ Hartree on water, but the orbital energy error was "ten times worse than that by QHNet (33.76)", approximately 330x10$^{-6}$ Hartree.

This observation implies that only when the Mean Absolute Error (MAE) of the Hamiltonian is sufficiently small can we ensure that the corresponding orbital energy deviation is also small. In the comparison between QHNet and PhiSNet, we observed similar situations. This somewhat suggests that using existing Hamiltonian error measurements cannot definitively reflect the errors in the eigenvalues. These evidence suggests that it is a nontrivial and long-standing challenge in the domain of Hamiltonian prediction to design a proper metric on the matrix space so that reflects the metric of derived quantities e.g. energy, for which we will investigate in future work.

Finally, concerning your comment that "the presentation of the paper needs further improvement," we have already stated in our rebuttal that "We will also add further explanations about the integration of these two components and how this combination contributes to the overall model. We hope that these revisions will make our paper more comprehensible to readers with various backgrounds." The PDF we provided in the global rebuttal is merely an initial visualization, as the current NeurIPS review process only permits a one-page PDF that includes images and tables but no text.

We are immensely grateful for your assessments. We will address each point in our response accordingly.

Weakness 1:

Thanks for your feedback and suggestions. We provide a PDF document in the global rebuttal section, which includes a figure delineating the distinction between the off-the-shelf model used for Hamiltonian prediction and the DEQH model. We hope this visual aid will enable readers to gain a clearer and more intuitive understanding.

Furthermore, in the revised version, we will include additional diagrams and illustrations to better depict the workflow of the DEQH model and its components. This will include detailed flowcharts and schematic representations that highlight the key processes and interactions within the model. We are grateful for your insights, which have been instrumental in improving the clarity and quality of our work.

Weakness 2:

Thank you for your valuable feedback. We agree that adding more benchmark results would provide a more comprehensive evaluation of our DEQHNet model's performance. In response to your suggestion, we include additional benchmark results for SchNOrb [1], PhiSNet [2], and DeepH [3] on the MD17 dataset as follows.

We would like to note that due to the design of DeepH, the data used is re-labeled by OpenMX [4], which may lead to different results compared to other models. Furthermore, considering the high training cost of PhiSNet, we also provide PhiSNet results from the QHNet article for a more balanced and comprehensive comparison.

These results will be incorporated into the revised manuscript to provide a more robust and thorough evaluation of DEQHNet, thereby offering readers a clearer understanding of its performance relative to other state-of-the-art methods.

[1] Schütt K T, Gastegger M, Tkatchenko A, et al. Unifying machine learning and quantum chemistry with a deep neural network for molecular wavefunctions[J]. Nature communications, 2019, 10(1): 5024.

[2] Unke O, Bogojeski M, Gastegger M, et al. SE (3)-equivariant prediction of molecular wavefunctions and electronic densities[J]. Advances in Neural Information Processing Systems, 2021, 34: 14434-14447.

[3] Li H, Wang Z, Zou N, et al. Deep-learning density functional theory Hamiltonian for efficient ab initio electronic-structure calculation[J]. Nature Computational Science, 2022, 2(6): 367-377.

[4] Ozaki T, Kino H. Numerical atomic basis orbitals from H to Kr[J]. Physical Review B, 2004, 69(19): 195113.

Question 1:

Thanks for your question regarding the runtime and memory usage of DEQHNet compared to previous methods. As discussed in Section A.10 of our Supplementary Material, the training time of DEQHNet on the QH9 dataset is approximately 1.5 times that of QHNet. This is primarily due to the need for DEQ models to perform fixed-point iteration solving, which results in a higher runtime compared to models that do not incorporate a self-consistency mechanism.

Regarding memory usage, DEQHNet requires extra blocks to process node features constructed from the Hamiltonian and overlap matrix inputs. This results in a slightly larger memory footprint compared to previous network structures. In our experiments, DEQHNet only had a 28.5% increase in parameters compared to QHNet, demonstrating that it improved accuracy without a significant increase in parameters.

We are currently quantifying these requirements more precisely and will include a detailed comparison of memory usage in the revised manuscript.

Question 2:

Thanks for your insightful question. The theoretical foundations of our DEQH model are rooted in the common practice in Density Functional Theory (DFT) of iteratively solving the electronic Schrödinger equation. The process begins by initializing a Hamiltonian, solving the generalized eigenvalue problem, and then constructing a new Hamiltonian for the next iteration. We treat this iterative property of the Hamiltonian as a fixed-point iteration problem. This allows us to leverage the characteristics of DEQ in solving fixed-point problems to impose self-consistency on the network for Hamiltonian prediction.

The results from the QH9-stable-ood experiment demonstrate that our approach can significantly reduce the Mean Absolute Error (MAE) of the Hamiltonian, suggesting an enhanced capability for generalization. Additionally, the DFT Acceleration ratio presented in Appendix A.8 further reflects the generalizability of our model to a certain extent. We hypothesize that introducing DEQ enables the network to learn the iterative mechanism of solving the Hamiltonian, rather than directly learning the solution to the Hamiltonian. This shift could allow the network to better capture inherent physical laws, thereby improving generalizability.