HyperINF: Unleashing the HyperPower of the Schulz's Method for Data Influence Estimation

We thank the reviewer for the valuable feedback! We have updated the manuscript according to the suggestions, and provided further clarifications as follows:

W1: Clarification of the main contributions

The first contribution of this work is leveraging Generalized Fisher Information (GFIM, Equ. 4) in the data attribution problem, which yields a novel low-rank formulation of the data influence estimation function (Equ. 5). Secondly, we apply Schulz’s method (Equ. 6) to improve the accuracy of matrix inverse approximation. The mathematical formulas of Schulz’s method are from very old linear algebra literature, while it has not been used in large-scale neural network training or data attribution problems. By combining the two techniques, we can achieve improved efficiency and accuracy, which then transfer to better empirical performance. In comparison, the prior work (DataInf, [1]) only proposed to apply sherman-morrison formula upon previously proposed influence functions.

## W2: Convergence analysis on Schulz’s method We refer the reviewer to Appendix C, where we include a proof of convergence of Schulz’s method. Indeed, we agree with the reviewer that not all theoretical improvements can be demonstrated useful empirically. However, we show with comprehensive empirical results that our proposed method outperforms all the other baselines in various real-world data attribution tasks.

W3: Balance of Quality and Diversity

Firstly, we want to clarify that HyperINF is proposed to improve the accuracy and efficiency of the influence function computation on large-scale models. We demonstrate from comprehensive experiments that our method brings a large improvement in various real-world tasks. The discussion on the balance of quality and diversity may be beyond this scope. Also, our work mainly focuses on task-specific fine-tuning or instruction tuning instead of multi-task learning or general-purposed training. In that case, we claim the relevance to the target task and the quality of data points would indeed have a great impact on the downstream performance. Regarding the criticism of [2] and the results from [3], one potential explanation is the whole training data distribution is too far from the target task, which makes random selection a very strong baseline. Besides, [3] did not report the selection ratios in their paper, which makes their results less convincing.

We consider the study on the balance of data quality and diversity can be an intriguing topic, especially on multi-task learning. We will leave it for future work.

Q1: Convergence experiments among different methods and Leave-One-Group-Out error experiments

We conduct the correlation test with Leave-One-Group-Out (LOGO) retraining scores and present the results in Appendix D.2. To prevent the large computations incurred from retraining LLMs, we retrain the model by excluding each group of data (Leave-One-Group-Out) instead of each individual datapoint (LOO). the details can be found in Appendix D.2. The results demonstrate greatest correlation with the LOGO, which indicates its superior accuracy on data attribution.

Q2: Running time comparisons

We refer the reviewer to Appendix D.1, where we provide both the time costs on CPU and GPU across various LoRA ranks. We observe that the efficiency of three algorithms ranks largely differently between GPU and CPU. On CPU (Figure 6), DataInf introduces the least time overheads while HyperINF incurs the most amount of extra time costs. In addition, the time costs from DataInf and LiSSA increase quadratically with LoRA rank $r$ while HyperINF increases linearly. Alternatively, on one single GPU (Figure 7), the time costs from all algorithms are almost constant across LORA ranks, and HyperINF costs the least amount of time, followed by DataInf. In comparison, LISSA requires ~4x more time costs than HyperINF and DataInf. That difference demonstrates that HyperINF is more efficient and compatible with modern GPU computing.

We hope our answers resolve most of your concerns. If you have any further questions, don’t hesitate to contact us. Your valuable suggestions have helped us improve the manuscript!

[1] DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Models

[2] Less: Selecting influential data for targeted instruction tuning

[3] Rethinking Data Selection at Scale: Random Selection is Almost All You Need

We thank the reviewer for the valuable feedback! We have updated the manuscript according to the suggestions, and provided further clarifications as follows:

Q1: Empirical evidence justification for Lemma 1

We agree with the reviewers that our assumptions made for Lemma 3.1 are idealized and hard to verify in real large language model training. Instead, we want to demonstrate that in some conditions, the generalized fisher information matrix (GFIM) and fisher information matrix (FIM) are equal in expectation, which motivates our low-rank formulation of the information function (Equ. 5). We also thank the reviewer for the great references! We include a detailed discussion on the limitations of GFIM and FIM approximations in Appendix I. In practice, FIM and GFIM could have some differences (Figure 17), while it does not impact the empirical performance of our method according to the improvement from our comprehensive experiments. How to derive a more accurate low-rank approximation of Hessian matrices within tractable computations is an important and compelling research topic. We will leave it for future work.

Q2: Potential explanation for the strong performance by random baseline

We agree with the reviewer that the random sampling baseline demonstrates strong performance in some cases but we want to clarify that HyperINF could outperform it in most of the tasks. We provide potential explanations as follows: (1). Since all the influence-function-based methods could ignore the interactions between various data points, which could suffer from duplicated data points and lead to suboptimal performance from the lack of diversity. (2). The training dataset could already include relevant and high-quality data points. Therefore, the data selection may not bring much improvement above random sampling (e.g. Hellaswag).

Q3: Practical applications concerns for multi-task

We want to clarify that our work mainly focuses on task-specific fine-tuning or instruction tuning instead of multi-task learning or general-purposed training. Thus, we only conducted data selection experiments on LLM finetuning with one specific target domain, where the $D_{train}$ and $D_{val}$ are from the same distribution. The multi-task data attribution problem is a more complex setting, where each target could either overlap or conflict with other ones. We consider it as a very intriguing direction for data attribution and we will leave it to future work.

Q4: Evaluation datasets selection

We provide some insights from our data selection experiments. Firstly, we find the influence-function-based methods could bring more improvement on more challenging tasks (e.g. QASC, LogiQA) while having marginal improvements on easier ones (e.g. Hellaswag). Another insight is that the data attribution method would be less effective when all the training data points are very high-quality or the whole training dataset is far from the target task.

On the split of validation and training data, we randomly sample $D_{val}$ from the same distribution of $D_{train}$, without applying quality control. However, we expect using a well-curated high-quality subset as the validation set ($D_{val}$) could lead to better data attribution performance.

Q5: Strong performance on dense finetuning than LoRA-finetuning

We thank the reviewer for the detailed and interesting observation! We provide a potential explanation as follows: since we use the dense gradient to estimate the influence function, we expect HyperINF could yield more accurate scores, while other baseline methods could suffer from large approximation errors. Thus, it will select more relevant and high-quality datapoints for finetuning, which makes the difference more distinguishable, especially when selecting a small portion of data points (5%).

We hope our answers resolve most of your concerns. If you have any further questions, don’t hesitate to contact us. Your valuable suggestions have helped us improve the manuscript!

Dear reviewer 7qzk,

Thank you again for your efforts reviewing our paper and providing valuable feedback! Since tomorrow is the last day of the author-reviewer discussion phase, we hope you can take some time to look into our rebuttals and additional results. Let us know if you have other concerns and questions!

Sincerely,

the authors

We thank the reviewer for the valuable feedback! We have updated the manuscript according to the suggestions, and provided further clarifications as follows:

W1: Choice of objective function

We agree with the reviewer that the GFIM and FIM approximation of the Hessian matrix is only applicable when the loss function is a log-likelihood function, which is the mainstream objective function for auto-regressive model training. Given that a large portion of foundation models are trained in an auto-regressive way, our method can be widely applied to a wide range of tasks and models.

W2&Q1: Theoretical convergence analysis on Schulz’s method

We thank the reviewer for the suggestion on a more rigorous theoretical analysis of our method! We provide a detailed convergence analysis of Schulz’s method in Appendix C, where we show the approximation error decreases quadratically.

W3: Analysis of Computing Time Costs

We thank the reviewer for this intriguing observation! We have also discovered this discrepancy between theoretical complexity and real GPU hours, and we are happy that we may have found the answer. According to Appendix D.1, we provide both the time costs on CPU and GPU across various LoRA ranks. We observe that the efficiency of three algorithms ranks largely differently between GPU and CPU. On CPU (Figure 6), DataInf introduces the least time overheads while HyperINF incurs the most amount of extra time costs. In addition, the time costs from DataInf and LiSSA increase quadratically with LoRA rank $r$ while HyperINF increases linearly. Alternatively, on one single GPU (Figure 7), the time costs from all algorithms are almost constant across LORA ranks, and HyperINF costs the least amount of time, followed by DataInf. In comparison, LISSA requires ~4x more time costs than HyperINF and DataInf. That difference demonstrates that HyperINF is more efficient and compatible with modern GPU computing.

Q2: Convergence test among different methods

For the evaluation on convergence ability and speed, we refer the reviewer to Section 4 and Figure 1, where we compare the approximation error of the inverse Hessian-vector product from three different algorithms. We further conducted a complete study with matrices drawn from different distributions in Appendix H. The results demonstrate that HyperINF, based on Schulz’s method, consistently shows superior stability and convergence performance, which is robust with various data distributions. In comparison, the other methods can only converge in a few cases, indicating their sensitivity to different data distributions and the scale of matrices.

W4: Marginal improvement than DataInf

The approximation error of DataInf algorithms is $O(d^2)$. In LoRA finetuning (Table 2&4), the number of tunable parameters per layer is relatively small, whereas DataInf could yield a small approximation error which has less impact on the rank of data samples. In the dense finetuning case (Table 3), HyperINF could have a larger improvement above DataInf when we have a larger amount of tunable parameters.

W6: Missed negative sign in Equ. 3

In our settings, we consider the loss function $l(\cdot)$ to be a negative log-likelihood function, rather than a log-likelihood function. Therefore, we believe Equ. 3 should be valid without a negative sign.

W7: Color-blind plot

We thank the reviewer for the considerate suggestion! However, it is a bit hard to make Figure 2 color-blind-friendly. In that case, we have included a table (Table 5) with detailed accuracies in Appendix D.

We hope our answers resolve most of your concerns. If you have any further questions, don’t hesitate to contact us. Your valuable suggestions have helped us improve the manuscript!

Dear reviewer GxRJ,

Thank you again for your efforts reviewing our paper and providing valuable feedback! Since tomorrow is the last day of the author-reviewer discussion phase, we hope you can take some time to look into our rebuttals and additional results. Let us know if you have other concerns and questions!

Sincerely,

the authors

We thank the reviewer for the valuable feedback! We have updated the manuscript according to the suggestions, and provided further clarifications as follows:

W1: Clarification of the main contributions

The first contribution of this work is leveraging Generalized Fisher Information (GFIM, Equ. 4) in the data attribution problem, which yields a novel low-rank formulation of the data influence estimation function (Equ. 5). Secondly, we apply Schulz’s method (Equ. 6) to improve the accuracy of matrix inverse approximation. The mathematical formulas of Schulz’s method are from very old linear algebra literature, while it has not been used in large-scale neural network training or data attribution problems. By combining the two techniques, we can achieve improved efficiency and accuracy, which then transfer to better empirical performance. In comparison, the prior work (DataInf, [1]) only proposed to apply Sherman-Morrison formula to previously proposed influence functions. We consider our contribution is not incremental compared to DataInf.

W2: Fair comparisons among different methods on convergence tests

We thank the reviewer for the good advice! We updated Figure 1 with the comparison of the inverse Hessian-vector product $H^{-1}v$ from all three methods. We also conducted a complete study with matrices drawn from different distributions in Appendix H. The results demonstrate that HyperINF, based on Schulz’s method, consistently shows superior stability and convergence performance, which is robust with various data distributions. In comparison, the other methods can only converge in a few cases, indicating their sensitivity to different data distributions and the scale of matrices.

W3: Choices of LORA ranks and number of training epochs

For LLM finetuning experiments, we use LORA rank $r$=$64$ since the variance in data quality could be more distinguishable if the model has more finetuning capacity. In the mislabeled data detection task, we have also presented the results with LORA rank $r$=$8$ and $64$ in Appendix D. We refer the reviewer to Figure 3, 4, 5 for details.

We train the model on the full dataset for 1 epoch since it costs the same amount of FLOPs as 5 epochs on 20% selected data points. We also clarify this in Line 398-399 that, with 5% datapoints, HyperINF can achieve a comparable performance as the full dataset with 20$\times$ less tokens and $4\times$ less FLOPs.

W4: Computational complexity of DataINF

Since Equ. 23 and 24 are from one of the baseline methods [1], we refer the reviewer to the detailed discussion in DataINF [1].

We hope our answers resolve most of your concerns. If you have any further questions, don’t hesitate to contact us. Your valuable suggestions have helped us improve the manuscript!

[1] DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Models

Dear reviewer v6An,

Thank you again for your efforts reviewing our paper and providing valuable feedback! Since tomorrow is the last day of the author-reviewer discussion phase, we hope you can take some time to look into our rebuttals and additional results. Let us know if you have other concerns and questions!

Sincerely,

the authors

We thank the reviewer for the valuable feedbacks! We provide the further clarifications as follows:

W1: Clarification on Lemma 3.1

We indeed leverage the GFIM (i.e. MatFIM) following the same idea as Yang et al. [1], which makes our Lemma 3.1 similar to Lemma 1 in [1]. However, we further extend GFIM on large-scale data attribution and yield a low-rank influence function formulation (Equ. 5) in matrix form. We agree with the reviewers that our assumptions made for Lemma 3.1 are idealized and hard to verify in real large language model training. Instead, we want to demonstrate that in some conditions, the generalized fisher information matrix (GFIM) and fisher information matrix (FIM) are equal in expectation, which motivates our low-rank formulation of the information function (Equ. 5). We also thank the reviewer for the great references! We include a detailed discussion on the limitations of GFIM and FIM approximations in Appendix I.

W2: Fair comparisons between LiSSA and others

We indeed adopt the implementation of LiSSA from Koh and Liang (2017), could the reviewer clarify how to make the comparison more fair? For a fair comparison and complete study on the convergence test, we include the experiment results with matrices drawn from different distributions in Appendix H. The results demonstrate that HyperINF, based on Schulz’s method, consistently shows superior stability and convergence performance, which is robust with various data distributions. In comparison, the other methods can only converge in a few cases, indicating their sensitivity to different data distributions and the scale of matrices.

W3: Clarification on our main contributions

We thank the reviewer for recognizing the empirical improvement by applying our proposed method! We hereby want to clarify the main contribution of our work as two points: the first contribution of this work is leveraging Generalized Fisher Information (GFIM, Equ. 4) in the data attribution problem, which yields a novel low-rank formulation of the data influence estimation function (Equ. 5). Secondly, we apply Schulz’s method (Equ. 6) to improve the accuracy of matrix inverse approximation. While the mathematical formulas of Schulz’s method are from very old linear algebra literature, it has not been used in large-scale neural network training or data attribution problems. By combining the two techniques, we can achieve improved efficiency and accuracy, which then transfer to better empirical performance. In comparison, the prior work (DataInf [1]) only proposed to apply sherman-morrison formula upon previously proposed influence functions.

Q1: Complexity of using LiSSA and Schulz’s method

We compare the error on the inverse Hessian-Vector Product $H^{-1}v$ since LiSSA is not used to approximate $H^{-1}$ but directly estimate the inverse Hessian-Vector Product $H^{-1}v$. For the complexity comparison, we refer the reviewer to Table 1 for more details.

We hope our answers resolve most of your concerns. If you have any further questions, don’t hesitate to contact us. Your valuable suggestions have helped us improve the manuscript!

[1] An Efficient Fisher Matrix Approximation Method for Large-Scale Neural Network Optimization

[2] DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Models

Thank you for your response.

My point is that LiSSA is guaranteed to converge in theory. The issue that LiSSA diverges in Figure 1 originates from Koh and Liang (2017), whose interpretation of LiSSA differs from its original proposal by Agarwal et al. (2017). Suppose a Hessian matrix $H$ is invertible, what is refered to as LiSSA by Koh and Liang (2017) is merely the use of $H\^{-1} = \sum\_{t=0}\^{\infty} (I - H)\^{t}$, an equality that holds if and only if $\\|I - H\\|\_{2} < 1$. However, the version of LiSSA implemented in this submission, following (Koh & Liang, 2017), does not account for this condition, leading to its divergence, as shown in Figure 1.

Isn't it unfair to say that a method does not work while it is not correctly implemented? One correct implementation for LiSSA is: let $A = H / (\\|H\\|\_{F}+1)$, initialize $H\^{-1}\_{0} = I$ and proceed with $H\^{-1}_{t+1} = I + (I-A)H\^{-1}\_{t}$. Then, $H\^{-1}\_t / (\\|H\\|\_{F}+1)$ converges to $H\^{-1}$. In other words, $H$ should be scaled so that LiSSA converges, and we can retrieve the result at any time by re-scaling.

Besides, the purpose of the convergence test is confusing. In theory, the Schulz’s method and LiSSA are guaranteed to converge, and datainf is clearly not designed to converge. So, why bother to compare a converging method, such as Schulz’s method, to a non-converging method, like datainf? why not implement LiSSA correctly and include other converging methods, such as conjugate gradient that has also been mentioned by Koh and Liang (2017)？Whether or not a method converges can already be determined in theory.

---

Agarwal, N., Bullins, B., & Hazan, E. (2017). Second-order stochastic optimization for machine learning in linear time. Journal of Machine Learning Research, 18(116), 1-40.

Koh, P. W., & Liang, P. (2017). Understanding black-box predictions via influence functions. In International conference on machine learning (pp. 1885-1894). PMLR.

Dear Reviewer mprn,

Thank you again for your efforts reviewing our paper and your engagement in discussions! Since the rebuttal phase comes to the end, we hope our rebuttals and additional results have resolved your concerns. We will appreciate it if you can consider to adjust your score if our responses are satisfying. Thanks!

Sincerely,

the authors

We thank the reviewer for the feedback and suggestions. We provide further clarifications as follows:

## W1&W2: Empirical evidence justification for Lemma 3.1 and FIM approximation We agree with the reviewers that our assumptions made for Lemma 3.1 are idealized and hard to verify in real large language model training. Instead, we want to demonstrate that in some conditions, the generalized fisher information matrix (GFIM) and fisher information matrix (FIM) are equal in expectation, which motivates our low-rank formulation of the information function (Equ. 5). We also thank the reviewer for the great reference! As the reviewer suggested, we provide a detailed discussion on the limitations of GFIM and FIM approximations in Appendix I.

Q1: Performance comparison with EK-FAC

We have added a new comparison to EKFAC for LLM finetuning data selection. Since it only supports the influence computation on FFN layers, we only apply it to dense finetuning experiments. Specifically, we use all the gradients on Linear modules in the last layer of Llama-2-7B to compute the influence score by EKFAC algorithm. We present the results on QASC dataset as follows, where HyperINF outperforms EKFAC by a large margin. We promise to include the complete results on all datasets in the updated version.

We hope our answers resolve most of your concerns. If you have any further questions, don’t hesitate to contact us. Your valuable suggestions have helped us improve the manuscript!

Dear reviewer h5zQ,

Thank you again for your efforts reviewing our paper and your engagement in discussions! Since the rebuttal phase is towards the end, we hope our rebuttals and additional results have resolved your concerns. We will appreciate it if you can consider to adjust your score accordingly. Thanks!

Sincerely,

the authors

We thank the reviewer for the thoughtful feedback and valuable suggestions! We provide the clarifications as follows:

W1&Q4: Clarification on the main contributions

The first contribution of this work is leveraging Generalized Fisher Information (GFIM, Equ. 4) in the data attribution problem, which yields a novel low-rank formulation of the data influence estimation function (Equ. 5). Secondly, we apply Schulz’s method (Equ. 6) to improve the accuracy of matrix inverse approximation. The mathematical formulas of Schulz’s method are from very old linear algebra literature, while it has not been used in large-scale neural network training or data attribution problems. By combining the two techniques, we can achieve improved efficiency and accuracy, which then transfer to better empirical performance. In comparison, the prior work (DataInf, [1]) only proposed to apply the Sherman-Morrison formula to the previously proposed influence function. We consider our contribution is not incremental compared to DataInf.

Q1: Limited improvement over DataInf

We agree with the reviewer that HyperINF could perform comparably as DataINF with LORA finetuning on datasets with little noise. However, it shows a large improvement over DataINF with dense finetuning (Table 3). We provide a potential explanation based on the error analysis: The approximation error of DataInf algorithms is $O(d^2)$ according to [1]. With LoRA fine-tuning, the number of tunable parameters per layer is relatively small, whereas DataInf could yield a small approximation error which has less impact on the rank of data samples. In the dense finetuning case (Table 3), HyperINF could have a larger improvement above DataInf when we have a larger amount of tunable parameters.

Q2: Worse performance with dense finetuning than low-rank finetuning

According to the original LoRA paper (Table 2 in [2]), using LoRA finetuning could possibly lead to better performance than dense finetuning (FT).

Q3: Different distributions of matrix used in convergence test

We agree with the reviewers that difference algorithms could show various convergence performances on matrices drawn from different distributions. We have updated the paper to include more holistic convergence test results in Appendix H with five different matrix distributions. We also include the results from Neumann series and Successive Over Relaxation (SOR), while HyperINF consistently converges to a desirable error, while other methods could be sensitive to the distributions and scale of the targeted matrices.

We hope our answers resolve most of your concerns. If you have any further questions, don’t hesitate to contact us. Your valuable suggestions have helped us improve the manuscript!

[1] DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Models

[2] LoRA: Low-Rank Adaptation of Large Language Models

Dear reviewer NbyJ,

Thank you again for your efforts reviewing our paper and providing valuable feedback! Since tomorrow is the last day of the author-reviewer discussion phase, we hope you can take some time to look into our rebuttals and additional results. Let us know if you have other concerns and questions!

Sincerely,

the authors

W1&W2&Q1&Q2: Clarification on the main contributions

We thank the reviewer for recognizing the importance of our target problem! The main problem we want to solve is the hessian-inverse approximation with satisfying accuracy and efficiency. We did follow the influence function formulation for data attribution while incorporating Generalized Fisher Information instead of FIM in previous works.

The first contribution of this work is leveraging Generalized Fisher Information (GFIM, Equ. 4) in the data attribution problem, which yields a novel low-rank formulation of the data influence estimation function (Equ. 5). Secondly, we apply Schulz’s method (Equ. 6) to improve the accuracy of matrix inverse approximation. The mathematical formulas of Schulz’s method are from very old linear algebra literature, while it has not been used in large-scale neural network training or data attribution problems. By combining the two techniques, we can achieve improved efficiency and accuracy, which then transfer to better empirical performance. In comparison, the prior work (DataInf, [1]) only proposed to apply Sherman-Morrison formula to the previously proposed influence functions. We consider our contribution is not incremental compared to DataInf. We have updated the paper to include a list of contribution in Line 81 - Line 93.

Q3: Ablation on the distribution of matrices used in convergence test

&Q4: Comparisons to other matrix inversion techniques

We agree with the reviewers that difference algorithms could show various convergence performances on matrices drawn from different distributions. We also have updated the paper to include more holistic convergence test results in Appendix H with five different matrix distributions. We also include the results from Neumann series and Successive Over Relaxation (SOR), while HyperINF consistently converges to a desirable error, while other methods could be sensitive to the data distributions and the scale of the targeted matrices.

[1] DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Models