HyperINF: Unleashing the HyperPower of Schulz's Method for Data Influence Estimation

We thank the reviewer for their valuable feedback and constructive suggestions! We address each concern below:

Q1: Comparison between DataInf and HyperINF

According to Table 1, we acknowledge that HyperINF incurs higher complexity in dense finetuning compared to DataINF, while it can lead to better efficiency with LoRA finetuning with a large LoRA rank $r$. We also note that DataINF can lead to large approximation error according to Fig. 1, which leads to suboptimal empirical results on LLM training (Fig. 2, Table 2, Table 3, Table 4).

Nevertheless, we acknowledge that DataINF is an efficient strategy for large-scale dense finetuning, instead of other iterative approximation algorithms. We will add it into discussion in the updated version.

Q2: Model size scalability

Due to resource constraints, our current experiments are only conducted on 7B models. Yet, the theoretical and complexity analysis can be generalized to any scales of models: (1) Schulz’s method converges independently of dimension (Appendix D); (2) GFIM approximation improves efficiency for LoRA finetuning on larger models (e.g., 13B).

We agree with the reviewer that adding the empirical results across various model scales can strengthen the current paper. We will add the experimental results on 1B LLMs in the updated version of the manuscript.

Q3: Task-dependent Data Selection Impact

We thank the reviewer for bringing up the interesting discussion!

The improvements from data selection methods vary across different tasks (Table 3, Table 4), probably resulting from various data quality in the training datasets. For example, on Hellaswag and LogiQA, all the data selection methods can only achieve comparable or marginally improved performance compared to random selection, perhaps because the data quality within its training set is already quite high. Also, the sample-level data selection tends to extract similar high-quality datapoints, which might risk duplications or even overfitting if we repeat the selected subset for many epochs.

We will add the discussion in the updated version of the manuscript.

Q4: Minor comments

1. We will correct the title to "Data Selection for VLM Instruction-Tuning".
2. We thank the reviewer for the pointer! We will add the work in the related work section to acknowledge methodological overlaps.
3. We will fix the redundant citations to Grosse et al. (2023) in the updated version.

Q1: Theoretical Limitations on Real-world Settings

We thank the reviewer for this insightful observation about the i.i.d. zero-mean gradient assumptions in our GFIM-based approach. We acknowledge that these are indeed strong theoretical assumptions that merit careful discussion.

• Theoretical Justification: The zero-mean gradient assumption becomes increasingly valid as models approach convergence during training, where the expected gradient approaches zero. For the i.i.d. assumption, while individual gradients may exhibit some correlation, our empirical analysis in Figures 19-20 (Appendix J.3) demonstrates that gradient matrices maintain near-full column rank across diverse settings, suggesting sufficient linear independence for our method to be effective.

• Empirical Robustness: Importantly, our extensive experiments across LLM/VLM fine-tuning and mislabeled data detection demonstrate that HyperINF performs robustly even when these assumptions are violated in practice. This suggests that while the theoretical analysis relies on these assumptions, the practical performance is more forgiving to assumption violations. When Assumptions Hold Better: The assumptions are more likely to be satisfied in: (1) later stages of training when models are near convergence, (2) well-regularized models, and (3) scenarios with sufficient data diversity. We observe that HyperINF performs particularly well in these settings.

• Comparison to Alternatives: We note that existing influence function methods also rely on similar or stronger assumptions (e.g., convexity assumptions in classical IF methods), and our empirical results show significant improvements over these baselines even in challenging real-world scenarios.

Q2: Adapting to the black-box (API-based) LLMs

Since our method is built on the traditional influence function formulation [1], which cannot be directly applied to black-box APIs since they require gradient computation with respect to internal model parameters that are inaccessible.

However, we appreciate this important practical question given the prevalence of API-based LLMs. Some promising direction could be Gradient-free approximations with zero-order approximation etc. We will leave it to future research.

[1] Understanding Black-box Predictions via Influence Functions. Koh and Liang, 2017.

We thank the reviewer for their valuable feedback! We provide the sample analysis as below:

Q1: Direct results demonstration

We thank and agree with the reviewer's opinions for providing some 'highly influential' examples. Therefore, for the Mislabeled Data Detection task, we provide the top-ranked data point (i.e., high-quality data, where its label is correct ) and bottom-ranked data point (i.e., low-quality data, where its label is corrupted to be incorrect) from HyperINF on MRPC task.

• Top-ranked: {'sentence1': 'Microsoft , which acquired Virtual PC from Connectix in February , said a fix for the problem is not around the corner .', 'sentence2': 'Virtual PC , which Microsoft acquired from Connectix Corp. in February , will not run on the G5 , the company said .', 'label': 0}, label 0 denotes the these two sentences are not semantic equivalent, whose label is correct, not corrupted.

• Bottom-ranked: {'sentence1': '" We acted because we saw the existing evidence in a new light , through the prism of ... Sept . 11th . "', 'sentence2': '" We acted because we saw the evidence in a dramatic new light , through the prism of our experience on 9 / 11 . "', 'label': 0}. The label should be 1, since these two sentences are semantic equivalent but the label is corrupted to 0. And HyperINF succeeds in detecting the wrong-labeled data point.

We will provide comprehensive sample analysis example in our revised version.

We thank the reviewer for the feedback and insightful questions! We provide corresponding clarifications as follows:

Q1: Comparison between DataINF and HyperINF

We not only care about the efficiency but also emphasize the importance of the accuracy of Hessian inverse approximation. We agree with the reviewer that DataINF has the best theoretical efficiency compared to other methods, including HyperINF under the dense-finetuning context. However, we refer the reviewer to Fig. 1, which demonstrates that DataINF and Lissa can both lead to large approximation errors when the scales of tunable parameters and datasets are increased. In comparison, HyperINF exhibits a better convergence ability. For efficiency considerations, we note that with LoRA fine-tuning (i.e., the main scenario considered in DataINF), HyperINF can improve the theoretical complexity according to the LoRA rank $r$ (Table 1). Additionally, HyperINF adopts matrix multiplications, which further improve empirical efficiency on modern GPUs (Fig. 7).

Q2: Confidence intervals for large experiments

Since Table 3 and Table 4 present the results on Llama-7B and VLM (CLIP+Llama-7B) tuning, thus, the large computation costs prevent us from repeating these experiments several times. However, we have provided the confidence intervals for all smaller experiments on Roberta-large (Table 2), which demonstrates a concrete understanding of the benefit from HyperINF.

Q3: Implementation on different LoRA ranks

We agree with the reviewer that more complete experiments with all LoRA ranks could bring better understanding. However, because of the computation limits, we choose the largest rank $r=64$, which may lead to large accuracy improvements (according to [1]). Therefore, we can observe more differences in the downstream performance. To study the impact of LoRA ranks, we provide the results for $r=8,16,64$ on smaller-scale experiments with Roberta-large and GLUE benchmark (i.e. the settings of the original LoRA paper [2]). We refer the reviewer to Figure 2,3,5 for more details.

Q4: Clarification on dense finetuning

For the dense-finetuning experiments, we select data using the gradients from the last layer, then tune all the model parameters on the selected datapoints. We only use the gradients from the last layer as they are the most impactful ones according to [3].

We hope our answers resolve most of your concerns. If you have any further questions, don’t hesitate to contact us. Your valuable suggestions have helped us improve the manuscript!

[1] A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA

[2] LoRA: Low-Rank Adaptation of Large Language Models

[3] ShortGPT: Layers in Large Language Models are More Redundant Than You Expect