Taming Hyperparameter Sensitivity in Data Attribution: Practical Selection Without Costly Retraining

We thank the reviewer for the comments and questions. We address your individual concerns below.

Are there any findings or insights that are more surprising

We would like to highlight that our paper has several novel/surprising findings with practical implications. These findings are discussed in detail in Section 2 and 3. For instance, we demonstrate that parameters like training epoch have NOT been considered as hyperparameters in prior literature, but they significantly affect the performance of a wide range of data attribution methods. As another example, by studying the entanglement of hyperparameters, we have provided a compelling explanation for an open question raised in the TRAK paper: the counterintuitive LDS drop with respect to increasing projection dimension can be counteracted by introducing a proper regularization. In short, our work extends beyond the observation that hyperparameter selection is highly case-specific, which is true, but also shed light on concrete behaviors of attribution performance when hyperparameters are tuned with evidence from the first systematic research in this direction.

or could any of the results be suggested as actionable steps for practitioners

We kindly refer the reviewer to the end of Section 1 and Section 5 for some discussion on the implications of our results. Specifically, (1) we highlighted that hyperparameter tuning is a unique challenge in data attribution given the high computational cost associated with the evaluation metrics—to the best of our knowledge, ours is the first study to identify this unique challenge. (2) We pointed out that future methodological development of data attribution should have a more systematic treatment on hyperparameter selection. (3) Our proposed method in Section 4 provides both a practical method for selecting the regularization term in many popular data attribution methods (including IF, TRAK, LoGra), and an example for analyzing the hyperparameters.

In addition, some of our novel findings in Section 3 have straightforward implications for practitioners. For example, when using random projection in gradient-based methods, one should carefully tune the regularization term (possibly with our proposed method).

In our next revision, we will include a dedicated paragraph to discuss these implications in one place.

it focuses on a single parameter for one method, and I am not convinced of its generalizability to other datasets

We would like to clarify that our surrogate indicator algorithm applies to a wide range of settings instead of one method or a limited number of datasets. As we discuss in Section 2 and Appendix, state-of-the-art data attribution methods that involve inverting FIM (or GGN) and introduce random projection can be categorized as a variant of our proposed IFFIM, which include but are not limited to TRAK and LoGra. Our discussion of IFFIM can be easily generalized to these methods. Further, following the reviewer’s suggestion, we include an additional larger experimental setting on text data (GPT-2 on WikiText-2), which supports the effectiveness of our algorithm to select a good regularization coefficient. The results are summarized in the table below.

We highlight that our study now covers experiments on three different modalities (image, text, and music), as well as three types of data attribution methods (IFFIM, TRAK, and LoGra). We are actively expanding our experimental coverage and will include additional results in the revised version to further support generalizability.

We appreciate the positive evaluation as well as the constructive feedback given by the reviewer. We address each point in Weaknesses and Questions below.

The proposed surrogate indicator is only applicable to the λ parameter in the influence function method, and cannot be extended to other key hyperparameters (such as projection dimension) or other types of attribution methods, with obvious limitations. In addition to λ, the projection dimension and the number of training rounds have also been shown to have a significant impact on performance. Why didn't you try to estimate them using a surrogate-like approach? Is it because of mathematical difficulty, non-differentiability, or other practical obstacles?

We would like to clarify that the regularization strength is one of the most influential and sensitive hyperparameters according to our empirical study, which justifies our special attention to it. Regularization strength is also applicable to a wide range of state-of-the-art attribution methods, including but not limited to TRAK and LoGra (as discussed in Section 2 and Appendix). Additionally, we would like to point out some practical difficulties when applying similar surrogate indicator analysis on other hyperparameters such as projection dimension and training epoch. While we include some discussion of gradient projection and its influence on the effectiveness of the surrogate indicator in Section 4, projection dimension is difficult to analyze directly, as its impact is highly dependent on the choice of random projection matrices. Similarly, to predict the effect of tuning training epochs, we have to carefully consider a variety of external factors such as optimization methods and learning rate, which complicates theoretical modeling and makes surrogate-based analysis less tractable. Furthermore, theoretical understanding of other hyperparameters are indeed interesting topics, but we would like to bring up another significance of our work: it includes the first systematic large-scale study highlighting the problem of hyperparameter selection in data attribution, which provides important future research directions.

The experimental indicators are single, and many results are based on the synthetic indicator LDS. End-to-end verification has not been performed in actual tasks (such as noise detection and training data screening), and it cannot be proved that the selected hyperparameters can bring better downstream performance. Have you considered other performance indicators? All current experiments use LDS as the only metric. Have you considered using downstream performance as an evaluation metric in data denoising or data selection tasks? If it cannot be extended to practical tasks, the practicality of this paper will be questioned.

Following the reviewer’s suggestion, we include an evaluation of our proposed algorithm on downstream settings. We focus on the task of data selection, where we measure the performance drop of models when the most positively influential training data are removed (more drop indicates better attribution performance). The results are shown in the table below. We observe that our proposed algorithm generally achieves visible model performance drops, justifying the effectiveness of the algorithm in downstream tasks.

The proposed indicators are based on complex partial derivative conditions and high-order tensor expressions, which lack interpretability for non-theoretical researchers and have a high threshold for actual adoption.

While we recognize the reviewer’s concern regarding interpretability, we have taken care to design our algorithm with a clear conceptual intuition and practical implementation. In brief, the algorithm is simply seeking a range of regularization where the surrogate indicator is close to neither of its bounds (0 and 1) when averaged over the test set. To further facilitate understanding, we also provide an intuition of the surrogate indicator here. For simplicity, we let $v$ denote the gradient of $f$ at $z’$ and $\theta^\ast_S$. By observing the definition of terms $t_{k, z’, \lambda}$ where $k = 1, 2, 3$, the surrogate indicator can be viewed as the cosine “angle” between $v$ and $(F_S+\lambda I_p)^{-1} v$. Here, $(F_S+\lambda I_p)^{-1} v$is related to the $\theta$ update direction in IFFIM, as in Equation 2. When the surrogate indicator is close to 0, this “angle” is overly large, suggesting a significant misalignment between the $\theta$ update direction and the gradient descent direction $v$. This indicates that the update direction is not reliable. In this case, increasing $\lambda$ helps intuitively because it helps condition the inverted matrix $F_S$ better. On the other hand, when the surrogate indicator is close to 1, $F_S$ is overshadowed by the regularization term $\lambda I_p$, meaning that the attribution method may become dominated by the regularization and lose sensitivity to the underlying data structure.

Are theoretical indicators generalizable? The derivation of the surrogate is based on a series of technical assumptions. Do these assumptions hold true in large-scale deep models or strong non-convex situations? It is recommended to provide further empirical support or hypothesis sensitivity analysis.

We mainly focus on Assumption B.6 and Assumption B.8. We would like to state that the constants $C_1$, $C_2$, and $C_3$ generally stay at the 1e0 level instead of scaling visibly with the smallest eigenvalue. Due to computational limitations, currently we only conduct the test in the CIFAR2+ResNet9 setting, which already contains a certain level of non-convexity. We will add similar experiments and analysis on other larger settings. Note that we compute $C_1$ with respect to the smallest eigenvalue (instead of all eigenvalues), which is reasonable because only the $C_1$ term related to the smallest eigenvalue occurs in our theoretical derivation. Our results are that $C_3 = 0.4082$ is enough for all test points with $\varepsilon_g = 0.5$. Further, 90% of the test points yield $C_1 = 0.3121$ and $C_2 = 51.9933$. In comparison, the smallest eigenvalue is only about 1e-4.

The author identified the pain points of hyperparameter tuning, but the solution he finally gave is more like an "ad hoc fix" rather than a systematic solution to the problem.

We would like to highlight that the surrogate indicator and the related algorithm we gave is both theoretically interpretable and practically useful. As hyperparameter selection in data attribution is different from ordinary hyperparameter selection in machine learning, our solution has a more general implication that could inspire further research in this direction, and we believe that the theoretical analysis that supports our solution could be generalized to solve similar problems in future works.

Is there a risk of "misleading selection"? The author's method attempts to fit the LDS trend with the surrogate curve, but is there a situation where the surrogate indicator fails to be monotonically in some areas, resulting in the misselection of suboptimal λ? Have the authors considered uncertainty estimates or confidence bounds to aid in the selection?

We acknowledge that we do not take uncertainty into account when formulating the surrogate indicator and the algorithm, but we would like to state that the uncertainty is minimal and unlikely to affect the reliability of the selected hyperparameters in practice. To justify this, we re-run the experiment in CIFAR2+ResNet9 setting 50 times with different training seeds for the ensemble model. The mean and standard error (SE) of LDS are shown in the table below, which demonstrates that the error is generally negligible compared with the mean. Additionally, we also observe that in 49 among these 50 trials, our algorithm selects the same regularization term. We also notice that the surrogate indicator curves across all 50 trials are monotonic.

We appreciate the insightful comments by the reviewer.

Section 4.1 suffers from poor presentation and organization. The notation is confusingly introduced in separate remark paragraphs after the main theorem statement, making it difficult to parse the theoretical contributions. This section would benefit from significant revision to include: (1) clear interpretation of the theorem statement, (2) intuitive explanations of the key insights, and (3) a proof sketch for Theorem 4.3 (and other theoretical results if space allows).

We acknowledge that Section 4.1 introduces a dense set of new notations and theoretical results within a compact space. Following the suggestion of the reviewer, we will rewrite Section 4.1 in mainly the following aspects. First, we will move the introduction of key notations (e.g. $t_{k, z’, \lambda}$) to precede Theorem 4.3, and provide intuitive explanations for each. In particular, we will interpret $\alpha_{z’, i}$ and $g_{z’}$ by relating them to the leave-one-out (LOO) interpretation of data attribution, and clarify other terms like $t$, $r$, and $o$ by expressing them as inner products of more elementary components. Then, we will adjust Theorem 4.3 and add an interpretation of the result that is grounded in the meanings of the key terms from Equation 8.Finally, we will summarize the core ideas and key proof steps in a clear proof sketch below Theorem 4.3, incorporating relevant intermediate results from the Appendix. We also consider splitting Section 4.1 into smaller subsections for better readability. These modifications will be included in the final version of our paper.

While the proposed heuristic in Section 4.2 demonstrates effectiveness in the experiments of Section 4.3, there are concerns about its generalizability. The fixed threshold of 0.5 may work well primarily because the experimental evaluation relies on relatively clean datasets (MNIST and CIFAR-10). I suspect this threshold would require significant adjustment for different types of datasets or domains. This concern is supported by the notably worse performance on the MAESTRO dataset, suggesting that the method's robustness across diverse data characteristics remains questionable.

We include an additional larger experimental setting on text data (GPT-2 on WikiText-2), which validates the effectiveness of our algorithm to select a good regularization coefficient. The results are summarized in the table below.

We highlight that our study now covers experiments on three different modalities (image, text, and music), as well as three types of data attribution methods (IFFIM, TRAK, and LoGra). These experiments support that a fixed threshold 0.5 enables our algorithm to reliably select effective regularization strengths across diverse settings.

There is also a hyperparameter in the metric of LDS, the subset size . I wonder whether varying will also significantly change the results?

We included the discussion of the subset size in LDS with empirical evidence in Appendix C.3. As a high level summary, we observed that while the magnitude of LDS could vary with the subset size, the overall monotonicity and trend as a function of $\lambda$ remains similar. This observation suggests that our method remains robust across different subset sizes.

We appreciate the positive evaluation as well as the constructive suggestions given by the reviewer.

The proposed theoretical strategy, while appealing in principle, is less convincing in the case study. The theoretical derivation involves several assumptions (e.g., the LDS being uni-modal and concave in , technical assumption in Appendix B.3) and heuristics (e.g., in lines 263–264) that may not hold broadly in practice. The empirical results in Figure 2 show that the hyperparameter values determined by the proposed algorithm are not always close to the optimal value.

Rather than aiming to find an exact optimal regularization—which is intractable and costly in practice—we focus on providing an efficient, empirically validated method (the surrogate indicator and the algorithm) for selecting a strong candidate. The results we include demonstrate that our method outperformed the default choice (no regularization) and baselines (the spectra-based) in terms of stability and overall performance. Furthermore, our empirical studies under multiple different settings justify that the LDS curve is very likely to be uni-modal and concave in practice.

It is unclear how easily this theory-driven approach for hyperparameter selection can be generalized to other TDA methods beyond IF.

As discussed in Section 2 and Appendix, many state-of-the-art data attribution methods such as TRAK and LoGra can be categorized as variants of IFFIM. The experiments in Section 4 and Appendix focus on the effectiveness of our method with IFFIM and TRAK. We also include an experiment of LoGra under GPT-2 + WikiText-2 setting, as shown in the table below. These results, obtained under multiple different settings, demonstrate the strong applicability of the method in practice.

Theorem 4.3 introduces a large number of new definitions with little intuitive explanation. Please clarify the interpretation of the terms in Equation 8 in Theorem 4.3. A more intuitive explanation would improve the accessibility of the result.

To aid understanding, we now offer an intuitive explanation of both sides of Equation 8. We first focus on the surrogate indicator, which is also the RHS of Equation 8. For simplicity, we let $v$ denote the gradient of $f$ at $z’$ and $\theta^\ast_S$. By observing the definition of terms $t_{k, z’, \lambda}$ where $k = 1, 2, 3$, the surrogate indicator can be viewed as the cosine “angle” between $v$ and $(F_S+\lambda I_p)^{-1} v$. Here, $(F_S+\lambda I_p)^{-1} v$is related to the $\theta$ update direction in IFFIM, as in Equation 2. When the surrogate indicator is close to 0, this “angle” is overly large, suggesting a significant misalignment between the $\theta$ update direction and the gradient descent direction $v$. This indicates that the update direction is not reliable. In this case, increasing $\lambda$ helps intuitively because it helps condition the inverted matrix $F_S$ better. On the other hand, when the surrogate indicator is close to 1, $F_S$ is overshadowed by the regularization term $\lambda I_p$, meaning that the attribution method may become dominated by the regularization and lose sensitivity to the underlying data structure. Similarly, the LHS of Equation 8 measures the “angle” between $v$ and the ground-truth direction that involves $g_{z’}$ (by applying Lemma B.3). Hence, Theorem 4.3 is in essence stating that when the “angle” between $v$ and $(F_S+\lambda I_p)^{-1} v$ is of comparable level as the “angle” between $v$ and ground-truth direction (not too big or too small), LDS could reach its maximum. We will provide a similar and more detailed interpretation in the final version of our paper.

The purpose of Proposition 4.7 is unclear. The derivation steps from the theoretical results (Theorem 4.3, Proposition 4.7) to Algorithm 1 are not clearly articulated. How do Theorem 4.3, the surrogate indicator, and Proposition 4.7 connect to the derivation of Algorithm 1? Please describe this connection more explicitly.

We establish a clearer connection between Theorem 4.3, Proposition 4.7 and Algorithm 1 here. Overall, Theorem 4.3 inspires the formulation of the surrogate indicator as well as Algorithm 1, while Proposition 4.7 justifies a desirable property of the surrogate indicator, which is helpful for Algorithm 1. In concrete, the surrogate indicator is introduced as exactly the RHS of Equation 8 in Theorem 4.3. The reason we drop LHS of Equation 8 is that it involves retraining-related terms $g$ and $\alpha$, whose computation is costly. From the above discussion of the interpretation of Equation 8, we notice that a good regularization strength commonly lies in the region where the surrogate indicator is close to neither of 0 and 1. Further, Proposition 4.7 guarantees that the surrogate indicator and LHS of Equation 8 both lie in the interval [0, 1] under certain assumption, which suggests that a fixed threshold of the surrogate indicator such as 0.5 could be adopted to select the regularization, which is exactly how Algorithm 1 works. Finally, Proposition 4.7 ensures that both sides of Equation 8 are comparable as they both fall in the interval [0, 1].

What is the computational cost of estimating the optimal hyperparameter using Algorithm 1, and how does it compare to computing the LDS metric under similar conditions?

Notably, directly computing LDS requires retraining, making it prohibitively expensive for large-scale models. However, our method only requires gradients at test examples, which can be obtained easily through inference passes. Under CIFAR2 + ResNet9 setting, searching for a same number of hyperparameter costs 13.7s using Algorithm 1, while costing 9m12.6s when ground-truth data in LDS are computed on the test set and the hyperparameter is selected by computing LDS directly. We obtain a 40.3x speedup over LDS-based selection under the same computational condition, demonstrating the practical efficiency of our approach.

Possible typos

We appreciate the reviewer’s careful reading and will ensure all identified typos are corrected in the final version.

The finding that earlier model checkpoints may produce better results than final ones for some TDA methods (line 111) is quite interesting. I wish the authors gave possible explanations for this.

While a complete explanation requires thorough investigation, our hypothesis is that this phenomenon stems from numerical issues due to vanishing gradient norms in later epochs. We also include an empirical fact here that supports our hypothesis.