Final-Model-Only Data Attribution with a Unifying View of Gradient-Based Methods

Thank you very much for your review. It appears from your originality score of 1 that originality is your main concern. As we discuss below, we think that the works you cited to support this assessment are actually complementary to our contributions rather than similar to them. We would be happy to discuss these works further with you. We also think that the last weakness you identified (lower cosine similarities) is not a weakness of our paper, and we respond to your concern about data size.

Setting

Although the setting is claimed to be new, it is actually just a variation of saying "re-training is so expensive and we can't compute gradients".

To clarify possible misunderstanding: We do assume it is possible to compute gradients of training instances, as otherwise (gradient-based) further training and the gradient-based approximations to it would not be possible. And the motivation for considering the FiMO setting is not just that re-training is expensive. We are considering the scenario where the original training process is inaccessible altogether (e.g., a model downloaded from Hugging Face). Given this constraint, our main contributions are as summarized in lines 51-54 of the introduction: We formulate an alternative gold standard that also operates under this constraint and show the relationships of existing gradient-based methods to this gold standard.

Works that you cited

So literature like [1] implicitly already discusses similar situation. There is also work [4] utilizing "unlearning" as TDA, which also resemble your core idea.

The main idea of both [1] and [4] is to reverse the roles of training and test data and use the effect on a training point of training on a test point [1], or unlearning a synthesized image [4], to approximate influence in the opposite direction (the usual one). While a valuable idea, we see it as complementary to our contributions summarized above. Specifically, this idea could be applied to yield reversed versions of the gradient-based methods that we study, but it does not address the question of an alternative gold standard for the FiMO setting or its relationships with gradient-based methods.

In terms of further training, works like [2] also discuss the possibility of "further LoRA fine-tuning as TDA". ... The ensemble idea they propose also match the expected value notation in equation (3).

We also think that [2] is complementary to our main contributions. We mentioned in Appendix F, lines 1219-1221 that LoRA fine-tuning could be used to make the further training gold standard more efficient, and this could be done using the specific method of [2], which we are happy to cite for this purpose. However, [2] does not address the question of a gold standard for the FiMO setting or its relationships with gradient-based methods. In addition, while the idea of ensembling has been proposed to improve practical TDA methods (for example TRAK by Park et al. (2023), which predates [2]), our work is the first to also propose it for a gold standard itself, to make it more reliable.

Lower cosine similarities

Looking at Figure 2, I will be hesitate to claim the cosine similarity between these baseline TDA methods and their further training counterparts are actually similar...Especially the setting is more complex like (e)(f). The curves in Figure 5 are even harder to explain.

• We agree with you (see point 1 in lines 377-378 and its elaboration in Appendix F). We would only say that the first-order-like methods (Grad-Dot, Grad-Cos, DataInf) are “actually similar” to further training when the amount of further training is low and the cosine similarity is high enough, and perhaps not at all in Figure 2(e)(f). Figure 5 looks less pleasing because it is an ablation showing that an alternative adjustment method (see lines 303-306 and Appendix D.3) works less well.
• In any case, we do not see these lower cosine similarities as a weakness of our paper because we are not advocating for a particular TDA method and are merely reporting how well current methods approximate further training. From this perspective, we might say that they reflect weakness of the current methods and open up possibilities for future research.

Data size

The training and testing data size seem to be too small to draw insightful and consistent conclusion.

• First, we used the full training set for further training.
• Second, we evaluated influence scores only for a subset of training instances ($|\mathcal{L}| =$ 50 or 100) because the purpose of the experiment was to compare to the further training gold standard. The latter constrains $|\mathcal{L}|$ because the number of further trainings is proportional to $|\mathcal{L}|$. Moreover, 50 or 100 training instances compares favorably to previous retrospective work on influence functions (Bae et al. (2022) used 20 training instances (see their Appendix C.2), Schioppa et al. (2023) used 32 in their Section 5.2).
• Third, regarding test instances, we evaluated 100 to assess variation across test instances; this number also compares favorably to Schioppa et al. (2023) who used 16. Figure 2 and similar figures show that the error bars resulting from 100 test instances are reasonable and our conclusions are reliable.

Your questions

Apart from cosine similarity, could you also consider correlation coefficient or other measure to give us a more comprehensive view comparing between further training and those gradient-based baselines?

Indeed, we use Spearman correlation as an alternative measure in Appendix E.1, Figure 6, with qualitatively similar results.

Just to make sure, is the very original IF already within the so-called FiMO setting?

Yes, and one of the points we are making is that influence functions are more suited to the FiMO setting than to the full retraining setting.

For the low quality of the gradient-based approximation, it can just because the approximation is not good. Under this setting, we might also need to assume the further training objective is not deviating form the original one too far, which is usually hard to justify in real world. Will it be interesting to just "come up with a good enough further training obj" for better attribution?

The problem is not that the further training objective deviates from the original training objective, since we have shown that the gradient-based methods approximate further training, and we compare them only to further training, not original training. We think it may be interesting to further investigate how the further training algorithm affects the quality of this approximation. We touch on this in Appendix E.1, Figure 4, where we used Adam as the optimizer instead of SGD.

How does the damping term in second-order method affect the quality of attribution?

For the second-order methods CG and LiSSA, which use a damping parameter $\lambda$, please see Appendix D.4, line 1042 and onward for discussion of our choice of $\lambda$. We observed that cosine similarities increase with $\lambda$.

Conceptually, how different it is between your further training and the "two-stage retraining" introduced in [1]?

We mention in Appendix C, lines 913-915 that the main difference is our proposal of averaging to mitigate the randomness of training.

Thank you very much for your review. We respect your strong position on what TDA can encompass, what the objective can be, and therefore what the gold standard should be. We will say however that if one takes the unavailability of the training procedure as an unchangeable fact in the FiMO setting (as is usually the case with a model downloaded from Hugging Face), then we think that one should at least consider an alternative objective that also respects this constraint (i.e., quantifying sensitivity rather than contribution), and accordingly an alternative gold standard.

We do want to ask you the following: Could your concern be allayed if we used different terms to distinguish our setting from the standard TDA problem? For example, we could always qualify TDA as FiMO-TDA from the end of Section 2 onward, and we could refer to further training as a “silver standard” rather than a gold standard.

For your other comments, we think that they can be largely addressed by moving content from the appendix to the main paper (as allowed for camera-ready papers), as detailed below.

when leaving out a data point (line 1024), the authors subtract the loss on that point in each gradient update rather than removing it from the dataset entirely—this is quite different from further training without a data point and needs justification (or at least mention) in the main text.

We are happy to move the referenced paragraph in Appendix D.3 to the main text. As mentioned there, one justification is that [28] used a similar procedure (but they did not actually justify it). Another reason is that it seems to also mitigate the randomness of stochastic mini-batch training, where the mini-batch that a data point appears in/is left out of is random. (This could have been the reason for [28] as well.)

second-order methods approximate further training under fixed regularization (as in the derivations authors provided), while further training is done without regularization.

As mentioned in line 125, we wanted further training to be like typical neural network training, so we did not apply regularization. For the second-order methods and specifically CG and LiSSA, which use a regularization/damping parameter $\lambda$, we explored varying $\lambda$ to improve the quality of approximation to further training. As mentioned in Appendix D.4, lines 1042 and onward, we found that a larger value of $\lambda = 0.01$ was better than $\lambda = 0.001$ used in prior work. We can mention this in the main text as well.

the choice of cosine similarity as the evaluation metric (line 317) is not intuitive and needs explanation in the main text.

We are happy to move Appendix D.5 to the main text to elaborate on cosine similarity.

I'm not sure what the authors meant in line 865. Further training could change the solution even when at the model is optimal under a fixed dataset. The gradient of the empirical loss being 0 does not imply the gradients is 0 for all individual data points.

In the convex case of Appendix A, the gradient of the empirical loss being 0 is a sufficient condition for optimality, and thus $\Delta\theta = 0$ is an optimal solution to (1) when $\theta^f = \theta^*$ is optimal. You are right that the gradient is likely non-zero at each individual data point, and this might be an issue with stochastic optimization algorithms, but this is more an artifact of that class of algorithms. Here we are making more of a theoretical statement.

Dear Reviewer NTqP,

Thank you very much for thoughtfully considering our rebuttal and being wiling to raise your score.

We understand your main concern. In the first part of Section 3 (to the end of page 3), we believe we are indeed reframing the problem, as one of quantifying "sensitivity" of the model and "predicting the effect of leaving one data point out during further training." Thus, we proposed in our rebuttal above to use the term "FiMO-TDA" to further distinguish our problem setting from standard TDA. We do not think that this would require significant changes to the manuscript. It would require replacements and single-sentence clarifications where the terms "TDA" and "FiMO" appear, as follows:

• To the last paragraph of Section 2, we will add the sentence "We will use the term 'FiMO-TDA' to refer to TDA in the FiMO setting and to clearly distinguish it from the standard TDA problem." Note that the title of Section 3 is already "Final-Model-Only TDA."
• First sentence of Section 3 will be changed to "We first consider the question of how FiMO-TDA should ideally be done, i.e., what could serve as a “gold standard” method, where the gold standard also respects the FiMO constraint." The acronym "FiMO-TDA" and phrase "where the gold standard also respects the FiMO constraint" will also be added to the similar sentence in the introduction, lines 28-29.
• Line 94: "TDA" --> "FiMO-TDA"
• Line 109: "the FiMO setting" --> "FiMO-TDA"
• Line 159: "the FiMO setting" --> "FiMO-TDA"
• Line 238: "Under the FiMO setting" --> "For FiMO-TDA"
• Line 277: "in the FiMO setting" --> "for FiMO-TDA"
• Line 310: "TDA" --> "FiMO-TDA"
• Line 373: Add "(FiMO-TDA)" after "TDA"
• Line 376: "FiMO" --> "FiMO-TDA"
• Line 378: "TDA" --> "FiMO-TDA"

We hope that, if you could be so kind as to re-read the first part of Section 3 and review the above changes, that they can address your concern.

I thank the author for their comment. While I agree that addressing these issues is a step towards the right direction, I have reviewed Sections 1-3 again and believe the proposed changes are insufficient to address my concerns. Therefore, I would like to maintain my current score. I continue to have concerns about the use of "TDA" (even within the FiMO framework) for the reasons I outlined above, as I believe it may be misleading in this context. Instead of using the term TDA, I think that the work should be formulated directly in terms of predicting the effect of further training. I don't think further training can be the golden standard of TDA in any setting. If something is to be the gold standard, it should show a correlation with retraining-based metrics (e.g., actually removing the data point(s) and retraining the model from scratch).

Thank you very much for your positive review (bordering on Strong Accept) and recognition of our contributions! Below we respond to your concerns and questions.

While the paper explicitly mentions the cost, this method is extremely expensive nonetheless -- making it impractical for most users.

To be clear, we are proposing further training as a gold standard and as such, we did not intend it to be computationally efficient, nor are we proposing it as a practical TDA method. The suggestions given in Appendix F (and work cited by Reviewer hy33) could ease implementation of further training in future work.

although it is an artifact of the FiMO setting, sensitivity after having trained the model might not fully capture the real impact of the training example as the model may have saturated for that input already.

We agree that saturation is possible. Nevertheless, we did find in Appendix E.2 that further training influence scores can be useful for detecting mislabelled examples and giving insight into what the model has learned.

What's the dependency on the further training algorithm? What if the employed learning is really detrimental to the model?

We addressed this question partially in Appendix E.1, Figure 4, where we used Adam as the optimizer instead of SGD, with qualitatively similar results.

What would be the right amount of further training? Can there be a generalized guidance based on empirical evidence?

Indeed, we pose this as an open question in line 380 and elaborate on it in Appendix F, but as such is difficult to provide a concrete answer.

Thank you very much for your review and for recognizing the value of our positioning/organizing the TDA literature and unifying gradient-based methods within it. Below we first discuss why the low cosine similarities are not a weakness of our paper. We then respond to your concern about computation and your other questions.

[Scope]: The empirical studies on non-tabular datasets have low cosine similarity on all methods. … Lines 1140-1153 in Appendix E.1 discuss this issue but this poor approximation quality is still a weakness in non-tabular domain.

We agree that the low cosine similarities on non-tabular datasets are a less desirable finding for researchers in this area. From the perspective of approximating further training, we think they reflect weakness of the current methods and motivate the development of higher-quality methods (see point 1 in lines 377-378 and its elaboration in Appendix F). However, we do not see this as a weakness of our paper because we are not advocating for a particular TDA method and are merely reporting how well current methods approximate further training.

[Computational Time]: The experiments took about 3000 GPU hours (Appendix E) on (relatively) small CNN and BERT models. The Appendix F mentions this and suggestions are made to reduce the number of seeds or using parameter efficient approaches like LoRA. However, the effects of LoRA have been left for future work and that would still not address the problem of computational time for widespread and meaningful adoption.

As discussed in Appendix D.6, most of the computational time was needed because we aimed to implement and compare to the further training gold standard. Our focus in this work was to establish this gold standard and as such, we did not intend it to be computationally efficient, nor are we proposing it as a practical TDA method. The suggestions given in Appendix F (and work cited by Reviewer hy33) could ease implementation of further training in future work.

[Minor Comments]: From line 87-90, it's argued that FiMO is applicable to scenarios with TAA and CPA settings. While, this is true, the increased access in these cases may be able to provide deeper insights?

Yes of course, if increased access is available, then we encourage practitioners to take advantage of it if they can. We can clarify that while FiMO is applicable to these settings with greater access, it may not be the most recommended.

[Number of Random Seeds]: Figures 3(b) and 7(f) show the impact of number of random seeds averaged on the maximum cosine similarity for the SST-2 dataset -- the general suggestion is that a lower number of random seeds (10-20) can be used as the highest impact is seen there. However, in these cases, DataInf seems to be benefitting from multiple seeds. This is explained in the Appendix. However, this adds to the computations. A clarification on the following two points will be useful -- (i) why DataInf is uniquely sensitive, and (ii) whether the steadily rising curve is worth the extra compute cost.

1. DataInf is not uniquely sensitive. The Grad-Dot curve almost coincides with it. We remark in lines 1130-1131 that the first-order-like methods (including DataInf) are generally more sensitive to the number of seeds. It might appear this way because these methods can achieve higher maximum cosine similarities than second-order methods, but it requires more averaging of further training runs to make this apparent.
2. Whether the extra computational cost is worthwhile is user-dependent and not something we can answer in general. It does appear that most of the gain is achieved at ~20 seeds (this might be more apparent if we plotted on a linear scale).

[Pseudocode/Implementation]: If there are some institutional constraints on sharing the code, it may be good to provide some example pseudocode for reproducibility.

We will revisit the question of sharing code with our institution and we are hopeful that we will be able to do so for the purpose of reproducibility. In particular, we hope to be able to share our specific code for further training and the indices of the left-out training instances in $\mathcal{L}$.

Thank you for the responses, I am in agreement with points that you have shared and believe that this will highlight the value of this work further. Regarding the "further training" vs "two-stage retraining", I intended to suggest that the detailing in comparison should not just be at a conceptual level i.e. the difference in the methods (averaging) and it should also be highlighted that your overall contributions are still novel findings. It was more of a suggestion to frame the comparison in a broader way.

Given the overall state of the paper and the. discussions, I am inclined towards acceptance and am raising my score accordingly.