We thank the reviewer for their detailed comments and praise of the paper. We are happy to address their minor concerns below. We also politely draw their attention to our extra experiments, added in response to their comments.

Applicability to transformers&diffusion models. The reviewer correctly notes that our chief contributions are conceptual and theoretical, introducing d-TDA, explaining how it can be operationalised with unrolled differentiation and IFs, and rigorously motivating IFs in a deep learning context within this framework with Thms 2 & 3. Whilst we provide a preliminary empirical exploration (see especially Figs 2, 3, 4 and 7), a full study of our methods’ applicability to larger models like LLMs must be left to future work.

That said, we do agree that our messaging might be clearer with experiments on different, bigger architectures. For this reason, we have added additional experiments applying our d-TDA methods to transformers and diffusion models. For the vision transformer experiments, we show that distributional (mean) influence can better identify training datapoints that will improve the final test loss fo a single model when removed from the training set. By removing 10% of datapoints, we can improve the test loss from 0.715 to 0.509 (compare to 0.528 for regular TDA with a fixed seed). For latent diffusion trained from scratch on a dataset of 50000 Artbench images (256x256 resolution), we compare top predicted Wasserstein shift and mean shift samples, illustrating that they affect the ground-truth retraining distribution in the predicted manner. Thanks for prompting this; we hope this addition strengthens the paper in the reviewer's eyes.

Assumptions. The reviewer is right to observe that the assumptions used for our theoretical work are much less restrictive than those commonly used to derive IFs in the literature (namely, convexity). They are also very standard for theoretical work involving SGD. Whilst we do not claim that they will always be universally applicable for deep learning, we suggest that they are actually remarkably general. For instance, bounded loss gradient assumptions could be enforced by considering optimisation on a bounded domain, which is the case in practice when using finite-precision floating-point numbers. We also emphasise that A1-A6 should be interpreted as sufficient conditions for unrolled differentiation to converge to IFs. There could be pathological situations where e.g. SGD itself does not converge (in which cases, deep learning training wouldn't perform), in which case our theoretical results will certainly not hold. We respectfully suggest that this should not be construed as a weakness of our analysis – it is more of a reflection of the mathematical character of the problem.

Relationship between Thms 2 and 3. Thanks for the question. These are parallel justifications with some relation to one-another. Roughly speaking, the relationship is that, after running SGD with a small learning rate for a sufficiently long time, the distribution over final model weights will converge towards a set of delta-functions at the loss function minima (~ a low temperature Boltzmann distribution, under some assumptions on the batching noise). In this case, when the loss is perturbed, the best possible transport map (in terms of KL-divergence) will simply translate these delta functions to their new locations. Thm 3 shows that this best map is given by influence functions. Thm 2 shows that unrolled differentiation converges to the same IF formula. Hence, when training for sufficiently long we get that ‘optimal transport map’ ≈ ‘influence functions’ ≈ ‘unrolled differentiation’. We will add this discussion to the text; thanks for prompting this.

Citations. Thanks for the extra references, which we have now added.

Once again, we warmly thank the reviewer for their comments. If the reviewer thinks the newly added experiments and explanations strengthen the paper, we respectfully ask that they consider raising their score.

We thank the reviewer for their thoughtful feedback, and are pleased that they identify the work as interesting, clear and technically rigorous. We address all their comments and questions in detail below.

Few numerical results. The reviewer is right to note that our chief contributions are conceptual and theoretical, offering some preliminary empirical evidence via UCI concrete and MNIST but deferring more exhaustive experiments to future work. Our main intention is to introduce and explore the mathematical foundations of d-TDA – especially, as a lens to understand the unreasonable effectiveness of IFs in deep learning. Given how widely influence functions are being used in deep learning, and the limitations of current interpretations, we believe this is a major theoretical contribution. Future papers can further explore applications and improvements to IFs based on insights and evaluations suggested by framework.

That said, we do agree that the paper could benefit from some additional empirical results. For this reason, following the reviewer’s comments, we have added additional experiments applying our d-TDA methods to transformers and diffusion models, including the suggested data pruning task with vision transformers. For the vision transformer experiments, we show that distributional (mean) influence can better identify training datapoints that will improve the final test loss fo a single model when removed from the training set. By removing 10% of datapoints, we can improve the test loss from 0.715 to 0.509 (compare to 0.528 for regular TDA with a fixed seed). For latent diffusion trained from scratch on a dataset of 50000 Artbench images (256x256 resolution), we compare top predicted Wasserstein shift and mean shift samples, illustrating that they affect the ground-truth retraining distribution in the predicted manner.

Comparison to standard (non-distributional) TDA. Thanks for the comments. We again emphasise that our chief goals are to 1) introduce a new training data attribution framework that properly takes into account stochasticity, and 2) show how existing standard methods such as IFs can be properly motivated and used within this mathematically rigorous framework, in contrast to the non-distributional TDA setting. Whilst we do spend some time comparing distributional influence to its regular counterpart (see especially Fig 7 in the appendix, where ‘mean influence’, albeit distributional, is the current standard practice), our principal focus is not new data attribution methods per se, but rather a novel way of thinking about existing and future training data attribution methods.

That said, we agree showing the practical limitations of applying IFs to a single model with a fixed random seed would be beneficial. We believe the newly added data pruning experiments with vision transformers on CIFAR-10, showing empirical improvements from using distributional influence, illustrate this point.

Questions. Thanks for the great questions.

How many samples are needed for d-TDA? One can adopt a distributional perspective with only 1 sample. In this case, the original sample is interpreted as distributed according to the measure for the original dataset. Meanwhile, the modified sample, obtained by offsetting with IFs or unrolled differentiation, is (approximately) distributed according to the measure for the new dataset. Through Thms 2 and 3, we make this statement mathematically rigorous for the first time. More pragmatically, if computing e.g. Wasserstein or mean influence, we typically use 5-100 samples, with full details for each experiment reported in the Appendix. Also note that practitioners often heuristically average over a small ensemble of IFs to improve performance. Hence, one could argue that we are really just suggesting a more rigorous way to use this existing set of samples, at essentially identical computational cost.

Does d-TDA work with other optimisers? Yes. Unrolled differentiation can be extended to arbitrary optimisers, provided their updates are differentiable. We believe our Thm 2 can be extended in kind. In fact, following the reviewer’s comments, we have verified that this is the case for SGD with momentum. However, we leave a full exploration of the theoretical details to important future work, following standard practice in the literature by initially focussing on SGD (Bae et al, 2024).

We again thank the reviewer for their time and comments. With the above clarifications in mind, as well as the extra experiments and theory added in response to their comments, we respectfully ask that they consider raising their score and recommending acceptance.

References. (Bae et al., 2024) Training Data Attribution via Approximate Unrolled Differentiation, https://arxiv.org/abs/2405.12186

We thank the reviewer for thoughtful comments and for finding our work technically sound and novel. We address their main concerns below and highlight our new diffusion and transformer experiments, added in response to their feedback.

Motivation for d-TDA

Why is distributional influence interesting? It is widely acknowledged that conventional TDA methods are limited by their inability to account for stochasticity in training (K et al., 2021, Revisiting Methods for Finding Influential Examples). It is difficult to distinguish changes to model behaviour due to training data modifications from changes due to sampling randomness. Furthermore, usual derivations of IFs do not account for non-convex losses with multiple minima (Bae et al., 2022, If Influence Functions are the Answer, then What is the Question?) which are typical in deep learning. In contrast, in the d-TDA setting IFs can be naturally derived without these restrictive assumptions.

Why not regular TDA with a fixed seed? This is an important question that we will dedicate a discussion to in the paper. Even with a fixed seed, chaotic training dynamics can push training to a different parameter space region when perturbing a dataset. To aggrevate this, in standard deep learning training with a fixed batch-size, training on a subset completely offsets at what iteration each datum is presented, even with a fixed seed. There are no guarantees that in these settings IFs would give us an answer to the counterfactual retraining question with a fixed seed. This is why prior work relies on convexity assumptions, where everything converges to the same point. In contrast, our work rigorously shows that IFs do something meaningful even in this challenging setting. We just have to interpret IFs as approximate resampling instead.

To illustrate this, we will add an experiment where we vary the peak learning rate at the beginning of training, showing the break-down in correlation between IF prediction and retraining with the same seed. Nonetheless, we illustrate that IFs are good at approximately resampling from the target distribution in all cases. Furthermore, our new vision transformer data pruning experiments illustrate that distributional (mean) influence does better in practice than applying IFs to a single model with a fixed seed.

We also think that the distributional properties of training may be of independent interest. Practitioners might be interested in selecting the data to reduce the variance for the output for a given unlabelled input, or they might want to do data selection to ensure it leads to improvements in validation loss for 90% of training runs. Our d-TDA methods allow practitioners to curate the dataset to target both these desiderata, or yet more complicated distributional properties. We demonstrate this with new Latent Diffusion and vision transformer experiments on 50000 Artbench (256x256 resolution) and CIFAR-10 datasets respectively.

We agree that exploring the full scope d-TDA applications is an important future research direction. However, our paper's chief goal is to build the mathematical foundation of influence functions for deep learning through the lens of d-TDA.

Takeaways from theorems and position on IFs

Purpose of Thm 3. Thm 3 shows that IFs are the optimal KL-divergence approximation to the d-TDA question if the fully trained model weights obey a Boltzmann distribution with the energy given by the loss – a reasonable approximation which arises from approximating the batching noise in SGD (Mandt et al., 2017, Stochastic Gradient Descent as Approximate Bayesian Inference). In this setting, IFs provide precisely the KL-divergence-optimal asymptotic transformation to map samples trained with the original dataset to the samples trained with the perturbed dataset. The isotropic Gaussian noise is a simplifying relaxation, akin to how prior works have studied convex losses (albeit arguably a weaker one). We include Thm 3 to demonstrate that IFs have another distributional motivation (similarly to Thm 2). By slightly simplyfing the setting compared to Thm 2, we obtain a stronger result: IFs are the optimal distributional approximation in terms of KL-divergence. It is not intended to be interpreted as practical guidance for training models.

Position on IFs (Q2). The core motivation for this work is that IFs appear to work well in downstream tasks, but lack theoretical and conceptual motivation in deep learning due to stochastic training and non-convex losses. Our paper provides a justification for IFs in these settings by showing that they can be interpreted as resampling. Hence, our paper is pro-IFs in that it provides new strong motivation for their use through links to unrolled differentiation.

Minor points

Related work of Luo et al. (2023). Forward-mode unrolled differentiation in deep learning was proposed even earlier by (Franceschi et al., 2017, Forward and reverse gradient-based hyperparameter optimization), whom we cite. We claim that our paper is the first such application to TDA (hence ‘first application [...] for the computation of the response’). We have clarified this; thanks.

Empirical evidence that LOO is just noisy. Please see Fig. 6 and App. C.1, in which provide the suggested experiments.

Figures 2, 4 and 5. We apologise that parts of the figures were unclear to the reviewer. The captions are dense due to the intial 9-page limit.

Fig 2. We consider neural networks trained on UCI concrete, with either the full dataset or a subset. We consider 4 measurements: the 1D neural network output on 4 chosen ‘query samples’. Each is represented by a different colour. Left: we show actual outputs trained on a subset (x axis) plotted against against predicted outputs (y axis) for each measurement (colour). The prediction for the outputs trained on a subset (“retrained”) are made by applying unrolled differentiation to models trained on the full dataset. Each sample corresponds to a single random seed. Here, a perfect prediction would lie on the black $x=y$ line – meaning predicted matches ground-truth retraining. Most samples lie far away from the line. Centre: Repeats the left panel, but shows the mean over all random seeds. We see that the mean of each measurement distribution predicted with unrolled differentiation (circles) actually lies close to the $x=y$ line, certainly closer than the mean of the original measurement distribution of a model trained on all the data (diamonds). Right: we show that the true retrained distribution (colour, line) is closer to the predicted retrained distribution (colour, shaded) than the distribution trained on the full dataset (black). We will make sure this is explained more thoroughly in the caption.

Fig 4. As in prior work, ‘query point’ refers to an example (input) for which we compute the measurement function. We now introduce the term explicitly.

Fig 5. As the y-label indicates, the top plot shows the Pearson correlation between IF-predicted and UD-predicted measurement changes (y-axis) against training time (x-axis). 5 differently-colored lines correspond to runs with different learning rates. Three smallest learning rates give similar correlations, so they overlap. We have made this visually clearer. The dashed lines are spurious and have been removed.

Also, is it true for neural networks that their Hessian is bounded and Lipschitz?

This is a common assumption in deep learning theory (far weaker than convexity). For instance, it holds if we consider optimisation on a bounded domain (which is virtually always true in practice due to finite-precision arithmetic).

"If the [model] weights converge to a saddle point ... the final parameters will become sensitive to any infinitesimal perturbation of the loss function." I'm pretty sure this isn't true.

We mean that a gradient flow trajectory converging to a saddle point will in general (not always) be unstable under infinitesimal perturbations to the loss function. One can construct careful perturbations for which this is not the case. To illustrate, consider $L(\theta) = \theta_1^2$. The set $\{\theta \in \mathbb{R}^2: \theta_1=0\}$ are all minima, but any infinitesimal perturbation $L(\theta) + \epsilon \theta_2^2$ collapses the minimum set to a single point. We have now phrased this more carefully in the text.

Motivation for assumptions A4 and A5. A4 is intuitive as it's necessary for unrolled differentiation to not diverge; if unrolled differentiation diverges then we certainly cannot expect it converges to IFs. Likewise, A5 is needed to make sure that minima cannot be arbitrarily flat so IFs themselves do not diverge. We stress A1-A5 are intended to describe sufficient conditions for unrolled differentiation to converge to IFs. Whilst reasonable for many deep learning applications, we certainly do not claim that they will hold universally. Indeed, situations where they do not hold may be of independent interest for understanding when IFs break down. We'll make sure to expand the discussion of the motivation of each assumption.

Bounding nonzero Hessian eigenvalues away from zero. A5 refers to the nonzero Hessian eigenvalues on the set of critical points (‘minimum manifold’), rather than at a single location $\theta$ – hence why the assumption is nontrivial.

Missing definition. The set $\mathcal{S}^g_\mathcal{L}$, to which ‘minimum manifold’ refers, is defined in line 339; we made the connection clearer.

We thank you for your thorough feedback. If anything remains unclear, please let us know. We hope our clarifications and new experiments warrant recommending acceptance.

Thanks for the detailed response and for continuing to engage so wholeheartedly with the reviewing process! We completely understand your confusion, and agree that the text needs refining to be clearer on these points.

To answer your question: the paper intends to do both of (1) and (2). In our minds, the core contribution of the paper is that IFs appear naturally if you try to tackle data attribution in a rigorous, distributional manner for general, non-convex loss functions -- or, as you aptly put it, they are 'secretly an approximation to d-TDA' (1). Of course, properly introducing d-TDA (2) is a prerequisite for this, which we also think may be of independent interest. Taking the distributional interpretation of IFs to its logical conclusion, a secondary contribution is to come up with distributional influence: metrics to measure how effective TDA methods are at estimating changes in distributions over model outputs (2). Distributional influence can be used to benchmark how effective methods like IFs and unrolled differentiation (or others) are at this job.

As a minor point, which may be helpful for understanding our contributions: we emphasise that we use ‘distributional TDA’ -- a framework for thinking about TDA with stochastic training algorithms -- distinctly from ‘distributional influence’ -- a measure of how important individual training examples are in a distributional sense.

We agree that (1) should be the main focus of the paper. (1) is where we dedicate most of our mathematical work. However, we think that (2) is also a notable contribution. A proper distributional TDA framework is essential to even understand contribution (1), and moreover our extra experiments show that distributional influence can be practically useful. For a dataset pruning task with a ViT trained on CIFAR-10, distributional (mean) influence can reduce the test loss from 0.725 to 0.509 — compared to 0.528 with non-distributional ‘fixed-seed’ variant — for a single model trained with the same seed. In this sense, distributional influence is very much a 'new, real algorithm' that can be practically useful -- even if we do not necessarily recommend that it should be used in every possible instance (e.g. LLMs).

As a subtle but important point: we emphasise that (2) should not be interpreted as a weakness of IFs per se, because one can actually use IFs to compute distributional influence. As we say in the text (lines 361-366), you can approximate samples from the retrained model by taking samples from the original model and offsetting each of them by $r_{IF}/N$. Regular IFs are thus fully compatible with d-TDA, and with computing distributional influence. It is not a case of 'IFs vs d-TDA', but rather 'IFs for d-TDA'. When we talk about 'classical IF/TDA' methods, we mean using IFs naively without following the algorithm in lines 150-156. Indeed, even the regular usage of IFs with just one seed can actually be seen as a special case of distributional influence: namely, unbiased estimate of mean influence with a single sample. This may help explain why they work in practice for non-convex losses.

With the above discussion in mind, we would like to suggest renaming the paper to:

• Why Do Influence Functions Work? A Distributional Perspective
• Influence Functions as Distributional Data Attribution
• The Distributional Data Attribution Task and Why Influence Functions already Solve It or similar, subject to discussion with reviewers and other authors.

We will refine the text to more clearly convey what our contributions are. As an illustrative example, we have sharpened the 'Core Contribution' paragraph (L53-L62) as follows:

Core contributions: 1) We introduce distributional training data attribution (d-TDA), a framework for studying data attribution in stochastic deep learning settings. We show that d-TDA helps explain the limitations of leave-one-out retraining, and the effectiveness of influence functions (IFs). 2) In particular, we prove that IFs actually solve certain distributional TDA tasks, approximating a) the limiting distribution of unrolled differentiation and b) the best possible transport map between the final (Boltzmann) distributions of model weights, under select assumptions. 3) Lastly, within the d-TDA framework, we propose distributional influence, which quantifies the importance of training examples by how much their inclusion/exclusion affects the distribution over model weights and outputs (Section 3). We show that distributional influence can be used to benchmark TDA methods and that it captures interesting information missing from its regular predecessor. We provide preliminary empirical evidence that our novel distributional influence variants are effective in data pruning tasks.

We again thank the reviewer for their thoughtful feedback – it has helped improve the clarity of the writing!

Thanks for continuing to respond in so much detail!

This rephrasing makes a lot more sense to me. I think presenting the paper along the lines proposed here really alleviate a lot of my issues, and I've increased my score to vote for an accept.

The only point above that I think could use more support in the paper is"We show that d-TDA helps explain the limitations of leave-one-out retraining" (either more experiments / deeper discussion of literature / logical arguments). But it's not totally unsupported by the paper, so I don't think this is a big issue. And overall I think the flow of "surprise! IFs are distributional, and by the way, doing explicitly distributional TDA might be of separate interest" is a great contribution to this literature.

(Edit: I don't have any complaints about the current title of the paper, so I'd encourage the authors to update it only if they want to!)

We thank the reviewer for their favourable comments on the text. We appreciate they describe our perspective on IFs and data attribution in deep learning as “highly theoretically interesting” and “theoretically sound” -- key goals of the paper. We are happy to address their concerns and some points of minor misunderstanding below, also emphasising the extra transformer and diffusion experiments added in response to their helpful feedback.

1. the IF calculation to become even more computationally expensive

We stress that a core contribution of the paper is showing how standard influence functions (IFs) can already be interpreted as resampling within a distributional framework (Thms 2 and 3). Concretely, offsetting trained model weights by an IF enables one to (approximately) sample from the distribution of models trained with a modified dataset. Our paper makes this claim mathematically rigorous. Hence, our work shows that standard influence functions applied to a single model are already distributional. In this sense, d-TDA is not inherently more expensive than IFs: d-TDA simply provides a new interpretation of what IFs actually do in deep learning. Without this distributional interpretation, IFs must be derived with unrealistic assumptions on the loss function: e.g. convexity or the presence of a unique minimum (e.g. the PBRF approximation in (Bae et al., 2022)).

The distributional reinterpretation allows us to in addition define novel measures of distributional influence. In our minds, the primary utility of of distributional influence is for evaluating methods like IFs in deep learning. Since IFs are inherently distributional, we think distributional evaluations are appropriate to verify how well they approximate actual resampling. As the reviewer correctly notes, calculating e.g. Wasserstein influence does indeed require an ensemble of samples – as few as 5 suffices in some of our experiments. However, when evaluating TDA methods, it is already standard practice to average TDA results over a small number of models (Park et al., 2023). Hence, our algorithm does not actually substantially increase the cost compared to usual in these cases.

2. It is not clear how could we sample seeds for actual distribution, which leads to concern whether the ground-truth distribution is achievable

Could the reviewer kindly clarify what they mean by “how could we sample seeds for actual distribution”? If you are asking how we sample from the “ground-truth” training distribution trained on a given dataset: to obtain a sample, we change the seeds for the training sources of randomness – 1) the data batch ordering, 2) model initialisation, and 3) any other sources of stochasticity (e.g. dropout masks). This is what we report as grount-truth samples in e.g. Fig. 2.

3. ‘Little evidence to show the distribution is really necessary for data attribution in practice.’

The reviewer is right to note that this initial work focuses on introducing the conceptual foundations of d-TDA and understanding its mathematical properties. Whilst we provide some empirical demonstrations, we do indeed defer a full–scale experimental evaluation of downstream applications as important future work. We again stress that the crux of our paper is about showing that influence functions inherently answer a distributional question; we do not argue that everyone should always estimate the whole distribution for every downstream task.

That said, we agree that the paper might benefit from a clearer demonstration of the downstream utility of distributional influence. Therefore, in response to the reviewer's feedback, we added an extra dataset pruning experiment comparing distributional influence against regular influence on a subset selection task with SWIN Vision Transformers on CIFAR-10. Our results show that, by using distributional (mean) influence, we can select a better subset of examples to reduce the test loss than when we compute influence on a single model with a fixed seed. Our distributional subset selection removing 10% of data improves the final test loss from 0.725 to 0.509 (compared to 0.528 with non-distributional variant). We use 10 samples to estimate the distribution.

Questions:

Do you think the actual distribution of influence function is achievable in practice?

Yes. Our novel theoretical results show that you can approximately sample from the distribution over retrained models with as few as 1 training run and its corresponding IF (see Thm 2). Moreover, taking a small ensemble of models (~5 models), one can effectively capture changes to the distribution over model measurements (Fig 2 right). We again stress that training with multiple seeds is the default practice when evaluating methods like IFs (see Appendix C.2).

Whether you think the estimation of [influence on parameters] is always necessary, since we sometimes just need to estimate the training samples contribution to a specific prediction on a testing sample.

Thanks for the comment. We certainly agree that there is an important place in the literature for empirical debugging and explanation tools, like those referenced by the reviewer. Such techniques are sometimes framed as analogous to IFs when used for TDA. However, our paper is more concerned with the following mathematical question: how does the distribution over final model predictions change depending on the dataset? Whilst we agree that fast, heuristic alternatives to unrolled differentiation and IFs might suffice in certain down-stream applications, these techniques do not necessarily address this question. There are many cases where we really care about understanding how the model would behave if retrained, e.g. copyright concerns (Mlodozeniec et al., 2024) or understanding LLMs (Grosse et al., 2023). As such, we need to develop principled methods that aim to answer this question. In this sense, we consider these works to be orthogonal to our work.

I like your interpretation on the leave-one-out training, but what is the useful measurement for real-world experiment of data attribution?

We are pleased the reviewer appreciates our contribution that LOO is not inherently incompatible with IFs: a distributional interpretation reveals that the problem is just a low signal to noise ratio (SNR), and with enough samples IFs can indeed approximate the LOO distribution. For evaluation purposes, we believe leave-many-out retraining (as is standard practice in the literature (Park et al. 2023)) is a useful alternative for real-world experiments that has a sufficient SNR. We report such experiments in Appendix C.2.

In practice, how your d-TDA can be significantly different from classical TDA to identify problematic training data samples?

Fig 3 already demonstrates an example of a datapoint whose exclusion does not substantially modify the mean model behaviour, but which does drastically modify the distribution over model outputs in e.g. the Wasserstein sense. We provide further examples for MNIST in Fig 4. Training examples that are visually and intuitively ‘influential’ are sometimes missed by mean TDA. In response to this comment, we have now also added experiments comparing top Wasserstein shift vs mean shift samples on Latent Diffusion Models on 50000 Artbench images (256x256 resolution). Once again, we find that most important examples do differ between Wasserstein and mean influence.

---

We again warmly thank the reviewer for their feedback. We hope we have resolved all their questions and concerns, but invite them to let us know if any remain. If satisfied with our clarifications and the extra experiments, we respectfully ask them to consider raising their score.

References

(Park et al., 2023) TRAK: Attributing Model Behavior at Scale, https://arxiv.org/abs/2303.14186 (Bae et al., 2022) If influence functions are the answer, then what is the question?, https://arxiv.org/abs/2209.05364 (Mlodozeniec et al., 2024) Influence Functions for Scalable Data Attribution in Diffusion Models, https://arxiv.org/abs/2410.13850 (Grosse et al., 2023) Studying Large Language Model Generalization with Influence Functions, https://arxiv.org/abs/2308.03296