Efficient Data Selection at Scale via Influence Distillation

We thank the reviewer for sharing their valuable insights. Below find our response to the questions:

1. The choice of number of landmarks

We appreciate the reviewer’s comment. As shown in Figure 3 (left), we have empirically studied the effect of the number of landmarks on the performance of Llama2-7B. The results indicate that performance stabilizes around 4096 landmarks. For other models, we did not perform any tuning and simply used the same value (4096) across the board.

Determining the optimal number of landmarks is indeed an important and interesting question. One potential approach is to incrementally add landmarks and, at each step, approximate the gradient of each landmark using the other ones. Once the approximation quality reaches a certain threshold, the process can be stopped and that number can be used as the chosen landmark count.

We explored this idea empirically by implementing a procedure to adaptively select landmarks for Llama2-7B. The method proceeds as follows:

1. Begin with a small set of landmarks L, initialized with 32 randomly selected samples. Compute and project their gradients using a Rademacher-based projection (similar to LESS [1]).

2. For each sample in L, approximate its gradient as a linear combination of the other samples (excluding itself). We use kernel ridge regression (KRR) with an RBF kernel to perform this approximation. To ensure that each sample is not used in approximating itself, we set the diagonal of the Gram matrix to zero.

3. Compute and report the average cosine similarity between the true and approximated gradients.

4. If the size of L is less than 8192, add one new random sample to L, compute and project its gradient, and repeat from step 2. Otherwise, stop the procedure.

Running this experiment, we observe that the average cosine similarity improves smoothly as the number of landmarks increases. The table below presents the average cosine similarity values for several different sizes of the landmark set L:

These results suggest that setting a threshold of 0.8-0.9 for approximation quality would naturally lead to selecting a number of landmarks consistent with what we used in the paper.

Importantly, this procedure does not incur additional computational cost, as the gradients of the landmarks must be computed anyway for Influence Distillation.

We thank the reviewer for this insightful and valuable suggestion. While we believe that this method needs further investigation to establish its effectiveness, we agree that including this or a similar approach for selecting the number of landmarks would strengthen the paper.

[1] https://arxiv.org/pdf/2402.04333

2. Discussion on other baselines

We thank the reviewer for their constructive feedback and for suggesting several relevant baselines for comparison and discussion. This will help better position our work and clarify its contributions. We provide a point-by-point response below. We will include these results and discussions in the next revision of the paper.

• IFD is a heuristic-based filter that relies on metrics like perplexity and is independent of the target model's training dynamics. Previous work (e.g., the RDS+ paper [6]) has already shown that such heuristics are generally outperformed by more sophisticated data selection methods like RDS+ with a large margin. Given our focus on model-aware influence and our competitiveness with state-of-the-art data selection methods, we believe a direct experimental comparison would be less informative for demonstrating the specific advantages of our approach.

• DSIR is a model-agnostic method that uses importance resampling and does not leverage the target model’s parameters for selection. This places it in a different category from model-aware influence function methods like ours. This distinction is already discussed in the Related Work section.

• MATES trains a small, separate model to predict the influence of data points during the main model's training. A key limitation is the substantial cost and complexity of training this auxiliary model, which must be done repeatedly. By contrast, our Influence Distillation technique is a far more streamlined framework that requires no auxiliary models.

• IDEAL is an interesting concurrent work (published after the NeurIPS deadline), which addresses a different problem: Its goal is to find the optimal mixing of data from different domains or sources. Our method, in contrast, selects the most influential individual data points from a given data pool.

• QUAD & TAROT & LESS: We agree that QUAD, TAROT, and LESS are relevant baselines. TAROT and LESS, however, require computing the gradients for each sample in the pool, which means they are significantly slower than Influence Distillation due to our efficient landmark-based gradient approximation. QUAD, on the other hand, though closely related, focuses on pre-training settings, which differs from our focus on supervised fine-tuning. Due to the significant computational resources and time required to implement and run these methods fairly, we only have included a comparison with LESS in the next point below. We will include comparisons with QUAD and TAROT as well in the next revision of our paper.

Overall, this detailed examination of prior/concurrent work suggests that our original submission already compares against the most relevant baselines, and does not impact our claims of novelty or performance. We thank the reviewer again for their detailed additional references, which we will incorporate into the next revision of our work.

[6] https://arxiv.org/pdf/2503.01807

3. Comparison with LESS [1]

We thank the reviewer for this valuable comment. As noted in the last paragraph of Appendix F (also mentioned in Section 5.1, Baselines), we have included a brief discussion of the LESS method. Specifically, as an ablation, we replaced JVP embeddings with actual projected gradients. Figure 7 (right) in the appendix shows that this yields a gradient recovery of 0.9 (in terms of average cosine similarity) with 4096 landmarks, making the approach, in this case, very similar to LESS which uses exact (projected) gradients. The last row of Table 2 shows that this variant performs on par with Influence Distillation using JVP embeddings.

To provide a more complete comparison, we also ran the 10k/200k selection experiment on Llama2-7B using the LESS method (with one seed due to its high computational cost). In this setup, gradients are computed after a 10k-step warmup, projected down to 8192 dimensions, and compared to the target gradients. The top 10k examples are then selected based on similarity—following the original LESS procedure. Below we present a comparison between uniform sampling, InfDist with JVP embeddings, and LESS.

While LESS achieves higher accuracy on average, it is significantly more computationally expensive. Our method requires approximately 872 TeraFLOPs (TFs) for embedding and selection, whereas LESS requires around 8400 TFs—approximately 10x more.

We thank the reviewer again for this insightful suggestion and will include these results in the main body of the next revision.

[1] https://arxiv.org/pdf/2402.04333

We thank the reviewer for their comments. Below please find our response to the issues raised:

1. Unconstrained weights being irregular

The reviewer’s observation is indeed correct, and we also observe this in the “Discussion” paragraph around line 157 of the paper. In response to this issue, we have introduced a regularization scheme for the influence weights that (1) enforces the weights to sum to the size of the source dataset, (2) constrains them to be non-negative, and (3) applies L2 regularization to prevent overly large values. This leads to what we refer to as “Robust Weights” (introduced around line 166), which effectively address the concerns raised by the reviewer. An example of the resulting weight distribution is shown in Figure 2 (middle).

2. The cost of warm-up

We acknowledge this limitation, as noted in the Limitations section of our paper. However, we believe it is not a major concern for two reasons: (1) as the training pool grows, the cost of a brief warm-up on a small random subset becomes negligible compared to the cost of embedding the full dataset; and (2) the warm-up phase can be shortened (as demonstrated in Appendix A) or further compressed—for example, using Low-Rank Adaptation (LoRA) [1], as done in the LESS method [2]. A more thorough exploration of warm-up optimization is left for future work.

To provide a clearer picture of the current runtime, we estimate the cost of our (unnecessarily long) warm-up phase. In the setting of Table 1, the warm-up cost for Llama2-7B is approximately 420 TeraFLOPs (TF). Including this, the total runtime of Influence Distillation with 4096 landmarks is about 1292 TF, which is still 2.2x faster than the RDS+ baseline.

[1] https://arxiv.org/pdf/2106.09685

[2] https://arxiv.org/pdf/2402.04333

3. The cost of gradients and Hessian

We agree that the naive version of Influence Distillation would be prohibitively expensive due to the need to compute and store full gradients and Hessians. This is why Section 4 of our submission is dedicated to addressing exactly these challenges. Specifically: (1) we approximate gradients using a theoretically grounded landmark-based method, which enables efficient and accurate estimation of gradient similarities between source and target samples; and (2) we argue that explicit computation of Hessians is unnecessary, as the second-order term becomes negligible under practical learning rates. This significantly reduces the computational burden while maintaining performance, shifting the pareto frontier of runtime vs accuracy as shown in Figure 1.

We thank the reviewer for their constructive feedback. Below we address the raised concerns:

1. Discussion on the baselines and related work

We would like to emphasize that, compared to training on the full dataset, Influence Distillation offers significant computational savings. For instance, in the Llama2-7B experiments, full training on 200k examples requires approximately 8400 TeraFLOPs (TF), whereas Influence Distillation with 4096 landmarks requires only 1292 TF for embedding, selection, and training on the 10k subset. This results in a 6.5x speedup while recovering the accuracy of full training across the average of six tasks (see Table 1).

When compared to other baselines included in our paper, we argue that Influence Distillation is Pareto-superior—that is, it is both faster and at least as accurate on average across all tasks. This trade-off is illustrated in Figure 1.

Regarding a comparison with the LESS method, please see the next point below

We appreciate the reviewer’s suggestion and thank them for highlighting these related works. The two additional methods (AlpaGasus and DS2) involve evaluating each sample in the pool using one or more large-scale language models, such as ChatGPT, which results in substantially higher computational cost due to the size of the models involved. Moreover, these methods are generally model-agnostic and do not offer the same level of theoretical grounding as our approach.

Finally, we thank the reviewer for this valuable comment. We will incorporate the new results and include a discussion of AlpaGasus and DS2 in the next revision of the paper.

2. Comparison with LESS [1]

We thank the reviewer for this valuable comment. As noted in the last paragraph of Appendix F (also mentioned in Section 5.1, Baselines), we have included a brief discussion of the LESS method. Specifically, as an ablation, we replaced JVP embeddings with actual projected gradients. Figure 7 (right) in the appendix shows that this yields a gradient recovery of 0.9 (in terms of average cosine similarity) with 4096 landmarks, making the approach, in this case, very similar to LESS which uses exact (projected) gradients. The last row of Table 2 shows that this variant performs on par with Influence Distillation using JVP embeddings.

To provide a more complete comparison, we also ran the 10k/200k selection experiment on Llama2-7B using the LESS method (with one seed due to its high computational cost). In this setup, gradients are computed after a 10k-step warmup, projected down to 8192 dimensions, and compared to the target gradients. The top 10k examples are then selected based on similarity—following the original LESS procedure. Below we present a comparison between uniform sampling, InfDist with JVP embeddings, and LESS.

While LESS achieves higher accuracy on average, it is significantly more computationally expensive. Our method requires approximately 872 TeraFLOPs (TFs) for embedding and selection, whereas LESS requires around 8400 TFs—approximately 10x more.

We thank the reviewer again for this insightful suggestion and will include these results in the main body of the next revision.

[1] https://arxiv.org/pdf/2402.04333

3. Discussion on approximations

We would like to thank the reviewer for this insightful comment. As suggested, we first include an ablation measuring the similarity between approximated and non-approximated influence weights.

We note that computing the similarity between all pairs of source and target gradients without projection is computationally intractable for our data scale. Specifically, it requires $O(|S| \times |T|)$ gradient computations, as we can only store a constant number of gradients in memory. Additionally, given the high dimensionality, the source data pool must be large to yield meaningful similarities. Therefore, even for the non-approximated weights, we apply random projections to reduce the gradient dimensionality to 8192, using Rademacher projections (as in LESS).

For this experiment, we consider the Llama2-7B model, the 200k Tulu V2 training dataset, 32 random MMLU target samples, and 4096 landmarks. We compute both the approximate Influence Distillation weights and the "true" second-order influence weights (with both gradients and Hessian-vector products projected to 8192 dimensions), with learning rate 2e-5, which we used for our fine-tuning experiments. We find that the two sets of weights have a Pearson correlation of around 0.7, which aligns with the theory presented in Theorem 1: while the low-rank gradient approximation may yield a weak recovery of individual gradients, their similarities with target gradients are preserved due to the high dimensional structure.

As for the observation that using weights during training does not improve performance, we believe this is largely due to the regularization tuning procedure. As mentioned in the paper, we choose the regularization strength $\lambda$ such that the final solution has exactly 10k non-zero weights. In practice, many of these weights are small. In the experiment described above, approximately 20% of selected samples had weights less than 5, while another 20% had weights larger than 30, with some samples with weights as high as 160. When training with these weights, the low-weight samples contribute very little and are effectively ignored, reducing the advantage of weighting. A potential alternative would be to allow a larger number of non-zero weights and then select the top-k most influential samples, but we leave this to future work.

We thank the reviewer again for raising this important point and will include these new findings and discussions in the next revision of the paper.

4. Impact of landmark selection

We repeat the experiments from the appendix Figure 7 (Left) with different landmark selection methods. Specifically, we compute the average cosine similarity of the landmark-based approximated gradients with true gradients when using JVP embeddings. We fix the number of landmarks to 4096, and compare the following landmark selection methods: (1) uniformly at random, (2) leverage score-based sampling in the JVP embeddings space, (3) longest samples, and (4) shortest samples. The results are presented in the following table (the model is Llama2-7B):

These results show that deliberately choosing longest or shortest samples significantly degrades gradient approximation quality. On the other hand, choosing landmarks uniformly at random is on par with leverage score-based sampling, as mentioned in the Hyperparameters paragraph in Section 5.1.

We thank the reviewer for this feedback, and will include this study in the next revision.

5. Performance degradation when #landmarks=8192

We acknowledge that the reviewer’s observation is correct: as the number of landmarks increases, the quality of gradient recovery should improve, and consequently, the performance of Influence Distillation is expected to increase. This trend is indeed reflected in the $*general*$ behavior shown in Figure 3.

However, as noted in Table 1, certain tasks—such as GSM8k and Codex—show a high variability and are particularly noisy. We believe the slight performance drop observed at 8192 landmarks is likely due to this noise.

6. Transferability of embeddings between models

To investigate this point, we conducted an experiment to study how the average cosine similarity between approximated and true gradients of Llama2-7B is affected by the choice of embedding model. Specifically, we compare the performance of the following embeddings: JVP from Llama2-7B (same as the target model), JVP from Llama3.2-3B, JVP from Qwen2.5-1.5B, and JVP from Qwen2.5-3B. We fix the number of landmarks at 4096. The results are summarized in the following table:

Comparing these results with the recovery achieved by RDS+, we conclude that JVP embeddings are highly transferable across models. Notably, JVP embeddings computed from Qwen2.5-3B approximate the gradients of Llama2-7B with similar quality to those obtained directly from Llama2-7B itself.

We appreciate the reviewer’s constructive comment and will include these results and a discussion in the next revision of the paper.

7. Performance of raw base models

We thank the reviewer for the suggestion and will include these results for all models in the next revision. For the purpose of this rebuttal, we have computed the results for Llama2-7B, which are presented in the table below:

As expected, without instruction tuning, the base model performs poorly on all tasks.

8. Typo: fine-tuning

We appreciate the reviewer’s attention to detail and will correct this typo in the next revision.

We would like to thank the reviewer for their valuable comments. Please find the answers to your concerns below.

1. Extension to pre-training and the no-target case

Our current focus is indeed on the supervised fine-tuning phase, rather than pre-training or reinforcement learning. This choice reflects our goal of improving training efficiency in realistic fine-tuning settings, where large amounts of instruction-tuning data are available and training cost is a bottleneck.

That said, we believe that several components of our method are applicable to pre-training scenarios as well:

Gradient approximation via JVP embeddings remains effective in pre-training after a short warmup period, as the gradients have been shown to exhibit low-rank structure after this initial phase [3]. This suggests that the embedding-based similarity approach could be extended to the pre-training stage with minimal overhead.

In cases where there is no explicit target distribution, there are two possible adaptations worth exploring:

(i) One could set the source and target sets to be the same (i.e., $S=T$). In this case, both sets share a common gradient matrix $G$, which can be written as $G=CL$ (similar to the paper). This allows for efficient computation of pairwise gradient similarities as $G^TG=L^TC^TCL$. This structure could enable selection of influential samples using, for example, leverage score-based sampling.

(ii) Alternatively, one could define a small, high-quality subset of examples as the target distribution and select pre-training data based on their influence relative to that set.

While these directions are promising, they would require additional experiments to fully validate. We view this as an exciting avenue for future work and thank the reviewer for this suggestion.

[3] https://arxiv.org/abs/2403.03507

2. Transferability of weights

We agree that this is an interesting experiment. To investigate this, we compute sample weights for different target sets as described in the paper, and analyze their pairwise Pearson correlations. For this analysis, we use the Llama2-7B model with 1024 landmarks and a data pool of 200k examples. As shown in the table below, the computed weights for four of the six tasks exhibit high correlation (0.5–0.8), suggesting that sample weights computed for one task may transfer well to others.

To further assess cross-task generalization, we use the sample weights computed for MMLU to select a 10k subset from the training pool. We then train a model on this subset and evaluate it on each of the remaining tasks. The results are presented in the table below, comparing Influence Distillation and uniform sampling (the results are averaged over three seeds):

These results show that, even though the samples were selected for MMLU, Influence Distillation still outperforms uniform selection in most tasks when evaluated for three seeds.

3. Study on size of T

To study the effect of the target set size, we vary the number of samples per category in the MMLU target distribution. MMLU contains 57 categories, and in the main paper we use 5 samples per category (285 samples in total) as the target set. For this ablation, we repeat the Llama2-7B experiment using 1, 2, 3, and 4 samples per category, resulting in target set sizes of 57, 114, 171, and 228, respectively.

These results suggest that Influence Distillation is relatively robust to the size of the target dataset in this setting. We thank the reviewer for this valuable comment and will include these findings—along with similar analyses for other target datasets—in the next revision of the paper.

4. Divergence of Target and Source

In this case, it intuitively would be hard to perform well on the target distribution for which we don’t have any data. We believe that this would be a limitation of all data selection methods.

We thank the reviewer for their insightful feedback. Here are our answers to their questions.

1. Comparison with LESS [1]

We thank the reviewer for this valuable comment. As noted in the last paragraph of Appendix F (also mentioned in Section 5.1, Baselines), we have included a brief discussion of the LESS method. Specifically, as an ablation, we replaced JVP embeddings with actual projected gradients. Figure 7 (right) in the appendix shows that this yields a gradient recovery of 0.9 (in terms of average cosine similarity) with 4096 landmarks, making the approach, in this case, very similar to LESS which uses exact (projected) gradients. The last row of Table 2 shows that this variant performs on par with Influence Distillation using JVP embeddings.

To provide a more complete comparison, we also ran the 10k/200k selection experiment on Llama2-7B using the LESS method (with one seed due to its high computational cost). In this setup, gradients are computed after a 10k-step warmup, projected down to 8192 dimensions, and compared to the target gradients. The top 10k examples are then selected based on similarity—following the original LESS procedure. Below we present a comparison between uniform sampling, InfDist with JVP embeddings, and LESS.

While LESS achieves higher accuracy on average, it is significantly more computationally expensive. Our method requires approximately 872 TeraFLOPs (TFs) for embedding and selection, whereas LESS requires around 8400 TFs—approximately 10x more.

We thank the reviewer again for this insightful suggestion and will include these results in the main body of the next revision.

[1] https://arxiv.org/pdf/2402.04333

2. Accuracy throughout training and the size of the target set

We would like to note that our 10k example selection setting follows the setting introduced in [2], where 10k examples were selected from the same dataset.

To be complete, we repeated the Llama2-7B experiment on MMLU using Influence Distillation, Uniform sampling, and RDS+. In this run (with a single seed), we saved model checkpoints every 25 steps to better track performance over time. Below, we report MMLU accuracies throughout training:

We also repeated the experiment using full training on the entire dataset. Our results show that full training reaches an accuracy of 48.5 only around step 1100. In contrast, Influence Distillation achieves the same accuracy at step 150 – over 7x faster.

We thank the reviewer for this insightful comment. We will include these results in the next revision.

[2] https://arxiv.org/pdf/2503.01807

3. Placement of proofs in the main body

Thank you for the suggestion regarding the placement of proofs. Upon review, we note that the main text does not currently include full proofs; the closest instance is the four-line Taylor expansion leading to Equation 2. We agree that this would be better suited to the appendix and will make this adjustment in the next revision.

Thank you to the authors for their detailed responses. The additional experimental results further demonstrate the effectiveness of the proposed Influence Distillation (InfDist) framework. However, although InfDist requires significantly lower computational cost than LESS, it does not outperform LESS. Nonetheless, I believe InfDist is a valuable contribution to the research community, and I would be pleased to see this paper accepted. Since a rating of 4 is already considered positive, I will maintain my score.