Dear Reviewer MZJ6,

Thank you for your efforts in reviewing our work, we are grateful for your insights. We provide answers to the questions raised, below:

• Although this work conducted experiments on LLMs, I find it strange that they only used LLMs for classification tasks. It would make more sense to experiment with LLMs' generation tasks. This is my biggest concern, and if the authors can address this concern, I would be happy to raise the score.

  Thank you. We would like to point out that the current LLM benchmarks are not classification tasks, but generation tasks. Each of the 3 influential data identification tasks (Math With Reasoning, Math Without Reasoning, and Sentence Transformations) we considered from past work (Kwon et al, 2024) have 10 classes/categories of subtasks (e.g. Sentence Transformation can have 10 different types of natural language transformations and Math problems can have 10 different categories of word problems) but are still generation tasks. More details regarding these benchmarks are provided in Appendix B.1.4 and B.1.5. In general, we agree with the reviewer that one of the challenges associated with influential data identification in LLMs is the lack of more complex and varied benchmarks. This is currently an evolving field, and designing benchmarks is challenging because ground-truth influence labels need to accurately reflect the model's inductive bias for a test sample (i.e. the model should find the training samples most influential for a particular test example). Past work does this by making the train-test problem sub-categories very similar. We are now working on designing better LLM influence identification benchmarks for future work, and hope to test our outlier gradient analysis methods on these benchmarks as well.

---

• Sensitivity to outlier detection algorithm. The performance of the method depends on the choice of outlier detection algorithm and its hyperparameters. While the paper explores different options (iForest, L1/L2-norm), a more systematic analysis of this dependency would be valuable.

  Thank you for the great question.

  Regarding outlier analysis algorithms, in the paper, we have utilized 4 outlier analysis algorithms: iForest (main paper), L1 norm (main paper), L2 norm (main paper), and OneClassSVM (Appendix C.9) for outlier gradient analysis. For all 4 of these algorithms, the general trend of outlier gradient analysis improving model performance against competitive baselines in the noisy learning regime can be observed. Our primary aim in choosing these algorithms was their high computational efficiency and minimal number of hyperparameters.

  Regarding hyperparameter sensitivity, we had provided details and additional experiments on hyperparameters in Appendix C, which we also discuss below. More specifically, our outlier analysis algorithm has two hyperparameters: (1) the trimming budget $k$ which is the number of samples to remove, and (2) the hyperparameters of the outlier detection algorithm being used (e.g. iForest).

  1. For the first hyperparameter of the trimming budget $k$, we conduct additional experiments while varying the value of $k$ from 2.5% to 12.5% for all 4 noise settings of CIFAR-10N. These results are provided in Table 5 of Appendix C.2. As can be seen, the highest values across each noise regime are obtained by outlier gradient analysis (L2 norm thresholding at 12.5% for Aggregate and Random; and L2 norm thresholding at 2.5% for Worst), indicating its broad suitability. These results also show that the budget of 5% is a good choice for the trimming budget, leading to desirable performance in most cases.

  2. The second hyperparameter choice is only a facet of the iForest outlier analysis algorithm and constitutes the number of trees being used in the algorithm. Note that this is because for the L1 and L2 norm thresholding approaches, we do not have a norm threshold that needs to be chosen manually, since setting the budget automatically decides the threshold. For the iForest algorithm, we have provided results for varying the number of tree estimators on CIFAR-10N in Table 8 of Appendix C.4. As can be observed, the performance of outlier gradient analysis remains stable across the board when the number of trees/estimators are varied, indicating low sensitivity of this hyperparameter to final results obtained.

---

Thank you once again for helping improve our paper, we appreciate it.

Dear Reviewer r7gY,

Thank you for your thoughtful review and feedback, we appreciate it. We have answered questions raised, below:

• The paper would be in stronger position if it included comparisons with: sample selection methods for learning with noisy labels and data pruning for training efficiency

  Thank you for the great suggestion. Along with comparisons with other influence function baselines and simpler noisy label correction methods, we had compared with a number of sample selection methods designed for noisy learning in Appendix C.7 (Table 11). Approaches for noisy learning can be categorized into (1) methods that either change the loss function or model architecture or (2) those that identify noisy samples and remove/relabel them for improving model performance (Algan & Ulusoy, 2021). While the main paper has results for the latter category, we compare with the former category on CIFAR-10N (all 3 noise settings) in Table 11 of Appendix C.7. As can be observed, outlier gradient analysis is the top performer across these methods as well. While we had thought of also comparing with training data efficiency methods, we could not undertake a fair comparison as several methods opt for reducing the size of the training set as much as possible while ensuring that performance on the reduced set remains as close to the original set. However, the goal in our work is to specify a small trimming budget and increase performance as much as possible, leading to the research questions between these approaches being fundamentally different.

---

• For the LLM influential data identification task, can iForest estimators scale to different tasks with many classes?

  Currently, each of the 3 influential data identification tasks (Math With Reasoning, Math Without Reasoning, and Sentence Transformations) we considered from past work (Kwon et al, 2024) each have 10 classes/categories of subtasks (e.g. Sentence Transformation can have 10 different types of natural language transformations and Math problems can have 10 different categories of word problems). More details regarding these benchmarks are provided in Appendix B.1.4 and B.1.5. While the reviewer makes a great suggestion of scaling to even more classes, one of the challenges associated with influential data identification in LLMs is the lack of more complex and varied benchmarks. This is currently an evolving field, and designing benchmarks is challenging because ground-truth influence labels need to accurately reflect the model's inductive bias for a test sample (i.e. the model should find the training samples most influential for a particular test example). Past work does this by making the train-test problem categories very similar. We are now working on designing better LLM influence identification benchmarks for future work, and hope to test our outlier gradient analysis methods on these benchmarks as well.

---

• Have you explored using other outlier detection algorithms beyond iForest and L1/L2 norms?

  In the paper, we have utilized 4 outlier analysis algorithms: iForest (main paper), L1 norm (main paper), L2 norm (main paper), and OneClassSVM (Appendix C.9) for outlier gradient analysis. For all 4 of these algorithms, the general trend of outlier gradient analysis improving model performance against competitive baselines in the noisy learning regime can be observed. Our primary aim in choosing these algorithms was their high computational efficiency and minimal number of hyperparameters.

---

• The approach currently requires computing gradients for all training samples. Why not consider approximation or sampling technique?

  The reason we did not opt for approximating or sampling gradients is the ease with which they are available for model deep learning models trained via backpropagation. Basically, we can obtain gradients in one-pass as the model is training post each backpropagation step. Owing to the ease of gradient access, we did not optimize this further via approximation. Moreover, approximation/sampling would lead to some reduction in performance as opposed to using the original first-order gradients. As an aside, the Hessian is not available during training since it contains second order information.

---

• Moving table of runtime evaluation to the main text and minor typos:

  Thank you for pointing these out. We will incorporate these suggestions into the revision as requested.

---

Thank you once again for all your time and effort in reviewing our work, and helping strengthen our contributions.

Dear Reviewer 91NL,

Thank you for your insightful review and feedback. We answer the questions raised, below:

• Q1: Could you elaborate on the computational specifics of Outlier Gradient (L1) and Outlier Gradient (L2)?

  Thank you for your question. Our outlier gradient approach (Algorithm 1) takes in as input a trimming budget $k$ and an outlier analysis algorithm $\mathcal{A}$. Here, $\mathcal{A}$ can be any outlier analysis algorithm, including L1 and L2 norm thresholding. For these two algorithms, we compute the gradient of the training samples, and then simply select the top-$k$% (i.e. based on budget) norm values as outliers. For L1 (or L2) norm thresholding, the L1 (or L2) norm values need to be computed, but this is a computationally efficient simple tensor operation.

---

• Q2: Why is there an additional summation symbol in Equation 1?

  The additional summation symbol is simply aggregating the loss for each of the validation/training set samples (i.e. samples from either $V$ or $T$) on which training sample $z_j$'s influence is being measured. As the loss is additive, the derivative of the loss is also additive, and hence, can be summed over to compute the overall loss contributions made by the individual training sample ($z_j$) we are computing influence for. This summation over the loss is used in past work on influence function for model performance measurement, such as [1,2], among others.

---

• Q3: Avoiding the computation of the Hessian matrix can accelerate calculations, but it is strange that only calculating the third term of Equation 1 improves accuracy

  Thank you for the great question. One intuitive reason for this (that also past work has found) is that the Hessian is not mandatory for influence analysis in all cases. While (Koh and Liang, 2017) pioneered influence functions based on the Hessian, other work has considered influence functions without relying on the Hessian such as TracIn [3], TracIn-Last [4], VAE-TracIn [5], BoostIn [6], Hydra [7], etc. Note that TracIn here is the Gradient Tracing baseline we compare with in the paper as well. TracIn and its variant approaches are simply calculating a vector inner product on the gradient space without computing the Hessian. Our outlier gradient analysis approach undertakes the Hessian-free in a novel manner by discovering detrimental training samples using the outlyingness of the gradient terms.

---

• The key hypothesis should be validated on real datasets:

  For real-world datasets (with a large number of classes) and large models, visualizing the gradient space (which will be very high-dimensional) requires undertaking aggressive approximations and is not possible in 2D/3D without losing useful information that describes outlyingness. Thus, we visualized the gradient space for simpler models and synthetic datasets as these are controllable and allow us to demonstrate our hypothesis, while real-world datasets could be affected by unknown factors. However, we would like to emphasize that the success of our methods on downstream real-world datasets (all the experiments in our paper) showcases the benefits and efficacy of outlier gradient analysis (thereby validating it) on real-world datasets and large models as well.

---

• The article lacks a discussion on the limitations of the methodology:

  Thank you for the suggestion. We can definitely aim to add more discussion on the limitations of our approach in the main paper. While outlier gradient analysis is useful in cases where training data can be noisy, it might not be as useful if the data is already very high quality and there are no outlying gradient samples. However, this might not be the case in the real-world unless some steps have already been taken to ensure high data quality. Furthermore, outlier analysis algorithms have a fundamental limitation of how to specify the budget for outlier detection, which is a non-trivial hyperparameter optimization problem. While this is a common problem with little consensus across the entire field of outlier analysis, our methods inherit this limitation as well (although our methods work well for different budget thresholds, as shown in additional experiments in Appendix C.2).

---

Thank you once again for your help in strengthening our paper, we appreciate it.

---

References:

1. Understanding black-box predictions via influence functions. ICML 2017.
2. DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Models. ICLR 2024.
3. Estimating training data influence by tracing gradient descent. NeurIPS 2020.
4. First is better than last for language data influence. NeurIPS 2022.
5. Understanding instance-based interpretability of variational auto-encoders. NeurIPS 2021.
6. Adapting and evaluating influence-estimation methods for gradient-boosted decision trees. JMLR 2023.
7. Hydra: Hypergradient data relevance analysis for interpreting deep neural networks. AAAI 2021.

Dear Reviewer p9JH,

Thank you for your insightful review, comments, and suggestions. We answer the questions raised, below:

• Essential References Not Discussed:

  Thank you for providing these references on gradient-based outlier/anomaly detection, we appreciate it. As the reviewer correctly points out, these are closely related to our work in a technical sense (but explore different research questions), and we will be sure to include and discuss these in the revision.

---

• Lack of Discussion on Failure Cases:

  We can definitely aim to add more discussion on the limitations of our approach in the main paper. While outlier gradient analysis is useful in cases where training data can be noisy, it might not be as useful if the data is already very high quality and there are no outlying gradient samples. However, this might not be the case in the real-world unless some steps have already been taken to ensure high data quality. Furthermore, outlier analysis algorithms have a fundamental limitation of how to specify the budget for outlier detection, which is a non-trivial hyperparameter optimization problem. While this is a common problem with little consensus across the entire field of outlier analysis, our methods inherit this limitation as well (although our methods work well for different budget thresholds, as shown in additional experiments in the Appendix C.2).

---

• Further Analysis of Outlier Detection Algorithms:

  Thank you for this suggestion. In our paper, we have utilized 4 outlier analysis algorithms: iForest, L1 norm thresholding, L2 norm thresholding, and OneClassSVM (Appendix C.9) for outlier gradient analysis. For all 4 of these algorithms, the general trend of outlier gradient analysis improving model performance against competitive baselines in the noisy learning regime can be observed. Also, note that our primary aim in choosing these algorithms was their high computational efficiency and minimal number of hyperparameters. More complex approaches generally tend to be slower (e.g. those based on deep learning) and thus, wouldn't be as useful for outlier analysis of the gradient space. However, we are happy to include any other useful outlier detection methods that the reviewer would like us to.

---

• How sensitive is your method to the choice of hyperparameters, such as the number of trees in iForest or the threshold for L1/L2 norm methods? Please provide more guidance on how to choose appropriate hyperparameter values for different datasets and model architectures. A discussion on the robustness of the method to hyperparameter settings would be valuable.

  Thank you for the great question. We had provided details and additional experiments on hyperparameters in Appendix C, which we also discuss below. More specifically, our outlier analysis algorithm has two hyperparameters: (1) the trimming budget $k$ which is the number of samples to remove, and (2) the hyperparameters of the outlier detection algorithm being used (e.g. iForest).

  1. For the first hyperparameter of the trimming budget $k$, we conduct additional experiments while varying the value of $k$ from 2.5% to 12.5% for all 4 noise settings of CIFAR-10N. These results are provided in Table 5 of Appendix C.2. As can be seen, the highest values across each noise regime are obtained by outlier gradient analysis (L2 norm thresholding at 12.5% for Aggregate and Random; and L2 norm thresholding at 2.5% for Worst), indicating its broad suitability. These results also show that the budget of 5% is a good choice for the trimming budget, leading to desirable performance in most cases.

  2. The second hyperparameter choice is only a facet of the iForest outlier analysis algorithm and constitutes the number of trees (as the reviewer correctly pointed out). Note that this is because for the L1 and L2 norm thresholding approaches, we do not have a norm threshold that needs to be chosen manually, since setting the budget automatically decides the threshold. For the iForest algorithm, we have provided results for varying the number of tree estimators on CIFAR-10N in Table 8 of Appendix C.4. As can be observed, the performance of outlier gradient analysis remains stable across the board when the number of trees/estimators are varied, indicating low sensitivity of this hyperparameter to final results obtained.

---

We would like to thank the reviewer again for all the time and effort spent on the review, and helping improve our work.