Outlier Gradient Analysis: Efficiently Identifying Detrimental Training Samples for Deep Learning Models

Dear Reviewer A4t7,

Thank you for your review, we are grateful for your feedback and insights. We discuss the concerns raised, below:

• Outlyingness of Sample Gradients:
  • The reviewer mentioned that some meaningful outliers may represent rare but valuable patterns within the training set and should not be excluded. While we agree with the reviewer's general sentiment, if some patterns occur in the training set but are not present in the validation set (i.e., distribution/concept shift), these patterns can be regarded as detrimental, and removing these samples should lead to performance improvement.
  • We would also like to clarify and emphasize that our method is not removing outlier samples (which at times can be beneficial to training), but outlier gradient samples (which we show can negatively affect model performance). Our approach is backed by the intuition from the influence function formulation of Eq. 1, which also utilizes sample gradient influence in determining whether the sample is beneficial or detrimental to training.
• Adapting to Validation Set: Thank you, we had undertaken experiments in the current version of the paper (Appendix C.8, Table 12) where we utilize and adapt the validation set for our outlier gradient analysis method for the same reasons suggested by the reviewer. Since we can utilize any outlier algorithm for outlier gradient analysis, we employ the OneClassSVM semi-supervised outlier algorithm to adapt it to the validation set (which provides supervision information) along with the training set. We employed the distribution shift experimental framework from [1] on the Folktables ACS-Income dataset where the training and test distribution are either time shifted, location shifted, or both time and location shifted. Our results indicate that our outlier analysis approach outperforms the other methods across all three distribution shift settings (please refer to Table 12 in Appendix for these results).
• Text-to-Image Tasks:
  • While we wanted to undertake these experiments, unfortunately, the DataInf authors [2] have not released their benchmarks for the text-to-image tasks using diffusion models (see codebase here [3]). It was fairly non-trivial for us to undertake this comparison as a number of custom subjects and data splits are used in these experiments and are not available at the moment. However, we will aim to add these experiments as the benchmarks are released (in future preprint versions of our paper) and work with the DataInf authors to obtain more information regarding these tasks.
  • We would also like to emphasize that the main advantage of our work is the high computational efficiency obtained via the equivalent transformation to outlier analysis of the gradient space. Empirically, our method can achieve better or similar performance compared to other baselines but much more efficiently (as our results also show).
• 2-D Gradient Features for MLP: Thank you, we can provide more information regarding this. Basically, for the simple 2-D half moons dataset, our MLP architecture is as follows: we first have a hidden linear 2x2 layer with two neurons (stacked with ReLU activation) whose output passes to another 2x2 hidden linear layer with two neurons (stacked with ReLU activation). Then, the output from this second layer goes to another linear 2x1 output layer with the sigmoid function applied for generating predictions. As this final layer has 2 parameters, we compute the sample gradient with respect to this output layer for ease of visualization as the gradient will be two-dimensional here too. Where visualization is not an issue, gradients from more layers can be combined/concatenated as well (as done in other experiments in our paper).

References:

1. Chhabra, Anshuman, et al. "What Data Benefits My Classifier?" Enhancing Model Performance and Interpretability through Influence-Based Data Selection." ICLR (2024).
2. Kwon, Yongchan, et al. "DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Models." ICLR (2024).
3. https://github.com/ykwon0407/DataInf

Dear Reviewer PgCg,

Thank you for your review, we appreciate your feedback and advice. Below we discuss the weaknesses listed:

• Writing: We apologize for any lack of clarity. Could we request the reviewer to provide us with instances where writing can be improved and we shall make those changes right away in the revision? We would like to take this opportunity to learn from the reviewer and any other excellent papers; thank you.
• Small-scale Datasets: Thank you for your point. For fair comparison and due to multiple experiment runs, as some influence functions are costly to compute (for instance, exact Hessian takes cubic time in the worst case), we resorted to standard datasets used in the literature for influence functions (i.e. those used in computationally efficient influence function work such as DataInf [1]). However, we also undertook experiments with a subset of ImageNet in the current version of our paper-- these are provided in Appendix C.6, Table 10. As can be seen from these results, outlier gradient analysis is tied as the top performer while being much more computationally efficient than the other approach achieving the same performance (i.e. DataInf). For an academic group, we have limited computational resources and are unable to afford repeated experiments on industry-scale datasets. We hope the reviewer will consider this point as well. In future work, we can aim to experiment with larger datasets wherever possible.
• Title and Paper Goal: We apologize for the lack of clarity. Our main contribution in this work lies in uncovering the fact that detrimental samples (for instance, noisily labeled samples) that negatively affect model performance appear as outliers in the gradient space. We do this by crafting a very general hypothesis (Hypothesis 3.2), by drawing inspiration from how influence functions estimate whether training samples are beneficial or detrimental. The general nature of Hypothesis 3.2 results in wide applicability, as the extensive experiments we undertake on vision models, NLP models, and LLMs show. Moreover, no other work has explicitly shown this potential link (via influence functions) between gradient outlyingness and sample benefit/detriment, although this has been implicitly observed in some past work [2,3]. Note that any outlier algorithms can be used to detect the outlying gradients (we use 3 approaches in this work). This contribution and aim forms the basis for our title. However, based on the above explanation if the reviewer would like us to make any adjustments to the title for clarity, we are happy to do so.
• Gradients and Hessian: Thank you for the question! We would like to clarify that our work is not replacing the Hessian with gradient terms. However, there are a few reasons why our approach is able to attain performance gains in detecting detrimental training samples for deep learning models without relying on the Hessian (and only using gradient terms):
  • First, our work is grounded in only the detrimental data detection task, and is not a general-purpose influence function. However, for the detrimental data identification task, we know that detrimental samples are fewer than beneficial samples (due to ERM). Moreover, in the general influence function formulation (Eq. 1) we know that for each sample $z_j$ whose influence is being estimated, the Hessian term $H_{\theta}^{-1}$ remains the same and only the third term $\nabla_{\theta} \ell(z_j)$ changes with $z_j$. Therefore, the effect of being a detrimental sample should show up in the gradient space and as there are only a minority of such samples, they should appear as outliers. In this scenario, the Hessian does not contribute meaningful information (i.e. curvature) and the gradient space itself provides a good estimate for detrimental sample detection.
  • Second, in the original influence function formulation (Eq. 1), inverting the Hessian $H_{\theta}^{-1}$ implies that the loss function is strictly convex. This does not hold for deep learning models which are non-convex and here, the Hessian can lead to inaccurate estimates. However, the gradient terms are still valid in the non-convex case, so utilizing them can lead to accurate estimations, especially when the problem setting is restricted (such as detrimental sample detection).

References:

1. Kwon, Yongchan, et al. "DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Models." ICLR (2024).
2. Kim, SungYub, Kyungsu Kim, and Eunho Yang. "GEX: A flexible method for approximating influence via Geometric Ensemble." NeurIPS (2024).
3. Bejan, Irina, Artem Sokolov, and Katja Filippova. "Make Every Example Count: On the Stability and Utility of Self-Influence for Learning from Noisy NLP Datasets." EMNLP (2023).

Dear Reviewer NLkn,

Thank you for your insights and detailed review, we appreciate it. Below we discuss some of the concerns raised:

• Discussion on Hessian-free methods vs Outlier Gradient: Thank you for pointing this out. We will add more details about Gradient Tracing (also known as TracIn) [1] as well as other Hessian-free methods and compare with our approach in the revision. Essentially at a high-level, Gradient Tracing simply takes the inner product of the first and third gradient terms in the influence function formulation, basically removing the Hessian from the influence function of Eq. 1, that is, $\langle \sum_{v \in V}\nabla_{\theta} \ell(v), \nabla_{\theta} \ell(z_j) \rangle$. On the other hand, outlier gradient analysis employs an outlier detection algorithm $\mathcal{A}$ to operate on the gradient space of samples to detect detrimental samples, i.e. $\mathcal{A}(\sum_{v \in V}\nabla_{\theta} \ell(v), \nabla_{\theta} \ell(z_j))$. Our formulation is more general as a result.
• Other Datasets with Real-world Noise: Thank you for the concrete suggestion for improvement. Despite significant computational overhead, we had run experiments with ImageNet in Appendix C.6, but constitute the same setting of artificial label noise (i.e. just flipping the sample labels to introduce noise). Here too, (as Table 10 in Appendix C.6 shows), outlier gradient analysis is one of the top-performing baselines. Moreover, we would also like to mention that the CIFAR-10N and CIFAR-100N datasets [2] are not artificial label noise datasets, as the labels for these were obtained by multiple crowdsourced workers via Amazon Mechanical Turk. Hence, this is actually mimicking a scenario with real-world label noise introduced via human annotation.

References:

1. Pruthi, Garima, et al. "Estimating training data influence by tracing gradient descent." NeurIPS (2020).
2. Wei, Jiaheng, et al. "Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations." ICLR (2022).

Dear Reviewer yKaF,

Thank you for your detailed review, we really appreciate your feedback and insights. With regards to the concerns raised, we provide some counter points of discussion:

• Re-interpretation of existing methods:

  • Our main contribution in this work lies in uncovering the fact that detrimental samples (for instance, noisily labeled samples) that negatively affect model performance appear as outliers in the gradient space. We do this by crafting a very general hypothesis (Hypothesis 3.2), by drawing inspiration from how influence functions estimate whether training samples are beneficial or detrimental. Owing to the general nature of Hypothesis 3.2, it is non-trivial to prove this for all classes of learning models. However, this results in wide applicability, as the extensive experiments we undertake on vision models, NLP models, and LLMs show. Moreover, no other work has explicitly shown this potential link (via influence functions) between gradient outlyingness and sample benefit/detriment, although this has been implicitly observed in some past work [1,2]. While any outlier algorithms can be used to detect the outlying gradients (we use 3 approaches in this work), we do not feel that this contribution constitutes a re-interpretation of existing outlier analysis approaches as our focus is entirely different.
  • The reviewer also mentioned that shedding light on why and how the first-order gradient is enough to anticipate the influence of training samples would be a much more significant contribution. However, this is equivalent to the question "why are influence functions suitable for detecting detrimental samples?" as even the original influence function formulation is based on the first-order Taylor expansion to estimate model performance without retraining. For complex models, higher-order extensions might provide more accurate estimation; however, they also require more computational effort. For practical usage, first-order information is used. In our paper, we provide a re-interpretation of influence function for detrimental data detection, which significantly reduces the computational cost for large models.

• First-order vs Higher-order gradients: Thank you for the question-- there are a few cases where first-order estimates can be better than the Hessian containing second-order derivatives of the loss function.

  • First, we would like to emphasize that our work is grounded in only the detrimental data detection task, and is not a general-purpose influence function. However, for the detrimental data identification task, we know that detrimental samples are fewer than beneficial samples (due to ERM). Moreover, in the general influence function formulation (Eq. 1) we know that for each sample $z_j$ whose influence is being estimated, the Hessian term $H_{\theta}^{-1}$ remains the same and only the third term $\nabla_{\theta} \ell(z_j)$ changes with $z_j$. Therefore, the effect of being a detrimental sample should show up in the gradient space and as there are only a minority of such samples, they should appear as outliers. In this scenario, the Hessian does not contribute meaningful information (i.e. curvature) and the gradient space itself provides a good estimate for detrimental sample detection.
  • Second, in the original influence function formulation (Eq. 1), inverting the Hessian $H_{\theta}^{-1}$ implies that the loss function is strictly convex. This does not hold for deep learning models which are non-convex and here, the Hessian can lead to inaccurate estimates. However, the gradient terms are still valid in the non-convex case, so utilizing them can lead to accurate estimations, especially when the problem setting is restricted (such as detrimental sample detection).

• Other LLM tasks: Thank you again for the great suggestion. Unfortunately, data valuation for LLMs is an evolving research area, and currently the field is lacking in exhaustive benchmarks. Due to the general generative nature of LLMs, it is non-trivial to assess influence of samples currently without labeled influential prompt identification benchmarks. This is why we utilize the LLM benchmarks from prior work [3] for this paper, however, we plan to extend analysis to more general downstream tasks for future work in this domain.

• Covariate Shifts: Thank you for the excellent point. We had undertaken experiments for covariate shift in the current version of the paper (Appendix C.8, Table 12) by employing the distribution shift experimental framework from [4] where the training and test distribution are either time shifted, location shifted, or both time + location shifted. Here, we utilized the semi-supervised OneClassSVM outlier detection algorithm (to adapt to the validation set) for outlier gradient analysis on the Folktables ACS-Income dataset and found that our approach outperformed the other methods across all three covariate shifts (please refer to Table 12).