Memorization Through the Lens of Curvature of Loss Function Around Samples

We thank the reviewer for their time and consideration, and answer the questions here.

1. Scalability: To capture the curvature of training samples in every epoch, additional backward passes are required(n=10 for these datasets), which requires an additional 10X the training time to capture the metric proposed. It can be partially circumvented by calculating the curvature per sample every few epochs, but can still continue to be computationally expensive.

Authors: While it is true that we need upto n=10 extra forward and backward passes to calculate curvature as shown in the paper, it is still far more efficient than the baseline for calculating curvature scores, Feldman&Zhang, that requires training upto 20,000 models (equivalent to n=20,000 for us). Additionally, we show through our hyperparameter runs in the next answer, that we can use a smaller n (upto n=5) to get reliable results. We also show from our ImageNet model, that we can calculate curvature every 4 epochs, resulting in the extra computation reducing to just about 1x (double the standard runtime). Further, we perform analysis in Section 4 to show the earlier epochs give a much higher match with cosine similarity score, and resource constrained users can use this to determine the best subset of epochs to perform curvature calculations on. We believe this allows us to scale our method, as evidenced by our ability to show results for ImageNet dataset without using massive GPUs.

2. Limited to memorization: The baseline by Feldman & Zhang calculates both the memorization scores for the training examples, and influence scores for test examples. There is no natural extension of curvature as a metric to influence scores.

Authors: We thank the reviewer for helping us think in this direction. We feel that using contrastive loss and calculating curvature on this loss might help us get influence scores. We leave this exploration for future work, but thank the reviewer for identifying another application of our method. We want to highlight that if we were to use contrastive loss, most methods mentioned in the prior work section would not be applicable (SSFT [Maini1], CL [NorthCutt1], Inconfidence [Carlini1], FZ [Feldman2], AUM [Pleiss1], Learning Time) since they require logits to work. However, since curvature only requires a loss, we would still be able to capture the scores.

Feldman2 - Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33:2881–2891, 2020.

Maini1 - Pratyush Maini, Saurabh Garg, Zachary Lipton, and J Zico Kolter. Characterizing datapoints via second-split forgetting. Advances in Neural Information Processing Systems, 35:30044–30057, 2022.

NorthCutt1 - Curtis Northcutt, Lu Jiang, and Isaac Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 70:1373–1411, 2021a.

Carlini1 - Nicholas Carlini, Ulfar Erlingsson, and Nicolas Papernot. Distribution density, tails, and outliers in machine learning: Metrics and applications. arXiv preprint arXiv:1910.13427, 2019a.

Pleiss1 - Geoff Pleiss, Tianyi Zhang, Ethan Elenberg, and Kilian Q Weinberger. Identifying mislabeled data using the area under the margin ranking. Advances in Neural Information Processing Systems, 33:17044–17056, 2020

Response to Reviewer: Part 1/2

We thank the reviewer for their time and consideration. We would like to address the overall review here, and then go into the details of each question after that.
 

We want to emphasize that in this paper we wanted to introduce the idea of measuring memorization with input loss curvature. We believe that extreme memorization and overfitting is one of the most significant insights into deep neural networks, as evidenced by the extensive work done in references [Zhang1, Feldman1, Feldman2]. We establish curvature as a reliable and efficient metric to measure memorization, and show both quantitative and qualitative corroborations of the reliability of curvature as a memorization metric.
 

For practical applications, we show three in the paper which we believe are quite significant. One is detecting mislabeled samples, and that is an area of significant research [Maini1, Northcutt1, Carlini1]. Secondly, we show an as of yet unobserved failure mode on CIFAR100 and ImageNet, two of the most widely used datasets to test different algorithms for vision algorithms, that of duplicate samples with differing labels. This shows how curvature can be used to find failure modes especially for datasets used in the industry that may not have been vetted by many research works. Thirdly, we also note the outliers in FashionMNIST dataset show a bias towards high contrast clothes, despite them being not more prototypical of clothing in real life. This highlights that our analysis can reveal biases present in the dataset.
 

There are many downstream applications of being able to measure and study memorization, and the reviewer lists some that we are currently working on as extensions. However, this work forms the bases of all those future explorations, which we and hopefully the community will pursue. We feel that this manuscript is self-contained in its claim, justification and example applications, and hope that the reviewer would reconsider the paper on its own merit, instead of the potential applications that can exist as a result of this manuscript.
 

References:
Zhang1 - Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. “Understanding deep learning requires rethinking generalization”. In ICLR 2017,

Feldman1 - Vitaly Feldman. Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pp. 954–959, 2020.

Feldman2 - Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33:2881–2891, 2020.

Maini1 - Pratyush Maini, Saurabh Garg, Zachary Lipton, and J Zico Kolter. Characterizing datapoints via second-split forgetting. Advances in Neural Information Processing Systems, 35:30044–30057, 2022.

NorthCutt1 - Curtis Northcutt, Lu Jiang, and Isaac Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 70:1373–1411, 2021a.

Carlini1 - Nicholas Carlini, Ulfar Erlingsson, and Nicolas Papernot. Distribution density, tails, and outliers in machine learning: Metrics and applications. arXiv preprint arXiv:1910.13427, 2019a.
 
 
 

Response to Reviewer Part 1/3

We thank the reviewer for their time and consideration, and answer the questions here:  

1. Novelty: The proposed method has a high degree of overlap with Garg and Roy 2023 [R2], where the similar per instance score is used for determining the curvature. Further, the score in their paper is able to and long-tailed and rare samples, which is the application demonstrated in this paper. It would be great if authors can please clarify the novelty of this work in comparison with Garg and Roy 2023 [R2].

   Authors: Thank you for your question. Garg and Roy [R2] indeed was a source of inspiration for this work. We found their analysis of curvature for creating coresets interesting, and were inspired to see if we could use it as a measure of memorization. We found that their method did not work out of the box, and we had to make significant changes to establish a good metric, in essence reverting back to Hutchinson’s trace estimator form for calculating curvature [Hutchinson1]. Their scores also failed to capture the failure mode of duplicated samples with differing labels. We would like to highlight the following differences:

   • In this manuscript, we focus on using curvature as a method for measuring memorization of samples, and focus on the high curvature samples. In contrast, Garg and Roy 2023 [R2] focus on low curvature samples, and their application to data efficiency for coreset creation.

   • We note that their method of calculating curvature does not give reliable results:

     • They used the adversarial direction to estimate curvature. We removed the adversarial direction assumption and returned to traditional Hutchinson’s trace estimator form, and used random Rademacher vectors instead for more reliable results. The cosine similarity with FZ scores on CIFAR100 for Garg and Roy [R2] is 0.17, while our method achieves 0.82.

     • To further improve estimation, we average our results over 10 different random Rademacher variables while [R2] use a single direction. We get a good match for n=5,10,20 with FZ scores as seen in the hyperparameter tuning results from Table 3 in Appendix B.

     • Their method only calculates curvature at the end of training, and the following table shows that that gives unreliable scores. For instance, the match with FZ scores for curvature calculated at the end of training, as shown in the table below, is only 0.18. We average curvature during training to get reliable results, in order to allow for different decision boundaries that have been learned at different epochs as training progresses. This is the result of studying curvature dynamics during training in Section 4.4 and is quite evident if we form a GIF of the decision boundary in the input space. We have added that to the supplementary material, also found here. We can see from the first few epochs in the GIF, that the network is trying out different hypotheses in the form of different decision boundaries, on the way to the final one. Averaging over all these hypotheses allows us to be robust to any particular one and shows much more reliable results.

     • We show the results for using Garg and Roy’s method, our method limited to calculating curvature at end of training, and our method as outlined in the paper and show that we get considerably better matches (better cosine similarity CS) with FZ scores for both the most memorized samples, and for the entire dataset, both with (wd1) and without (wd0) weight decay.

       Method	Top 5K FZ CS (wd0)	Top 5K FZ CS (wd1)	CS with FZ for all data (wd0)	CS with FZ for all data (wd1)
       Garg and Roy [R2]	0.07	0.24	0.10	0.17
       Ours n=10 EOT	0.06	0.28	0.12	0.18
       Ours	0.82	0.90	0.73	0.82

     • We provide insights and strong empirical evidence for the use of high curvature samples in studying memorization of deep neural nets, and utilize it to identify a new failure mode that all other methods, including [R2], fails to capture, as shown in Figure 19 in the Appendix.

   We thank the reviewer for helping us consolidate the differences, and have summarized this discussion in the section in Appendix D4.

3. Missing Concrete Application: The section on curvature dynamics for training provides interesting insights. However I couldn’t find any specific experiments which demonstrate its practical applicability.

   Authors: Thank you for pointing out the insights in Section 4.4. The major reason for writing section 4.4 was to help the readers to understand why we need to average our scores every epoch as opposed to say, calculating curvature at the end of training. In that way, it serves as the base that informs our design of metric for measuring memorization. We realize that this was not coming across from our writing, and thank the reviewer for identifying the need for a better explanation. We have cleaned up Section 4.4 considerably, and believe it reads a lot better now. We have reduced the content to highlight the following key takeaway, and believe that it is now a lot easier for the reader to follow.

    
   Takeaway: At the early epochs of training, the boundary changes very rapidly and significantly. In fact, every epoch, the boundary can curve around a different subset of samples. This means that measuring the curvature at 1 epoch only will be unstable. Averaging the curvature over all of training helps us iron out this effect and get reliable curvature results. This is quite evident if we form a GIF of the decision boundary in the input space, and we have added that to the supplementary material, also found here. We can see from the first few epochs in the GIF, that the network is trying out different hypotheses in the form of different decision boundaries, on the way to the final one. Averaging over all these hypotheses allows us to be robust to any particular one and shows much more reliable results.

4. I am curious if can the above method be used to identify the samples which have inconsistent captions for the Vision Language Based Methods?

   Authors: We believe this is possibly a very good application of our method we have not yet tried. We focused on vision datasets in this paper, and hope that this would encourage others to try our method in different domains.  

5. Further, it’s claimed in the paper that FZ scores can’t be used to find samples that are duplicate images with different samples. Can you please provide a concrete reason for this?

   Authors: We wish to clarify that we did not mean that FZ scores cannot calculate this, just that the FZ scores that have been released do fail to catch duplicate samples with different labels. We have edited the paper to make this clear. We cannot recreate FZ scores to try and improve their methodology to catch this failure mode since it requires training 20,000 models. However, we can think of some reasons why they may have failed to capture it. FZ scores are calculated by looking at the change in expected probability of correct prediction when the sample is removed from the training set. To make this computationally easier, the models perform subsampling of the dataset, and it is possible that the considered subsets missed the duplicate samples. It is also possible that each of the 20,000 models was not trained to full convergence for efficiency reasons.
    

6. Is it possible to provide any theoretical results regarding the correctness of the curvature scores for finding the noisy labeled samples etc?

   Authors: In this manuscript, we use qualitative and quantitative results to show that curvature is a good measure of memorization. We are working on more theoretical findings as part of future work.

References

Hutchinson1 - Michael F Hutchinson. A stochastic estimator of the trace of the influence matrix for Laplacian smoothing splines. Communications in Statistics-Simulation and Computation, 19(2):433–450, 1990.

R2 - Isha Garg and Kaushik Roy. Samples with low loss curvature improve data efficiency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20290–20300, 2023.

We thank the reviewer for taking the time to write detailed feedback. We answer the questions here:  

1. The main weakness of the paper is that it does not provide a comprehensive discussion on what it means to say that an example is being memorized. For example, the work of Feldman & Zhang calls those points to be memorized where the model only performs well when those points are present in the training set. What is the precise notion of memorization that this paper aims to capture?

Authors: This is a very relevant question, and we attempt to share the way we think of our metric with the reviewer here. The memorization that our manuscript is aiming to capture is conceptually quite similar to what Feldman and Zhang intend to capture with their metric. Specifically, both methods ask how sensitive the model is to a particular input. Feldman & Zhang propose to answer this by looking at probabilities in presence and absence of the sample, and we propose to answer this by measuring the sensitivity of the model to perturbation in the input, calculated via curvature.
 

A far more intuitive understanding of why we chose curvature arises from correlating the curvature of loss around a sample to the curvature of decision boundary around the sample in the input space. The classical machine learning illustration of overfitting, such as the one found here, shows that the network overfits by bending the decision boundary around certain sample points in the input space, indicating that these samples are being memorized by the network. In the example shown, the samples that would be counted as most memorized, would be the ones with the curviest decision boundaries around them, establishing a straightforward link between memorization score and curvature of decision boundary around the sample.
 

Previous work such as [Dong1, Srinivas1] has shown the link between curvature of decision boundary and curvature of loss, in that regularizing loss curvature results in a flatter decision boundary. This link means that our metric estimates the curvature of the decision boundary around a sample in the input space, which makes intuitive sense as a measure of memorization.
 

If the reviewer feel that this would add clarity and intuitive visualization of our metric to the readers, we are happy to add this to the manuscript.

 
References:
Dong1 - Bin Dong, Haocheng Ju, Yiping Lu, and Zuoqiang Shi. "CURE: Curvature regularization for missing data recovery." SIAM Journal on Imaging Sciences 13, no. 4 (2020): 2169-2188.

Srinivas1 Suraj Srinivas, Kyle Matoba, Himabindu Lakkaraju, and Francois Fleuret. "Efficient training of low-curvature neural networks." Advances in Neural Information Processing Systems 35 (2022): 25951-25964.

2. It appears that the paper claims that their curvature-based memorization score aligns well with Feldman & Zhang’s memorization score as the two scores have a high correlation (cosine similarity). Such aggregate-level correlations can be misleading, especially while studying memorization behaviors which are very much tied to individual examples (e.g., see Section 3.5 in https://arxiv.org/pdf/2310.05337.pdf).

Authors: We agree with the reviewer, and this is a justified question. In order to inform the readers, we have added a qualifying line to state the limitations of cosine similarity in the paper. However, we list two points in support of why we used cosine similarity below:

• First, we show in Table 4 in the Appendix D, curvature scores calculated only at the end of training do not show a high cosine similarity match with FZ scores, with the cosine similarity being only 0.18 compared to our match of 0.82 when averaged over all epochs. We know that the curvature scores calculated at the end of training do not capture the duplicated samples, and are therefore not reliable. The fact that we get visually more reliable results by averaging over epochs, and our cosine similarity match rises significantly (from 0.18 to 0.82), leads us to believe that the cosine similarity metric still captures meaningful correlations.
• Additionally, we show an independent link between FZ scores and curvature that does not rely on cosine similarity in Figure 8: We plot the curvature of the 5000 most memorized samples as per FZ scores and show that the curvature of FZ score identified memorized samples is considerably higher at all epochs.
   
  These two reasons were why we decided to proceed with the cosine similarity score in the paper.

8. Section 4.3: Can you elaborate on why ROC curves are more reliable?

   Authors: Metrics such as accuracy are very brittle to imbalance in datasets. For instance, if we had a dataset with very few samples of the positive class, such as in our case with mislabeled samples, predicting all negative would give us a hundred percent accuracy, despite nothing having been learnt. This can be avoided by using AUROC curves. We preferred AUROC curves over PR curves as PR curves over represent the smaller class, in our case the mislabeled class. There can be memorized samples in the dataset that were not synthetically mislabeled as well, and we wanted to give a more holistic comparison. However, we realize that this may be more application and user dependent than we estimated, and upon looking closer, we think that giving a generic recommendation is likely not the best. We have removed this line from the text, and thank the reviewer for taking the time to check the minutiae of our paper.  

9. How is the threshold for the curvature of loss selected?

   Authors: In the paper we do not specifically calculate a threshold, we report the AUROC (Area Under ROC curve), as is commonly practiced in literature. AUROC/AUC is obtained by varying the threshold. The Area under the ROC curve gives a more holistic view than the ROC metric at some threshold. However, given an ROC plot there are several ways in literature to select a suitable threshold. For example, one method is to choose a threshold that maximizes the true positive rate (TPR) while minimizing false positive rate (FPR) i.e. the threshold that results in the top left point of the ROC curve. Another method chooses a threshold such that FPR is below a certain value or TPR is above a certain value. There are other works that inject known label-altered data and track their metric to identify the threshold. Any of these methods could be used if the user wants a threshold for their use case. However, it is common practice to just report AUROC numbers to show good separability between two classes.  

10. Does averaging the curvature over epochs remove the specific epoch signals? Isn't it better to study the difference between curvature evolution during time than taking an average?

    Authors: Thank you for your question, this is exactly what we were hoping to answer with the analysis done in Section 4.4. Different examples are memorized at each epoch as the model makes different boundaries in order to learn the data since there can be many correct boundaries. For instance, the boundaries at epochs, 20, 30 and 40 are different in Figure 7. There is inherent noise in how the network reaches the minima during learning as it samples different minibatches stochastically, which results in different curvature scores for samples at each epoch. Averaging over epochs allows us to average over all the decision boundaries the model looked at during its training and make aggregate decisions, which is more reliable. To support our claim, we can show that curvature scores calculated at the end of training are not good enough, and we revamp Section 4.4 to make this point clearer. We provide a link to a GIF to illustrate the development of input space curvature. From the early epochs in this GIF, we can note the different hypotheses tried out by the network, evidenced by changing decision boundaries.  

11. Figure 8: Which type of sample triggers the peak in curvature, and do memorized points contribute more to that?

    Authors: Figure 8 plots the average curvature of the training set across epochs. The curvature of all samples peak around the overfitting epoch, as explained in Section 4.4. However, from the same figure, we can see that the curvature of the 5000 most memorized samples according to an independent measure (FZ scores) remains higher than the average curvature across the dataset. Hence, we can indeed conclude that the peak is contributed to more by the memorized samples than the rest of the dataset.
     

12. Figure 8: Why is the curvature of test points much higher than memorized points in CIFAR-100? And why don't we see that in ImageNet?

    Authors: We believe this is due to ReNet18 being significantly overparameterized for CIFAR100 compared to ImageNet. Thus, with CIFAR100 the model is able to overfit to the training dataset a lot more than on ImageNet, thus widening the gap between train and test examples.

Response to Reviewer, Part 1/3

Thank you for your encouragement, and for mentioning your willingness to reconsider your review. We address each of the questions below:

1. Regarding the toy example formulation, and the clarity of section 4.4:
   Question: Toy example in section 4.4: The number of training points is far less than in the test set, which is not a similar case to your image datasets. The training data does not have enough points to accurately represent the underlying distribution. Additionally, adding noise to it makes it more challenging for the model to learn the decision boundary correctly.

   Authors: The imbalance in the design of this toy example is intentional. Our goal was to evaluate what the decision boundary looks like when networks memorize the data. The few, noisy training data encourages the network to memorize the training data. But, having clean abundant test data allows us to also measure generalization of the underlying true data distribution under extreme memorization and noisy samples. This setup allowed us to understand how curvature behaves with extreme memorization, with the following key takeaway:

   At the early epochs of training, the boundary changes very rapidly and significantly. In fact, every epoch, the boundary can curve around a different subset of samples. This means that measuring the curvature at 1 epoch only will be unstable. Averaging the curvature over all of training helps us iron out this effect and get reliable curvature results. This is quite evident if we form a GIF of the decision boundary in the input space, and we have added that to the supplementary material, also found here. Please look at the early epochs from the GIF, we can see that the neural network is trying out different decision boundaries to try to minimize the loss, since multiple different boundaries give good results. Averaging over these different hypotheses was necessary to not rely too much on one particuaar hypothesis.

   We thank the reviewer for identifying the need for a better explanation, and we have cleaned up Section 4.4 considerably, and believe it reads a lot better now. We have reduced the content to highlight this key takeaway, and believe that it is now a lot easier for the reader to follow.  

2. Figure 4: It would be better to display mislabeled examples side by side.

   Authors: Thank you for the suggestion, we have made this change to Figure 4.  

3. Figure 5: The labels on the axes of the figure are not clear, and I can't interpret the results.

   Authors: Thank you for identifying this readability issue, we have increased the font size and improved the readability of graphs in Figure 5  

4. Having a distribution over the corrupted samples would be more informative.

   Authors: We have added a density plot visualizing the distribution of input loss curvature for clean and mislabeled samples in the appendix (See Figure 19).
    

5. Appendix F: It does not explain any correlation between memorized sample by FZ and your method.

   Authors: We show the quantitative comparison of our score with FZ scores using a cosine similarity metric in Table 1. However, we use Appendix F to emphasize a different point. We display the most memorized samples as per FZ scores, in order to highlight that despite being 3 orders of magnitude more computationally expensive than our method, they do not find the failure case of duplicated samples with differing labels that we found with our analysis. This is significant since the duplicate samples are most definitely memorized by a model if the model gets close to a 100% training accuracy. We highlighted this better in the text in Appendix F.  

6. Introduction (Regarding “weakly labeled or have noisy annotations”): Can you apply your method in unsupervised settings?

   Authors: One advantage of our method is that computing curvature of sample only requires a loss, as opposed to other methods that require logits. This renders our method scalable towards unsupervised settings, unlike SSFT or FZ scores. Thank you for highlighting an advantage of our method.  

7. Table 2: Did you use weight decay for other methods as well? Can you show the result of FZ method?

   Authors: Yes, all results in Table 2 are using weight decay. For FZ scores however, we were limited to using the results the posted on their GitHub repository as it is not possible to recreate their experiments as they calculated their scores by training 20,000 models per dataset. For this reason, it is also not possible to add that as a baseline to the mislabeled sample detection in Table 2.