We thank the reviewer for their valuable feedback, we address their questions below. Tables and Figures are provided at Rebuttal Page clickable link.

1. Which other architectures and dataset sizes?

A: Please see our response to reviewer ZrxQ Q1. Additionally we also ran the the long tail experiments, visualized in Figure 3,4,5,6 in the rebuttal page. See the Q2 for a more in-depth explanation.

2. Other memorization-related signals used in curriculum learning or adversarial robustness:

A: We would like to clarify that we provide two experiments, first mislabeled detection, second duplicate detection to capture mem. properties. Additionally, we ran the exps below

• Adversarial Robustness (Fig 7 rebuttal page):
  • We provide a visualization of adversarial distance versus memorization and CSL scores. Adversarial distance refers to how easily a sample can be perturbed to change its classification. The results show that samples with low adversarial distance (more vulnerable) tend to have higher memorization and CSL scores. This suggests that CSL and memorization capture similar properties and are strongly related to model robustness.
• Easy vs. Hard-to-Learn Samples:
  • Fig 1 (on the Rebuttal Page (link above)) shows images with low CSL scores from CIFAR100, interpreted as easy or typical examples (class prototypes). Fig. 2 shows high CSL images from CIFAR100 capturing atypical and likely memorized examples. Fig 3,4 presents the same for ImageNet. The high CSL visualizations clearly show we identify memorized (but not mislabeled) examples in such cases. This visualization is similar to [3] who show similar figures with their proposed proxy of curvature.
• Long Tail Behavior
  • Clean CIFAR100 and ImageNet datasets are also long tailed [11]. Fig 5 and Fig 6 plots the histogram of CSL on CIFAR100 and ImageNet showing that CSL captures long tail behavior
• Additional Results Under Higher Noise (Table 2 rebuttal page)
  • We evaluate CSL under higher label noise settings.
  • CSL maintains its ability to distinguish clean from mislabeled examples, indicating its robustness across different noise levels.

3. On Other learning dynamics method?

A: Please note, Second Split Forgetting time (SSFT), Forgetting time, learning time, in-confidence (1 - confidence) and Curvature are learning dynamics based methods and clearly we see that CSL out performs them in both mislabeled detection and duplicate detection (Tables 2 and 3 in the main paper). Regarding similarity with memorization scores we have added forgetting frequency, loss sensitivity, final epoch loss to the (Table 1 rebuttal page) as additional baselines.

4. Supplementary Material

A: Feldman and Zhang [11] ran Inception on CIFAR10/100 and ResNet50 on ImageNet. Additionally, we have added ResNet50 to ImageNet experiments (Table 1 rebuttal page) and Inception on CIFAR100. To save space we kindly ask the reviewer to see our response to reviewer ZrxQ Q1.

5. There are several recent works that explore similar ideas but in a broader context,.. would improve the positioning of the paper.

A: Thank you for bringing this to our attention. While these works share similar goals, they differ from ours. We will include them in the related work section. [7] measures performance across multiple models on a single input, while CSL analyzes the distribution of memorization scores within a single training run, with no added overhead. In contrast, [7] is significantly more computationally expensive. [8] addresses how memorization and spurious correlations hinder generalization and proposes MAT to mitigate them. Our work introduces CSL, a fast, theoretical metric for estimating memorization, which we apply to detect mislabeled and duplicate data. [9] explores how memorization fades when examples are removed. CSL, on the other hand, focuses on efficiently estimating how memorized each sample is, offering both theoretical grounding and empirical utility. [10] proposes data splitting for robust OOD training. CSL instead offers a lightweight and accurate method for measuring memorization. Additionally please see our response to reviewer WoPj Q5.

[3] Garg et al. "Memorization Through the Lens of Curvature of Loss Function Around Samples." ICML 24.
[7] Kaplun et al. "Deconstructing Distributions: A Pointwise Framework of Learning." ICLR 23
[8] Bayat et al. "The Pitfalls of Memorization: When Memorization Hurts Generalization." ICLR 25
[9] Jagielski et al. "Measuring Forgetting of Memorized Training Examples." ICLR 23
[10] Pezeshki, et al. "Discovering Environments with XRM." ICML 24
[11] Feldman & Zhang "What neural networks memorize and why: Discovering the long tail via influence estimation" NeurIPS 20.

We thank the reviewer for their valuable feedback, we address their questions below. Tables and Figures are provided at Rebuttal Page clickable link.

1. There is no description of how CSL is used to detect mislabeled samples (Section 7.1) and duplicate samples (Section 7.2). I also cannot find it in Appendix B.1 or B.5. Is this simply thresholding the CSL for each sample?

A: Thank you for bringing this to our attention, we will clarify this in the revision. We do indeed use thresholding on each sample’s CSL, giving us a binary detector whose performance is measured and reported using AUROC.

2. The expectation over samples. Starting from Theorem 5.3, there is an introduction to an expectation over training samples (). Looking at the proof quickly I believe that it is an expectation over the entire training data distribution. This already makes me curious about how this theorem would be used because we often care about memorization of a particular sample, not an expectation over a distribution. However, on L246, Theorem 5.3 is being interpreted as applying to “a group of samples” like  which is a subset of all training data . It is a bit unclear to me why this interpretation is valid, given the theorem holds in expectation for the entire training distribution. This is fairly important as it is used to motivate the experiments showing relationship between CSL and “Mem Score” (Figure 6 and 8). I might be missing something here.

A: We would like to clarify that we meant arbitrary random subsets. Consider selecting a subset U at random; then T can be set as $\max T_{z_i}$ for $z_i \in U$. Since Equation (10) holds for such a distribution, we obtain the stated result. We will revise the text to clarify this point and explicitly indicate that the subsets under consideration are random.

3. Assumption on L257: “it can be assumed that is constant across different subsets .” This assumption may need more explanation for why it is reasonable. My guess is that because depends on Frobenius norm and the singular values of the samples batch, which are arguably irrelevant to the difficulty to learn by neural network. This assumption may need more explanation for why it is reasonable. My guess is that because depends on Frobenius norm and the singular values of the samples batch, which are arguably irrelevant to the difficulty to learn by neural networks.

A: The reviewers explanation is intuitive and correct. More formally, note that $\kappa_g$ is not dependent on the training mini batch but is dependent on the choice of input made for computing the gradient norm w.r.t to the input, which is user controlled. The the input gradient bound for a given input is independent of the choice of the training mini batch. Now choosing $\max_u \kappa_{g_u},~u \subset S$, i.e. max value over the subsets, we can get a worst case upper bound constant.

4. Is there a particular reason for choosing cosine similarity as a metric instead of Pearson's correlation coefficient or mean square error? Correlation coefficient seems the most natural to me.

A: We used cosine similarity because it was employed in prior methods and setups [3]. We have also added Pearson correlation in Table 1 (see the rebuttal page linked above). Any magnitude greater than 0.25 is considered a statistically significant correlation. Thus, the takeaways are similar between cosine similarity and Pearson correlation. Please also see our response to Q1 for reviewer ZrxQ.

5. I believe that related works have been mentioned throughout the paper. I personally would like to see more discussion of the prior literature in more details.

A: Thank you for the suggestion, other reviewers have also expressed similar concerns. We will expand our Related Works section to discuss more thoroughly (1) stability-based memorization and its computational challenges; (2) learning dynamics; and (3) broader loss trajectory works for membership inference attacks. This should clarify further how our approach (CSL) integrates with and advances prior research.

6. I feel like Lemma 5.1 can be generally useful, and it’d be a nice contribution too (if it’s not been shown before)

A: Thank you for recognizing the broader utility of Lemma 5.1. To our knowledge, it has not been explicitly presented before. We will emphasize this contribution and place it more prominently in the revised manuscript.

7. I could not find how  l^z is defined. Is it loss when the sample z is removed from the  training set?

A: Yes, $l^{\setminus {z_i}}(z_i)$ is the per-sample loss for ($z_i$) under the model trained on the leave-one-out dataset ($S^{\setminus {z_i}}$). We will clarify that notation explicitly in the revised manuscript.

[3] Garg et al. "Memorization Through the Lens of Curvature of Loss Function Around Samples." ICML 24.

We thank the reviewer for their valuable feedback, we address the questions below. Tables and Figures are provided at Rebuttal Page click here.

1. A primary concern is the authors' use of memorization scores precomputed by Feldman & Zhang (2020) while employing different model architectures .... scores.

A: We agree with the reviewer. Feldman & Zhang used Inception for CIFAR100 and ResNet50 for ImageNet. We have added ResNet50 (same as Feldman & Zhang) and Inception (for CIFAR100 same as Feldman & Zhang) in Table 1 (rebuttal page above) along with more baselines (we reran Inception model, due to a minor bug fix, and present the updated scores). The choice of ResNet18 was dictated by its use in prior proxy such as curvature [3]. Additionally, we have included results using multiple architectures in Appendix B.2. These were already part of the appendix but not clearly referenced in the main text; we will fix this in the revised version. Our findings are inline with [3, 4, 5, 6] which have shown that memorization is often a dataset-level property, given the model has sufficient capacity. Also, please see our response to fZEA Q3.

2. Furthermore, ... Without explicitly comparing against such straightforward alternatives, the paper fails to convincingly demonstrate the unique value of CSL over simpler approaches.

A: We have provided the final sample loss, loss sensitivity and forgetting frequency as a baselines (also suggested by reviewer fZEA) in Table 1 (rebuttal page above) . As noted by reviewer woPj, we have also added Pearson correlation $\in (-1, 1)$, any magnitude > 0.25 is considered statistically significant correlation. And clearly we see $CSL$ has statistically significant result generally and outperforms other methods. Also, please see our response to reviewer fZEA Q3.

3. Lemma 5.1, which states .. weight matrix configurations and with no bias.

A: We would like to clarify that Lemma 5.1 indeed holds for a general feed forward neural networks. While the proof in Appendix C.1 begins with the case of a linear layer for simplicity, the result extends to arbitrary feed forward networks (see steps eq. 23 -> 24 this is holds for a general feed forward NN).

Regarding bias, this is a notational simplification. The inclusion of bias can be handled without loss of generality by augmenting the input vector with an additional identity row, a common technique to account for bias. As such, Lemma 5.1 holds for feed forward networks with and without bias. We will revise the appendix to make this generalization more explicit.

4. The theoretical framework relies on .. assumptions that may not hold in practical deep learning settings .. irregular.

A: First, our theoretical results do not assume convexity. In fact, our analysis applies to widely used unbounded, non-convex, cross-entropy loss, as noted in the paper. Additionally also for any non-convex bounded loss.

Second, the assumptions (e.g., smoothness, stability, bounded gradients) are well-supported in practice. Lipschitz continuity and smoothness of deep networks have been studied and validated in Virmaux & Scaman [1], and uniform stability of SGD has been established by Hardt et al. [2]. Please see our detailed discussion in lines (303-317). Therefore, we believe the theoretical framework remains broadly applicable to standard deep learning practice, especially for networks trained with SGD.

5. Additionally, the paper is unclear about ... minibatches might affect the derived bounds.

A: Thank you for raising this important point. We understand the potential confusion, and we’d like to clarify. Our proof relies on lemma 5.1, since lemma 5.1 does not depend on the choice of minibatch, our proof can also be extended to SGD with minibatch size larger than 1.

6. I also don't understand ... data subsets.

A: We would like to clarify that we meant arbitrary random subsets. Consider selecting a subset U at random; then T can be set as $\max\{T_{z_i} : z_i \in U\}$. Since Equation (10) holds for such a distribution, we obtain the stated result. We will revise the text to clarify this point and explicitly indicate that the subsets under consideration are random.

7. Regarding Missing references w.r.t MIA:

A: Please see our response to reviewer WoPj Q5.

[1] Virmaux & Scaman Lipschitz regularity of deep neural networks: analysis and efficient estimation. NeurIPS 18
[2] Hard et al. Train faster, generalize better: Stability of stochastic gradient descent. ICML, 16
[3] Garg et al. "Memorization Through the Lens of Curvature of Loss Function Around Samples." ICML 24.
[4] Garg & Roy. "Samples with low loss curvature improve data efficiency." CVPR 23
[5] Lukasik et al. What do larger image classifiers memorise? arXiv preprint arXiv:2310.05337, 2023
[6] Ravikumar et al. "Unveiling privacy, memorization, and input curvature links." ICML 24