Efficient Representativeness-Aware Coreset Selection

W1: Prior Work Already Identified Representativeness Degradation

Thank you for this important clarification. We acknowledge that the CREST paper (arXiv:2306.01244) has already identified the issue of coreset representativeness degradation, and explicitly demonstrated that coreset gradient fidelity can deteriorate even under periodic reselection.

While our paper initially framed this as a novel observation, we agree that CREST precedes our work in formally highlighting this problem. We will revise the statement of our first contribution in the final version to accurately reflect this prior work.

That said, our approach differs in both motivation and methodology. CREST uses second-order curvature information (i.e., Hessian-preconditioned gradients) as a supervision signal to guide coreset reselection. In contrast, we propose a lightweight signal-to-noise ratio (SNR)-based criterion as a proxy for monitoring coreset representativeness.

More importantly, our method is not merely an alternative trigger: we reinterpret representativeness degradation from the perspective of overfitting to a stale coreset, and show empirically and theoretically that SNR strongly correlates with this overfitting behavior. This leads us to design a simple, scalable, and efficient metric for dynamic coreset monitoring that avoids the computational overhead of second-order methods.

Thus, while we share similar concerns with CREST, our contribution lies in introducing the SNR-based perspective, providing a theoretical grounding, and offering an efficient implementation that performs well across tasks.

We will explicitly acknowledge the connection to CREST in the revised manuscript and clarify that our main contribution lies in the proposal and validation of the SNR-based representativeness criterion.

W2: Incremental Nature of Using SNR as a Trigger

Thank you for the feedback. While we use GradMatch as the primary base method in our experiments, this is due to its popularity and strong empirical performance, not because our method is tied to it.

Our core contribution is the SNR-based representativeness detection framework, which is model-agnostic and can be integrated with any dynamic coreset selection algorithm as a triggering mechanism. ERACS is not a modification of any specific selection algorithm but a general plug-in for monitoring and improving coreset quality over time.

To demonstrate this generality, we applied our ERACS triggering mechanism to multiple coreset methods—including GradMatch, GLISTER, InfoBatch, and DDSA—without changing their selection logic. As shown in Table below, ERACS consistently improves accuracy across all baselines. These results clearly indicate that our method is not a narrow enhancement to GradMatch, but a flexible and broadly applicable scheduling strategy.

Table: Accuracy improvements by adding ERACS triggering to different coreset selection methods.

As shown above, integrating our SNR-based ERACS trigger into various existing coreset methods leads to consistent accuracy improvements—despite not altering their selection mechanisms. This highlights the generality and robustness of our framework in improving training effectiveness by adaptively detecting representativeness degradation.

We will clarify this point further in the revised manuscript and include these cross-method results to reinforce that ERACS offers a general, efficient, and scalable solution for adaptive coreset scheduling.

Q1: Behavior of ERACS Under Noisy Gradients

Thank you for raising this important point. To evaluate the robustness of ERACS under noisy training conditions, we conducted additional experiments on CIFAR-10 by injecting synthetic label noise into the training set.

Specifically, we adopted a label flipping strategy to inject noise: for 10% of the training samples, we randomly replaced their ground-truth labels with incorrect ones, drawn uniformly from the remaining 9 classes. This approach simulates real-world scenarios where label corruption occurs in a class-agnostic fashion and induces instability in gradient directions during training.

Table: Test accuracy (%) under 10% label noise on CIFAR-10. ERACS achieves the best robustness among all baselines.

We then compared ERACS with existing coreset methods under this noisy setting. The results show that ERACS remains stable and consistently outperforms other baselines, even when gradients are corrupted by label noise. This suggests that our SNR-based criterion is not overly sensitive to noisy setting.

Q2: Applicability Beyond Classification

Thank you for this forward-looking question. Although our current manuscript focuses primarily on image classification, the ERACS framework is inherently general and can be extended to other learning paradigms such as regression and language modeling.

To validate the generality of ERACS, we conducted experiments on:

• Regression tasks using the California Housing dataset.
• Language modeling tasks using RoBERTa fine-tuned on SNLI.

In both cases, we compared ERACS against the baseline training as well as GradMatch-based coreset selection. Below, we summarize the results:

Regression Task: California Housing (RMSE ↓)

ERACS consistently reduces RMSE compared to both the baseline and GradMatch, demonstrating its effectiveness on regression tasks.

Language Task: SNLI with RoBERTa (Accuracy ↑)

ERACS improves classification accuracy on natural language tasks, showing its utility beyond vision.

These results suggest that our SNR-based representativeness criterion is broadly effective across model types (e.g., ResNet, FT-Transformer, RoBERTa) and task types (classification, regression, language).

We will explicitly incorporate these results and discussions into the revised manuscript, and highlight them in the Limitations and Broader Impact sections.

Q3 Clarification Request: Rephrase Discovery Claim as Quantitative Replication

Thank you for the valuable comment. We agree that the phenomenon of coreset representativeness degradation in dynamic coreset selection has been previously mentioned in related works. We will cite and properly acknowledge these prior contributions, such as CREST (arXiv:2306.01244), in the revised manuscript.

Our contribution does not lie in claiming the original discovery of this phenomenon, but rather in offering a new interpretation from the perspective of overfitting to a stale coreset. Building on this interpretation, we propose a lightweight, scalable, and model-agnostic SNR-based criterion to monitor coreset representativeness and adaptively trigger reselection. Our method is compatible with a wide range of dynamic coreset selection algorithms and offers an efficient solution to this long-standing issue.

In the revised manuscript, we will rephrase the contribution accordingly to reflect these clarifications: emphasizing our novel interpretation and practical solution, rather than claiming the phenomenon itself as new.

W1: More Discussion on Representativeness, SNR Justification and other criteria.

Intuitively, coreset sizes are typically small relative to the full dataset and the model capacity. As training progresses, the model can easily overfit to the small coreset, leading to a situation where gradients from the coreset lose their informativeness. In this regime, the gradient updates become increasingly aligned with noise rather than meaningful signal. This phenomenon is naturally captured by the signal-to-noise ratio (SNR) of coreset gradients—when representativeness deteriorates, the SNR drops, signaling that the coreset no longer provides sufficient training signal.

Theoretically, we provide analysis showing that SNR provides an upper bound on expected gradient discrepancy. Specifically, low SNR correlates with small informative variation in gradients, indicating that the coreset may no longer reflect the structure of the full data distribution.

Empirically, we tested several alternative criteria for triggering coreset reselection, Please refer to Reviewer 5gdu – W2 and Q1: Answer (1) for detailed experiment results and analysis.

W2: Details of Clarity in Cost Evaluation and Computation

Hardware. All experiments on CIFAR-10 and CIFAR-100 were conducted using a single NVIDIA RTX 4090 GPU. For large-scale experiments on ImageNet, we utilized a server equipped with 8 NVIDIA RTX 4090 GPUs.

Model Architectures. Our primary architecture for image classification tasks was ResNet18.

Optimization Setup. We used SGD as the default optimizer with momentum = 0.9, Nesterov enabled, learning rate = 0.025, and weight decay = 5e-4. For learning rate scheduling, we adopted CosineAnnealingLR with $T_{\text{max}}=300$ epochs.

Loss Function. We used standard cross-entropy loss without sigmoid activation.

Training Procedure. Each experiment was run for 300 epochs with a batch size of 128.

We will include these details in the revised version to improve clarity and reproducibility.

Q1: Difference of the Proposed Method from GradMatch

While we use GradMatch as the primary base method in our experiments, this is due to its popularity and strong empirical performance, not because our method is tied to it.

The core contribution of our work is the SNR-based representativeness detection framework, which is model-agnostic and can be integrated with any dynamic coreset selection algorithm as a triggering mechanism. ERACS is not a modification of any specific coreset selection method, but a general plug-in for adaptively monitoring and improving coreset quality over time. We will revise the manuscript to make the distinction between GradMatch and ERACS clearer.

To demonstrate this generality, we applied the ERACS trigger to multiple coreset selection methods—including GradMatch, GLISTER, InfoBatch, and DDSA. Please refer to Reviewer 5gdu – W2 and Q1: Answer (2) for detailed experiment results and analysis.

Regarding the reviewer’s suggestion on Figure 1, We will make changes in the revised version.

Q2: Can the Illustration Extend Beyond Linear Regression?

Notation

Let

• $\mathcal{S} = \{z_i = (x_i, y_i)\}_{i=1}^{m}$ be a fixed subset.

• $\mathcal{L}(\theta; z)$ be the per-sample loss with gradient
  $g(\theta; z) = \nabla_\theta \mathcal{L}(\theta; z) \in \mathbb{R}^{d}$.

• The empirical risk
  $F(\theta) = \frac{1}{m} \sum_{i=1}^{m} \mathcal{L}(\theta; z_i)$
  with minimiser $\theta^\star = \arg\min_\theta F(\theta)$ and deviation
  $\delta = \theta - \theta^\star$.

• The averaged single-sample gradient and its signal-to-noise ratio:

  $G_m(\theta) = \frac{1}{m} \sum_{i=1}^{m} g(\theta; z_i)$

  $\text{SNR}(\theta) = \frac{ \| \mathbb{E}[g(\theta; Z)] \|_2 }{ \sqrt{ \frac{1}{m} \cdot \text{Tr} \left( \text{Cov}[g(\theta; Z)] \right) } }$

Assumptions

1. $L$-smoothness (or $B$-Lipschitz gradient) in a neighbourhood of $\theta^\star$:
   $\left\| \nabla F(\theta) - \nabla F(\theta^\star) \right\|_2 \le L \left\| \theta - \theta^\star \right\|_2$
   For non-smooth losses (e.g. ReLU, hinge) replace $\nabla$ with a sub-gradient and $L$ with a Lipschitz constant $B$.

2. Non-vanishing gradient variance: there exists $\sigma_g^2 > 0$ such that
   $\mathrm{Tr} \left( \mathrm{Cov}_{\mathcal{S}}[g(\theta; Z)] \right) \ge \sigma_g^2$
   for every $\theta$ in the same neighbourhood (guaranteed by input/label noise, model misspecification, etc.).

Proposition

Under assumptions (1)–(2),
$\mathrm{SNR}(\theta) \le \sqrt{m} \cdot \dfrac{L}{\sigma_g} \cdot \left\| \delta \right\|_2$,
so as $\left\| \delta \right\|_2 \to 0$, we have $\mathrm{SNR}(\theta) \to 0$.

Proof sketch.

• Because $\nabla F(\theta^\star) = 0$, smoothness gives
  $\left\| \mathbb{E}[g(\theta; Z)] \right\|_2 = \left\| \nabla F(\theta) \right\|_2 \le L \cdot \left\| \delta \right\|_2$

For any differentiable or sub‑differentiable model with standard loss functions, the above derivation remains applicable and theoretically sound.

Q3: Are There Other Effective Representativeness Signals?

We conducted a controlled experiment to compare our SNR-based method (ERACS) with three alternative approaches based on different information modalities discussed in Section 2.1, including geometry and uncertainty. we tested several alternative criteria for triggering coreset reselection, Please refer to Reviewer 5gdu – W2 and Q1: Answer (1) for detailed experiment results and analysis.

Major1；Minor2、3、4；Misspellings1:

Thank you for your suggestion. We will correct them accordingly in the revised version.

Major2: Linear Regression Example Does Not Inform Coreset Quality

Our goal in this section is to illustrate how SNR evolves as optimization progresses — specifically, how representativeness degrades as overfitting to a fixed coreset occurs. The example is not intended to identify which coreset is “good,” but rather to support the motivation for monitoring representativeness over time. The actual selection of a good coreset remains the responsibility of the base selection algorithm (e.g., GradMatch), while our framework focuses on determining whether a previously selected coreset remains valid.

Major3: Clarify Practical Computation Cost of SNR

Thank you for the thoughtful question. We would like to clarify that the definition of $\hat{\sigma}_t$ in Appendix A—based on the spectral norm of the full empirical covariance matrix—is used solely for theoretical analysis, where a rigorous formulation is needed to establish our guarantees.

In practice, we adopt a diagonal approximation of the covariance to avoid the prohibitive $O(d^2)$ memory and $O(d^3)$ computation cost. This approximation reduces the complexity to $O(d)$, making the SNR trigger highly efficient in real training scenarios. Such approximation is widely adopted in deep learning, such as the well-known techinique Batch Normalization, see the first paragraph of Section 3 in [r1] for details.

Major4: Clarify Coreset Size and Selection Process in Algorithm 1

Yes, you are correct.

Minor1: Dual Motivation of SNR—Cost vs. Overfitting Control

Thank you for the observation. Our primary motivation for using SNR is to monitor and prevent overfitting to a stale coreset. However, in practice, computing full-data gradient discrepancy is too expensive. Thus, proposing a lightweight alternative like SNR is also an important part of our contribution. We will revise the abstract to reflect both aspects.

Minor5: Why Might SNR Fail in Non-Classification Tasks?

We mentioned that SNR may be less effective in non-classification tasks primarily because we thin such tasks (e.g., regression) may require more precise gradient information—both in direction and magnitude. Since SNR only captures the ratio of mean to variance, it may not fully reflect the nuanced gradient behavior needed to evaluate representativeness in these settings.

But preliminary results on regression tasks (please refer to Reviewer pdTr – Q2) suggest that SNR can still effectively guide coreset reselection and lead to noticeable performance gains. We plan to include these findings in future versions of the paper.

Minor6: Clarify Future Directions—Adaptive Thresholding and Proxies

We clarify the intended meanings as follows:

• Adaptive thresholding refers to allowing the model to automatically learn appropriate threshold values, instead of relying on manually specified ones.
• Alternative proxies refers to other potential metrics that could be explored for evaluating the representativeness of the coreset.

Minor7: Explain or Reference 1/2-net in Appendix A

This result is based on Corollary 4.2.11 from Vershynin's High-Dimensional Probability, Chapter 4, Section 4.2, "Nets, Covering and Packing".

Corollary 4.2.11 (Covering numbers of the Euclidean ball)
For any $\varepsilon > 0$, the covering number of the unit Euclidean ball $B_2^n$ satisfies:

$(1/\varepsilon)^n \le \mathcal{N}(B_2^n, \varepsilon) \le (2/\varepsilon + 1)^n$

The same upper bound also holds for the unit sphere $S^{n-1}$.

In particular, setting $\varepsilon = 1/2$ yields a $1/2$-net on the unit sphere with cardinality at most $5^n$, meaning that every unit vector lies within Euclidean distance at most 0.5 of some point in the net.

[r1] Ioffe, Sergey, and Christian Szegedy. "Batch normalization: Accelerating deep network training by reducing internal covariate shift." ICML, 2015.

W1: Further Discussion on Representative Degradation

Thank you for the insightful comment. While we agree that coreset selection methods may sometimes favor easy or high-density samples, we believe the representativeness degradation observed during training has deeper causes that deserve further discussion.

First, taking GradMatch as an example, when an $\ell_2$-norm regularization term is included (as is common in practice), the selection is no longer biased solely toward easy samples. In fact, a subset of harder examples---often with large gradient---are preferentially included due to the $\ell_2$ regularizer.

Second, we posit that the degradation in representativeness is not primarily due to the selection of easy samples, but rather due to the intrinsic limitations of deep neural network training in high dimension nonlinear space. In almost all deep learning settings, the number of samples is typically much smaller than the dimensionality of the data distribution, and the samples always lie in a highly complex nonlinear space. Thus, even a well-chosen coreset cannot maintain long-term representativeness throughout training, since it cannot fully determine the classification surface, which will be reflected in over-fitting. Consider a 2D example with linearly separable samples. In such a case, we can select a few (e.g., 10-20) points to represent the whole dataset, the representativeness may remain robust over time since the classifier determined by the coreset can be very close to the one trained on the full dataset. However, such stable coverage is infeasible in deep networks with millions of parameters and complex loss landscapes. Therefore, we argue that representativeness degradation is not only about "sample difficulty" and more importantly about the intrinsic limitations of deep neural network training in high dimension nonlinear space.

We hope this perspective helps clarify that representativeness degradation is a complex phenomenon that cannot be attributed solely to the selection of easy samples.

W2 and Q1: Comparison with the results of papers after 2023

Thank you for pointing this out. We appreciate your suggestion to include comparisons with more recent coreset selection methods. In response, we conducted a broader literature survey and identified two state-of-the-art methods published after 2023:

• InfoBatch (ICLR 2024) [r1]: Proposes a lossless training acceleration approach through unbiased dynamic pruning. It conducts soft pruning based on per-sample loss and rescales gradients to maintain unbiased estimation of full-dataset gradients.
• DDSA (ICML 2025) [r2]: Introduces a dynamic coreset selection framework that combines importance-based selection and selective augmentation. Samples are chosen every epoch based on a combination of high loss and low confidence, and high-confidence samples are further augmented to boost generalization.

We implemented both methods and evaluated them on the CIFAR-10 dataset with a fixed coreset budget of 10% of the training data. Specifically, we applied both methods with periodic coreset reselection every 20 epochs and compared them against their respective variants that use our proposed SNR-based trigger instead.  The results are summarized below:

Table: Comparison with recent coreset methods using SNR-based reselection trigger (CIFAR-10)

These results demonstrate that even for non-gradient-based coreset methods, our proposed SNR-based coreset representativeness detection framework can be seamlessly integrated and consistently improves model performance.

We will include these results in the revised manuscript if accepted.

W3 and Q2: Comparison of Distillation Methods under Fair Storage Restrictions

Thank you for raising this important and practical concern. In deep learning, GPU memory (rather than disk storage) is often the primary constraint during training. Since mini-batch training is standard, dynamic coreset selection does not significantly increase memory usage per iteration as the full dataset can be stored in the disk.

Moreover, static coreset selection methods often require a larger coreset size to achieve comparable performance, which may result in higher memory usage during every training step. To examine this trade-off, we conducted the following experiments that take both storage and accuracy into account.

We implemented an additional experiment focusing on the overall storage usage throughout training. Specifically, in our ERACS configuration using the best trade-off threshold (as reported in the main paper), the full dataset is only accessed during 26 coreset reselection steps out of 300 total epochs. For the remaining 274 epochs, only a 10% coreset is used for training. This means that the average effective dataset size used over time is approximately 17.8% of the full dataset.

To ensure a fair comparison, we evaluated the static coreset selection method CRAIG that uses a static 17.8% subset of the dataset, and trains on it for the full 300 epochs. The results, shown below, demonstrate that ERACS still achieves higher accuracy under equivalent storage conditions.

Table: Comparison under equalized average storage budget (17.8% of full dataset).

The result suggests that dynamic coreset selection methods like ERACS provide a compelling middle ground: they offer significant storage savings compared to full-data training, while outperforming static coresets in accuracy. While static approaches are more efficient in terms of raw storage, the dynamic strategy captures training dynamics more effectively—a promising direction worthy of continued exploration in coreset research.

W4 and Q3: Results Across Diverse Model Architectures, Different Optimization Methods, and Varied Training Scenarios.

Thank you for the suggestion. Since in the rebuttal we are not allowed to give the visualization of intermediate results as Figure 1 in the main text, we present the final results across diverse optimization methods, model architectures, and training scenarios. We will include detailed plots and analysis in the revised version.

We first evaluated ERACS under different optimizers on CIFAR-10 with a coreset budget of 10% using ResNet18. As shown in below, our method consistently outperforms the baseline (GradMatch) across SGD, Adam, and RMSProp, confirming that the observed representativeness degradation and our solution are not tied to a specific optimizer.

Table: Accuracy comparison of ERACS and baseline under different optimizers (ResNet18 on CIFAR-10).

To demonstrate the generality of ERACS beyond image classification, we extended our experiments to both text and regression tasks.

In the vision domain, we repeated our experiments using VGG16 on CIFAR-10. In the text domain, we applied ERACS to fine-tuning RoBERTa on the SNLI dataset. As shown in the table below, ERACS consistently improves accuracy across all tested architectures and modalities, and exhibits similar overfitting dynamics during late-stage training—further supporting our hypothesis on representativeness degradation.

Table: Accuracy comparison of ERACS and GradMatch under different model architectures.

To further assess applicability to regression problems, we evaluated ERACS on the California Housing dataset using three model architectures: MLP, ResNet, and FT-Transformer. We report root mean square error (RMSE), where lower values are better. ERACS achieves consistently lower RMSE than both the baseline and GradMatch, demonstrating its effectiveness in regression settings as well.

Regression Task: California Housing (RMSE ↓)

Table: RMSE comparison on regression task. Lower is better.

[r1] Qin, Ziheng, et al. "InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruning." The Twelfth International Conference on Learning Representations.

[r2] Yang, Suorong, et al. "When Dynamic Data Selection Meets Data Augmentation." arXiv preprint arXiv:2505.03809 (2025).

W1: Lack of statistical significance data

Thank you for your suggestion. In the early stages of our experiments, we also conducted multiple runs and observed relatively small standard deviations. Based on this, we fixed the random seed in later experiments and reported the results accordingly.

To further address your concern, we have re-run all experiments in Table 2 five times and computed the mean and standard deviation of test accuracy for each method. The updated results are shown below:

Table: Updated accuracy results with standard deviation (mean ± std over 5 runs) across datasets and budgets.

As shown in the updated table, our proposed method consistently demonstrates superior performance and stability across various datasets and budget settings, validating the statistical significance and robustness of the results.

W2 and Q1: Extension and Comparison of Coreset Representativeness Evaluation Metrics

We designed two experiments to evaluate and compare the effectiveness of different coreset representativeness metrics:

(1) Comparison of alternative metrics for triggering coreset reselection
We evaluated four metrics for coreset reselection on CIFAR-100 using a 10% coreset budget and report the resulting test accuracy, all applied on top of the GradMatch coreset selection algorithm:

Table: Comparison of coreset reselection triggers based on final test accuracy.

The Geometric Dispersion Score [r1] computes the average minimum distance from unselected points to the coreset in feature space. A high GDS indicates poor coverage and degraded representativeness. We trigger coreset reselection when GDS exceeds a predefined threshold.

The Uncertainty Score [r2] monitors the overall uncertainty of the coreset based on entropy. A lower average entropy indicates that the model has already absorbed most of the information from the coreset, making it less informative. When the Uncertainty Score drops below a predefined threshold, we trigger coreset reselection to restore representativeness and maintain training effectiveness.

The GradientDiff (Coreset vs. 10% Full) measures the L2-distance between the average gradients of the coreset and a randomly sampled 10% subset of the full dataset. A large gradient difference indicates degraded representativeness and prompts coreset reselection.

While each of these methods helps detect representativeness degradation, our proposed ERACS method consistently achieves higher accuracy by more effectively monitoring overfitting, thereby providing a more reliable signal of coreset quality.

(2) Comparison between SNR and a baseline method (GradientDiff (Coreset vs. 10% Full)) when integrated with different coreset selection algorithms

We conducted experiments on the CIFAR-10 dataset with a coreset budget of 10% to evaluate the generalization and adaptability of SNR as a trigger for coreset representativeness degradation across multiple coreset selection methods, including GLISTER, GradMatch, InfoBatch, and DDSA. GradientDiff serves as a simple probabilistic baseline, estimating representativeness by comparing the gradient shift between the coreset and a random subset of the full data.

GLISTER[r3] and GradMatch[r4] are two dynamic coreset selection methods discussed in the main paper, and we additionally compare our approach with the latest methods, InfoBatch and DDSA.

• InfoBatch (ICLR 2024) [r5]: Proposes a lossless training acceleration approach through unbiased dynamic pruning. It conducts soft pruning based on per-sample loss and rescales gradients to maintain unbiased estimation of full-dataset gradients.
• DDSA (ICML 2025) [r6]: Introduces a dynamic coreset selection framework that combines importance-based selection and selective augmentation. Samples are chosen every epoch based on a combination of high loss and low confidence, and high-confidence samples are further augmented to boost generalization.

Table: Comparison of coreset trigger metrics based on final test accuracy.(cifar10, 10% dataset, the original method selects the core set once every 20 epochs)

As shown, while GradientDiff (vs. 10% Full) provides modest improvements over fixed-interval reselection, its gains are consistently smaller than those achieved with our proposed SNR metric. In all four coreset methods evaluated, the SNR-based trigger (ERACS) more effectively detects representativeness degradation and enables timely reselection, resulting in more stable accuracy improvements. This demonstrates the generality and robustness of our approach.

Q2: Analyze of Overfitting Dynamic Behaviour

We share the same viewpoint as you in identifying the dynamic of overfitting to a fixed coreset when representativeness is not monitored. In our experiments, we observe that when coreset selection is performed at fixed intervals (e.g., as in GradMatch), the model tends to overfit the current coreset. Due to the use of relatively small learning rates typical in deep learning training, the gradients on the coreset do not vanish entirely. When we extend the application window of a selected coreset, the coreset loss continues to decrease to a small value, while the test loss first decreases and then increases, signaling overfitting. Upon reselecting the coreset, the coreset loss resets to a higher value and resumes decreasing, while the test loss also starts to decrease again. This pattern suggests that the newly selected coreset restores representativeness and improves generalization performance.

Table: Illustration of overfitting dynamics under fixed coreset selection schedule.

For example, after the y-th coreset selection at epoch x, test loss decreases initially but eventually rises, and after the (y+1)-th selection at epoch x+T, it drops again. This evidences a repeating cycle of overfitting to the coreset – performance degradation – recovery via new coreset selection when no representativeness-aware mechanism is applied.

However, we emphasize that precisely verifying repeated absolute overfitting—i.e., when the gradient converges to zero—is challenging in deep learning applications. This difficulty arises because, in deep neural network training, the learning rate needs to be gradually decreased to ensure training stability. As a result, it is impossible to fully overfit the coreset an infinite number of times.

What our proposed method adds is the ability to detect the onset of overfitting early—e.g., before test loss starts to rise—by tracking the signal-to-noise ratio (SNR) of gradients. This allows for timely replacement of stale coresets and leads to more stable and efficient convergence.

We agree that analyzing this dynamic in more depth is valuable, and we will include a visualization and tabular analysis in the revised version to support this explanation.

[r1] Sener, Ozan, and Silvio Savarese. "Active learning for convolutional neural networks: A core-set approach." arXiv preprint arXiv:1708.00489 (2017).

[r2] Coleman, Cody, et al. "Selection via proxy: Efficient data selection for deep learning." arXiv preprint arXiv:1906.11829 (2019).

[r3] Killamsetty, Krishnateja, et al. "Glister: Generalization based data subset selection for efficient and robust learning." Proceedings of the AAAI conference on artificial intelligence. Vol. 35. No. 9. 2021.

[r4] Killamsetty, Krishnateja, et al. "Grad-match: Gradient matching based data subset selection for efficient deep model training." International Conference on Machine Learning. PMLR, 2021.

[r5] Qin, Ziheng, et al. "InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruning." The Twelfth International Conference on Learning Representations.

[r6] Yang, Suorong, et al. "When Dynamic Data Selection Meets Data Augmentation." arXiv preprint arXiv:2505.03809 (2025).