Progressive Data Dropout: An Embarrassingly Simple Approach to Train Faster

Reviewer 3Tz5

We thank Reviewer 3Tz5 for the detailed feedback and valuable suggestions. Below, we directly address each point, clarifying misconceptions and reinforcing our contributions through additional empirical evidence.

---

Originality concerns and omission of recent related works.

We respectfully disagree that our work lacks originality or adequate distinction from recent literature. While we acknowledge the rapidly evolving field, our approach fundamentally differs from dynamic pruning methods like InfoBatch or Instance-dependent Early Stopping (IES). Unlike methods focused on pruning difficult examples or samples, our Progressive Data Dropout (PDD) explicitly targets computational efficiency through progressive reduction of data.

To address the reviewer's concern, we have empirically compared PDD directly against these recent methods, along with those suggested by Reviewers bHp2 and FSoK, clearly demonstrating our method’s distinctiveness and superior performance. We compare against DataDiet both ELN and Forget [1], IES [2], Early Stopping and InfoBatch [3]. We follow the official implementations and again use EE to compare efficiency. Our method outperforms all compared baselines in both accuracy and EE. These are listed below as Accuracy / EE. Higher the better for both. For all methods, we report fine-tuning a model on CIFAR-100 (pretrained on ImageNet) for 200 epochs.

---

Questionable baseline settings (short training schedules).

We acknowledge this limitation on CIFAR-10. We ran our experiments on CIFAR-100 and ImageNet experiments for the extended schedule. We initially employed shorter schedules to efficiently explore multiple architectures and standardized comparison models. However, in response, we conducted additional experiments using standard-length training runs (200 epochs) on CIFAR-10 with EfficientNet, ResNet, and MobileNet v2 architectures.

Preliminary results indicate that Progressive Data Dropout (PDD) persists in providing sustained computational savings while improving accuracy. We used uniform optimization parameters across models for fairness, although this could have slightly affected overall accuracy.

Accuracy / Effective Epoch Table (CIFAR-10):

Fine-tuning pretrained models (200 epochs):

---

Lack of detailed wall-clock time evaluation.

While our primary metric (Effective Epochs, EE) provides hardware-independent comparability, we list a few wall clock savings estimates below. These experiments were run on a NVIDIA 4060 RTX GPU with 8GB RAM.

Wall Clock Time:

Despite minimal optimization efforts in our current implementation, these clear reductions confirm practical efficiency. Further code optimization could lead to even greater wall-clock time savings. We also note that in the GPT-2 experiment (below), SRD reduced wall-clock time by more than half while maintaining competitive performance. The GPT-2 experiment was run on 2 x NVIDIA A-100 GPU with 80 GB RAM.

We will add wall-clock times to the paper, but want to point out that EE is a hardware-independent way of reporting efficiency. As we can see between these two experiments using different GPUs results in wildly different wall clock times.

Q: Practical performance and robustness of SMRD with approximated schedules

We thank the reviewer for raising this important concern regarding the practicality of SMRD when using an approximated data reduction schedule rather than the oracle version derived from DBPD. We have some approximations listed in Section I of the Appendix.

To address this, we conducted additional experiments using a logarithmic decay schedule for SMRD, which is entirely derived without referencing DBPD. This approximation is simple, easy to implement, and does not require any task- or model-specific tuning.

Logarithmic Decay:

$$\hat{f}(x; \alpha) = \frac{1}{\log(x + \alpha)}$$

Critically, we observe that:

• Logarithmic SMRD is competitive with the oracle SMRD in both accuracy and effective epochs.
• In the appendix, we listed this in comparison to EfficientNet from scratch on CIFAR-100
• Similar trends hold across other architectures such as EfficientNet, MobileNet v2 and ResNet, showing that our approximation generalizes well. All models are pre-trained on ImageNet.

Most importantly, SRD and DBPD already perform strongly on their own. In particular, SRD remains a simple yet highly efficient method, showing consistent gains across image classification, self-supervised learning (MAE), and language modeling (GPT-2, shown above in previous weakness).

---

We thank the reviewer again for the detailed feedback, which has helped clarify our contributions and strengthen the final paper.

We thank Reviewer FSoK for their insightful comments and valuable suggestions. Below, we directly address the reviewer’s points, providing clarifications and empirical evidence to resolve misconceptions and reinforce our position.

---

Hyperparameter impacts not thoroughly explored.

Since we wanted to ensure minimal hyperparameter tweaks for each task, model, or dataset, we only performed the tweaks for one model on one dataset and consistently stuck to those same values. We report the same for CIFAR-100 on two new models as well as on ImageNet for one model showing similar trends to Table 4 in the paper.

Threshold sensitivity (CIFAR-100):

Threshold sensitivity (ImageNet - ResNet-50):

For SRD, we only used it as the most trivial way of data dropping and hence did not conduct an extensive study on this. However, based on the reviewer’s suggestion, we test SRD on CIFAR-100 for varying thresholds and report them below.

SRD Threshold sensitivity (CIFAR-100):

---

Lack of comparison to baselines like early stopping.

We appreciate this suggestion. Early stopping terminates training prematurely to avoid overfitting but does not strategically alter data utilization. Based on this suggestion and Reviewers bHp2 and 3Tz5, we compare it to a suite of methods (active learning, data pruning, early stopping) listed below. We follow the official implementations and again use EE to compare efficiency. Our method outperforms all compared baselines in both accuracy and EE. These are listed below as Accuracy / EE. Higher the better for both. For all methods, we report fine-tuning a model on CIFAR-100 (pretrained on ImageNet) for 200 epochs.

---

Interpretation of Table 1 (varying effective epochs).

We thank the reviewer for this excellent suggestion. Comparing methods at a fixed compute budget (equal EE) is indeed informative. However, this is a computationally intensive experiment, and due to the wide range of additional results included in the rebuttal (e.g., full-length runs, wall-clock time, NLP, and SMRD approximations), we were unable to complete it in time.

Matching EE across methods also requires tuning thresholds for two methods while keeping one fixed, a non-trivial process, as each method responds differently to threshold changes.

That said, we ran a preliminary experiment aligning EE across CIFAR-100 for ResNet-50. All models are fine-tuned from ImageNet pre-training. Results reported as Accuracy / EE / Threshold value:

Even under equal EE, PDD variants outperform the baseline. We still see that SMRD is the best performing variant. We are extending this study to more models and will include full results in the final version. Additionally, training curves (accuracy vs. number of samples) will be added to the supplementary materials for deeper comparison.

---

Practical usability of SMRD due to dependency on DBPD.

We agree that SMRD’s practicality is inherently limited. Although initially informed by DBPD’s schedule, we have derived simple mathematical approximations of the dropout schedule which closely replicate the oracle SMRD performance (Appendix Section I). In particular, we consider log decay as our closest approximation. Logarithmic Decay:

$$\hat{f}(x; \alpha) = \frac{1}{\log(x + \alpha)}$$

Empirical experiments demonstrate that our approximations still clearly outperform baselines, validating practical usability. More importantly, SRD and DBPD still outperform the baselines in most scenarios and are practical.

---

Data dropout potentially increasing risk of overfitting?

We appreciate the reviewer’s concern and are happy to clarify. While it may seem that seeing fewer examples per epoch could increase overfitting, our data dropout method is designed to reduce redundancy, not diversity. By selectively removing well-learned examples, we encourage the model to focus on underfit or more informative samples.

This is similar to pruning unnecessary updates during training. Importantly, our final revision phase reintroduces the full dataset, ensuring no information is permanently lost and mitigating any risk of forgetting.

Empirically, we observe that data dropout consistently improves generalization and accuracy across datasets and models, despite reduced per-epoch exposure. This suggests it serves as a form of regularization, injecting stochastic variation and helping prevent overfitting to a fixed subset.

In short, data dropout enhances generalization by improving efficiency, without sacrificing full data coverage over training.

---

Learning rate scheduler and training procedure.

We use a standard StepLR scheduler across all experiments, with a decay factor of 0.98 applied every epoch (step size = 1). The learning rate schedule is synchronized with the number of training epochs, not the number of samples seen. So, it remains consistent across all methods, regardless of how much data is dropped in a given epoch. As for the second part of the question: the difference between SRD and SMRD is entirely at the data level. These techniques determine which samples are included per epoch but do not interact with the model's parameters, gradients, or learning rate. So while the number of samples per epoch may vary, the learning rate schedule is unaffected and consistent across variants.

---

Impact of total number of training samples.

We conducted experiments across a wide range of datasets. For example, CIFAR-10 and CIFAR-100 consist of 50,000 training samples, whereas ImageNet has 1.2 million. We also tested on a simulated long-tailed version of CIFAR as well as fine-grained datasets such as Airplane and CUB based on Reviewer bgJ1’s suggestion. The simulated long-tailed CIFAR-10 has 12,440 samples, while Airplane and CUB have 10,000 and 5,994 training samples respectively.

Results reported as Accuracy / EE.

Long-Tail Classification, CIFAR-10, ρ = 100
Dataset generated using the same recipe used by LDAM-DRW [4] and all results use Focal Loss [5] for ease of implementation to our code base.

We actually see big improvements in the long-tailed scenario, but most importantly, see similar improvements on all datasets regardless of size. Thus, PDD remains beneficial across varying dataset sizes.

---

[1] Deep learning on a data diet: Finding important examples early in training. Neurips 2021. [2] Instance-dependent Early Stopping, ICLR 2025. [3] InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruning, ICLR 2024. [4] Learning imbalanced datasets with label-distribution-aware margin loss. Neurips 2019. [5] Focal loss for dense object detection. CVPR 2017.

We deeply appreciate Reviewer FSoK’s thoughtful questions and comments, as addressing these points clarifies misconceptions, strengthens our argument, and significantly improves the manuscript.

We sincerely thank Reviewer bHp2 for their thoughtful and constructive feedback. Below, we respond to each comment in detail. While time constraints limited us to reporting results on a subset of models, we will include the full set of results in the final version of the paper.

Empirical evaluation limited to image classification.

We agree that our method lacks sufficient diversity. However, we clearly demonstrate its generalizability across a variety of architectures and datasets (CIFAR-10, CIFAR-100, ImageNet), as well as self-supervised learning (with MAE, which is extremely time-consuming). While our focus remains on image tasks for clarity and practicality, we recognize the reviewer's suggestion that increasing diversity would be helpful. Following the Reviewer bHp2’s suggestion, we used GPT-2 pre-training with SRD on the Fine-Web Edu dataset [8] (comparable in nature yet much larger than OpenWebText). Since the task is different and modifying DBPD and SMRD for this particular task would need more time, we used our most simple approach of SRD. We set a really high threshold due to the amount of data that we would have to go through and could not commit to more experiments due to compute limitations. Limited time prevented us from trying even more thresholds. Despite the simplified implementation, SRD provided substantial computational savings, more than halving the number of steps and wall-clock time:

GPT-2 Results:

---

No evaluation on long-tailed class distribution or label noise.

We appreciate the suggestion but would like to emphasize that PDD fundamentally focuses on computational efficiency, rather than robustness to class imbalance or noise. Nevertheless, following reviewer suggestions, we performed the following experiments.

Long-Tail Classification CIFAR-10, ρ = 100 where ρ is used to denote the ratio between sample sizes of the most frequent and least frequent class
Dataset generated using the same recipe used by LDAM-DRW [1]. The baseline below corresponds to using Focal Loss [9]. We use Focal Loss [9], for it's ease to work with our current codebase. Results reported as * Accuracy / Effective Epochs (EE)*.

We see improvements of over 12% while also reducing the effective epochs (EE) by more than a third. We are happy to add these results, but note that a full-scope comparison of long-tailed experiments is beyond the scope of this paper.

---

Label Noise:

We evaluated our method on noisy label classification using the Noisy CIFAR-10 dataset, following the setup from [2]. Again, we use [2] for the ease of implementation with our current codebase.

• Noise rate: 17.3%
• Noise type: Random
• Epochs: 50

Results:

We observe that DBPD achieves strong accuracy with significantly fewer effective epochs.

---

Theoretical reasoning for random dropout effectiveness over difficulty-based methods.

We appreciate this insightful question. While it may seem counterintuitive, random dropout can outperform difficulty-based dropout due to its regularizing properties and improved sample diversity during training. Difficulty-based dropout, though principled, can prematurely eliminate low-loss (potentially clean or informative) samples, leading to overfitting on harder or noisy examples and reducing generalization.

In contrast, random dropout introduces stochasticity into the data stream, which aligns with principles from curriculum learning and stochastic optimization, where varied or randomized sample exposure helps models escape narrow local minima and enhances generalization [3, 4, 5]. From a theoretical perspective, this stochastic exposure acts as a form of implicit regularization, similar to what has been observed in small-batch SGD training and randomized curricula [6, 7].

Empirically, we find that random dropout consistently performs robustly across architectures and datasets, validating the intuition that simpler, less biased sampling often leads to better generalization, especially when revisited data is used in the final phase of training.

---

Parallels between PDD and forgetting–relearning dynamics.

We appreciate this connection and agree that PDD embodies a form of transient data-level forgetting, analogous to neuron- or connection-level forgetting in prior literature (Ramkumar et al., 2023; Zhou et al., 2022). Our method implicitly leverages such dynamics by periodically revisiting previously dropped data, thereby enhancing generalization. We provide theoretical justification to this above.

---

Efficiency gains diminishing with increasing dataset size.

We acknowledge that efficiency gains relatively decrease with larger datasets since the redundancy per epoch reduces. However, absolute computational savings remain significant and practically meaningful. We would like to point out two things:

1. The results in Table 1 use thresholds optimized for both speed and accuracy. For accuracy alone, higher thresholds would give us even bigger improvements. For example, our ImageNet experiments show that setting thresholds to 0.8 could give us an additional accuracy boost of over 2%.

2. As demonstrated in the self-supervised training results on MAE and the GPT-2 results above, experiments with massive data and epochs, despite marginal performance drop, compute saved is significant. Even modest percentage improvements at scale translate into substantial absolute savings.

---

Ideal dataset characteristics for PDD.

PDD benefits mid-sized to large datasets, where computational savings become significant. We also tested on smaller fine-grained datasets, following the reviewer's suggestion like CUB and Flowers, observing continued computational savings without accuracy degradation. This clarifies its suitability across scales and complexities. These results will be explicitly included in our revision. Results reported as Accuracy / EE.

---

Statistical significance and number of random seeds.

We report one such study in Section I of the supplementary material. We saw similar results across 3 different models on all three classification tasks. Our conclusion was that there is no significant statistical anomaly to worry about.

---

We appreciate the reviewer’s detailed comments and questions, as addressing these clarifies misconceptions and strengthens our contribution significantly.

References

[1] Cao, K., Wei, C., Gaidon, A., Arechiga, N. and Ma, T. (2019) Learning imbalanced datasets with label-distribution-aware margin loss. Neurips. [2] Wei, J., Zhu, Z., Cheng, H., Liu, T., Niu, G., & Liu, Y. (2022) Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations. ICLR. [3] Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009). Curriculum learning. ICML.
[4] Hacohen, G., & Weinshall, D. (2019). On the power of curriculum learning in training deep networks. ICML.
[5] Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., & Tang, P. T. P. (2017). On large-batch training for deep learning: Generalization gap and sharp minima. ICLR.
[6] Hardt, M., Recht, B., & Singer, Y. (2016). Train faster, generalize better: Stability of stochastic gradient descent. ICML.
[7] Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2021). Understanding deep learning (still) requires rethinking generalization. Communications of the ACM. [8] https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu [9] Lin, T. Y., Goyal, P., Girshick, R., He, K., & Dollár, P. (2017). Focal loss for dense object detection. CVPR.

We sincerely thank Reviewer bHp2 for their valuable and constructive feedback. Below, we address each comment and question individually. Due to the limited time, we only report results on some models but will add the complete results to the final version.

Short training schedules (30 epochs on CIFAR-10 ResNet) limit interpretability of results.

We acknowledge that CIFAR-10 had a short schedule. We report results below for a longer schedule. However, CIFAR-100 and ImageNet were run for standard settings and we see consistent improvements across models for these datasets as well. We initially employed shorter schedules to efficiently explore multiple architectures and standardized comparison models. However, in response, we conducted additional experiments using standard-length training runs (200 epochs) on CIFAR-10 with EfficientNet, ResNet, and MobileNet v2 architectures.

Preliminary results indicate that Progressive Data Dropout (PDD) persists in providing sustained computational savings while improving accuracy. We used uniform optimization parameters across models for fairness, although this could have slightly affected overall accuracy.

Accuracy / Effective Epochs (EE) Table (CIFAR-10 with 200 epochs):

Fine-tuning models pretrained on ImageNet (200 epochs):

---

Missing comparisons to strong active learning baselines (Data Diet, coreset selection).

We appreciate this suggestion. While PDD is complementary yet distinct from active learning and pruning methods, comparisons are insightful. Based on your suggestion (and Reviewers FSoK and 3Tz5), we conducted comparisons by comparisons against a suite of methods. We compare against DataDiet both ELN and Forget [1], IES [2], Early Stopping and InfoBatch [3]. We follow the official implementations and again use EE to compare efficiency. Our method outperforms all compared baselines in both accuracy and EE. These are listed below as Accuracy / EE. Higher the better for both. For all methods, we report fine-tuning a model on CIFAR-100 (pretrained on ImageNet) for 200 epochs.

---

Limited task diversity; NLP tasks (pretraining small GPT-2).

While our focus remains on image tasks for clarity and practicality, we recognize the reviewer's suggestion that NLP would be helpful. We also used PDD for MAE self-supervised pre-training. Following the reviewer's suggestion, we used GPT-2 pre-training with SRD on the Fine-Web Edu dataset [4] (comparable in nature yet much larger than OpenWebText). Since the task is different and modifying DBPD and SMRD for this particular task would need more time, we used our most simple approach of SRD. We set a really high threshold due to the amount of data that we would have to go through and could not commit to more experiments due to compute limitations. Limited time prevented us from trying even more thresholds. Despite the simplified implementation, SRD provided substantial computational savings, more than halving the number of steps and wall-clock time:

We emphasize again our main experiments remain in image classification, and will revise the claims in our paper to reflect this.

---

Hyperparameter sensitivity (dropout thresholds).

We deliberately minimized hyperparameter tuning for generalizability. Below we report additional results for CIFAR-100 and ImageNet (both from scratch), in addition to the existing results in Table 4 of the main paper:

Threshold sensitivity on CIFAR-100

---

Threshold sensitivity on ImageNet (ResNet-50)

For SRD, we only used it as the most trivial way of data dropping and hence did not conduct an extensive study on this. However, based on the reviewer’s suggestion, we test SRD on CIFAR-100 for varying thresholds and report them below.

SRD Threshold sensitivity (CIFAR-100):

---

Writing style (GPT-like, itemized lists).

Thank you for highlighting this stylistic issue. This was a personal choice, but we will revise the manuscript to reduce excessive itemization and improve readability, ensuring greater clarity throughout.

---

Performance under strong regularization or augmentation (Mixup, CutMix)

This is an insightful direction. Using stronger regularization or augmentation reduces the effect of the current set thresholds for our methods. However, as mentioned in the paper, we report on 0.3 threshold for a balance of speed and accuracy. However, if the focus is on accuracy, we can obtain that with a higher threshold as shown below. These results are on CIFAR-10 for 200 epochs. We use CutMix style augmentation with $\alpha$ set to 1.

---

Could dropout schedules become adaptive based on training signals?

We acknowledge this as a potential improvement. However, as the core of the paper talks about an "embarrassingly simple" to train faster and adding training signals to make this adaptive increases the complexity of the method. We show results of 2 adaptive ideas below whilst maintaining the idea of being "embarrassingly simple". For 200 epochs, we start with a threshold of 0.9 for the first 20 epochs and progressively reduce the threshold after every 20 epochs. This would be akin to how a learning rate scheduler works with a different decay rate. We also do the opposite, where we start with a threshold of 0.3 and increment this by 0.1 every 60 epochs.

We tested two simple adaptive schedules, preserving our method's "embarrassingly simple" nature, yielding strong performance:

• ResNet-50 (CIFAR-100) 0.3 starting threshold with 0.1 increment every 60 epochs: 79.96 % accuracy, 23 EE.
• ResNet-50 (CIFAR-100) 0.9 starting threshold with 0.1 decrease every 20 epochs: 70.77% accuracy, 15 EE.
• ResNet-50 (CIFAR-100) static 0.3 threshold: 79.92% accuracy, 21 EE.

We notice similar performances between the first dynamic one and the currently used static one. However, we acknowledge there is potential for improvement in the dynamic one in terms of scheduling.

---

We sincerely appreciate the insightful feedback, greatly helping us strengthen our manuscript.

[1] Deep learning on a data diet: Finding important examples early in training. Neurips 2021. [2] Instance-dependent Early Stopping, ICLR 2025. [3] InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruning, ICLR 2024. [4] https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu

Response to Reviewer bHp2 (Follow-up Comment)

We sincerely thank the reviewer for the thoughtful follow-up and for considering an upgrade to their recommendation. We are glad the new results were compelling and appreciate the prioritization of suggestions. Below, we respond to each of the three key areas raised:

---

1. Theoretical interpretation of the method

We acknowledge the importance of a stronger theoretical interpretation of Progressive Data Dropout (PDD). While a formal distributional estimation analysis is still ongoing, we had previously grounded our method in existing literature on curriculum learning and stochastic regularization.

We mentioned:

Random dropout can outperform difficulty-based methods by introducing beneficial stochasticity, broadening sample diversity during training. In contrast, difficulty-based heuristics risk prematurely dropping easy yet informative samples, which may hurt generalization.
This aligns with established ideas in curriculum learning [3, 4], where sample ordering and exposure influence generalization, and with stochastic optimization theory [5, 6, 7], where randomness acts as an implicit regularizer to prevent overfitting and escape sharp minima.

To further empirically explore this idea, we conducted a toy experiment on EfficientNet trained over 50 epochs on CIFAR-100. We logged sample usage across epochs and recorded how often each sample survived training. We then aggregated this by class label.
Here’s what we observed:

• The least used class label (label 1) appeared 19,885 times across 50 epochs.
• The most used label (label 3) appeared 55,421 times.
• For context, in standard training without dropout, each class would be seen 50 × 5,000 = 250,000 times.

This early experiment suggests that PDD naturally induces class-level skew in its empirical distributions over time. Studying such skewed distributions and their learning dynamics could be a valuable direction for future theoretical work. We plan to further explore how this affects gradient properties and convergence in upcoming revisions.

---

2. Large-scale NLP pretraining (e.g., C4, WikiText-103)

We completely agree with the reviewer that testing on large-scale datasets like C4 or WikiText-103 would dramatically strengthen the paper. Unfortunately, given the limited time and computational resources, we were unable to perform these experiments during the rebuttal phase. We hope to include such studies in future versions of this work.

---

3. Task-specific tweaks for NLP applicability

As the reviewer rightly notes, our preliminary GPT-2 experiment used only SRD, which was easier to adapt to the pretraining setting. However, we are currently extending this to implement a DBPD-style dropout schedule for GPT-2, which would offer a more direct application of our full method to language modeling.

We will add these results as soon as we have them. We’re optimistic that this variant will offer both compute savings and improved generalization over long training runs.

---

We thank the reviewer again for their feedback and for considering an upgrade. We hope the combination of theoretical motivation, new experimental insights, and our ongoing work address these points satisfactorily.

---

References
[3] Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009). Curriculum learning. ICML.
[4] Hacohen, G., & Weinshall, D. (2019). On the power of curriculum learning in training deep networks. ICML.
[5] Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., & Tang, P. T. P. (2017). On large-batch training for deep learning: Generalization gap and sharp minima. ICLR.
[6] Hardt, M., Recht, B., & Singer, Y. (2016). Train faster, generalize better: Stability of stochastic gradient descent. ICML.
[7] Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2021). Understanding deep learning (still) requires rethinking generalization. Communications of the ACM.