Q3: Potential improvements in clarity (minor)

A3: Thank you for your thorough review and valuable suggestions. We have addressed each of these points in the revised manuscript to improve clarity.

• Eq. 2 defines the second-order difference of the loss function $L_i(w^{(t)})$ for a given sample $i$ at the current training epoch $t$: $\Delta^2 L_i(w^{(t)}) = [L_i(w^{(t)}) - L_i(w^{(t-1)})] - [L_i(w^{(t-1)}) - L_i(w^{(t-2)})] = L_i(w^{(t)}) - 2L_i(w^{(t-1)}) + L_i(w^{(t-2)}).$

• The use of different optimizers and architectures is intended to demonstrate the applicability of the proposed IES method under various settings.

• We have corrected the typos in L137, L328, L368, L423, and L1004 in the original manuscript.

• Concerning the statement in L295-L296, "As shown in Figure 1, our method can reduce the number of training instances in backpropagation by over 40%", the reference to Figure 1 was incorrect. In the revised manuscript, we have updated the reference to Figure 4 and rephrased the statement to be more precise: ''As shown in Figure 4, our method reduces the number of training instances in backpropagation by approximately 40%''

Q4: Extensive experiments in Speedups

A4: Thank you for the suggestion. We have included an additional scenario where the computation speedup is increased to 3x, and the results are now in Appendix F. The brief results are shown below:

As shown in the table, the experimental results jive with your intuition: IES demonstrates increased effectiveness compared to other methods as we further increase the speedup.

We now turn to address the questions raised by the reviewer.

Q5: Clarification

• We agree that FLOPs can be a reliable metric for comparing computational costs. However, in our experiments, we found that the actual running time ratio better reflects the real-world computational overhead, as many of the computations are performed on the CPU (often with limited threads) in serial execution. Furthermore, although we ensured that each comparative method was run with exclusive access on identical nodes, we cannot guarantee that all tasks were run on nodes with the same configuration. Therefore, for fairness, we reported the average ratio of running times for a given task.

• We did not use the IES method during fine-tuning because our goal was to simply evaluate the transferability of the trained models.

• The baseline (No removal) does not incorporate conventional early stopping. We compared conventional early stopping in the speedup experiments presented in Table 4.

We hope these explanations address your concerns and provide a clearer understanding of our experimental setup and choices.

We would like to thank the reviewer for your detailed review and insightful comments. Below, we address the questions you have raised.

Q1: Reasoning behind second-order information

A1: We agree with your point that the statement "the choice of using the second-order difference as the removal criterion for IES has been extensively validated through experiments" (L537-538 in the original manuscript) is too strong. We have removed the word "extensively" from this statement in the revised manuscript. Furthermore, as per your suggestion, we have added third-order information to Figures 3 and 6 in the revised manuscript. The updated figures demonstrate that the coefficient of variation (CV) for third-order and second-order loss values are close, reflecting a similar degree of dispersion. Considering the lower computational and storage costs of second-order differences, along with their slightly better experimental results, we opted to use second-order differences as the criterion for IES in the manuscript.

Q2: Relationship to Dynamic Data Pruning

A2: When exploring the commonalities between the two strategy, we agree that our IES method could be viewed as a variant of dynamic data pruning that adaptively adjusts the removal rate based on the model's current learning status on training data. However, we humbly emphasize that IES differs from traditional data pruning methods in its ability to adaptively remove samples as the model training progresses, rather than following a predefined removal rate.

Regarding the comparison with dynamic data pruning methods, we initially focused on a SOTA method to demonstrate the performance of our proposed selection criterion. To further clarify, we have conducted additional experiments comparing IES with another dynamic data pruning criterion ϵ-greedy [C1]. The results are as follows:

As demonstrated, our proposed selection criterion consistently maintains higher performance compared to the dynamic data pruning method.

Furthermore, in response to your constructive suggestion, we have included a comparison of the wall-time between IES and the dynamic data pruning method (InfoBatch) from our manuscript in Appendix D. The brief results are presented below:

CIFAR-10:

CIFAR-100:

The results show that, at similar removal ratios, IES saves more wall-time on CIFAR-10, while achieving comparable results on CIFAR-100. This difference may be attributed to the shorter pipeline of our proposed method.

Q6: Future direction

A6: You raised a very interesting point. To address this issue, we tracked 2,500 samples that were excluded earliest during the training of the CIFAR-10 dataset. Throughout the entire training process, we monitored the number of transitions these samples underwent from mastered to unmastered states, during which the model's performance changed significantly. The results showed that these samples had an average of 13.14 unmastered-to-mastered transitions, with a maximum of 26 transitions and a minimum of 0 transitions. Additionally, we tracked the loss changes for this subset of samples:

The results indicate that although the frequencies at which these samples were re-included in training varied, with some samples never being re-included in training, the model's learning performance on these samples consistently approached (or surpassed) its performance on the entire training set. This can be attributed to the adaptiveness of the IES method, where the mastered state of samples always depends on the current stage of model learning and changes dynamically. Therefore, we tend to believe that, in contrast, predefining a static subset would not improve model learning and may instead lead to brute-force memorization of that subset.

Thank you for your valuable time and effort in reviewing our manuscript. Your insightful suggestions have greatly contributed to improving our manuscript.

Best regards,

Authors

Reference:

[C1] Accelerating Deep Learning with Dynamic Data Pruning. arXiv.

Dear Reviewer fBn3,

We are genuinely grateful for your prompt response. We have made our best efforts to address the concerns raised by all reviewers and have included additional experiments to evaluate our method. We also sincerely apologize for any lack of clarity in our manuscript that may have led to misunderstandings about the algorithmic details of IES.

In the IES method, excluded samples can be re-included in the training process if they no longer meet the "mastered" criterion. This is reflected in Algorithm 1, where lines 3-5 compute the "mastered" status of instances through forward propagation, and line 10 performs backpropagation only on the not-yet mastered instances ($\mathcal{D}^{(t)}$). The main text also mentions this property of IES (e.g., lines 268-269 and 416-419). We have denoted the full training set as ($\mathcal{D}^{(0)}$) and referred to it as the "full training set" in the algorithm to emphasize this point.

Regarding the 2,500 samples mentioned in our response to Q6, this number represents the top 5% earliest excluded samples from the training set, which we selected as a case study to analyze the IES method. The analysis reveals that, on average, these samples undergo 13.14 transitions from non-mastered to mastered states throughout the training process. This means that although these samples were considered well-learned early on, they are re-included in the training process and are subsequently excluded again upon reaching the "mastered" criterion.

Thank you again for your valuable comments.

Best regards,

Authors

We would like to thank the reviewer for the detailed review and constructive comments. We sincerely appreciate your time and effort in reviewing our manuscript. Below, we address the questions you have raised.

Q1: Label Noise

A1: Thank you for your insightful and constructive suggestion. We agree that an analysis of learning with noisy labels is crucial to evaluate the robustness and practicality of our proposed IES method. To address this, we attempt to discuss this issue under Typical Learning with Noisy Label scenarios and Epoch-wise Double Descent scenarios, respectively.

Typical Learning with Noisy Labels

We validate the performance of the IES method and the baseline method (without removal) under typical learning with noisy labels settings, specifically, on the CIFAR-10/CIFAR-100 datasets with 20% and 40% symmetric and instance-dependent label noise.

CIFAR-10

CIFAR-100

The experimental results indicate that the IES method degenerates to the baseline method (without removal) across all tested label noise rates, noise types, and datasets. This suggests that during the training process, no training sample satisfies the master criterion before the model overfits to the noisy labels and its performance declines. The core idea behind the IES method is that once the model has mastered an instance, the training on it should stop. However, when a certain proportion of label noise exists in the dataset, memorization of mislabeled samples may affect the model's ability to learn stable patterns, making it difficult for the model to truly master any samples before the early stopping point.

Epoch-wise Double Descent

Epoch-wise Double Descent refers to the phenomenon where, when the training samples contain a certain amount (usually low) of label noise, as training progresses, the model's generalization performance first rises, then falls, and then rises again, with the generalization performance after the second rise being superior to the first peak. In this label noise scenario, the model needs to prolong training to achieve better generalization performance compared to conventional early stopping. We validate the performance of the IES method and the baseline method (without removal) under typical Epoch-wise Double Descent settings, specifically, on the CIFAR-100 datasets with 10% symmetric and instance-dependent label noise.

We now turn to address the questions raised by the reviewer.

Q3: Question about Figure 1

A3: The difference in the rate of excluded samples between different datasets can be attributed to the varying model capacities and task complexities. It's important to note that the "master" criterion in our work indicates that the model's performance improvement on a particular sample has become very small. This serves as an estimate of the sample's contribution to the current parameter updates of the model. Due to the correlations among samples, requiring the model to "master" all samples may be unnecessary and impractical. The model only needs to well-learn certain samples to achieve sufficiently good generalization performance.

Moreover, as training progresses, the number of samples meeting the "master" criterion will eventually stabilize, with only minor fluctuations. This is because the contributions of samples to model parameter updates may be conflicting or not necessarily positive. In this situation, further extending the training will not significantly improve the generalization performance, as demonstrated by the following experiment on CIFAR-10:

As shown in the table, the number of "mastered" samples remains relatively stable after 150 epochs. This suggests that the model has reached a point where further training does not significantly change the "mastered" status of the samples or improve the generalization performance. Consequently, as an early-stopping-based method aimed at computational efficiency, we choose 150 epochs as an appropriate end point for training in Figure 1 to evaluate the effectiveness and efficiency of our method under a limited computational budget.

Q4: Training loss in Figure 4

A4: The training loss reported in Figure 4 represents the average loss computed over samples participating in training during each respective epoch.

Q5: Higher gradient norms, present in Figure 4

A5: The IES method adaptively selects instances that are not yet well-learned by the model. These instances are likely to have larger loss values and, consequently, larger gradient norms. By focusing on these instances, IES enables the model to learn more efficiently and make more substantial progress in each iteration. Higher gradient norms in this context are a sign of effective instance selection and do not imply instability. We argue that the main reasons for IES's ability to maintain training stability are as follows:

1. IES employs second-order differences and moving averages as criteria for determining whether an instance has been mastered, rather than focusing on the instantaneous magnitude of gradients. Moreover, the adaptivity of the IES method allows for the re-include of instances whose "mastered" status has changed, in the event of potential training instability.
2. IES retains the vast majority of instances in the early stages of training and gradually reduces the number of instances in an adaptive manner as training progresses. This progressive instance selection strategy helps to maintain the smoothness of the optimization process.
3. We have already employed methods such as learning rate decay during training, which can mitigate the potential instability risks caused by large gradient updates. Furthermore, the further use of gradient clipping can further enhance training stability.

We again thank the reviewer for the detailed and constructive feedback, which has helped us clarify key aspects of our method and improve the overall presentation of our work.

Best regards,

Authors

The experimental results show that the IES method can achieve lossless efficient training under the Epoch-wise Double Descent scenario. In the later stages of training, the model inevitably "well-learn" some instances due to the memorization effect. However, this does not affect the generalization performance of the final model (even slightly better). This behavior can potentially be explained by the fact that although ''well-learned'' instances may be forgotten as the model training overfits the mislabeled samples, the IES method allows these samples to adaptively re-include in training, thereby mitigating the negative impact of mislabeled samples. Consequently, in the typical scenarios of learning with noisy labels and scenarios of Epoch-wise Double Descent, the IES method appears to have no negative impact on model performance compared to the baseline.

This analysis is included in Appendix G of the revised manuscript.

Q2: Hyperparameters

A2: The "mastered" criterion introduces only two hyperparameters: a threshold $\delta$ and an order N (in our experiments, we select N=2 for the IES method). While acknowledging these hyperparameters, our sensitivity analysis shows that model performance remains stable across a wide range of $\delta$ values, suggesting that precise tuning is not required for these datasets. Importantly, when $\delta$ is set too low, IES simply degenerates to baseline training without any performance penalty. Conversely, when $\delta$ is set too high, IES's adaptive mechanism - which automatically re-includes "mastered" samples into training - helps preserve model performance.

To further validate the generalizability of IES, we conducted additional experiments on four different datasets, keeping the code and hyperparameter settings identical to those used for CIFAR-10 (except for dataset-specific dataloader and model architecture). The results are as follows:

Besides comparing with the baseline, we also compared with the InfoBatch method at corresponding removal rates. As shown, without hyperparameter tuning, the IES method achieves superior performance in three training tasks; for the STL-10 task, it achieves a 45% reduction in backpropagation samples while only decreasing test accuracy by 0.2% compared with full data training. The results demonstrate that IES can efficiently and effectively train models on a variety of tasks without requiring targeted adjustment of hyperparameters.

The robustness to parameter settings, along with the patterns observed across diverse datasets, suggests that the introduction of these hyperparameters does not significantly hinder the practicality, generalizability, or ease of use of the IES method.

Dear Reviewer 1YJ5,

We sincerely appreciate your detailed review and constructive comments on our manuscript. We are deeply grateful for your valuable time and effort.

We have carefully considered your two main concerns and made the following efforts to address them in the revised version:

1. We conducted additional experiments under label noise scenarios. The results demonstrate that the IES method maintains performance on par with the baseline under typical learning with noisy labels and epoch-wise double descent settings, without introducing additional negative impact.

2. We validated IES hyperparameter generalization on four additional datasets, where IES performs better than competing methods without tuning hyperparameters, indicating that the hyperparameters do not significantly affect the method's practicality.

We would greatly appreciate your further feedback on our revised version and our responses. If there are any unclear explanations, we are more than willing to provide further clarification.

Best regards,

Authors

Q4: Fairness

A4: We appreciate your specific experimental suggestions, which indeed help us evaluate the limitations of the IES method that we pointed out in the Limitations section. Based on your recommendations, we conducted a preliminary assessment of the fairness of training using the IES method in sensitive environments.

We utilized the CelebA face dataset as an adversarial dataset to investigate whether the IES method would introduce new biases during training when using male as the sensitive attribute and attractiveness as the target label, thereby affecting the model's fairness.

We compared the baseline method (without sample removal) and the IES method on the ResNet-18 model for the attractiveness classification task, evaluating the accuracy, recall (True Positive Rate), and Demographic Parity Difference (DPD) metrics on the male and female validation subsets. The results are as follows:

From the Demographic Parity Difference (DPD) metric, which evaluates fairness (the closer to 0, the better), the IES method is slightly lower than the baseline method (0.4544 vs 0.4613), indicating that its prediction results have slightly less disparity between the two gender subsets.

These results provide a preliminary indication that the IES method may introduce or amplify certain biases to some extent, negatively impacting the classification performance for different population subsets. However, since IES allows excluded samples to adaptively re-include in training, the overall fairness is slightly improved.

This analysis is included in Appendix I of the revised manuscript.

We now turn to address the questions raised by the reviewer.

Q5: World-clock Training Time of the 1-order Diff. Method Compare to the 2-order Diff.

A5: We compared the world-clock training time using the first-order difference criterion and the second-order difference criterion. Results are shown in the table below:

The results demonstrate that under similar backpropagation sample excluded ratios, the total running time of two methods is very close. Notably, compared to the first-order difference criterion, the second-order difference criterion only needs to store one additional epoch of loss values for the required calculations. The computational complexity also only increases from one arithmetic operation to three arithmetic operation (4.73s -> 5.16s), and the impact on the training time is very small (0.43s).

Q6: Training Time Reduction in Figure 5 Last Row

A6: As per your suggestion, we have modified the last row of Figure 5 in the manuscript to include training time reduction.

We extend our sincere thanks for your effort in reviewing our work and deeply appreciate your detailed comments towards improving our manuscript.

Best regards,

Authors

Reference:

[A1] Deep Learning via Hessian-Free Optimization. In ICML, 2010.

[A2] Variance Reduction in SGD by Distributed Importance Sampling. In ICLR, 2016.

[A3] GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training. In ICML, 2021.

Furthermore, we investigated the reasons why our IES method does not lead to catastrophic forgetting. Notably, the IES is a reversible method, which means that the removed samples have a chance to re-include in the training process if their second-order loss difference exceeds the threshold. Therefore, we tracked the average number of times the "early removed examples" were re-included in the training process, as shown in the following table:

Our proposed IES method allows these "early removed examples" to re-include in training for an average of about 13 times, with the most frequently re-include samples experiencing 26 training replays. We propose that this adaptive dynamic training mechanism contributes to the ability of IES method to effectively prevent "early removed examples" from being catastrophically forgotten during model training.

This analysis is included in Appendix H of the revised manuscript.

Q3: More Comparison

A3: We have conducted additional experiments on four diverse datasets: SVHN, STL-10, Oxford Flowers, and Stanford Cars. These datasets cover a range of image classification tasks, and the results are presented below:

For all datasets, our proposed IES method consistently achieves comparable or even slightly better accuracy than the baseline while saving a significant rate of mini-batches during training. Compared to the Small Loss & Rescale criterion in InfoBatch method, IES consistently maintains higher accuracy while saving more mini-batches.

Furthermore, as per your suggestion, we have included comparative experiments on high-level vision tasks, and the results have been added to Appendix E. The brief results are as follows:

Object Detection, Faster R-CNN (3 runs, mean±std):

Segmentation, DeepLab v3 (3 runs, mean±std):

These results demonstrate that our proposed "mastered" criterion can efficiently and effectively accelerate training compared to the Small Loss & Rescale criterion in InfoBatch method across both high-level vision tasks.

We would like to thank the reviewer for the detailed review and insightful comments. We sincerely appreciate your time and effort in reviewing our manuscript. Below, we address the questions you have raised.

Q1: Theoretical Analysis

A1: As mentioned in the Limitations section, we acknowledge that a theoretical analysis is not present in this work. Our primary focus was to empirically validate the effectiveness of the proposed IES method in helping models achieve better performance in an efficient manner. The empirical validations indicate the practical value and rationality of our method. These empirical validations cover classification, segmentation, and detection tasks, using different combinations of backbone and optimizers, and on nine visual datasets.

Additionally, we tracked various metrics during the training process, such as training loss, maximum eigenvalue of the Hessian matrix, and Sharpness-Aware Minimization (SAM) value. We utilized the Coefficient of Variation and experimental results to justify the selection of second-order differences as the criterion for sample removal. These measurements help explain our method's effectiveness.

The second-order difference of the loss function, $\Delta^2 L(w)$, used in IES captures the change in the rate of change of the loss function values. This can be seen as an approximation of the second-order directional derivative of the loss function, which is related to the curvature of the loss landscape and the Hessian matrix [A1]. The second-order difference of loss provides an indication of the local curvature of the loss landscape. In regions where the curvature is high, the gradient is likely to change rapidly, indicating that the model is still actively learning from the corresponding samples. Conversely, in regions where the curvature is low, the gradient is likely to be more stable, suggesting that the model's learning from these samples is slowing down.

Therefore, regarding potential directions for theoretical analysis, one could draw inspiration from previous works that have studied the use of gradient information for importance sampling. For example, Alain et al. [A2] analyzed importance sampling for SGD and proved that the variance-optimal sampling proposal $q^*(x)$ should be proportional to $p(x)||f(x)||_2$, where $p(x)$ is the data distribution and $f(x)$ is the gradient of the loss. Killamsetty et al. [A3] proposed selecting subsets to match the full gradient and proved convergence bounds under different conditions (Lipschitz continuity, smoothness, strong convexity).

Q2: Catastrophic Forgetting

A2: Thank you for your insightful and constructive suggestion. We agree that it's crucial to check whether our method might cause the model to suffer from catastrophic forgetting. Taking your advice, we have looked into whether the "early removed examples" in the IES method would be catastrophically forgotten by the model.

We define "early removed examples" as the first 5% of samples that are removed. We conducted experiments in a typical IES training environment with CIFAR-10, ResNet18, and SGD optimizer, which saves approximately 43% of the backpropagation samples in total 200 training epoch. We tracked the average training loss and accuracy of these "early removed examples" during the training process and compared them with the corresponding values of the entire training set. The experimental results are as follows:

The results demonstrate that the "early removed examples" are well learned (even better) by the model, and their training accuracy and loss are on par with other samples in the end of training. This implies that the model isn't catastrophically forgetting these "early removed examples".

Dear Reviewer TP1Y,

We are genuinely grateful for your prompt response. Your feedback has helped us significantly improve our manuscript.

Best regards,

Authors

The updates to the paper include 3 crucial experiments that I believe were missing from the original submission: (1) a study of catastrophic forgetting in the context of the proposed method, (2) a study on the introduced unwanted bias, and (3) a larger experimental study against SOTA. The authors study some of the limitations of their work, which IMO opens up the opportunity to directly address them in follow up works. Other points and questions were also addressed by the updated submission. I have thus updated my assessment to reflect the changes made to the paper.