We sincerely thank the reviewer for their valuable and constructive feedback. Below, we address the concerns raised.

Comparison with Dynamic Data Pruning Methods

Thank you for pointing out the relevant references, especially [1, 2]. To address your concerns, we conducted several experiments comparing recent dynamic pruning methods with static approaches, including DUAL pruning.

Before discussing the experimental results, let us first highlight two major differences between static data pruning and dynamic data pruning.

1. Compared to static pruning, dynamic pruning maintains access to the entire original dataset throughout training, allowing it to fully leverage all available information in the original dataset.
2. While both aim to improve training efficiency, their underlying goals differ slightly. Static data pruning seeks to identify a “fixed” subset that reduces the dataset size while preserving as much information about the original dataset as possible. This subset can then serve as a new, independent dataset, reusable across various model architectures and experimental setups. In contrast, dynamic data pruning enhances training efficiency within a single training session by pruning data dynamically on the fly. However, this approach requires storing the entire original dataset, making dynamic pruning less memory-efficient and not reusable.

Standard Training

We conducted experiments on CIFAR-10, 100 with ResNet-18, following the same hyperparameters as in Section 4.1 of our paper. All reported results are averaged over five runs. We first tested dynamic random pruning, which dynamically prunes randomly selected samples from the entire dataset at each epoch. Notably, dynamic random pruning significantly outperformed all static baselines, achieving test accuracies of 91.82% on CIFAR-10 and 72.8% on CIFAR-100 at a pruning ratio of 90%. We also evaluated [1, 2] and the results are provided here. Overall, dynamic methods consistently outperform static baselines. However, at lower pruning ratios (e.g., on CIFAR-10), DUAL can outperform dynamic methods under a similar computational budget.

As mentioned, we believe this performance gap stems from differences in accessible information: static methods are limited to 10% of the data, while dynamic methods use the full dataset. Consequently, the performance gap widens even further at aggressive pruning ratios, contrary to the reviewer’s expectations. To validate this, we plotted how often each sample was seen during training. The plot here shows that static methods are confined to a subset, while dynamic ones use nearly all data—rendering direct comparison somewhat unfair. Indeed, dynamic pruning methods might be better compared with scheduled batch-selection approaches, such as curriculum learning, rather than static pruning methods.

Label Noise Setting

We also evaluated these methods under label noise. In fact, [2] concluded that their method cannot prune any samples (corrupted or not) when label noise is introduced. Similarly, [1] tends to retain harder (and often noisy) samples, as it removes only easy examples during training. In contrast, DUAL effectively filters noisy samples, improving performance even beyond full-data training.

We conducted experiments on CIFAR-100 with a 40% label noise setting (full-train test accuracy: 52.74%) to verify this explanation. DUAL achieves over 70% test accuracy at a 50% pruning ratio, whereas InfoBatch achieves only 51.24% accuracy with a similar number of iterations. Under similar iterations, random dynamic pruning achieves 51.81% test accuracy, which still outperforms random static pruning (see Table 7 in Appendix). Lastly, IES [2] prunes only 1.7% of samples during training (consistent with the original report in their paper), resulting in 51.95% test accuracy. Furthermore, our static method can create fixed subsets in which nearly all noisy samples have been removed, resulting in high-quality datasets that can be preserved for future use.

Minor Comments

• For details on the Beta sampling method and its hyperparameters, please refer to the response to Reviewer YpaU.
• As suggested, we plotted test accuracy against total training time. The plots are available here. Results show that DUAL efficiently prunes data while achieving SOTA performance.

We will include these findings and additional discussions in the revised version.

---

[1] InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruning, ICLR 2024.

[2] Instance-dependent Early Stopping, ICLR 2025.

We appreciate your time and insightful comments. Below, we address the concerns and clarify any confusion raised.

1. Figures 2 & 3

First, we apologize for any confusion in Figures 2 and 3. Revised figures are available here (see Figure2_revised, Figure3_revised). In both figures:

• The Y-axis is the mean at epoch $T$: $mean_T(\mathbf{x},y)\coloneqq\bar{\mathbb{P}}=\frac{1}{T}\sum_{t=1}^T\mathbb{P}_t(y\mid\mathbf{x})$
• The X-axis is standard deviation at epoch $T$: $std_T(\mathbf{x},y)\coloneqq\sqrt{\frac{\sum_{t=1}^T[\mathbb{P}_t(y\mid\mathbf{x})-\bar{\mathbb{P}}]^2}{T-1}}$.

In Figure 2, we mislabeled the colorbars. Both the labels are revised as ‘Dyn-Unc Score’.

Figure 2a shows that if we calculate the Dyn-Unc score at epoch 60, the data points with the highest scores tend to evolve toward the top-right region by the end of training (epoch 90). However, this is not desirable for Dyn-Unc, as it prioritizes uncertain points (rightmost region) at the end of training (Figure 2b right). To effectively target that region earlier in training (e.g., at epoch 60), we should instead focus on the bottom-right region, since those points eventually move to the rightmost area by epoch 90.

Hence, we modify the Dyn-Unc score by multiplying it with 1-prediction mean, which helps us better sample this target region at an earlier stage, as shown in Figure 3. Also, the bold outline and the colorbar label of Figure 3 should be corrected to indicate Epoch 60.

Again, we sincerely apologize for the confusion this may have caused.

2. Computational Efficiency in ImageNet

First, we would like to clarify that CCS and TDDS require full training (90 epochs) to compute their scores on ImageNet. This is explicitly mentioned in Section 5.2 of TDDS [1] and in Appendix B of CCS [2]. The 60 epochs refer to the “post-pruning” training used for coreset, as described in [1]. For a fair comparison, we train 300,000 iterations on coresets, following the setup of CCS.

For further validation, we implemented CCS with AUM score calculated at epoch 60. As shown in the table below, the test accuracy is significantly lower than the original performance. Moreover, at low pruning ratios, the performance is even worse than random pruning (see Table 2 in our paper).

Therefore we can assert that our method is both time-efficient and superior compared to CCS. Also, we want to emphasize that saving 30 epochs of full ImageNet dataset training is a significant improvement. Training for 90 epochs on NVIDIA A6000 takes approximately 20h, thus reduction of 30 epochs can save about 6-7h.

There are a few subtle points for TDDS. In TDDS, the epochs used for full dataset training is 90, then they perform an exhaustive search to determine that around 30 epochs is optimal for pruning ImageNet by 70-90%. We evaluated their method across all pruning ratios using the reported score computation epoch. As a result, TDDS has a shorter overall training time when we ignore the exhaustive search; however, the important thing is that both their reported results and our reproduced experiments show that our method significantly outperforms TDDS in test accuracy.

3. Regarding Figure 4.

We can explicitly compare other baselines, such as CCS and TDDS, with DUAL in terms of total time spent as shown in Figure 4. However, many of the baseline plots are omitted because these methods require full training on the original dataset, which leads to excessive overlap in the curves. We will clarify this point more explicitly in the revision. We also plotted the test accuracy against the total time here, which hopefully addresses your concerns.

While we could also create a version of Figure 4 using ImageNet to compare our effectiveness with other SOTA methods such as CCS and D2, we present the CIFAR results in Figure 4 as it highlights the improvement achieved by our method.

4. Novelty of DUAL

The key novelty of DUAL is its time efficiency. Unlike most pruning methods that require full training to estimate sample difficulty, DUAL identifies informative samples much earlier. To address performance drops at high pruning ratios, we use Beta sampling to include easier samples. While the goal aligns with CCS and BOSS, our approach is distinct—it leverages prediction mean and a non-linear shift in the distribution mode. This allows us to prioritize harder samples at low or medium pruning ratios and easier ones at high ratios, contributing to our SOTA performance.

5. Minor comments

Thank you for pointing out the typo. We will fix them in the new version.

---

[1] Spanning training progress: Temporal dual-depth scoring (tdds) for enhanced dataset pruning, CVPR 2024.

[2] Coverage-centric coreset selection for high pruning rates, ICLR 2023.

Thank you for your constructive review and insightful suggestion.

Before we address your concerns regarding our beta sampling method, we would like to emphasize the novelty of the DUAL score lies in its time efficiency. Many existing pruning techniques require full training to estimate example difficulties, which makes the pruning process more expensive than training on the original dataset. Our DUAL can identify important samples more quickly, reducing the total time cost to less than that of a single training run. Beta sampling is applied to mitigate performance drops at high pruning ratios by selecting a larger portion of easier samples.

Details for Beta Sampling Design

• The choice of Beta PDF.

  The domain of Beta distribution is [0, 1], which naturally aligns with the range of prediction means. As the PDF decreases at the tails, ensuring that samples with extreme scores have a negligible chance of being selected. While other distributions, e.g. Gaussian, could also be used for modeling, their support is $\mathbb{R}$, which means they can assign a non-negligible probability to values far outside the desired range unless the standard deviation becomes extremely small.

• The choice for hyperparameters of the Beta Distribution.

  When the pruning ratio is set to 0, $\alpha$ and $\beta$ is configured so that the mean of the Beta distribution ($\frac{\alpha}{\alpha + \beta}$) aligns with the prediction mean of the highest-scoring sample. This helps target high score samples at low pruning ratios.

  To include easier samples at the high pruning ratios, we set parameters $\alpha$ and $\beta$ to depend on the pruning ratio. While BOSS changes them so that the Beta distribution's mode scales linearly with the pruning ratio $r$, we employ a non-linear scaling by raising $r$ to the power of $c_D$. This approach creates a PDF that is almost stationary at low pruning ratios and moves to the easier region polynomially as the pruning ratio increases.

  The hyperparameter $c_D$ is chosen based on the relative complexity of the dataset. We assume that the larger the dataset and the more samples per class there are, the easier the whole dataset is. Higher $c_D$ decreases $\beta$ and thus increases the mean and decreases the variance of Beta distribution. For a more difficult dataset, easy data has to be sampled more, so $c_D$ should be larger (refer to Fig. 15 of App. C).

  Recall that the variance of the Beta function is given by $\frac{\alpha \beta}{(\alpha + \beta)^2 (\alpha + \beta + 1)}$. According to our definition of the constant $C$ in Equation (5), where $C=\alpha+\beta$, increasing $C$ leads to a lower variance. This results in a more focused sampling distribution that concentrates on a specific region, improving the effectiveness of the sampling process by reducing unnecessary spread. The impact of the $C$ value is illustrated here.

  In conclusion, we fix a moderate value of $C$ as 15 across all experiments, and tune only $c_D$ based on the guidelines outlined above, which is not extensive. Furthermore, as demonstrated in Section 4.4.1 of our paper, the choice of hyperparameters including $c_D$ remain robust across a wide range of values.

Comparison with Dyn-Unc + Beta Sampling

First, we clarify that the performance gain by adding a beta sampling is higher for DUAL, not for Dyn-Unc. Table 4 should be corrected as Dyn-Unc+beta (+25.66) and DUAL+beta (+31.51) for CIFAR-10.

Beta sampling applied to Dyn-Unc utilizes the prediction mean at epoch 200 where Dyn-Unc is computed. Here, all samples are almost learned thus their prediction means become concentrated to 1. Therefore, Dyn-Unc with Beta sampling selects a much more easier samples, resulting in an inferior performance compared to DUAL.

We illustrate selected samples here. This figure clearly shows that Dyn-Unc+Beta actually selects easier samples. Furthermore, we test the Dyn-Unc+Beta for all pruning ratios and confirm that it is consistently worse than DUAL+Beta. The performance table can be found here.

Correlation between DUAL and D2, CCS

We appreciate this insightful suggestion. We visualize the selected subset by each method and examine the intersection here. CCS applies AUM with stratified sampling and a hard cutoff (pruning ratio, cutoff ratio): (70%, 20%), (80%, 40%), (90%, 50%). This approach completely excludes the most difficult samples while always retaining the easiest ones. D2 uses the forgetting score for graph-based sampling, applying the same hard cutoff as CCS. However, since it removes samples with the highest forgetting scores, it fails to eliminate the most difficult ones. DUAL selects uncertain and difficult samples, thereby excluding the hardest ones while simultaneously expanding coverage toward easier samples using Beta sampling.

We sincerely appreciate your insightful and valuable feedback. We address the given concerns and questions below.

1. Experiment with a more challenging dataset

Thank you for your suggestion. Due to time constraints, we experimented on a randomly sampled 20% subset of iNaturalist 2017 on ResNet-50, achieving 30.17% test accuracy. We applied random pruning and our DUAL+Beta method to prune 50% of this subset. Selected 50% subset is re-trained from scratch, and random pruning shows 12.6% test accuracy, while our method achieved 18.07%, showing its potential on challenging datasets. We additionally conducted experiments on the long-tailed CIFAR-10 and CIFAR-100, following [1]. Competitors are Random, EL2N, Dyn-Unc, and CCS for all pruning ratios. The table is attached here.

2. Extension to nonlinear separable cases

The key insight of our analysis is that the prediction variance of uncertain samples is lower than that of easy samples at the early stage; so we use the prediction mean to diminish the score of easy ones. In the non-linear case, the decision boundary is likely more complex than a hyperplane. However, the gist of our approach should still hold: samples near the decision boundary would exhibit higher variance over training epochs, while those far from it should stabilize quickly. One possible extension is to analyze the behavior of the DUAL score in non-linear models, e.g., deep neural networks. One might focus on how the feature representation evolves over training and thus how uncertainty propagates. A potential approach is to generalize the method in the representation learning framework. Instead of tracking the prediction variance and mean, one could study these in an appropriate feature space, e.g., the penultimate layer of the network. Such extension is out of scope, but we consider this an interesting future direction.

3. Stability of DUAL+Beta

We clarify that it is different runs altogether, i.e., repeated Appendix C.2 Algorithm 1. Beta sampling selects easier samples that tend to be more typical, leading to more stable performance than selecting the most uncertain and difficult samples. This phenomenon is observed in other methods used with beta sampling (see Table 4 in our paper).

4. Regarding Beta Sampling

We provide a detailed explanation for beta sampling in our response to Reviewer YPaU.

5. Under-performance of ResNet-50

We initially used ResNet-18's training parameters due to time constraints. We retrained ResNet-50 with batch size of 256, SGD with 0.9 momentum and 5e-4 weight decay, and a 0.3 lr with 3 epochs warm-up. With these parameters, full dataset test accuracy shows 80.1%. We also add a row indicating the case from R18 to R18 here, reflecting your suggestion.

6. Clarification of Table 4 & Figure 2.

Sorry for the confusion. Delta value for Dyn-Unc is 25.66, and for Ours is 32.14. We provide corrected Figure 2 and a detailed explanation in the response to Reviewer TH5d.

7. Missed essential reference

The SIM score [2] is composed of three factors: Class Separability (cluster overlap), Data Integrity ($\ell_2$-norm of feature representations), and Model Uncertainty (prediction consistency across independent models). All factors are computed at epoch 20 using a snapshot approach. As a result, stabilizing the SIM score requires 10 independent runs, incurring high computational costs. In contrast, our method leverages training dynamics to calculate the DUAL score within a single run, requiring fewer epochs. SIMS also proposes a ratio-adaptive sampling strategy, applying importance weights over the original score distribution. However, it assumes a normal distribution of scores, which does not hold in practice (see Fig. 2 of [2]). In contrast, our sampling method, by not relying on any specific score distribution, remains robust across diverse datasets. We will add a citation of SIMS [2] and this comparison in the new version.

8. Qualitative results on the pruned and kept samples

Thanks for your suggestion. We visualize pruned and selected samples by DUAL on ImageNet-1K here. DUAL tends to retain uncertain and challenging samples while discarding some typical instances and extremely difficult examples.

Other Questions

Q1) First, we emphasize that DUAL prioritizes samples with high uncertainty and high difficulty. Hence, the two cases you mentioned cannot be distinguished by the DUAL score alone. However, our Beta sampling favors samples with low difficulty and high uncertainty over those with high difficulty and low uncertainty.

Q3) As in our response to Review TH5d, ‘bottom-right’ is correct.

---

[1] Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss, NeurIPS 2019.

[2] Data pruning via separability, integrity, and model uncertainty-aware importance sampling, ICPR. Cham: Springer Nature Switzerland 2024.