Exploring Learning Complexity for Efficient Downstream Dataset Pruning

We appreciate the reviewer for the insightful and detailed comments. Please find our response below:

1. Role of the Learning Path [W1]

Thank you for pointing out the potential for misunderstanding. We first clarify that subnets with different numbers of parameters are deterministically produced with L1-based masking operation (line 197-199) instead of randomly. Such subnets with different capabilities constitute a viable learning path and allow us to distinguish between easy and hard samples as shown in Figure 2(a).

Importantly, we calculate the Learning Complexity score by approximating the definite integral of loss in the learning path. For efficiency, we sum the classification losses from subnets in the above learning path. Therefore, our method is designed to implement the learning path in a computationally efficient manner, which enables broader applications in practice.

2. Employment of pre-trained models for previous methods [W2]

Yes, the strategy can be directly applied to several previous methods. Here, we compare the performance of the original dataset pruning (FT) with the variants of pre-training models with average features as prediction head (PT), on several methods including Herding [2], k-CenterGreedy (kCG) [3] and Contextual Diversity (CD) [4]. The experiments are conducted with the fully pre-trained ResNet-18 model and we keep the same fine-tuning setting as in the manuscript.

As shown in the Table below, using our strategy can improve the average classification performance of these three pruning algorithms, over 5 downstream datasets and 9 pruning ratios. For example, the variant of CD outperforms the vanilla CD by 1.63% on average. However, those methods still perform worse than the random strategy, while our method obtains the best performance.

3. Comparison of FlexRand and the strategy from Dataset Quantization [W3]

Yes, the proposed strategy for alleviating distribution shift is similar but different from the idea in Dataset Quantization [1]. Specifically, the strategy in Dataset Quantization divides the full dataset into multiple bins of the same size, and the number of bins is defined as a hyperparameter. Differently, FlexRand divides the full dataset into two bins with different sizes: easy and hard bins, and we use the splitting hyper-parameter $\gamma\in(0,1)$ to split the dataset. In this way, samples in different bins have different probabilities of being selected, which enables adaptation in different settings. As recognized by Reviewer N2di, this FlexRand makes it robust across different data regimes, which is a valuable contribution to data pruning strategies.

Moreover, we empirically compare the performance of FlexRand and Dataset Quantization (DQ) using the same DLC score. The experiments are conducted with the fully pre-trained ResNet-18 model and we keep the same fine-tuning setting as in the manuscript. For the strategy in Dataset Quantization, we search the hyper-parameter within the default range {1, 5, 10, 20}, with the same method as the splitting $\gamma$ (see the 8th answer about hyper-parameter tuning). As shown in the Table below, FlexRand outperforms the strategy in DQ with better average classification performance over 5 downstream datasets and 9 pruning ratios. We add this comparison of different under-sampling strategies to Appendix C.2.4 of the manuscript.

I want to express thanks to the authors. Most of my concerns have been addressed.

Yet I still cannot understand the incorporation of the learning path very clearly. In the actual implementation, which is claimed to be computed efficiently, the classification losses are averaged. Although the concept of the learning path is interesting, it is not utilized in the implementation. I understand the average can be one of the implementations for the learning path, but it substantially weakens the significance of this concept. It can simply be replaced with models with different capabilities. In addition, you can still use the average of losses from different stages to reflect the learning path.

I suppose the learning path is one of the core ideas in this paper. If the authors can give a better utilization of the learning path in the implementation, I will increase the score.

Thank you for your positive and valuable suggestions. Please find our response below:

1. Results on pre-trained models with various sizes [W1]

Thank you for the suggestion. Indeed, we have employed pre-trained models with various sizes (including RN18, RN50, ViT-S, and ViT-B) in the main experiments. In Table 1, we present the results of models pre-trained by fully-supervised learning. In Appendix B.2.3, we present the results of models pre-trained by weakly-supervised learning. These results demonstrate the effectiveness of our method in different-sized pre-trained models.

2. Memory for multiple masked models storage [W2]

Thank you for pointing out the mistake in the limitation. We clarify that our method does not require storing all the masked models during the pruning. Instead, we dynamically generate the masked model using the l1_unstructured function in PyTorch. In particular, this function enables to pruning of a specific fraction of parameters for a given model. Hence, we calculate the outputs for all data points using the masked models sequentially and do not load the models simultaneously. Despite the sequential operation, our method only requires 1/35 of the computational times of previous methods, as shown in Table 1. To avoid any misunderstanding, we update the limitation in the revised version.

3. Discussion of potential failure cases [W3]

Thank you for the great suggestion. We conjecture that the effectiveness of our method relies on the model's capability. In particular, our method may fail to improve the performance if the pre-trained model cannot provide high-quality representations. To validate this, we conduct experiments with a ResNet-18 model pre-trained on a small dataset -- CIFAR-10. Obviously, the pre-trained model will perform poorly in producing the representations. We keep the same fine-tuning setting as in the manuscript and report the average classification accuracy over 9 pruning ratios.

As shown in the Table below, our method cannot outperform random selection in this case. Thus, the effectiveness of our method might be limited by the capability of pre-trained models (might be due to the small dataset). We add this discussion in the revised limitations.

Thank you for the positive and constructive feedback. Please find our response below:

1. Typos [W1]

Thank you for pointing out the typos. We have fixed these in the revised version.

2. Formulation of the pre-training weights masking [Q1]

Thank you for pointing out the missing details. Here, we provide a concrete formulation of the pre-training weights masking operation. Given the pre-training weights $\mathbf{\it{W}} \in \mathbb{R}^{n\*m}$ and masking ratio $r \in [0, 1)$, the masking matrix $\mathbf{\it{M}} \in \{0, 1\}^{n\*m}$ is constructed by:

where $\tau\_{r}$ is the $({n\*m\*r})$-th element in $\{W\_{1},...,W\_{n\*m}\}$ sorted by L1 norm in ascending order. Finally, the masked pre-training weights $\mathbf{\it{\hat{W}}}$ can be formulated as:

In our implementation, we utilize the l1_unstructured function in PyTorch to mask the pre-training weights. For clarification, we add this formulation in Appendix B of the updated manuscript.

3. Meaning of dotted lines in Figure 6(d) [Q2]

Thank you for pointing out the ambiguous description. In Figure 6(d), each dotted line denotes the results of preserving different percentages of data. In particular, the blue/green/orange dotted lines present the downstream classification accuracy with varying $\gamma$, when preserving 10%/20%/30% data respectively. For clarification, we add the above description in the caption of updated Figure 6(d).