Boosting Dataset Distillation with the Assistance of Crucial Samples

Q3: Comparison with SOTA methods like FTD and TESLA.

A3: The comparisons with FTD and TESLA are listed as follows. While these two methods are novel and provide good performances, it can be observed that our method holds a better performance in most cases. We have included this in Appendix G.

---

Q4: Could MSELoss lead to a bias in distilled data?

A4: Discarding a substantial proportion of sample losses in Eq.4 can help the optimization focus more on the hard-to-be-distinguished samples, making the distilled data helpful/biased in depicting a better decision boundary so that a model trained on these distilled samples can show comparable performances to the one trained on original data. This is consistent with the goal of dataset distillation described in the Introduction: Dataset distillation aims to learn a small set of synthetic examples from a large dataset such that a model trained on it can achieve a comparable test performance as one trained on the original dataset.

Thank you for the time and questions. Here we answer your questions point-by-point as follows.

Q1: More explanations of discarding samples.

A1: Thank you for the question. Actually, we try to explore "whether we should make a compromise to focus on the majority of data points around the center to ensure good performances on these data and what would happen if we discard these samples" (the last line in the first paragraph of Section 2.2). Motivated by this curiosity, we conduct extensive experiments on discarding easy samples or hard samples in Section2.2 (both the Figure 1 and Figure 5). The results on three datasets and four settings all suggest that dropping easy samples helps dataset distillation in all cases. And when the IPC is small, dropping hard ones can also help sometimes, which is consistent with your guess. Here, we provide two explanations and hope it can solve your puzzles.

• Recall that the goal for dataset distillation is to condense the large dataset into a smaller one such that a model trained on it can achieve a comparable test performance as one trained on the original dataset (see Introduction). Namely, we need to depict a good decision boundary for classification based on the distilled data. However, relying on easy samples solely may result in short-cut learning[a]. In contrast, hard samples are hard-to-be-distinguished samples (Eq.4) that usually exist along the decision boundary and thus can help to support a more explicit decision boundary. We think that's why hard samples can help to improve the dataset distillation performances.
• We have also explored it from both the perspective of data manifold and information in the third paragraph of Section 2.2. In short, we observed an increased manifold overlap between the generated images and the original dataset after discarding some easy samples (Figure 1 middle two images), thereby depicting a better representation of the manifold. We also find that making precise predictions for simple samples with hard samples is easy but the reverse is not. This indicates that harder samples are more informative than easier ones.

We have included this discussion in the Section B of the revision. We think this is a really interesting finding that makes our work unique and distinct. We hope it can bring more insights to scholars in dataset distillation.

---

Q2: Performance analysis.

A2: In Table 1 and 3, we have conducted experiments on 8 datasets under 20 settings. Compared to our baseline, our proposed method improves the baseline with an average relative improvement of 6.6% and an absolute improvement of 2.0% on 16 settings. When compared to SOTA approaches, our method has achieved the SOTA performance under 14 settings. Besides, the performances can be further boosted with more advanced studies, as shown in Table 4 and 8. For example, when combined with RFAD, our performance on CIFAR10, IPC=1 can achieve performance up to 54.8%, which is a new SOTA in Table 1.

What's more, we find the proposed module only fails to show superiority on small datasets like MNIST and F-MNIST but still holds a comparable performance. In contrast, it holds a superior performance on larger datasets such as T-ImageNet and ImageNet as shown in Table 1 and 3. Since the dataset distillation aims to solve the difficulty in processing large datasets, we believe performing better on large datasets is more important nowadays.

Dear Reviewer dBFm:

We thank you for the precious time and efforts in reviewing this paper. We have provided corresponding responses with elaborate discussions and extensive experiments on why dropping easy samples can help in dataset distillation, together with more discussion and comparisons with more studies. We have also include these in our revision (highlighted in blue). We hope to further discuss with you whether or not your concerns have been addressed appropriately. Please let us know if you have additional questions or ideas for improvement.

Looking forward to your reply.

Authors.

Q4: Comparison to 'Prune then distill'.

A4: 'Prune then distill' is a method that proposes to prune the dataset first and then conduct dataset distillation. Our method differs from it from two perspective:

• Compared to the vallina dataset distillation methods, the offline pruning will introduce extra time costs that you mentioned in the first question. In contrast, our method is an online one which just needs to rank the losses and average the topk losses, introducing almost no time costs as illustrated in the Answer to the fisrt question.
• The offline pruning may result in information loss by dropping samples before dataset distillation. Our online method ranks losses within batches during dataset distillation, it is after the samples can be represented by the learned samples (a.k.a 'NFR' loss is small) that these samples will be discarded. In this way, the information in easy samples can be maintained. The experimental comparisons are listed as follows. Note that since the 'prune then distill' adopts the distillation method MTT as their distillation method, here we provide MTT+Ours for a fair comparison in the table below, which indicates that our proposed method has a better performace. Detailed information can be found in Section G in the revision.

Table a: Test accuracy (%) comparison

---

Q6: Ablation study and the performance gain.

A6: The ablation study of each component is provided in the first paragraph (Ablation Studies on Each Proposed Module.) of Section 3.2 (ABLATION STUDIES). It is carried out on three datasets (Figure 3.a) and all the results indicate the effectiveness of both the proposed modules.

As for the performance gain, in Table 1 and 3, we have conducted experiments on $8$ datasets under $20$ settings. Compared to the baseline, our proposed method improves the baseline with an average relative improvement of $6.6$% and an absolute improvement of $2.0$% on $16$ settings, which is not marginal in the dataset distillation field (can be inferred by comparisons with SOTA methods RFAD, MTT and FRePo in Table 1).

Thank you for your time and valuable questions. We answer your questions point-by-point as follows.

Q1 : Discarding easy samples requires computing the NFR loss in FRePo twice, which would require longer running time for dataset distillation and make the baseline complex.

A1: The proposed method only requires to compute the 'NFR' loss once. To be specific, we first calculate the 'NFR' loss with Eq.4 to get the loss of each sample. Afterward, we sort the losses and average those with greater losses to obtain the final loss. Hence, compared to the baseline, we only added a step of sorting, whose time complexity is $\mathcal{O}(B \log B)$. $B$ is the batch size. This time cost for ranking is negligible compared to the forward and backward time costs. The python-like pseudo-code relative to the 'NFR loss' is listed below. Baseline is kernel_loss = mean_squared_loss(preds, labels).mean() , and ours is as follows:

kernel_loss = mean_squared_loss(preds, labels)
_, idx_ranked = top_k(kernel_loss, len(kernel_loss))
kernel_loss = (kernel_loss[idx_picked[:int((1-discarding_rate))*len(idx_ranked))]]).mean()

Furthermore, we have also provided the time cost comparison with our baseline in Appendix D, which writes "As for the training time cost, it is 2.5 hours (training time for 500,000 steps in total) under CIFAR10, IPC=10 setting while it is 2.4 hours for our baseline". This indicates that the proposed module introduces negligible extra computational and time costs.

---

Q2 : Discarding easy samples is heuristic and seems counterfactual.

A2: Thank you for the question . Actually, we hold the same curiosity with you on "whether we should make a compromise to focus on the majority of data points around the center to ensure good performances on these data and what would happen if we discard these samples" (the last line in the first paragraph of Section 2.2). Motivated by this curiosity, we conduct extensive experiments on discarding easy samples or hard samples in Section2.2 (both the Figure 1 and Figure 5). The results on three datasets and four settings all suggest that dropping easy samples helps dataset distillation in all cases. Here we provide some explanations below.

• The goal for dataset distillation is to condense the large dataset into a smaller one such that a model trained on it can achieve a comparable test performance as one trained on original dataset (see Introduction). In other words, we need to depict a good decision boundary for classification based on the distilled data. However, relying solely on easy samples may result in short-cut learning[a]. In contrast, hard samples are difficult-to-be-distinguished samples (Eq.4) that usually exist along the decision boundary thus can help to support a more explicit decision boundary. We think that's why hard samples can help to improve the dataset distillation performances.
• We have also explored it from both the perspective of data manifold and information in the third paragraph of Section 2.2. In short, we observed an increased manifold overlap between the generated images and the original dataset after discarding some easy samples (Figure 1 middle two images), thereby depicting a better representation of the manifold. We also find that making precise predictions for simple samples with hard samples is easy but the reverse is not. This indicates that harder samples are more informative than easier ones.

The above discussion has been included in Appendix B of the revision. We think this is a really interesting finding and it is the 'counterfactual nature' of our discovery that makes our work unique and distinct. We hope it can bring more insights to scholars in dataset distillation.

---

Q3: The data distribution in Fig. 1.

A3: In fact, we found that compared to the distilled samples generated by the baseline, the samples distilled by our method tend to be distributed not only in high-density spaces (central regions) but also in low-density spaces. This helps to increase the data manifold overlap with the original dataset.

Q5: Separate results of only using dropping strategy in Tab2, 3, 4, 8

A5: We list the results of using dropping/discarding strategy of Table 2, 3, 4, 8 in the following:

• Table 2 evaluates the cross-architecture generalization ability of the distilled data. This evaluation with only discarding operation can be found in Figure 3.d. To be short, removing easy samples can result in a small generalization ability degradation. However, this can be alleviated by our proposed ISA, as shown in Table 2 and Figure 3.d.

• Table 3 provides the evaluation of a larger dataset. The ablation studies of each proposed module on three datasets (CIFAR10, CIFAR100, Tiny-ImageNet) can be found in Figure 3.a. All the results validate the effectiveness of the proposed discarding strategy and ISA. We have also conducted experiments to provide discarding ablations on ImageNet datasets below. The results are consistent with ablation results in Figure 3.a that discarding some easy samples can help in dataset distillation.

Table b: Test accuracy (%) of adding discarding only (corresponding to Table 3 in main text)

• Table 4 and Table 8 provide experimental results of combining the proposed modules with other methods. We list the required results of adding discarding operation only to these methods below. Discarding can boost the performances in most scenarios. This is consistent with our findings.

Table c: Test accuracy (%) of adding discarding only (corresponding to Table 4 in main text)

Table d: Test accuracy (%) of adding discarding only (corresponding to Table 8 in main text)

Dear Reviewer R7gb:

We thank you for the precious review time and valuable comments. We have provided corresponding responses with elaborate discussions and extensive experiments on why dropping easy samples can help in dataset distillation, which we hope to address your concerns. We hope to further discuss with you whether or not your concerns have been addressed appropriately. Please let us know if you have additional questions or ideas for improvement.

Looking forward to your reply.

Authors.

Dear Reviewer R7gb:

Sorry to bother you again. As the rebuttal stage is drawing to a close (only 1-2 days left), we wanted to bring to your attention that we have not yet received any feedback from you yet. We have provided corresponding responses with elaborate discussions and extensive experiments on why dropping easy samples can help in dataset distillation a few days ago, which we hope to address your concerns. We hope to further discuss with you whether or not your concerns have been addressed appropriately. Please let us know if you have additional questions or ideas for improvement.

Looking forward to your reply.

Authors.

Thank you for your positive feedback and questions. Here we answer your questions point-by-point as follows.

Q1: The test accuracy of the proposed method doesn't consistently outperform existing techniques. When it does show an improvement, the margin is sometimes minimal.

A1: In Table 1 and 3, we have conducted experiments on $8$ datasets under $20$ settings. Compared to our baseline, our proposed method improves the baseline with an average relative improvement of $6.6$% and an absolute improvement of $2.0$% on $16$ settings, which is not marginal in dataset distillation field (can be inferred by comparisons with SOTA methods RFAD, MTT and FRePo in Table 1). When compared to SOTA approaches, our method has achieved the SOTA performance under $14$ settings. Besides, the performances can be further boosted with more advanced studies, as shown in Table 4 and 8. For example, when combined with RFAD, our performance on CIFAR10, IPC=1 can achieve performance up to $54.8$%, which is a new SOTA in Table 1.

What's more, we find the proposed module only fails to show superiority on small datasets like MNIST and F-MNIST but still shows to be comparable to other methods. In contrast, it holds a superior performance on larger datasets such as TinyImageNet and ImageNet as shown in Table 1 and 3. Since the dataset distillation aims to solve the difficulty in processing large datasets, we believe performing better on large datasets is more important nowadays.

---

Q2: Incomplete Review of Related Work

A2: Thank you for the suggestion. In this paper, we mainly discussed studies like MTT(CVPR2022), FRePo(NeuraIPS2022), RCIG(ICML2023). We have also included the discussion of the suggested work [1*][2*][3*][4*] and experimental comparison with the dataset distillation methods [3*] (IDC) [4*] (CAFE) in Section G of the revision. CAFE proposes to conduct feature alignment for a better dataset distillation performance while IDC is a novel method that analyzes the shortcomings of the existing gradient matching-based condensation methods and develops an effective optimization technique for improving the condensation of training data information. The comparisons are listed in the table below (IDC only provides its mean accuracy in its Table 10, here we only show the mean accuracy without standard deviation value). More results can be found in the revision. Besides, IDC also proposes to divide images into several parts to make full use of the storage budget (IDC+M). We also include this strategy (Ours+M) to make comparisons with IDC in table below.

Table a. Test accuracy (%) comparison.

Q3: Lack of Comparative analysis with data selection algorithms

A3: Thank you for the question. In this paper, we try to explore what kind of data is more critical during dataset distillation, "whether we should make a compromise to focus on the majority of data points around the center to ensure good performances on these data and what would happen if we discard these samples" (the last line in the first paragraph of Section 2.2). Motivated by this curiosity, we conducted extensive experiments on discarding easy samples or hard samples during training.

Compared to traditional data selection algorithms that find crucial samples first, requiring extra processing time and may lose some information in removed samples, our online method ranks losses within batches during dataset distillation. It is after the samples can be represented by the learned samples (loss in Eq.4 is small) that these samples will be discarded. In this way, the information in discarded samples can be maintained. Besides, the time cost for sorting losses is negligible, making our method more efficient than offline data selection methods. We have also compared our method with 'prune then distill'[a] which prunes the dataset first and then conducts dataset distillation. The experimental results show that when both are combined with MTT, we can achieve a higher performance. Besides, we have also compared our method with DREAM, who is also online method and proposes to conduct clustering (K-Means) to fetch representative samples before dataset distillation every certain iterations. After combining the proposed sample strategy with DREAM's baseline method for a fair comparison, our method can achieve $58.3$%(CIFAR10, IPC=1), $71.1$%(CIFAR10, IPC=10), $35.0$%(CIFAR100, IPC=1) while they are $51.1$%, $69.4$%, $29.5$% for DREAM. We also provide a figure illustrating the sampling differences between DREAM and Ours in Figure 9 of the revision. We hope it can solve your concerns. More detailed discussions can be found in Section G of the revision.

Table b. Test accuracy (%) comparison.

---

Q4: Why Data extension and not data augmentation/interpolation

A4: Thank you for the question. Both data augmentation and interpolation are techniques designed for dataset extension and differ from each other regarding the processing method. In other words, 'data extension' is the goal while 'data augmentation' and 'interpolation' are methods. In this paper, we named the title of Section 2.3 as 'Crucial sample extension with semantic augmentation' since we propose a method to "implicitly enrich harder ones in the semantic space through continuously interpolating between two target feature vectors" (the last paragraph of Introduction). We use semantic augmentation instead of data augmentation due to the reason that data augmentation typically refers to techniques that generate new training samples by applying various transformations or modifications to existing data, such as rotation, flipping, or adding noise. It is usually done before putting samples into models. In contrast, our proposed method conducts augmentation in semantic space. Experimental comparisons are also provided in Figure3.c, which indicates the superiority of the proposed semantic augmentation in the dataset distillation task.

---

References:

[a] Prune then distill: Dataset distillation with importance sampling, ICASSP 2023.

[b] DREAM: Efficient Dataset Distillation by Representative Matching. ICCV2023.

Dear Reviewer LMPF:

Thank you for the precious time and valuable suggestions to our paper. We have updated our revision to include your mentioned works and make comparisons with them, including more data selection methods and dataset distillation methods. And we also give the explanations on our performance gain in our last response. We hope these can address your concerns. As the rebuttal stage is drawing to a close, we hope to further discuss with you whether or not your concerns have been addressed appropriately. Please let us know if you have additional questions or ideas for improvement.

Looking forward to your reply.

Authors.

Dear reviewer LMPF:

Sorry for bothering you again. Since there is only a few hours left until the end of the discussion, we hope to further discuss with you whether or not your concerns have been addressed appropriately by our responses. Please let us know if you have additional questions or ideas for improvement.

Looking forward to your reply.

Authors.

We thank all the reviewers for helping us improve the paper. We just uploaded a newly revised version of our manuscript and marked the major changes in blue. In short,

1. We have included more discussion of why dropping easy samples can help in Appendix B. We have also included an example showing the differences between choosing the center vs focusing on harder samples in Figure 9. Many thanks to reviewer dBFm for helping us improve our paper.
2. We have added more comparisons and discussion with more studies in related work and Appendix G.

As the deadline of discussion period is getting closer, we hope other reviewers can reply to us whether the concerns have been addressed appropriately. We are more than willing to discuss with you.

Kind regards, Authors