Data Distillation Can Be Like Vodka: Distilling More Times For Better Quality

Thank you for your positive feedback on our work and for recognizing its novelty and significance in the field of dataset distillation. We appreciate your comments on the clarity of our writing and the robustness of our research motivation.

1. Additional Experimental Results on "DM+PDD": Certainly! We have conducted experiments with DM on CIFAR-10 with IPC=50. Note that DM does not distill the training dynamics but rather enforces the similarity between features extracted from synthetic and real samples by enormous models with random weights. This is why we did not consider DM initially in our experiments, as our goal is to capture longer-term training dynamics on full data. Our experiments show that for IPC=50, DM hits an accuracy of 63.0% while DM+PDD can reach 63.4%, showing a slight 0.4% improvement in performance. This mild improvement is expected, as DM does not capture any training dynamics of training on full data, and only captures similarity between features based on random weights.

5. Idea of Updating Previous Frozen Subset: We emphasize that our approach, while simple, is demonstrably effective and can be seamlessly integrated with any base method to enhance performance and scalability. This strength has been acknowledged by Reviewer S7ej.

Regarding the suggestion to update the previously frozen subset with a small learning rate, it is important to note that such an approach would void the scalability (i.e. generating larger synthetic data) enabled by our method since the samples distilled at previous stages need to be further optimized at later stages, requiring additional computational cost and memory consumption.

Nevertheless, we have conducted an experiment that allows images synthesized at early stages to be updated during late stages with a smaller learning rate (i.e., 10% of the original learning rate). We choose MTT as the base method and set up $5$ stages with a per-stage IPC of $10$, resulting in a total IPC of $50$. The trained model hits a test accuracy of 69.1%, which is inferior to our proposed method.

The decline in performance can likely be attributed to the fact that expert trajectories, i.e., the subsequent model checkpoints obtained by training on the full real dataset and synthetic samples (if applicable, from previous stages), are not derived from a consistent set of synthetic samples. Note that we first train on synthetic examples produced in previous stages to generate expert trajectories and synthesize data for the next stage. If the synthesized images of the previous stages are updated, the distilled samples from stage 1, which serve as the foundation for generating trajectories in stages 2 to 5, vary with each subsequent stage. When the samples synthesized in stage 1 are updated in stage 2, they gradually deviate from the initial synthetic samples (i.e., those employed to generate the expert trajectories for stage 2). Hence, training on the synthesized images that are being updated results in different training dynamics each time. This eventually leads to discrepancy and ultimately results in a decrease in performance. The above discussion has been incorporated into our manuscript.

We appreciate the reviewer's feedback and recognition of the clarity and effectiveness of our method. We address the specific concerns and suggestions as follows:

1. Missing Reference to DREAM [a]: Thank you for pointing out the missing reference. However, we would like to highlight that our submission to ICLR is concurrent with the public release (Aug 2023) of DREAM’s ICCV version. This timing meant we were not aware of the advancements presented in DREAM by the time we submitted our work.

Per your request, we have:

1. included and briefly discussed DREAM [a] in the revised manuscript (in Section 2; updates are highlighted in pink).
2. conducted experiments using DREAM [a] as the base method on CIFAR-10 (please refer to Question 2, and also the appendix).

However, this timing and the short period of rebuttal still result in a constraint on our ability to conduct additional experiments on CIFAR-100 and Tiny-ImageNet. We are committed to extending our research and will diligently work to incorporate these datasets into our analysis. We will share these expanded results as soon as they are available.

[a] DREAM: Efficient Dataset Distillation by Representative Matching, ICCV 2023

2. Experiment with DREAM as the Base Method: Certainly! We have conducted a set of experiments with DREAM as the base method on CIFAR-10 with IPC values of 10 and 50, respectively. The results are summarized in the table below. Using their provided implementation, we observed a slightly lower performance of DREAM, compared to what is reported in their paper (which might be due to the difference between computational devices and environments). However, using the same implementation and computational framework, we see that PDD can further improve the performance of DREAM beyond the current SOTA. These results have also been included in the Appendix and will be further incorporated into the main text.

3 & 4. Performance Comparison with DREAM We respectfully disagree that our method has a poor performance when IPC equals 10 due to the following reasons:

1. Firstly, we argue that it is not a fair comparison to contrast the accuracy of DREAM with the numbers in Table 1, as they deploy MTT or IDC as base methods.
2. Secondly, our method consistently improves the performance of base methods with substantial performance gaps. Specifically, PDD substantially improves the performance of IDC and MTT on CIFAR-10 by 0.4% and 1.6%, respectively. These improvements in performance are also acknowledged by Reviewer xT2K and S7ej.
3. Moreover, it is important to note that we only observe a smaller improvement for the combination of PDD and IDC on CIFAR-10 when IPC equals 10. This is likely due to IDC’s inability to distill longer training trajectories on real images (merely 4 epochs). Consequently, even extending to multiple stages is not enough for distilling more difficult images. For example, setting 5 stages approximately distills merely 20 epochs on real images, which is deemed insufficient. This is likely a limitation embedded in IDC, yet not in our PDD method.
4. Finally, the additional experiment results we provided in Question 2 show that PDD can also improve the performance of DREAM on CIFAR-10 with IPC=10 and 50.

Finally, we reiterate that our contribution lies in the revelation that a multi-stage synthesis approach can enhance the ability of base methods to capture training dynamics more effectively when compared to their single-stage counterparts. While distilling different subsets for different stages of training may seem intuitive, it has not been previously explored. We are the first to introduce this multi-stage synthesis framework, which is acknowledged as “intriguing and groundbreaking” by Reviewer xT2K and “interesting” by Reviewer S7ej. More importantly, our framework not only improves performance on standard IPCs, such as 10 and 50, but also represents a unique advance by enabling the generation of a large number of samples for the first time. This can equip any base method with the capability to significantly reduce the performance gap compared to training with complete datasets.

4. More Discussion on Benefits of PDD: Thank you for the suggestion to elaborate on the benefits of PDD in our manuscript. In the revised version, we discussed the advantages of our method again in both the introduction and conclusion sections (highlighted in pink). Specifically, our original introduction highlights that PDD captures the training dynamics of neural networks over longer intervals more effectively. We have added the following suggestion by the reviewer to our introduction: Importantly, PDD reduces the complexity associated with training the entire synthetic dataset simultaneously, and can serve as an effective base method for slimmable dataset condensation [a] which handles changes in the budget for storage or transmission.

We hope that these revisions and clarifications will address the concerns raised and demonstrate the value of our proposed PDD method. We appreciate the opportunity to improve our manuscript and thank the reviewer for their valuable input.

[a] Slimmable Dataset Condensation, CVPR 2023.

We sincerely thank the reviewer for the constructive feedback. We address your concerns point-to-point below.

1. Further explanation: We’d like to note that the details of our method are accurately specified in the paper. We will further elaborate below:

1.1. Explanation of PDD for Figure 1: Thank you for your feedback regarding Figure 1. This Figure is mainly the motivation of our method, and hence the details are specified later. To enhance the clarity, we have updated the caption of Figure 1 to: “Our proposed multi-stage dataset distillation framework, PDD, improves the state-of-the-art algorithms by iteratively distilling smaller synthetic subsets that capture longer training dynamics on full data. In the setting shown in the figure, PDD uses MTT as its base distillation method to incrementally generate $5$ synthetic subsets of size $10$, where each subset captures $15$ epochs of training on full data.“

1.2. Clarification about Base Method Modification We clarify that PDD seamlessly integrates with the base methods with hardly any modification and without introducing any additional hyperparameters (which is why our paper does not include discussion on extra hyperparameters):

For the first stage, we distill datasets using the base methods in their original form. Starting from the second stage, we distill with the same base algorithm. However, the model weights employed in these stages are those that have been trained on examples distilled in the earlier stages. For instance, we train the newly sampled networks with previous synthesized samples for IDC, and we start new collections of training trajectories from weights trained with previous samples for MTT. These operations are detailed in our pseudocode and also formulated in Equation (4). This ensures a smooth integration of PDD with existing frameworks, enhancing their effectiveness without the need for fundamental modifications to the base methods.

2. Practicality of Multi-Stage Training: Our method does not introduce additional hyperparameters: IPC per stage and the number of stages are determined based on the available budget, and PDD inherits all the rest of hyperparameters, such as learning rate, weight decay, number of training steps, etc., from the base method. This ensures that the practicality of our method is maintained by leveraging existing parameters, thereby simplifying the adoption and integration of PDD in various settings.

Furthermore, we wanted to emphasize that our method brings significant performance gain for the following reasons:

1. Our results have already outperformed current state-of-the-art methods in multiple scenarios, even when using less powerful base methods. For instance, on CIFAR-10 with an IPC of 50, our approach combining PDD and IDC surpasses DREAM, the current SOTA by 1.7%. Similarly, on CIFAR-100 with an IPC of 50, PDD + IDC outperforms DREAM by 0.5%. Notably, PDD can be also easily combined with DREAM to improve its performance and achieve SOTA, as we will report in our response to reviewer rp5C.
2. Across other settings, our methods consistently enhance the performance of the base methods, approaching the performance level of the current state-of-the-art benchmarks.

3. Discarding Easy-to-Learn Examples: Discarding difficult-to-learn examples is very effective in improving the efficiency of our method, and also improves its performance for smaller values of P. Table 6 shows that around 2/3 of examples can be discarded in the first stage and around 2/3 of examples can be discarded in the second and third stages. That is, only 1/3 of the (easiest-to-learn) data is required for the first stage and another 1/3 (easy but not easiest examples) is required for the second and third stages. Discarding difficult-to-learn examples not only considerably improves the distillation efficiency, but also improves the performance of dataset distillation, especially in earlier stages. This is because easy-to-learn examples contribute the most to learning in the early stages. Hence, focusing the distillation on such examples in earlier stages improves both the efficiency and the quality of the generated synthetic examples, as is evident in Table 6, P=1 and P=3.