Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation

Thank you for your valuable feedback. We have provided detailed responses to the issues raised and hope our explanations address your concerns. We welcome any further suggestions to improve our work.

Q1: The proposed techniques in the paper are not new, such as PCA and augmentation approximation networks.

A1: To the best of our knowledge, there has been no prior work integrating PCA or augmentation approximation networks into a dataset distillation framework. While these techniques might be individually familiar, their integration within a comprehensive framework for self-supervised dataset distillation is novel, as well as highlighting the advantage of our proposed method to be simple and effective. In brief, our method enhances its efficacy in distilling compact and transferable datasets, as detailed in Section 3.2 and 3.3 of the paper.

Q2: The proposed technique leverages data augmentation while minimizing bias, and similar ideas have been explored in self-supervised learning. It is important to compare it with other analogous methods [1][2][3].

A2: The goals between self-supervised learning (SSL) and dataset distillation (DD) differ fundamentally. SSL aims to learn feature extractors that generalize to downstream tasks, often benefiting from augmentation to improve representation quality, as seen in [1][2][3]. In contrast, dataset distillation aims to compress datasets into a smaller storage size while preserving their training performance. Specifically, KRR-ST [4], a self-supervised dataset distillation framework, leverage a self-supervisedly trained backbone to perform DD. It demonstrated that random data augmentation introduces gradient bias in bilevel optimization, deeming it incompatible with dataset distillation. To address this, we propose predetermined augmentations to avoid randomness while improves condensed performance, as elaborated in Section 3.3. This is a distinct approach from typical augmentation strategies in SSL.

Q3: Could the authors provide more details about the approximation networks, such as the number of networks used, structure, and layers?

A3: Our experiments use rotation augmentation as the default. The augmentation includes rotations of $0^\circ, 90^\circ, 180^\circ, 270^\circ$, requiring three distinct approximation networks to predict representation shifts for $90^\circ, 180^\circ, 270^\circ$ from $0^\circ$. These networks are lightweight, designed as 2-layer perceptrons with hidden layer sizes of 4 (for CIFAR100) and 16 (for TinyImageNet/ImageNet). As detailed in Section A.1, each network contains 4,612 parameters for CIFAR100 and 16,912 parameters for larger datasets. Additional implementation details can be found in Section 3.3 and Appendix A.1​.

Q4: Could the authors show a comparison of the distilled data sizes?

A4: We provide comparisons of distilled data sizes in Table 2 and Table 7, conducted on CIFAR100 and TinyImageNet, respectively. These tables demonstrate the superior performance of the proposed method across various memory budget sizes.

[1] Improving Transferability of Representations via Augmentation-Aware Self-Supervision. NeurIPS 2021 [2] Improving Self-supervised Learning with Automated Unsupervised Outlier Arbitration. NeurIPS 2021 [3] RSA: Reducing Semantic Shift from Aggressive Augmentations for Self-supervised Learning. NeurIPS 2022 [4] SELF-SUPERVISED DATASET DISTILLATION FOR TRANSFER LEARNING. ICLR 2024

We sincerely appreciate your thoughtful feedback and the time you have dedicated to reviewing our paper. Please let us know if you have any further suggestions or comments. If there are any additional questions or issues you would like to discuss, we are fully committed to engaging further to enhance our paper.

I apologize for my delayed response.

After reviewing other reviewers' comments, I have concerns about the proposed method's resource requirements. In contrast to the research aim of reducing training costs, the current approach necessitates much more resources.

If so, why do users not simply train on the original data directly, which would seem like a more straightforward and cost-effective approach? Given this limitation, I understand why the authors are hesitant to apply their method to ImageNet-1K with common settings, 224 * 224 and 1000 classes.

Q: Concerns about resource requirements and the cost-effectiveness of dataset distillation. Why not train directly on the original data?

A: This is an important question about the goals and procedure of dataset distillation.

As stated in the abstract: "Dataset Distillation becomes a popular technique recently to reduce the dataset size via learning a highly compact set of representative exemplars, where the model trained with these exemplars ideally should have comparable performance with respect to the one trained with the full dataset." This goal has been widely recognized and pursued in many recent works, including [1][2][3][4]. As highlighted in [3], "it is prohibitively costly to use all the examples from the huge dataset, which motivates the need to compress the full dataset into a small representative set of examples." This underscores the necessity of compressing large-scale datasets into smaller, more manageable ones.

Dataset distillation involves two major steps:

1. Synthesizing the distilled dataset – This step involves computationally intensive optimization to create a smaller, representative dataset.
2. Training a new feature extractor – The distilled dataset is then used to train a new model, with significantly lower computing requirements compared to training on the full dataset.

The concerns raised by Reviewer TeEU and Reviewer MMPZ regarding computational cost focus on the first step, which indeed requires more resources. However, this cost is a one-time investment. Once the distilled dataset is generated, it can be reused multiple times across different tasks or models, significantly reducing the cost for downstream training since the distilled dataset is much smaller than the original dataset.

In summary, by providing a compact yet highly effective dataset, dataset distillation facilitates efficient and flexible usage, aligning with these practical needs.

[1] Dataset Condensation with Gradient Matching (Bo Zhao et al., ICLR 2021) [2] Dataset Distillation by Matching Training Trajectories (George Cazenavette et al., CVPR 2022) [3] Self-Supervised Dataset Distillation for Transfer Learning (Dong Bok Lee & Seanie Lee et al., ICLR 2024) [4] Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matching (Ziyao Guo & Kai Wang et al., ICLR 2024)

Thank you for your valuable feedback. We have provided detailed responses to the issues raised and hope our explanations address your concerns. We welcome any further suggestions to improve our work.

Q1: unique challenges exist for self-supervised data distillation and how their method specifically addresses those challenges.

A1: Self-supervised dataset distillation presents unique challenges, particularly the instability caused by the randomness of data augmentation, which is a crucial element in self-supervised learning. Prior work [1] has highlighted this issue, noting its adverse effects on bilevel optimization. We address this challenge by introducing predetermined augmentations, as detailed in Section 3.3, which eliminate randomness and maintain consistent gradients during optimization. Additionally, to enhance storage efficiency, we parameterize the distilled images and their representations into low-dimensional bases, significantly reducing redundancy. Furthermore, we introduce lightweight approximation networks to model the representation shifts caused by augmentations, enabling compact and efficient storage of augmented data. The effectiveness of these components is validated through an ablation study in Table 3, where each proposed component is shown to contribute significantly to the overall performance of the distilled dataset.

[1] SELF-SUPERVISED DATASET DISTILLATION FOR TRANSFER LEARNING. ICLR 2024

Q1: I find that the experimental support is a bit lacking. As is common in Dataset Distillation works, it is generally good practice to show the scaling over different memory budges (N) on various datasets, rather than just a single dataset, in order to show generalizability.

A1: We appreciate the suggestion to extend the experiments to demonstrate generalizability over varying memory budgets. In addition to the results on CIFAR100 which we have provided in the main paper (cf. Table 2), as shown in Table 7 (in Appendix), we have conducted experiments on TinyImageNet with memory budgets $N=50, 100, 200$. Moreover, we further extended the experiment to $N=500$. The results below confirm that our method consistently outperforms baselines across a range of memory budgets, showcasing its scalability.

Q2: I noticed that the resolutions on ImageNet scale to 64 x 64 -- however recently, the field has shifted to higher resolutions such as 128x128 or even 512 x 512 -- I think it would be important to see if the method can scale well to larger resolutions.

A2: We recognize the importance of validating the method's performance on higher resolution datasets. To address this, we conducted an experiment on ImageNette (a 10-class subset of ImageNet) with 128x128 resolution. The results demonstrate that our method scales effectively to larger resolutions.

Q3: I think another important criteria that should be included is Applications -- as alluded to in the paper tasks like continual learning or neural architecture search (line 43) are important in the field, however none of these results were included in the main paper -- I think it is important to test the applicability of the method in order to determine significance and impact.

A3: We appreciate the reviewer’s insightful suggestion regarding the inclusion of applications like continual learning and neural architecture search. While our current work primarily focuses on demonstrating the generalizability of the distilled data through various downstream tasks, as shown in the linear evaluation results, we agree that exploring these applications would further strengthen the impact of our method. In future work, we plan to conduct experiments on continual learning and neural architecture search to validate the broader applicability of our approach. The proposed method ability to train a decent feature extractor, as demonstrated in the paper, provides a solid foundation for such tasks. We anticipate that the compactness of distilled dataset and generalization properties will prove beneficial in these scenarios as well.

Q4: Given that this approach involves multi-level optimization, I think efficiency metrics should be compared as well (time per step, GPU memory etc). -- This will demonstrate whether the gain in performance is justified over other methods when comparing the relative compute demands.

A4: Based on CIFAR-100 with $N=100$, we evaluated GPU memory usage and execution time. The results demonstrate that while our method requires approximately 1.5x more GPU memory than KRR-ST, it remains feasible within modern hardware constraints (e.g., NVIDIA RTX 4090 with 24GB memory). Furthermore, our method's execution time is comparable to other baselines, as shown below.

We sincerely appreciate your thoughtful feedback and the time you have dedicated to reviewing our paper. Please let us know if you have any further suggestions or comments. If there are any additional questions or issues you would like to discuss, we are fully committed to engaging further to enhance our paper.

Q1: The datasets in the experiments are CIFAR 100 and datasets with similar image attributes. I can understand it is possible to get a distilled dataset in a lab environment and the datasets are very feature-controllable. Do you have space to show that your experiment can be successful in other different scenarios? For example, some randomly taken images.

A1: The reviewer raises a valid concern about the generalizability of our method beyond controlled datasets like CIFAR-100. To address this, we had conducted additional experiments on ImageNet and TinyImageNet datasets, whose details and results are presented in Appendices A.3, A.4, and A.5. These datasets offer increased diversity and scale, and the results consistently demonstrate the robustness of our method across different scenarios. Importantly, our distilled datasets exhibit superior cross-architecture generalizability, as evidenced by linear evaluation performance across multiple feature extractors, including VGG11, ResNet18, AlexNet, and MobileNet..etc. The success on datasets like ImageNet, which closely resemble real-world data, highlights the potential for applying our approach to scenarios involving randomly collected images.

Q2: Though this is a memory saving method, a very large portion of the whole method is still computing intensive. Do you have any benchmark to show that the whole method could be executed in an efficient way?

A2: We appreciate the concern regarding computational intensity. Our methodology primarily focuses on achieving storage efficiency after the distillation process by employing parameterization. This approach decomposes images and representations into linear bases and coefficients, significantly reducing storage requirements. The combination of bases and coefficients involves matrix multiplication, which incurs minimal computational overhead. Additionally, the use of lightweight 2-layer perceptron approximation networks ensures computational simplicity. While our primary objective is storage efficiency, we recognize the importance of evaluating the computational cost during the distillation process. To address this, we benchmarked GPU memory usage and execution time using the CIFAR-100 dataset with $N=100$. The results, summarized below, indicate that our method requires approximately 1.5× more GPU memory than KRR-ST, yet remains manageable within the capacity of modern GPUs like the NVIDIA RTX 4090 (24GB). Furthermore, the computation time of our method is comparable to other baselines and does not impose a significant burden.

Q3: Difficult for practitioners to implement and tune the method without extensive expertise in self-supervised learning and dataset distillation

A3: We understand the concern regarding the accessibility of our method for practitioners. However, the core components of our approach—parameterization and approximation networks—can be implemented with just a few lines of code using popular deep learning frameworks like PyTorch or TensorFlow. Basically (cf. Figure 1), our pretraining step adopts the well-known and widely-adopted self-supervised learning scheme, Barlow Twins; our parameterization is based on the fundamental machine learning tool, PCA; our bilevel-optimization follows the practice of KRR-ST; and our approximate networks are simply multilayer perceptron while their training follows the typical supervised learning procedure, in which they are definitely not sophisticated. Detailed guidelines and implementation examples are provided in the supplementary materials to further assist practitioners in replicating the method.