Heavy Labels Out! Dataset Distillation with Label Space Lightening

Weakness 1

The generalizability of method to practical datasets like clinical and aerial images and evaluating its performance on ViT models.

Thanks for the reviewer's valuable comments. Our method is specifically designed for large-scale dataset distillation. We conduct experiments on the full-sized, original version of ImageNet-1K. We also conduct experiments on the aerial datasets AID[1] to show the effectiveness of our proposed method. The experiments are conducted under the settings of AID with IPC 1, 5, and 10. The performance results and the required extra storage costs are as follows (here, we adopt online soft label generation and regard the teacher model as the extra storage costs). The results show the effectiveness of our proposed method across different high-resolution datasets.

We have conducted cross-architecture evaluation experiments on Swin-V2-Tiny under the ImageNet-1K setting with IPC 10, as presented in the paper. Additionally, we perform supplementary experiments on ViT-B-16 under the same settings. The evaluation results indicate that our proposed method demonstrates a stronger generalization ability on transformer-based models. We will include the cross-architecture evaluation results on ViT-B-16 in the updated version of the paper.

[1] Gui-Song Xia, Jingwen Hu, Fan Hu, Baoguang Shi, Xiang Bai, Yanfei Zhong, Liangpei Zhang, and Xiaoqiang Lu. Aid: A benchmark data set for performance evaluation of aerial scene classification. IEEE Transactions on Geoscience and Remote Sensing, 55(7):3965–3981, 2017.

Weakness 2

An analysis of distillation costs (memory usage and training time).

Thanks for the reviewer's valuable questions and comments. A detailed comparison of the training costs between the proposed method and state-of-the-art methods is indeed of great importance, and we will include this result in the updated version of the paper. Here, we conduct experiments under the ImageNet-1K IPC 10 setting. All methods are evaluated based on the official code, and all experimental configurations, including hyperparameters, are set according to the official default values provided by the authors. Specifically, we measure the time required for each single downstream training iteration as well as the overall memory cost. All experiments are conducted on one single NVIDIA RTX A5000 GPU.

Weakness 3

The paper lacks comparisons with key state-of-the-art methods such as DREAM, DataDAM, and SeqMatch in Table 2.

Thanks for the reviewer's questions and suggestions. Our method specifically addresses the issue of high storage costs associated with soft labels in large-scale dataset distillation, such as the full-sized ImageNet-1K. In contrast, methods like DREAM, DataDAM, and SeqMatch primarily focus on the distillation of smaller datasets, such as CIFAR-10. Due to computational resource constraints, these approaches face challenges in scaling up to larger datasets, such as the full-sized ImageNet-1K. Moreover, our proposed method has already been compared with the current state-of-the-art methods for large-scale dataset distillation in the paper.

We greatly appreciate the reviewer’s comprehensive review. We hope our previous responses have satisfactorily addressed all concerns. Should any further questions arise or if there are topics that require more detailed discussion, we remain fully available and would be pleased to continue the discussion.

Weakness 1

Provide a more comprehensive comparison for the storage requirements across all methods.

Thanks for the reviewer's valuable suggestions. Our intention is to illustrate the storage under different soft label generation strategies. We apologize for any confusion this may have caused. The current state-of-the-art large-scale dataset distillation methods for soft label generation can be categorized into two main approaches: one involves pre-generating and storing the labels (e.g., Yin et al. (2024) and Shao et al. (2024)), while the other generates labels online during downstream training (e.g., Sun et al. (2024)). Our method primarily falls into the second category. Specifically, for the first strategy, the storage costs are from two parts. 1) The labels for each augmented image. Here, for each iteration, the synthetic images will apply well-defined augmentation strategies, e.g., CutMix, and each augmented image should be assigned with soft labels. 2) The information of the augmentation, e.g., the image ID, flip status, coords status for CutMix, for downstream reproduction. The other approach generates soft labels dynamically during the training process, so the additional storage required is limited to the teacher model. We will improve the presentation of Table 1 in the updated version to provide a clearer comparison. The actual storage costs comparison is as follows:

Weakness 2

Reproduce for the baseline methods on Places365-Standard.

Thanks for the reviewer's valuable suggestions. We reproduce the results for SRe2L on the Places365-Standard, the results are shown as follows. For G-VBSM, since it requires preparing a model pool of well-trained models with different architectures on the target dataset and selecting the most suitable model architecture combination, we are still in the process of preparation and evaluation. We will update the paper immediately once the results are available. The evaluation results show that our proposed method surpasses the SRe$^2$L for all different IPC settings on the Places365-Standard dataset.

Weakness 3 & Question 1

Ambiguous notations.

Thanks very much for the reviewer's thorough review. We apologize for any misunderstanding caused by the way we presented our formulas. We will make corrections in the next version.

1. We are sincerely sorry for this typo, $r_i$ here should represent the same as $r^{(i)}$, the text description of the class $i$.
2. We again apologize for the typo, and any misunderstanding caused. Line 199 should be "$v_I\in\mathbb{R}^{B\times d_f}$ and $v_T\in \mathbb{R}^{C\times d_f}$ refer to the normalized embedding for the input image and the text description, $v_T^{(i)}$ is for $i^{th}$ class."
3. Here, we intend to represent it as $W=[w^{(0)}, w^{(1)}, \dots, w^{(C-1)}]$, we are so sorry for the unclear representation, and we will correct all of the above ambiguous notations in the new version. Thanks again for the reviewer's thorough and careful reviews.

We deeply appreciate the reviewer’s thoughtful insights and hope that our response has resolved the issues raised. Should there be any further questions or points that require more in-depth discussion, we would be honored to continue the conversation and provide any needed clarifications.

Weakness

Thanks for the reviewer's valuable questions and suggestions. For current large-scale dataset distillation methods, at each downstream iteration, the synthetic images are augmented using well-defined strategies, such as CutMix. Each augmented image is then assigned a soft label to ensure the accuracy and informativeness of the knowledge transfer for downstream tasks. Consequently, the total number of soft labels is calculated as IPC × number of classes × number of downstream training epochs. In the case of previous methods, the number of training epochs for downstream training is set to 300; therefore, we calculate the storage requirement assuming 300 epochs. This results in a significant storage cost for maintaining the soft labels. Also, there still requires extra storage space to store the information of the augmentation parameters, such as the image ID, flip status, and coords status, for downstream reproduction. Thanks for the reviewer's valuable suggestions, we will modify and improve the figures in the new version.

Question 1

Results on more complex tasks.

Yes, here, we conducted additional experiments on semantic segmentation tasks. Following the definition of the dataset distillation, we select 0.09% of the dataset SA-1B, 10,000 images for downstream training. Here, the baseline performance is evaluated on the original SAM-B model (89.58M parameters, 358.32MB actual storage) to generate soft labels for the downstream model training. Our proposed method adopts SAM-B as the base model and applies our proposed label space lightening strategy to reduce the storage costs, finally with only 5.64M parameters (actual storage 22.56MB) trained and stored. For the baseline performance, the downstream model achieves mIoU of 49.46%, while our proposed method, with only 6.3% storage costs, achieves higher performance with mIoU of 50.75%.

Performance on semantic segmentation tasks. Here, SAM-B Directly Guided refers to directly adopting the SAM-B model to generate the soft labels for downstream tasks. Our method utilizes SAM-B as the base model with only 6.3% of the original costs to achieve better performance.

Question 2

Comparison with naive compressions such as quantization.

We sincerely appreciate the reviewer’s insightful comments. Indeed, a direct comparison involving label quantization is a crucial aspect that warrants careful examination. In response, we have conducted experiments under the ImageNet-1K IPC 10 setting on the NVIDIA RTX A5000 GPU, where we applied precision reduction to the labels for the previous state-of-the-art method and compared the results of our proposed method. From the results, it can be observed that there is no significant decline in downstream performance when reducing the precision to FP16 (50%) or INT8 (25%). However, a noticeable decline occurs when the precision is further reduced to INT4 (12.5%), which represents the limit for logit quantization. Despite this, we are able to compress storage to less than 0.1% with comparable performance.

We are truly grateful for the reviewer’s valuable time and constructive feedback. We hope our response has satisfactorily resolved the concerns raised. Should there be any additional questions or areas where more clarification is required, we would be pleased to engage in further discussion and provide any additional information.

We are truly grateful for the reviewer's constructive comments. We have implemented the following revisions in the paper according to the reviewer's feedback:

1. We have conducted a comparison with the logits quantization strategies, and have added the experimental details and results in Section 4.5 of the main text. The additions have been highlighted in red for the reviewer's convenience.
2. We have successfully adapted our method to segmentation tasks, and have included the experimental details and results in Section B.4 of the supplementary material. The additions have been highlighted in red for the reviewer's convenience.
3. We have revised Table 1 and provided additional explanations to make the importance of the topic being addressed in our paper more clear.
4. We thank the reviewer for their attention to detail. We have revised Figure 1 to ensure that the text is not covered by the symbols, making it more professional.

We sincerely thank the reviewer once again for the reviewer's comments and suggestions. We hope that our responses have addressed the reviewer's concerns, and we look forward to further discussions with the reviewer.

Weakness 1

W1.1 Replace CLIP with stronger supervised models or self-supervised models.

Thanks very much for the reviewer's valuable suggestions. Here, as mentioned in our paper, we adopt foundation models like CLIP for the following reasons: 1) it has been pre-trained on the massive data, and can readily adapt and generalize to various target datasets; 2) it has rich and feasible knowledge that can be leveraged during the training process; 3) it is open-source, such that it does not require extra storage space and can be accessed on demand. Benefiting from the aforementioned characteristics, the transformation to different downstream datasets can be achieved by LoRA, instead of finetuning or re-training the entire model. Also, the storage is significantly reduced as we only need to store some low-rank matrices.

Here, we would like to mention that other foundation models, like DINOv2, are also applicable in our proposed method as an alternative to CLIP for LoRA-Like Knowledge Transfer and other strategies. As shown in the following table, we also conduct the experiments under the settings of ImageNet-1K with IPC 10 for DINOv2 with LoRA-Like Knowledge Transfer.

In other words, we use CLIP in the main manuscript as a proof-of-concept validation of our proposed method. We will also add the experiments on other foundation models for further demonstration.

For other supervised models like ResNet and ViT, we do not adopt them for the following reasons: 1) it can not easily generalize to other datasets, and it may modify the architecture and re-train the whole model to satisfy the downstream requirements; 2) it cannot be accessed easily. As for arbitrary datasets, it can be difficult to find the pre-trained models for the specific target dataset; and 3) compared with the very little storage costs as the LoRA required, the storage costs for ResNet and ViT are increasing if we would like to process more datasets.

In addition, directly employing strong models for label generation is not necessarily beneficial for downstream tasks. Here, we utilize stronger models ResNet-50, ViT_b_16, CLIP (directly applied), and DINOv2 (directly applied) for label generation, the results are shown as follows. The experiments are under the settings of ImageNet-1K with IPC 10.

W1.2 Finetune the foundation models (CLIP or DINOv2) with LoRA on the distilled datasets.

Directly finetuning the foundation models with LoRA on the distilled datasets leads to overfitting, as the distilled dataset is significantly smaller compared to the original datasets. This also presents challenges in generalizing to downstream tasks. We also conduct experiments on CLIP for finetuning with LoRA on the distilled datasets. The evaluation result is 39.8 $\pm$ 0.1 compared with 43.7 $\pm$ 0.1 (finetuning on the original dataset) under the settings of ImageNet-1K with IPC 10.

Weakness 2

The effectiveness of the proposed method.

Thanks for the reviewer's valuable suggestions. In our paper, M specifically refers to the number of the parameters. We apologize for any potential misunderstanding this may have caused. We will revise the text to clarify this point and include the evaluation results for storage. Here, the actual storage for the classifier (around 4MB) is quite significant compared to the storage of the synthetic images (~50% of the storage of the images of ImageNet-1K with IPC 1). Moreover, the goal of our method is to achieve performance comparable to the state-of-the-art method, while using extremely little storage costs. Thus, a 4MB reduction in storage is quite important in our method. More importantly, in this context, Text-Embedding-Based Init accelerates the convergence and enhances the training speed of the projector. We will include metrics such as the loss during the training process in our updated version for reference. For the Image Update, its effectiveness varies under different settings. For instance, on the ImageNet-1K dataset with IPC 1, it demonstrates more significant improvements, achieving a gain of 1.0 (12.9 $\pm$ 0.3 with image update strategy compared with 11.9 $\pm$ 0.1 without image update strategy). Overall, it contributes positively to the downstream tasks. However, since our projector training phase narrows the distance between the observer model and the projector, the downstream performance gains may exhibit some variation.

(Cont.) The effectiveness of the proposed method

The ablation studies under the settings of ImageNet-1K with IPC 1

Question 1

Why there is no 'Size of Labels' for Yin et al. (2024) and Sun et al. (2024)

Thanks very much for the reviewer's insightful question. Yin et al. (2024) and Shao et al. (2024) employ a soft label generation strategy involving pre-generating and storing labels prior to downstream training, and Sun et al. (2024) generates soft labels online during downstream training. The pre-generation approach involves additional storage requirements in two aespects: (1) Soft labels for each augmented image: During each epoch, synthetic images will perform predefined augmentation strategies, such as CutMix, and each augmented image is assigned a corresponding soft label. (2) Augmentation details for downstream reproduction, e.g., image ID, flip status, and coordinates for CutMix to ensure reproducibility. The online generation strategy requires storing the teacher model(s). Table 1 presents a comparison of the storage requirements for soft labels in Yin et al. (2024) and Shao et al. (2024), along with the teacher model storage requirement for Sun et al. (2024). We sincerely apologize for any confusion that may have resulted from our initial presentation. In the revised version of our paper, we update Table 1 to clearly indicate the actual total storage requirements for each method, aiming to ensure better presentation to prevent any further misunderstandings.

We sincerely appreciate the reviewer’s time and thoughtful feedback. We hope our response has addressed your concerns. If there are any additional questions or areas that could benefit from further clarification, we would be more than happy to provide a detailed response and engage in further discussion.

Dear Reviewer KtxV,

We would like to inquire whether our previous responses have adequately addressed your concerns. We would greatly appreciate any further feedback from you and look forward to your response. If you have any additional questions or require more information, please do not hesitate to let us know. We would be more than happy to discuss further and provide any additional details needed.

Thanks very much for the your valuable time and suggestions regarding our work.