Leveraging Hierarchical Feature Sharing for Efficient Dataset Condensation

We thank Reviewer #4 (UGPZ) for the reviews and pointing out a missing related work. We address the raised questions and concerns below:

Q1: It lacks a discussion and comparison to a very relevant SOTA method, i.e., IDM [1], which also addresses the scalability of data condensation and works in a low GPU memory cost scenario. Thus, it might be incorrect to argue that the proposed method is the "first method to achieve such good performance with a low GPU memory loss" and such statements should be revised.

A1: We thank Reviewer #4 for pointing out a missing related work. As suggested by the reviewer, we have already added IDM to the Related Work section and added IDM as a new baseline in Table. 2 in the revised version. We also revised related statements in the paper. We would like to note that, after adding IDM, our proposed method still achieves better or comparable performance than other baselines.

Q2: The technical novelty is a bit thin. Specifically, the second contribution is relatively weak and more similar to a trick. It is interesting to know that over-budget generation and pruning may work, but this is more similar to remedying some of the shortcomings of the main idea rather than an independent contribution.

A2: We would like to note that the major novelty in HMN’s design comes from its hierarchical architecture. HMN is the first data container that considers the hierarchical structure of the classification system, and HMN achieves a better empirical performance. This makes us believe that HMN provides certain novelty and contribution to the DC area.

We agree with the reviewer that the second contribution (HMN pruning part) is more like a favorable property of HMN. However, we note that not only HMN but also other data parameterization methods, like HaBa, also have data redundancy in the condensed data containers (as discussed in Section 3.2.1). Unlike the HMN approach, it is non-trivial to prune redundant generated examples in factorization-based data containers. HMN allows better independence between different generated images, which makes HMN a different data container that allows for further improving efficacy by pruning redundant information.

---

We thank Reviewer #4 again for pointing out an important missing related work and bringing the novelty of pruning for discussion. We hope our response addressed your concerns regarding the related work and novelty and hope you can kindly consider updating the rating once you feel satisfied with the changes. We are happy to have further discussion and any questions that might arise!

Dear Reviewer UGPZ,

We greatly appreciate your time and efforts in reviewing our paper, and we would like to briefly summarize the response to your major concerns for your convenience. We also kindly refer you to our previous responses (both separate and general responses) for more detailed discussions.

---

Q: Lack of discussion on a related data condensation paper, IDM.

A: We have already added IDM to the Related Work section and added IDM as a new baseline in Table. 2 in the revised version. We would like to note that, after adding IDM, our proposed method still achieves better or comparable performance than SOTA methods.

Q: The second contribution is relatively weak and more similar to a trick.

A: We would like to note that the major novelty in HMN’s design comes from its hierarchical architecture. HMN is the first data container that considers the hierarchical structure of the classification system. We agree with the reviewer that the second contribution (HMN pruning part) is more like a favorable property of HMN, but this property enables HMN to further improve efficacy by pruning redundant information, which is different from previous data parameterization methods.

---

We have already revised the paper to include these points in the paper. We also kindly refer you to our previous responses (both separate and general responses) for more detailed discussions. We would like to check with you if our response and revised paper addressed your concerns and if we can provide more information to better address your concerns and clarify the claim in the paper!

We thank Reviewer #3 (RRfc) for the reviews and suggestions to improve the paper. We address the raised questions and concerns below:

Q1: Lack of experiments on higher resolution such as 128x128 (Image Woof/Nette) or 224x224 (ImageNet-10/100, a subset from ImageNet, as done in IDC)

A1: We thank the reviewers for suggesting additional evaluation on a high-resolution dataset like ImageNet to better demonstrate the effectiveness of HMN. We now include the ImageNet-10 evaluation result in Table. 2 (Section 4.1) in the revised version for a more extensive comparison. We evaluated HMN on ImageNet-10 1 IPC setting and compared it with IDC (other baselines are too resource-intensive to run). The evaluation results are summarized below:

From the table, we find that HMN still achieves strong performance on ImageNet-10, outperforming IDC.

Q2: Lack of describing the memory in detail. For example, how does the memory look like? What does the memory look like? How to initialize these memories, feature extractor, and decoder? (Q2)

A2: All memories are 3-D tensors. For CIFAR10, CIFAR100, and SVHN, memories have a shape (4, 4, Channels). For Tiny-imageNet, memories have a shape (8, 8, Channels). All values in the tensor are initialized with standard normal distribution. We included more details about the memory design of HMN in Section 3.1 and Appendix C.1 in the revised version.

Q3: The evaluation seems to be different from previous works. For example, HMN evaluates condensed images with a cosine annealing learning rate (LR) scheduler, while DSA or IDC uses a multistep LR scheduler. Thus, the comparison may not be fair.

A3: We thank the reviewer for pointing this out and agree with the reviewer that the evaluation with a multistep LR scheduler can still be useful. We report below the performance of HMN with a multistep LR scheduler on the CIFAR10 dataset as follows.

We find the difference due to the LR scheduler choice to be marginal, and the results with the multistep LR scheduler do not change the findings and conclusions of our evaluation. Our primary reason for choosing the cosine annealing LR scheduler in our evaluation is that it has fewer hyperparameters compared to the multistep LR scheduler (like milestones to reduce learning rate, and learning rate decay factor). We added more details to Appendix C.2 and D.4 regarding our hyperparameter settings and the rationale for any differences.

Q4: HMN stores parameters instead of images, this will incur extra resources for evaluation (to generate images). Is it correct?

A4: The reviewer is right that all data parameterization methods need to generate training data before model training. We would like to clarify that this generation overhead is relatively small, because it just needs one forward pass of HMNs. For example, on a 2080TI, the generation time for a 1 IPC, 10 IPC, and 50 IPC CIFAR10 HMN is 0.036s, 0.11s, and 0.52s, respectively (average number through 100 repeats).

We thank the reviewer for pointing this out for discussion, and we have already added this to the discussion in Appendix B in the revised version.

Q5: How many generated images per class (GIPC) used in Table 2?

A5: We listed GIPC for each setting in Table 5 (Appendix C). Since each instance-level memory corresponds to a generated image, the row “#Instance-level memory” indicates the GIPC for each setting. We have also further revised the writing in Section 3.1 and Appendix C.1 to reduce unnecessary confusion about GIPC of HMNs.

Dear Reviewer RRfc,

We greatly appreciate your time and efforts in reviewing our paper, and we would like to briefly summarize the response to your major concerns for your convenience. We also kindly refer you to our previous responses (both separate and general responses) for more detailed discussions.

---

Q: How does HMN scale to more complex datasets?

A: We now included the evaluation results on ImageNet-10 in Table 2. Our additional evaluation also shows that HMN also scales to complex datasets like ImageNet-10 and achieves better performance.

Q: The evaluation of HMN has a different setting. The training with HMN uses the cosine annealing LR scheduler, but the multi-step LR scheduler is more commonly used.

A: We report the numbers of training with HMN with a multi-step LR scheduler. We find the difference due to the LR scheduler choice to be marginal, and **the results with the multistep LR scheduler do not change the findings and conclusions of our evaluation.

Q: HMN seems to have a larger number of GIPC than conventional methods using images as data containers, which can increase the training time.

A: The reviewer's larger point is valid in that a larger GIPC size can impact training time. We would like to point out that this is a common feature of data parameterization methods, including IDC, HaBa, LinBa, and HMN. For example, LinBa generates 115 images per class with the 1 IPC setting for CIFAR10. However, data parameterization methods also achieve better accuracy. More GIPC can be treated as a tradeoff for better accuracy of data parameterization methods. We believe that this observation presents a promising direction for future research. We leave the challenge of reducing GPIC for data parameterization methods, while retaining accuracy, for future exploration.

Q: The baseline number of IDC is inconsistent with the numbers reported in the original paper.

A: We thank the reviewer for pointing out this mismatch. There turned out to be some typos in our paper, and we now correct all the numbers. We would like to note that the correction of those typos does not change the conclusion of our evaluation. HMN still achieves better results.

---

We have already revised the paper to include these points in the paper. We also kindly refer you to our previous responses (both separate and general responses) for more detailed discussions. We would like to check with you if our response and revised paper addressed your concerns and if we can provide more information to better address your concerns and clarify the claim in the paper!

Dear Reviewer #3 (RRfc),

We greatly appreciate your time and efforts in reviewing our paper again. Since the discussion period is about to end, we would like to gently double-check if our responses and revised paper have addressed your concerns. We are happy to provide more information if necessary!

Thanks,

Paper 4169 Authors

We thank Reviewer #2 (Y8Kv)’s time and effort for the reviews and highlighting some points in our paper that required further clarification. We address the raised questions and concerns below:

Q1: The authors highlighted that they are using a more efficient training loss "batch-based", and claim it is sufficient to outperform a better but more expensive training loss "trajectory-based". Can the proposed method also use "trajectory-based" loss to further improve the performance? Or is it the proposed method itself is heavier then others, so that it can't be optimized using the better loss? Note experiments are optional in the rebuttal.

A1: We thank the reviewer for bringing the scalability of different losses up for discussion, and we are happy to explain more about this.

The major reason that we do not choose a trajectory-based loss is that trajectory-based loss is not scalable enough not only with HMN (ours) but also with other data parameterization methods, like HaBa and LinBa.

In Table 7 (Appendix D.2), we discussed the memory usage of different parameterization methods. We find that HaBa with a trajectory-based loss consumes 48GB of memory even on the CIFAR10 dataset, and LinBa always exceeds the memory limitation. LinBa’s official implementation handles the memory issue by offloading GPU memory to CPU memory. However, CPU memory offloading leads to substantial computational overhead. For instance, in the CIFAR10 1 IPC task, LinBa requires 14 days to finish training on a 2080Ti, but training HMN with a batch-based loss takes only 15 hours with the same hardware. We thus choose the batch-based loss to train HMN.

Besides, to have a fair comparison, we compare HMN with other data parameterization methods with the same batch-based loss in Table 3 ( Section 4.2 ), and we find HMN outperforms other methods.

To sum up, training with a batch-based loss allows us to evaluate our methods on more complex datasets, like Tiny-ImageNet and ImageNet subset (see general response Q2), with reasonable resources and in a reasonable time. We agree with the reviewer that combining trajectory-based loss with HMN could be a line of future investigation if the hardware supported much higher GPU memory levels, much higher running times were acceptable (with CPU memory offloading), or more scalable trajectory-based methods were developed. We have revised Appendix D.2 to include more discussion on training with a trajectory-based loss.

Dear Reviewer Y8Kv,

We greatly appreciate your time and efforts in reviewing our paper, and we would like to briefly summarize the response to your major concerns for your convenience. We also kindly refer you to our previous responses (both separate and general responses) for more detailed discussions.

---

Q: Can we combine HMN with a trajectory-based loss? How is the scalability of the proposed method?

A: In Table 7 (Appendix D.2), we show that trajectory-based loss is not scalable enough not only with HMN (ours) but also with other data parameterization methods, like HaBa and LinBa (See original response for more details). Training with a batch-based loss allows us to evaluate our methods on more complex datasets, like Tiny-ImageNet and ImageNet subset.

In Table 7, we also show that our proposed end-to-end data condensation methods are more scalable than other SOTA data parameterization methods while achieving better performance. Our additional evaluation also shows that our proposed method also scales to complex datasets like ImageNet-10 and achieves better performance.

Q: What is the definition of IPC?

A: The abbreviation IPC stands for “Image Per Class”. This metric is used to measure how many tensors the storage budget is equivalent to in terms of the number of images. For example, 1 IPC for CIFAR10 stands for the storage budget: pixels of an image * IPC * class = 3 * 32 * 32 * 1 * 10 = 30720 tensors.

---

We have already revised the paper to include these points in the paper. We also kindly refer you to our previous responses (both separate and general responses) for more detailed discussions. We would like to check with you if our response and revised paper addressed your concerns and if we can provide more information to better address your concerns and clarify the claim in the paper!

We thank Reviewer #1 (mnZx)’s time and effort for the reviews and suggestions to improve the paper. We address the raised questions and concerns below:

Q1: 1- It is difficult to follow some of the ideas presented in the paper. For example, the paper didn't mention the abbreviation for IPC. Also, the paper didn't talk about whether the decoder parameters is part of the budget or not.

A1: We are happy to clarify the unclear presentation in the paper. The abbreviation IPC stands for “Image Per Class”. This metric is used to measure how many tensors the storage budget is equivalent to in terms of the number of images(same as SOTA data parameterization work [1,2]). For example, 1 IPC for CIFAR10 stands for the storage budget: Pixels of an image * IPC * class = 3 * 32 * 32 * 1 * 10 = 30720 tensors. Similarly, 10 IPC for CIFAR100 stands for the storage budget: Pixels of an image * IPC * class = 3 * 32 * 32 * 10 * 100 = 30720000 tensors. Same to the previous works, we treat those tensors as float32 tensors. We added the definition of 'IPC' the first time it was mentioned (Section 1) in the revised version.

Also, the paper didn't talk about whether the decoder parameters is part of the budget or not.

Both the decoder networks and memory tensors count towards the storage budget calculation. We also mentioned this in Section 3.1 and at the end of Section 4.1. We further highlighted this point in the revised version to eliminate potential confusion for readers.

Q2: 2- The paper demonstrated the effectiveness of their method on tiny datasets, and they didn't address the scalability of their technique. For example, the generated images seem very pixelated and abstract, how does their method perform in a more complex settings.

A2: We thank Reviewer #1 for bringing up scalability for discussion.

In Table 7 (Appendix D.2), we compare the scalability of end-to-end training algorithms of HMN with LinBa and HaBa, two data parameterization methods that have good performance. We find that HaBa with a trajectory-based loss consumes 48GB of memory even on the CIFAR10 dataset, and LinBa always exceeds the memory limitation. LinBa’s official implementation handles the memory issue by offloading GPU memory to CPU memory. However, CPU memory offloading leads to substantial computational overhead. For instance, in the CIFAR10 1 IPC task, LinBa requires 14 days to finish training on a 2080Ti, but training HMN with a batch-based loss takes only 15 hours with the same hardware. We believe that our end-to-end data condensation method not only achieves better performance than SOTA data parameterization methods but also is more scalable.

Moreover, we also conducted the evaluation on ImageNet-10 (see General Response Q2). The results show that HMN still achieves strong performance on a more complex high-resolution dataset.

For example, the generated images seem very pixelated and abstract.

We thank Reviewer #1 for pointing out the quality of the generated images. We think that it is a great question. We would like to clarify that the goal of data condensation is not to generate high-quality images, but to generate images representing the training behavior of the original dataset. Similar pixelation can also be found in other DC works, like DC, MTT, and HaBa. Pixelated images are fine as long as the resulting accuracy (performance) is still high for the level of compression desired. We have revised Appendix D.6 to include generated image quality discussion.

Q3: Table.1 What is the accuracy drop for randomly pruning 10% ?

A3: We present the accuracy of random pruning on the CIFAR10 10 IPC HaBa condensed dataset as an addition to Table 1. We can see that, compared to AUM pruning, random pruning consistently has a larger accuracy drop, but 10% AUM pruning does not cause an accuracy drop, which indicates the existence of redundancy in training data generated by HaBa.

Dear Reviewer mnZx,

We greatly appreciate your time and efforts in reviewing our paper, and We would like to briefly summarize the response to your major concerns for your convenience. We also kindly refer you to our previous responses (both separate and general responses) for more detailed discussions.

---

Q: What is the definition of IPC? Is the decoder parameters part of the budget?

A: The abbreviation IPC stands for “Image Per Class”. This metric is used to measure how many tensors the storage budget is equivalent to in terms of the number of images. For example, 1 IPC for CIFAR10 stands for the storage budget: pixels of an image * IPC * class = 3 * 32 * 32 * 1 * 10 = 30720 tensors.

Both the decoder networks and memory tensors count towards the storage budget calculation, which guarantees the fairness of the comparison to other DC methods.

Q: How is the scalability of the proposed method?

A: In Table 7 (Appendix D.2), we show that our proposed end-to-end data condensation methods are more scalable than other SOTA data parameterization methods while achieving better performance. Our additional evaluation also shows that our proposed method also scales to complex datasets like ImageNet-10 and achieves better performance.

Q: Does Algorithm 1. train the model for each subset selection? How this would scale for larger datasets?

A: Algorithm 1 retrains the model for each subset selection for different hard pruning rates. However, Algorithm 1 requires a relatively short computation time in practice. For example, DC with HMNs for CIFAR10 1 IPC needs about 15 hours on a 2080TI GPU, but the entire Algorithm 1 only costs an additional 20 minutes. Moreover, when we use HMNs for future training, we do not need to search for the best hard pruning rate again.

---

We have already revised the paper to include these points in the paper. We also kindly refer you to our previous responses (both separate and general responses) for more detailed discussions. We would like to check with you if our response and revised paper addressed your concerns and if we can provide more information to better address your concerns and clarify the claim in the paper!

Revision Summary:

We revised the paper based on the reviewers’ comments and suggestions. We also highlight significant changes in blue. Here is the summary of the major changes in the revised paper:

Here is the summary of the changes that address reviewer comments in the revised paper:

Section 1 (Introduction): We added the definition of 'IPC' the first time it was mentioned.

Section 2 (Related work): We added the introduction of IDM.

Section 3.3 (Methodology): 1. We highlighted the design details of HMNs, including the shape of memories, the number of generated images per class, architectures of feature extractors and decoder, and referred them to Appendix for more details; 2. We highlight the discussion of the computational cost introduced by Algorithm 1.

Section 4 (Evaluation): 1. We updated typos of IDC numbers; 2. We added IDM as a new baseline for comparison.

Appendix B (Discussion): We added discussions on the cost of training with data parameterization methods and the inference cost of HMNs.

Appendix C.1 (Experimental Setting): 1. We added more details on the design of HMNs, like how many GIPC that HMNs have. 2. We include the evaluation setting for ImageNet-10.

Appendix D (Additional Evaluation): 1. We updated the IDC numbers in Table 6. 2. We added more discussion on the limitation of training with trajectory losses. 3. We added a new subsection Appendix D.4 to conduct an ablation study on different learning rate schedulers.

Appendix D (Visualization): We added a discussion on the quality of generated images by HMNs.

---

Again, we thank the reviewers for their insightful feedback and suggestions, which largely help improve the quality and clarity of our paper. We hope that our clarifications, discussions, and additional evaluation address the reviewers’ concerns. We are happy to have any further discussions or answer other questions that may arise.

---

[1] Sachdeva, Noveen, and Julian McAuley. "Data distillation: A survey." arXiv preprint arXiv:2301.04272 (2023).

[2] https://github.com/snu-mllab/Efficient-Dataset-Condensation/

[3] Liu, Songhua, et al. "Dataset distillation via factorization." Advances in Neural Information Processing Systems 35 (2022): 1100-1113.

[4] Deng, Zhiwei, and Olga Russakovsky. "Remember the past: Distilling datasets into addressable memories for neural networks." Advances in Neural Information Processing Systems 35 (2022): 34391-34404.