Influence-Guided Diffusion for Dataset Distillation

Thank you very much for your positive feedback! We greatly appreciate your insightful question regarding the distribution of the generated data, which led us to discover new insights on applying our guided-diffusion method for dataset distillation. Below, we address your questions in sequence.

---

Q1(@Weakness1): The performance sensitivity to hyper-parameters should be examined, including how to conduct the hyper-parameter search.

A1: Thank you for your valuable suggestion. We have already analyzed the impact of the influence factor $k$ and deviation factor $\gamma$ in Appendix A.5, titled "Parameter Analysis", and in Figure 5 of the original paper. The optimal values for these factors were determined through a grid search. As shown in Figure 5, our DiT-IGD method is relatively sensitive to changes in both factors, which supports our motivation for adding a diversity constraint to the influence guidance. In contrast, since the baseline Minimax has already strengthened diversity through fine-tuning, our Minimax-IGD primarily relies on adjusting the influence factor $k$.

As for the guided range parameters, we drew on insights from previous guided diffusion work (line 266) to determine the them. This led us to search for the optimal range within steps $[25,50]$. For the length of the guided range, we tested values of 10, 15, and 20 steps, adjusting with a stride of 5 within $[25,50]$. Setting the length to 15 produced the best and most stable performance among various groups of $k$ and $\gamma$. Ultimately, we discovered that a guided range of $[30,45]$ achieved the best results. We provide a comparison for DiT-IGD on ImageWoof with IPC=100 as a reference.

---

Q2(@Weakness2): Whether the new method causes the collapse of the distribution of the generated data, though with the deviation guidance.

A2: In Section 4.5 and Figure 3, we provided t-SNE visualizations comparing the distributions of data generated by two baseline methods (DiT and Minimax) and our IGD-based methods with IPC=100. The figure shows that integrating IGD enhances diversity and alignment with the original dataset, supported by lower Wasserstein distances to the original dataset.

Our further experiments reveal that "focusing solely on diversity or alignment with the original dataset is insufficient for optimal effectiveness in DD scenarios". We compare the FID scores and coverage of surrogate datasets (IPC=100) generated for ImageWoof by different methods,

Coverage was assessed based on whether each original data point had a nearest neighbor in the surrogate dataset within a given threshold (e.g., 300 in the Inception V3 latent space). For fairness, we excluded data selected by the Random method from the original dataset during coverage calculation.

From the results, although the randomly selected dataset has the lowest FID and highest coverage, its performance was the worst. Similarly, while Minimax-IGD had worse FID and coverage than DiT-IGD, it performed better. These findings suggest that our diversity-constraint influence-guided objective is a more effective measure for DD than relying solely on distribution alignment.

Q3(@Weakness3&4): The paper should compare with other recent methods using pre-trained diffusion models.

A3: Thank you for the valuable suggestion and for introducing the insightful related work [1]. Although [1] was not originally proposed for the DD setting, it aims to improve the training efficacy of synthetic data generated by diffusion models. We also identified a related work, D$^4$M [2]. Together with our baseline method Minimax, these three works propose using distribution-matching-like objectives to fine-tune diffusion models. Here, we follow RDED’s evaluation protocol commonly used in dataset distillation (involving predicted soft labels) to compare our guided-diffusion methods with these three approaches on ImageNet-1K.

The results demonstrate that our guided-diffusion methods continue to achieve superior performance, supporting our claim that introducing informative guidance is essential for applying diffusion models in dataset distillation. We also observe that among the three distribution-matching-like fine-tuning methods, [1] achieves the best performance. We will include these evaluations in the later revised version of the paper and plan to explore integrating [1] with our influence-guided generation in future work.

[1] Jianhao Yuan, Jie Zhang, Shuyang Sun, Philip Torr, Bo Zhao. Real-Fake: Effective Training Data Synthesis Through Distribution Matching. ICLR 2024

[2] Duo Su, Junjie Hou, Weizhi Gao, Yingjie Tian, Bowen Tang. D4M: Dataset Distillation via Disentangled Diffusion Model. CVPR 2024

Dear Reviewer Qttw,

We are truly grateful for your constructive feedback and recognition of our work! Below is a summary of our responses to your questions:

• A kind reminder of hyper-parameter analysis in Appendix A.5 and a detailed searching process of the guided range are introduced.
• Additional experiments regarding data diversity and distribution coverage are discussed.
• Comparisons with recent approaches using pre-trained diffusion models are involved, demonstrating a continued superior performance of our method.

If there are any additional questions or thoughts, we are welcome to further discussion!

Best Regards

Thank you for your positive feedbacks. We address your questions in the following responses.

---

Q1(@Weakness1): There are writing issues in the math part.

A1: Here are responses to the issues you raised:

I. It is unclear how the derivation is transferred from stepwise (Eq4) to epochwise (Eq5).

Answer: Based on the original formulation of the trajectory influece (TracIn), in Eq. (4), the change in loss between subsequent parameter updates is approximated using a first-order Taylor expansion, measuring the influence stepwise for each parameter update, representing the contribution of a training pair $(x, y)$ to the validation loss on $(x', y')$. By analogy to the fundamental theorem of calculus, the sum of the influences of all training examples on a fixed test point $(x', y')$ is equivalent to the total reduction in loss on $(x, y)$ during training. Thus, for a particular training point $(x, y)$, the idealized influence can be approximated by summing the influence over all iterations where $(x, y)$ was used to update the parameters. In practical SGD-based training, since each training point is used once per epoch, TracIn approximates the influence of $(x, y)$ on $(x', y')$ by summing influences collected at checkpoints across epochs.

II. C duplicated defined on L096 and L110, some z are bold and some are not in Sec 2.2 and D is not clearly defined in L132.

Answer: Thank you very much for your attention to detail and for pointing out these issues. First, we update the notation for the conditional distribution from $p(x|C)$ to $p(x|*condition*)$ to avoid confusion. Second, we correct the missing bold formatting. Third, we clearly indicate that $D(.)$ represents the decoder function and that $u_i=D(z_i)$. All these revisions are updated in the current revised version of our paper.

---

Q2 (@Weakness2): The proposed method seems to have high computational and storage costs, particularly for gradient (L294) and similarity calculations (Eq. 8). Detailed costs should be provided.

A2: Thank you for the suggestion. We repectfully clarify that our method does not require high computational resources to generate surrogate datasets. All experiments conducted on ImageWoof, ImageNette, and ImageNet-1K can be completed using a single RTX 4090 with nearly 16 GB peak memory usage. Regarding storage costs, peak usage occurs before applying our proposed gradient-similarity-based checkpoint selection algorithm to retain representative checkpoints. This involves storing $E+1$ model checkpoint parameters, but after selection, peak storage is reduced to approximately $2K$ checkpoints. With ConvNet-6 as the surrogate in our default implementation, this cost nearly 295 MB for storing overall $51$ checkpoints' parameters and nearly 24 MB for storing $4$ selective checkpoints and nearly 24 MB for storing corresponding averaged gradients. Below, we provide a detailed analysis.

First, as outlined in L290-L295, we obtain $E$ checkpoints trained on $\mathcal{T}$ and retain $K$ representative checkpoints for subsequent influence guidance calculation (Eq. 7). As shown in Table 6, our checkpoint selection algorithm allows for only 4 or 5 representative checkpoints to achieve good performance. Moreover, based on our complexity analysis in Appendix A.3, the computational cost of this selection algorithm is comparable to training the surrogate model for $E$ epochs. Before starting generation for a specific class $c$, we calculate the average gradient $\bar{\nabla}_{\theta} \ell_c(X_c ; \theta_e^{\mathcal{T}})$ across each of the $K$ retained checkpoints for each class. This process is similar in computational load to training the model for $K$ epochs on the class c of the original dataset and storing $K$ corresponding averaged gradients. All these calculations are performed offline before data generation.

As for the diversity guidance, the similarity calculation for all generated samples is not computationally demanding because it is performed in the latent code space of the diffusion model rather than the original image space. The dimension of the latent code is only 4096.

More evaluations of the data generation time of our IGD methods can be found in our response A4 to the Reviewer brCT.

---

Q3 (@Weakness3): I would suggest revising the statement that this method is "training-free".

A3: Thank you for highlighting the need for greater clarity regarding this statement. The use of "training-free" was meant to emphasize our proposition that introducing informative guidance is effective for leveraging a well-trained diffusion model in DD tasks. However, we acknowledge that the term "training-free" may be ambiguous. Therefore, we remove the term in line 082 and revise "training-free guidance" in line 137 to "guided-diffusion generation".

Dear Reviewer hMXA,

Thank you for your positive assessment of our work. We sincerely appreciate your thoughtful comments and constructive feedback on our submission! Below, we briefly summarize our key responses:

• The derivation from stepwise to epochwise in Eq. (4) to Eq. (5) is clarified.
• Several notational issues in the math part are resolved.
• The computational and storage costs with a detailed analysis are reported.
• Ambiguous statements regarding the method being "training-free" are revised.

Could you please review our responses and let us know if you have any further questions? Your feedback is invaluable to us!

Best Regards

Dear Reviewer hMXA,

Thank you very much for your positive feedback on our responses! We especially appreciate your attention to the details and the mathematical formulation of our paper. The revisions based on your feedback certainly enhanced the clarity of the submission.

Best regards.

Q3(@Weakness4): Can IGD be used with other efficient diffusion sampling strategies like DPM solvers?

A3: Following your advice, we achieved a significant 50% reduction in average sampling time by applying the DPM solver to our IGD method (from 8.2 s to 4.3 s on an RTX 4090).

We used the official implementation of the DPM solver with the default 20 sampling steps. Notably, we observed negligible performance degradation or even slight improvement with fewer sampling steps. Below, we compare the average performance using DDIM with 50 steps and the DPM solver with 20 steps for distilling ImageNette with IPC=50:

We will include and further extend this useful supplementary evaluation in the later revised version of our paper and introduce the corresponding implementation with the DPM solver in our released code.

---

Q4(@Weakness2&5): The time consumption for training surrogate checkpoints and the generation time of IGD against other methods like RDED should be compared.

A4: Thank you for pointing out the need for greater statement clarity and your constructive suggestions. First, we want to clarify that our method's "training-free" claim means we do not require retraining the diffusion model, as done in Minimax. Rather than purely focusing on efficiency, we regard our influence-guided method as paving a new way for using diffusion for DD tasks by designing effective guidance to improve training efficacy.

Regarding your suggestion to compare time consumption with SOTA methods like RDED and model-inversion-based methods, all require a well-trained surrogate model for generating synthetic data and predicting soft labels. For example, to distil the ImageNet datasets, these methods need to train a ResNet model on the entire original dataset for over 100 epochs. In contrast, our method only requires training a simpler model, such as ConvNet, for 50 epochs.

For instance generation time, we acknowledge that RDED is inherently more efficient than diffusion-based methods for dataset distillation. RDED uses a strategy similar to core-set selection methods to choose informative patches from real images and generate data by stitching them together. This enables RDED to create a surrogate dataset with IPC=50 for ImageNette/ImageWoof in just a few minutes. By comparison, our method takes approximately 69 minutes with DDIM-50 and 35 minutes with DPM-20, respectively, when generating all classes without using parallel workers.

However, RDED’s reliance on core-patch selection also limits its ability to synthesize new content of distilled data, whereas our guided diffusion method can. This is reflected in our superior performance, especially in settings where only hard labels are used. Therefore, we believe exploring guided diffusion methods remains a promising direction for dataset distillation.

Thank you for your insightful questions and constructive suggestions. Specifically, we followed your suggestion to leverage the DPM solver in our method, which reduced sample generation time by 50% with negligible performance degradation. We answer you questions as below.

---

Q1 (@Weakness1): The compared methods, such as the latest RDED, are missing in Table 1.

A1: Thank you for pointing this out. We compare our method with RDED, and a state-of-the-art model-inversion-based method CDA [1], on the ImageNette, under two evaluation protocols below:

Hard-label evaluation protocol (our default):

Soft-label evaluation protocol (RDED's & CDA's default):

The results indicate that our method achieves superior performance under both evaluation protocols, despite being primarily designed for the hard-label protocol by default.

[1] Yin, Zeyuan, Zhiqiang, Shen. Dataset Distillation via Curriculum Data Synthesis in Large Data Era. Transactions on Machine Learning Research, 2024.

---

Q2(@Weakness3): How robust is the influence guidance calculation across different architectures?

A2: We have already evaluated the impact of using different-size models as surrogates for influence guidance calculation in Section 4.3, titled "Cross-Architecture Robustness of Influence Guidance," and in Table 4. The experimental results show that the effectiveness of influence guidance is insensitive to the choice of surrogate models.

Your suggestion to use a Swin Transformer, e.g., a non-CNN architecture, as a surrogate is valuable for a more comprehensive robustness evaluation. As a reference, we provid below the cross-architecture performance using Swin Transformer checkpoints to compute influence guidance for generating an IPC=50 synthetic dataset over ImageNette:

From the results above and Table 4, we observe: 1) the influence guidance effectiveness is generally insensitive to the surrogate model used; and 2) using a more complex model as the surrogate tends to slightly underperform compared to simpler models. Notably, the second observation aligns with findings from gradient-matching-based or training-trajectory-matching-based DD methods which also utilize gradient information. We hypothesize that complex, high-performance models as surrogates might inject less generalizable "short-cut" features into the synthetic data, leading to reduced performance.

Dear Reviewer brCT,

We sincerely appreciate the time and effort you have dedicated to reviewing our submission. As the discussion deadline approaches, we would like to provide a summary of our responses and updates:

• Leveraging the DPM solver achieving a 50% reduction in sample generation time with negligible performance degradation.
• Our method still performs better than RDED in Table 1's comparison.
• Insensitivity is observed when using the Swin Transformer as a surrogate.
• Detailed time comparison with RDED are reported.

Would you mind checking our responses and confirming if you have any additional questions? We welcome any further comments and discussions!

Best Regards

Thank you for your positive feedback! Your valuable question and suggestions regarding the evaluation of distribution diversity led new insights on supporting our influence-guided strategy for dataset distillation (DD). Below are our repones to your questions.

---

Q1(@Question1): The performance is still very far from the original datasets. What is the least IPC to achieve similar performance with full data?

A1: As a reference, our Minimax-IGD method achieves 90.8% test accuracy on a ResNetAP-10 model with IPC=400 (approximately 31% of the original dataset size) on the ImageNette dataset under RDED's evaluation protocol. This shows a 3.8% performance gap compared to the test accuracy on the full original dataset (94.6%). Further increasing the IPC only results in marginal improvements.

Empirically, the 3.8% error is largely due to "hard samples" that also result in relatively high test loss for a model trained on the full dataset, indicating these are marginal instances within the authentic data distribution. Due to the inherent objective of denoising diffusion models, synthetic data tends to be sampled from high-probability regions, resulting in poor coverage of these marginal instances. Moreover, as suggested by the definition of averaged gradient $\bar{\nabla}_{\theta} \ell_c(X_c ; \theta_e^{\mathcal{S}})$ (line 180), the influence guidance tends to contribute less to the influence promotion over marginal instances. Addressing this limitation will be a key focus of our future research.

---

Q2(@Question2): Which experiments show an improvement in diversity? The diversity should be measured in terms of FID/Recall values.

A2: In Section 4.5 and Figure 3, we provided t-SNE visualizations comparing the distributions of data generated by two baseline methods (DiT and Minimax) and our IGD-based methods with IPC=100. The figure shows that integrating IGD enhances diversity and alignment with the original dataset, supported by lower Wasserstein distances to the original dataset.

Our further experiments followed your suggestion reveal that "focusing solely on diversity or simple alignment with the original dataset is insufficient for optimal effectiveness in DD scenarios". We compare the FID scores and coverage of surrogate datasets (IPC=100) generated for ImageWoof by different methods,

Coverage was assessed based on whether each original data point had a nearest neighbor in the surrogate dataset within a given threshold (e.g., 300 in the Inception V3 latent space). For fairness, we excluded data selected by the Random method from the original dataset during coverage calculation.

From the results, although the randomly selected dataset has the lowest FID and highest coverage, its performance was the worst. Similarly, while Minimax-IGD has worse FID and coverage than DiT-IGD, it performed better. These findings suggest that our diversity-constraint influence-guided objective is a more effective measure for DD than relying solely on distribution alignment.

Q3(@Question3): Equation (7) uses cosine similarity instead of dot product; is it purely due to experimental results or based on some other hypothesis?

A3: As stated in lines 205-207, we replace the dot product with cosine similarity to primarily stabilize the magnitude of the influence guidance signal. Together with the dynamic scale factor $\rho_t$ defined in Eq. 8, this replacement allows our method to achieve good performance with the minimal tuning of the influence factor $k$ when using different surrogate models or diffusion samplers (as reported in Table 4 and our response A3@Reviewer-brCT).

Moreover, since Eq. 7 involves checkpoint parameters retained from different training stages, we empirically observed that using the dot product for gradient similarity causes loss from earlier checkpoints to dominate the influence guidance. Cosine similarity alleviates this issue effectively.

---

Q4(@Question4): How will the work be performed on different tasks apart from classification?

A4: As defined in Section 2.1, our work currently focuses on dataset distillation for image classification tasks. Given the limited time during the rebuttal stage, we respectfully provide evaluations centred on our primary focus and comparisons with related work of similar scope. We recognize the potential of this question and plan to explore the applicability of our approach to other tasks in future work.

---

Q5(@Weakness4): Is it possible to have a one-time distillation and use that distilled datasets to validate across models?

A5: Thank you for raising this important question, which relates to a fundamental criterion in DD research. The community often refers to this as the cross-architecture or unseen-architecture generalization capability of distilled datasets. This is one of our key motivations for introducing the guided-diffusion paradigm, allowing us to formulate the DD problem as learning a training-effective conditional distribution of the authentic distribution, thereby mitigating the distribution shift issues faced by earlier DD methods (line 108).

As noted in the implementation details (line 317), our method uses ConvNet-6 as the surrogate model for calculating influence guidance during one-time distillation by default. The results reported in Tables 1-3 are based on this default setting. These results demonstrate the strong unseen-architecture generalization of our one-shot synthetic dataset across both CNN and Transformer models.

Dear Reviewer Xr47,

We sincerely appreciate the time and effort you have dedicated to reviewing our submission, as well as your positive feedback on our work! Below is a summary of our responses and updates:

• A discussion regarding the least IPC needed for the full dataset performance is provided.
• Experiments measuring diversity with FID and distribution coverage have been included.
• The rationale behind using cosine similarity in Equation (7) is further clarified.
• The feasibility of using a one-time distilled dataset across unseen models is elucidated.

We would greatly appreciate it if you could check our responses and let us know if there are any additional questions. Your feedback is invaluable to us!

Best Regards

Thank you for your instructive feedback and valuable suggestions. Below are our responses to your questions.

---

Q1(@Weakness1): The introduction lacks a clear explanation of what "influence" entails or why it is used for guidance. Adding a figure to illustrate the motivation or differences from previous methods could be helpful.

A1: We have uploaded a revised version of the paper incorporating your constructive feedback. In lines 59-80 of the Introduction, we added a brief explanation of the influence function and reorganized the text to highlight its role in addressing the inherent challenges posed by the abstract nature of our training-effective condition for diffusion generation. Additionally, in the newly added Figure 5, we provide an intuitive illustration of the IGD sampling framework, contrasting it with the vanilla diffusion sampling method.

We respectfully look forward to your feedback on the updated content.

---

Q2(@Weakness2): How does the proposed training-free diffusion framework for dataset distillation differ from existing training-free approaches in AIGC?

A2: This is a key question related to our motivation for proposing the influence-guided paradigm for diffusion generation. In common diffusion-based AIGC applications, users primarily control data generation through text prompts (line 134). While this enables some degree of content specification, systematically defining a diverse set of high-quality prompts to effectively guide diffusion models in generating both diverse and training-effective data for dataset distillation (DD) remains an abstract challenge without a structured optimization methodology.

To address these challenges, we identify the influence function as an effective metric that measures the compatibility of generated data with generalized training-effective conditions. Building on this insight, our proposed IGD method is the first to systematically optimize data influence to generate high-quality surrogate datasets for DD tasks.

---

Q3(@Weakness3): The experiments only report results on ImageNet, without including results on classic datasets such as CIFAR-10 and CIFAR-100.

A3: We provide a comparison of our DiT-IGD method with other state-of-the-art DD methods designed for distilling high-resolution, large-scale datasets, including RDED [1] and SRe$^2$L [2], on CIFAR-10 as a reference.

Our method achieves comparable performance with RDED on CIFAR-10. However, for the primary focus of large-scale DD tasks, our method attains significantly better performance, as shown in Tables 2-3 and our response A1@Reviewer-brCT.

[1] Peng Sun, Bei Shi, Daiwei Yu, Tao Lin. On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigm. CVPR 2024

[2] Zeyuan Yin, Eric P. Xing, Zhiqiang Shen. Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective. NeurIPS 2023

---

Q4(@Question1): What is the relationship between the two proposed guidance and the "train-free" concept? Notably, even when these components are excluded from Equation 9, good results are still achieved.

A4: In this work, we frame the DD problem as learning a training-effective conditional distribution of the authentic distribution, thereby addressing the distribution shift issues encountered by earlier DD methods (line 108). As mentioned in A3, addressing this problem requires tackling the abstract nature of a generalized training-effective condition. To systematically optimize data influence and generate high-quality surrogate datasets for DD tasks, we introduce two effective guidance to steer the diffusion sampling process that provides influence promotion and diversity enhancement.

We respectfully disagree with the statement that "good results are still achieved even when these guidance components are excluded." Our comparative results in Tables 1 and 2 over ImageNet datasets show significant improvements over the two baselines, DiT and Minimax, when integrating our influence guidance and deviation guidance, as also acknowledged by reviewers brCT, hMXA, and Qttw. For instance, IGD enhances the average performance of DiT by 6.6% and provides a 5.1% boost for Minimax on ImageWoof when IPC ≥ 50. Furthermore, our ablation study in Table 5 shows the contributions of the two guidance components.

Thank you for your reply! We believe your further questions are related to Q3 (@Weakness3) and Q4 (@Question1) from our previous response. Below, we provide additional discussion on these two questions.

---

Further discussion on Q3(@Weakness3):

We would like to respectfully recall that the significant progress of early DD frameworks in distilling small datasets like CIFARs was acknowledged (lines 40-44). However, the limitations identified in lines 45-50 (e.g, cost increase with data dimension or high-frequency features) restrict their applicability to more practical, high-resolution, large-scale datasets (e.g., 224×224 ImageNets), which is the focus of our research (line 301). These observations motivated us to leverage diffusion models to generate near-real but training-effective synthetic data.

Thank you for mentioning DATM, a representative training trajectory matching DD method that achieves lossless performance on CIFARs. While DATM significantly outperforms our method or RDED (which aims to generate near-real distilled data) at IPC=10 for CIFAR-10, the test performance of 66.8% still has a significant gap compared to using the full dataset (e.g., 84.8%). More importantly, the core contribution of DATM is to emphasize the use of late trajectory matching to generate hard samples with fewer distilled features (closer to real images) for lossless distillation when IPC is high.

Due to limited time during the rebuttal phase, we continue to use CIFAR-10 as a reference here. The following results demonstrate that our method achieves approximately lossless performance with a 20% compression ratio (IPC=1000), comparable to DATM:

Implementation details will be released later in our official code.

---

Further discussion on Q4 (@Question1):

We apologize for not fully understanding your concern earlier. The term "training-free" was meant to emphasize the practicality of introducing informative guidance with a well-trained diffusion model in DD tasks, rather than enhancing distribution alignment through retraining, as done in Minimax.

Inspired by the remarkable ability of denoising diffusion models to learn a parameterized distribution that approximates the authentic distribution (line 144), a natural extension is to define a conditional distribution aligned with specific task requirements. A line of work known as "training-free guided-diffusion generation" based on energy-based models (EBM) (line 138) has verified the effectiveness of this strategy for guiding diffusion models to meet requirements that are hard to be described by text (e.g., segmentation-guided generation). This motivated us to adopt guided-diffusion frameworks to steer the diffusion model in generating data under training-effective conditions. Furthermore, the improvement achieved by our influence-guided generation over the Minimax method also verifies that our method can effectively complement DD frameworks like Minimax which require retraining the diffusion model.

We acknowledge that the term "training-free" may have caused ambiguity. To prevent any misunderstanding, we have removed the term in line 082 and revised "training-free guidance" in line 137 to "guided-diffusion generation" in the updated version of the paper.

We hope this response addresses your concern and further clarifies our motivation. If any questions remain, we would appreciate any specific suggestions or further questions you may have to help improve our clarity.

Dear Reviewer rhZM,

Thank you very much for the effort and continued attention you have given to our submission!

In Table 1, we have presented two representative DD methods, e.g., Distribution Matching (DM) and IDC, primarily designed for distilling small datasets but can also distil larger datasets, such as 10-class ImageNet subsets, with acceptable computational resources. However, both methods demonstrated poor performance. Moreover, given that these methods require tens of hours to distil even 10-class subsets of ImageNet, we found it impractical and unnecessary to include them for the distillation of ImageNet-1K in Table 2.

Additionally, while trajectory matching-based methods like DATM can achieve superior performance on small datasets such as CIFAR, their computational and time costs are prohibitive for distilling 224×224 ImageNet datasets. These costs primarily stem from:

• Constructing an expert trajectory pool requires training dozens of surrogate models on the full dataset, which is extremely time-consuming for large datasets.
• The trajectory-matching loss used to optimize distilled data involves unrolling the computation graph of multiple gradient descent steps, which is highly memory-intensive (≥ 100 GB) and time-consuming.

Due to these challenges, we considered employing these methods for distilling high-resolution datasets to be intractable and impractical. Given the limited time remaining in the discussion phase (less than 48 hours), we respectfully emphasize the difficulty of supplementing such a benchmark at this stage, as well as its divergence from the focus of our submission.

If our responses have addressed most of your concerns regarding our submission, we would respectfully request your kind consideration of increasing the rating. We truly appreciate your positive feedback on our submission!

Best regards.

Dear Reviewer rhZM,

As highlighted in our previous responses, adapting most low-resolution-oriented DD methods to distil high-resolution datasets presents scalability issues. We found it intractable to report their performance on ImageNet datasets evaluated in our submission. For reference, we have provided benchmarks for other state-of-the-art baselines (including two low-resolution-oriented methods DM and IDC) in Tables 1 and 2, which have been largely well-received by the other reviewers.

We agree with your valuable suggestion to test high-resolution-oriented methods on small datasets to better assess the generalizability of the methods. As per your recommendation, we have established a comprehensive benchmark comparing the performance of ConvNet using state-of-the-art high-resolution-oriented methods, including SRe$^2$L [1] and RDED, on CIFAR-10 and CIFAR-100 at various IPCs.

CIFAR-10:

CIFAR-100:

The results show that our method outperforms SRe$^2$L and RDED in most scenarios. Additionally, it achieves nearly lossless performance similar to DATM (e.g., at a 20% compression ratio). These findings, combined with our method's outstanding performance on ImageNet datasets, suggest that our approach is a unified solution that performs well across low-resolution and high-resolution datasets.

Thank you for your constructive suggestion! We have included these evaluations in Appendix A5 of the current revised paper. We would appreciate your feedback on the benchmark evaluation we have provided.

Best Regards.

[1] Zeyuan Yin, Eric P. Xing, Zhiqiang Shen. Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective. NeurIPS 2023

Dear Reviewer rhZM,

Thank you very much for your proactive attitude and continued engagement with our submission! We deeply appreciate your constructive suggestions, which have significantly helped improve our work.

In response to your previous comment titled "Last Question," we have added a benchmark evaluating our method and other high-resolution-oriented methods on smaller datasets (CIFAR-10 & CIFAR-100). We have included this valuable benchmark in Appendix A5 of the revised paper. Together with our method's strong performance on ImageNet datasets, this suggests that our approach provides a unified solution effective across both low-resolution and high-resolution datasets.

With the discussion period extended, we look forward to any further valuable feedback you may have on the additional evaluation. Any other suggestions or thoughts on future directions for this field from you are also invaluable to us!

Best Regards.

Dear Reviewer rhZM,

We are truly grateful for your kind recognition of our submission and the additional content! Your constructive feedback and suggestions have significantly improved our work!

We deeply appreciate your thorough review and proactive attitude to enhancing the paper, both of which are invaluable to the research community.

Wishing you all the best!

Best Regards.