MGD$^3$ : Mode-Guided Dataset Distillation using Diffusion Models

We appreciate the reviewer's positive assessment of our work, particularly our novel three-stage mode-guided approach, clear presentation, and contribution as a training-free dataset distillation method.

Addressing Concerns About Our Novelty

In dataset distillation, modes have been used to enhance diversity or sample from prototypes of the dataset distribution. Given these dataset prototypes, the goal is to generate a small dataset that maintains a similar distribution as the original dataset. Prior work [1] introduced noisy modes as $x_0$ instead of random noise; however, the high noise levels in the initial denoising steps often lead to outputs that are neither diverse nor fully representative. This explains why Disentangled Diffusion performs similarly to, or even worse than, simply using a pretrained diffusion model. The novelty of our approach lies in its efficient technical implementation, which requires no additional training and generalizes across various datasets and diffusion architectures, including Latent Diffusion (Class-to-Image), DiT, and Stable Diffusion (Text-to-Image), which is not the case of D$^4$M as shown in our experiments (Table 1 and Figure 4c).

[1] Su, Duo, et al. "D^ 4: Dataset Distillation via Disentangled Diffusion Model." CVPR . 2024.

Mode Guidance Theoretical Support

We aim to generate samples close to a given mode $m_i$ using mode-classifier guidance:

$\nabla_{x_t} \log p_t(x_t | m_i) = \nabla_{x_t} \log p_t(x_t) + \nabla_{x_t} \log p_t(m_i | x_t)$

Defining mode-classifier probability (how far is the current noise from the mode $m_i$) as:

$p(m_i | x_t) \propto \exp\left(-\frac{1}{2} |x_t - m_i|^2\right)$

Taking the gradient and introducing guidance scale $\lambda$:

$\nabla_{x_t} \log p_t(m_i | x_t) \approx -\lambda (x_t - m_i)$

Substituting into the classifier guidance equation:

$\nabla_{x_t} \log p_t(x_t | m_i) = \nabla_{x_t} \log p_t(x_t) - \lambda (x_t - m_i)$

With the score function related to noise prediction:

$\nabla_{x_t} \log p_t(x_t) \approx -\frac{\tilde{\boldsymbol{\epsilon}}_\theta(x_t, t)}{\sigma_t}$

Leading to modified noise prediction:

$\epsilon_\theta(x_t, t) = \tilde{\boldsymbol{\epsilon}}_\theta(x_t, t) + \lambda \sigma_t (m_i - x_t)$

Similar to Classifier Guidance, the model pushes toward images near mode $m_i$, with $\lambda$ controlling guidance strength.

Experiments with Other Mode Discovery Techniques

We tested more advanced mode discovery algorithms as suggested. GMM and Spectral Clustering improved our method in some settings:

Table 1: Experiments with suggested mode discovery methods on ImageNette.

Experiment with Complex distributions (imbalanced dataset)

We tested our method on class-imbalanced ImageNet-100 subsets. We created imbalanced datasets with "low sample" classes with 10-20 images and "normal" classes with 800 images. We vary low sample classes from 10 to 100 to generate a dataset with different degrees of imbalance and evaluate our method on IPC 10. The original test dataset is used to compute the accuracy. We reported the performance choosing 10 samples per class randomly of the imbalanced dataset,

Even when estimating modes from complex distributions with limited samples, our method maintains strong performance, consistently outperforming random selection. Notably, using a DiT trained on a balanced dataset achieves an accuracy of 23.3 ± 0.4, which our method surpasses even when 60% of the classes have low sample sizes.

Measuring Diversity and Representativeness Across MGD³ Components

Our three-stage process optimizes diversity and representativeness while maintaining realism. We include classification accuracy from an ImageNet-trained model to assess realism:

Mode Guidance improves diversity (0.71) and representativeness (0.82) but lowers accuracy (0.83), indicating reduced realism. Stop Guidance mitigates this, restoring accuracy (0.95) while preserving diversity (0.72) and representativeness (0.80). This achieves better trade-off than DiT (0.99 accuracy but significantly lower diversity and representativeness).

We appreciate the reviewer’s positive assessment of our work, particularly the recognition of our method’s state-of-the-art performance, its effectiveness in improving sample diversity, and its relevance to generative model-based dataset distillation. We also sincerely appreciate the reviewer’s positive evaluation of our claims and supporting evidence, which reinforces the validity and impact of our contributions. We address the reviewer’s insightful questions regarding the positive impact of the diffusion process on data quality and the question about mode selection in our response below.

Clarifying Mode Sampling During Inference:

Given a IPC = N, our method find N modes then we sample 1 image per mode.

Experiment using Latent Modes as Samples

As suggested by the reviewer, we tested using mode images directly as samples on ImageNette (IPC 10). Results in Table 1 show this performs poorly—worse than using the closest real sample to the mode. This occurs because the latent mode is an average of the cluster, creating a blurry image. MGD³ improves performance by using diffusion to generate real images near the mode rather than using the mode directly.

Table 1: Results for suggested experiment with latent modes.

Clarification about comparisons

Our method was implemented independently, not built upon baselines, and evaluated under identical conditions as existing approaches. We compared our approach with pretrained diffusion models (LDM, DiT) and incorporated these alongside Disentangled Diffusion for comprehensive evaluation. In text-to-image setups, we benchmarked against Stable Diffusion and D$^4$M. Notably, our method outperformed our closest competitor MinMaxDiffusion even without requiring the finetuning that MinMaxDiffusion needs for dataset distillation.

Addressing Mode Limitation Concerns in High-IPC Settings

When the selected IPC is larger than the true number of modes in the data for a class, our method captures fine-grained variations within each mode without affecting the diversity of the generated samples. This is controlled by the mode discovery algorithm, and we observe that our default k-means approach works well in practice.

We appreciate the reviewer’s thoughtful feedback and recognition of our contributions. Specifically, we are pleased that the reviewer highlights the novelty of our Mode-Guided Dataset Distillation using Diffusion Models (MGD³), its computational efficiency, the well-structured methodology, and the extensive experimental evaluation, including ablation studies and comparisons with strong baselines.

Clarifying Performance and Efficiency Trade-off

We acknowledge that while MGD³ achieves state-of-the-art performance across multiple benchmarks, IDC-1 slightly outperforms it in specific cases on ImageNet-100 (e.g., ResNet-18 with 10 samples). However, MGD³ consistently provides strong accuracy while significantly improving computational efficiency, making it more scalable than IDC-1. MGD³ achieves the best performance in several configurations (e.g., ConvNet-6 and ResNetAP-10 with 20 samples). We'll revise our statement to explicitly highlight this trade-off between accuracy and efficiency.

References

Although we cited [1] and [3] in the main paper, we will expand our discussion to clarify how MGD³ relates to works [1-4], particularly regarding efficiency and scalability, and include comparisons in Table 1(reviewer rCkP).

Impact Statement

Efficient dataset distillation reduces storage and computational costs while maintaining high model performance. MGD³ improves both accuracy and efficiency compared to prior methods, enabling large-scale dataset distillation with minimal performance trade-offs. This has significant implications for deep learning in resource-constrained environments, such as mobile AI and federated learning. By enhancing scalability, our work enables more effective model training with limited data while preserving diversity and representativeness.

The Impact Statement will be included in the final revision.

Hyperparameter Experiments

While our main experiments used fixed hyperparameters for consistency (still achieving SOTA results), we conducted additional experiments with varying hyperparameters for each dataset as requested. Results for different values of λ and t_stop on Nette, IDC, and ImageNet-100 with IPC 10 are shown below:

Table 2: Results with various mode guidance scale parameters (λ)

Table 3: Results with various values for $t_{stop}$

While performance can be improved for specific datasets by adjusting parameters, our default parameters result in higher average scores across datasets.

Discussion on ImageWoof

We included a discussion of ImageWoof results in the appendix (line 677-679).

Experiments with Other Mode Discovery Techniques

Please refer to the Response of reviewer y4kY.

Addressing Ultra-Low IPC Performance: MGD³'s Scaling Advantages

The results from existing works on dataset distillation suggest that generative methods for dataset distillation underperform in low-IPC setups (e.g., IPC = 1) compared to optimization-based methods, which explicitly optimize the small dataset but at a higher computational cost. However, generative models scale better to higher IPC (e.g., IPC = 50) while maintaining lower computational cost, i.e., within a given time frame, generative models can efficiently generate higher-IPC datasets compared to the optimization-based methods, making them more practical for scaling to larger datasets like ImageNet.

We appreciate the reviewers' recognition of our work as novel and acknowledging our work to address the key limitations of existing techniques without requiring any fine-tuning. We are also grateful that the reviewer highlighted the significance and impact of our approach, with mention of our work reducing the computational costs, thereby making the dataset distillation more accessible to researchers with limited resources. Additionally, we are pleased that the reviewer found our experimental validation thorough highlighting method's robustness and scalability, and found our method to be logically and clearly presented. Below, we provide answers to the reviewer's comments.

Model Scaling Experiments

In the main paper, we showed that our method achieved SOTA on ImageNet-1K (IPC-50) with ResNet18. As requested by the reviewer, we extended our experiments to larger models (ResNet-50 and ResNet-101) and showed consistent performance improvements across models of different scales.

Table 1: Results on ImageNet-1K with IPC 50. '-' means not reported.

Clarification about ImageNet-1k Baselines

Although the baseline accuracy may appear low in isolation, our method surpasses previous works on ImageNet. To provide proper context, we compare against the accuracy achieved when training the same network on the full dataset. For example, ResNet-18 reaches 69.8% accuracy with the full dataset, whereas our approach attains 86% of this performance while using only 3.9% of the data, demonstrating its efficiency and strong relative performance.