Taming Diffusion for Dataset Distillation with High Representativeness

Q1: The concern about the multi-component Gaussian mixture distribution assumption.

A1: Thank you for highlighting this. To clarify the assumption, we offer a more comprehensive explanation here. For current diffusion-based methods and ours, the pre-trained VAE are not conditional VAE, so both the training and inference processes of the VAE are not conditioned on the class label C. In theory, each latent variable is encoded by independent statistical parameters (mean and variance), resulting in an N-component Gaussian mixture distribution for N images. In practice, as shown in Equation (10) (in Appendix), the VAE loss includes a KL divergence term, which pulls each latent distribution moderately toward N(0, I). This results in a Gaussian mixture with overlapping components, indicating that the overall mixture distribution can be well-approximated by M (< N) effective components.

We give the visualization of the VAE space in the first column in Appendix B.4-Figure A4. Although the number of components is smaller than N, it remains challenging to fit the distribution, which is not in conflict with our method. We will add the above analysis in the updated version.

Q2: More comparison with other SOTA methods on CIFAR-10, CIFAR-100.

A2: We did not include comparisons with more SOTA results on CIFAR because the methods adopt varying validation settings, making us unable to perform direct comparisons. Since our method is geared towards more practical scenarios involving large-scale datasets, we follow the validation setting commonly adopted by these methods that support large datasets—training for 400 epochs on CIFAR.

For completeness, we also provide a comparison with [a], which achieves SOTA results on CIFAR and is trained for 1000 epochs in their paper. We directly obtained the 10 and 50 IPC distilled subsets from their official GitHub repository and evaluated them using the same validation code as in our setting to ensure a fair comparison. Our results outperform theirs under the same setup. We will include this comparison in the updated version.

| Dataset | IPC | $a$ (ResNet-18) | Ours (ResNet-18) | $a$ (ResNet-101) | Ours (ResNet-101) | |-|-|-|-|-|-| | CIFAR10 | 10 | 36.4±0.3 | 41.3±0.1 | 30.4±0.6 | 35.8±0.6 | | CIFAR10 | 50 | 55.4±0.4 | 70.8±0.5 | 41.9±0.6 | 63.9±0.4 | | CIFAR100 | 10 | 46.6±0.1 | 49.4±0.2 | 31.9±0.5 | 46.0±0.5 | | CIFAR100 | 50 | 61.0±0.2 | 65.7±0.3 | 57.5±0.5 | 66.6±0.2 |

Q3: The experiment for IPC=1 on CIFAR-10 and CIFAR-100.

A3: We provide a comparison with RDED in the table below, which achieves excellent results on CIFAR under the 1 IPC setting.

Q4: The specific VAE model, and the role of diffusion models.

A4: Thanks for pointing this out, We adopt the pre-trained DiT and VAE from [b]. The role of the VAE encoder in the framework is to encode the image to the latent space, while the DiT model is to inverse the latent for sampling new latent representation and then denoise the new latent representation to the VAE space. The VAE decoder is used to decode the sampled latents to pixels. Besides, as shown in Lines 149-155 left column, the motivation for using diffusion models lies in their ability to enhance realism and cross-model generalization, which are the key qualities of a high-quality distilled dataset. Based on this, as described in Lines 110–164 right column, our main motivation is that we identify three key limitations in current diffusion-based methods, primarily because of conducting optimization within the VAE latent space. This motivates us to explore a more effective optimization space, aiming to provide a better paradigm for diffusion-based methods. We will include further clarification in the updated version.

[a] Shao, Shitong, et al. "Generalized large-scale data condensation via various backbone and statistical matching." CVPR. 2024.

[b] Peebles, William, and Saining Xie. "Scalable diffusion models with transformers." ICCV. 2023.

Q1: Figure 1 and 3 less informative

A1: Thanks for the suggestions. The “blue lines” are the “blue contour lines” which indicate the probability density of the distribution. We will revise the caption to clarify this. Figure 1 is intended to convey two key messages: (1) The VAE latent space exhibits significantly lower normality compared to our mapped (inversion) space. The “blue contour lines” allow us to clearly observe the structure and concentration of the latents (blue dots). Denser and more centralized contours reflect regions with higher latent density, which are the areas we should pay more attention to. (2) We visualize representative latent points generated by our method. In Figure 1(b), we show the latents sampled in the mapped (high-normality) space, demonstrating that our sampling process closely matches the desired noise distribution. To further validate the fidelity of the mapping, Figure 1(a) presents the corresponding latents in the original VAE space after DDIM sampling. The blue contour lines help illustrate that our sampling process successfully captures the structure of the VAE latent space, and effectively concentrates in high-density regions. Similarly, Figure 3 visualizes n latents in the VAE space—comparing those generated by the default DiT model and our proposed method. It demonstrates that our approach yields more representative and diverse latents. We will provide more detailed explanations and clearer visual indicators to improve interpretability.

Q2: Deep discussion or empirical validation on why better alignment can be achieved in the inversion space

A2: The reason behind this is that a Gaussian mixture in the VAE space is more difficult to fit than a single Gaussian distribution in the inversion space. This is because compared to a Gaussian mixture with an uncertain number of components and unknown structure, a single Gaussian is a simpler and well-defined parametric form.

To prove our method achieves better distribution alignment, we visualize 10 generated latents in the VAE space for the same class ("Goldfish") using both D4M and ours, as shown in Appendix Figure A1 and Figure 1 in the main paper. The results show that D4M, which performs optimization directly in the VAE space, tends to select latents near the edges of the distribution. In contrast, our method produces a more representative and diverse set of latents, better capturing the overall structure of the distribution. For the quantitative comparison of accuracy, we refer to three key results: the D4M in Table 1, the default DiT generation in Table 3 (row 1), and our method with domain mapping but without group sampling in Table 3 (row 2). Together, the last has the highest accuracy, which demonstrates the effectiveness and necessity of our domain mapping strategy. Finally, to further support our claims, we also include a visual comparison of generated images for D4M, the default DiT, and ours. These qualitative results clearly illustrate the advantages of our approach in producing more representative samples.

Q3: Ablation study on domain mapping and group sampling

A3: We have the ablation studies in Section 6.1.

For the group sampling, we present the results in Table 3 (rows 2–9). These include our method without group sampling (row 2), as well as ablations that include only partial components of $L_{T, C}$. This allows us to verify the importance of group sampling and to assess the individual contributions of each component in $L_{T, C}$.

For the domain mapping, as discussed in lines 366–374, left column, we use the results in Table 3 (rows 1–2) to verify its importance. We compare the default DiT generation with DDPM (row 1) and domain mapping with DDIM (row 2), with the same configuration for all other steps. The results clearly demonstrate the advantages of domain mapping with DDIM (Line 375-384 left column). We will give a more clear description and annotation.

Q4: Performance on hard labels

A4: Please refer to Q1 of the response for xaqL.

Q5: Clarifying C for Equation 2

A5: C denotes the class condition, which means the same as other equations. We will provide a clearer explanation of this in the updated version.

Q6: Discussion of different Inversion Timesteps

A6: As discussed in Lines 378–415, left column, there is a trade-off between maintaining the Gaussian assumption and preserving image structural information across different time steps t. Please refer to Q3 of the response for xB8L with more details. The choice of inversion steps is guided by empirical observations. As shown in Figure 5, using 24–31 inversion steps consistently achieves SOTA accuracy, making it a reasonable and effective choice.

Q7: Storage of decoder

A7: In Appendix Figure A3, the decoder size has already been accounted for in our computation under “DiT weight.” The storage size of the VAE weights is 320MB. We will include this detail more clearly.

Q1: primarily focuses on optimal subset selection

In line 027-044 right column, we identify three key limitations in diffusion-based methods due to their reliance on optimization in the VAE latent space. This motivates us to identify a more effective space by DDIM inversion, aiming to provide a better paradigm for diffusion-based methods. Our contribution lies in the entire pipeline designed for dataset distillation, including DDIM inversion, distribution matching, group sampling, and generation—not merely the sampling. Each component is carefully designed to cohesively operate in the pipeline to ensure that the distilled subset is both compact and highly representative.

The process in the inversion space is not input selection. It is a principled generative procedure. As discussed in Sec. 4.3, we first map the original latent distribution to a Gaussian distribution due to Lemma 4.1, enabling efficient sampling via its high-normality. We first generate latents that probabilistically follow the distribution of the entire class, as shown in line 260–266 left column. Building on this, group sampling then supports parallel generation of candidate subsets (each maintaining distributional alignment), from which we identify the most representative one. The whole process tightly integrates sampling and optimization, enabling effective and distribution-aware subset generation tailored for diffusion-based synthesis, and goes beyond simple input selection.

Q2: The computation overhead for DDIM inversion

Please refer to Q6 of the response for Reviewer VDRK.

Q3: Extreme high T incur accuracy degradation

As discussed in Line 378–415 left column, there is a trade-off between maintaining the Gaussian assumption and preserving image structural information across different steps t. When t is small (such as 20), the distribution is a mixture of Gaussians as shown in Figure A4, and our distribution matching with a single Gaussian (Sec. 4.3) is not able to accurately describe the Gaussian mixtures, leading to certain performance loss. When t becomes large such as 40, although our distribution matching can accurately represent the real distributions which becomes more normal (Figure A4), the real distributions suffer from more significantly structural information loss due to adding more noise, which in turn degrades the performance of DDIM inversion. Thus, it is a trade-off between maintaining the Gaussian assumption and preserving image structural information.

Q4: Using cross-selection in group sampling

To validate cross-selection effect, we experiment on Tiny-ImageNet using 10 representative subsets with 10 IPC (performing group sampling 10 times; average accuracy: 44.07; individual results shown in Figure A2). We randomly combine the subsets, as shown in the table below, all three combined results yield lower accuracy than individual subsets. As discussed in Line 240–244 right column, this is because our $L_{T,C}$ is designed to ensure that the entire subset aligns well with the desired distribution, rather than fitting individual latents. Cross-combining latents from different subsets breaks this design principle without any distribution alignments.

Q5: The role of $L_{T, C}$, is $L_{T,C}$ complexity for usage and reproducibilit

Indeed, $L_{T,C}$ ​ is a static evaluation metric, which is easy to compute with basic mathematical operations, and the sampling process is efficient (Please refer to Q3-(1) of the response for xaqL). For $\lambda_\mu$, $\lambda_\delta$ and $\lambda_\gamma$, we set them to make the corresponding metrics on the same scale. We provide results for different $\lambda$ in Table below, which shows that the performance on $L_{T,C}$ is relatively robust to $\lambda$.

Q6: Choice of m across the datasets with different sizes

We empirically set the value of m. The table illustrates how accuracy varies with different m across datasets of different scales. As expected, increasing m improves accuracy, as a larger candidate pool offers greater diversity and a higher chance of including representative sets. However, when m becomes sufficiently large, the performance gains plateaus—indicating that the marginal benefit of adding more candidates diminishes, as the top-performing candidates become increasingly similar. We choose m at the saturation point, typically between 1e5 and 1e7, which can work well across different datasets.

Tiny-Imagenet, 10IPC:

CIFAR10, 10IPC:

Q7: Comparison with Minimax

Please refer to Q1 of the response for xaqL.

Q1: Comparison to MiniMax with hard labels

As noted in Lines 297–300 right column, we did not include Minimax in the main table as it focuses on small subsets of ImageNet-1K. For large datasets, it requires extra training of multiple diffusion models with high computational cost. They only report results for ResNet-18 on ImageNet-1K, and do not support the other three datasets and other architectures used in our table. For a fair comparison with their results, we give the comparison with Minimax using their main setting on ImageWoof. Minimax is the SOTA method with hard labels on ImageWoof. Our method with 224×224 resolution outperforms Minimax Diffusion with 256×256 resolution.

Q2: Why is Gaussian mixture "hard-to-fit"

"hard-to-fit" means that a Gaussian mixture is much more difficult to fit than a single Gaussian, due to its uncertain number of components and unknown structure. In contrast, a single Gaussian is a simpler and well-defined parametric form. In addition, in Fig. A1, we present the visualization of fitting the Gaussian mixture using K-means clustering under the D4M setting. It can be observed that some outliers near the edges of the distribution are selected, indicating that the Gaussian mixture is not well captured by the clustering. This further supports our point that the Gaussian mixture is relatively hard to fit.

Q3: About group sampling

Thanks for the valuable feedback. We explain the group samping more clearly, and then address your comments. Rationality of group sampling: We propose an efficient sampling design based on the mapped high-normality property. As discussed in Line 266-270 left column, the distribution of n latents (for n IPC) may still deviate from the desired distribution due to limited size n. To address this, we sample multiple subsets in parallel and choose the most representative subset with the closest distribution to the desired distribution, as detailed below.

(1) Computational efficiency: We report the runtime of the group sampling process in the table, demonstrating its simplicity and efficiency. As discussed in Lines 302-306 left column, this is because multiple subsets can be sampled in parallel on the GPU instead of sequential sampling, and the operations such as random sampling and mean/variance computations are very lightweight and efficient in current computation frameworks, making the entire process highly efficient.

m = 1e5, A100 40G:

(2) About analytical solutions: Group sampling is straightforward and efficient, taking just a few seconds to sample millions of subsets and select the best one. The mentioned analytical method to sample N-3 and solve the rest for accurate distribution matching may require additional complex solvers with high computation cost. Furthermore, with the solved examples, although the statistics of the distribution may match perfectly, the overall distribution of all samples may not be a Gaussian distribution. It is possible that the designed examples are far away from the sampled examples to match the desired distribution which is in the middle between them.

(3) Increasing/decreasing variance: Our domain mapping (Sec. 4.2) and distribution matching (Sec. 4.3) try our best to build an easy-to-fit distribution of real data. Then we generate synthetic samples following the real data distribution, to ensure the training performance on synthetic samples. Changing the variance may lead to a distribution deviation with samples not similar to real ones and degraded training performance. We further perform the experiment to verify the influence of variance. As shown in the table, we adjust the variance of the distribution by ±50% in sampling. The results show that neither increasing nor decreasing the variance leads to higher accuracy.

(4) Statistics may be biased for low IPCs: As the law of large numbers suggests, smaller sample sizes indeed leads to higher statistical bias. This also provides a reasonable explanation for the observed drop in accuracy at low IPC. This is a general issue for dataset distillation works. To mitigate this, our group sampling generates m subsets and selects the most representative one with the closest distribution. Thus, although one subset may suffer from large statistical bias, it is possible to find another subset with smaller bias. The results in Table 3 show our best performance compared with baselines in low IPC, showing the effectiveness of our method in addressing the bias issue.

Q1: no clear definition of normality

Thanks for pointing this out. In our context, normality refers to the degree of the latent space data conforms to a normal distribution. A higher level of normality indicates that the latent distribution more closely resembles a normal distribution. This usage is consistent with that in statistical normality test, which assess whether sample data is drawn from a normal distribution. We will add the clarification in the updated version.

Q2: not informative Figure 1

Please refer to Q1 of the response for phyT.

Q3: Why not use the embedding space of a pre-trained model

First, the scalability to multiple architectures in Lines 149-155 left column are the motivation why we use diffusion models. We agree with prior works (RDED, D4M, Minimax) that realism and cross-model generalization are key qualities of a high-quality distilled dataset, which can be effectively achieved with the help of diffusion models.

Based on this, as shown in Lines 110–164 right column, our main motivation is that we identify three key limitations in current diffusion-based methods, primarily because of conducting optimization within the VAE latent space. This motivates us to explore a more effective optimization space, aiming to provide a better paradigm for diffusion-based methods.

Certain previous approaches that rely on the embedding space of a pretrained model or a small set of pretrained models, can introduce architecture bias, which may limit their generalization capabilities in large scale datasets or different independent architectures. In contrast, we identify a better optimization space that is agnostic to any specific architecture, allowing a single distilled dataset to be directly used across a wide range of models for users. As shown in the Tabel A1 in the appendix, our method can achieve SOTA performance across various models just with one-time generation cost.

Q4: Sampling-based method instead of gradient-based optimization for group sampling

We propose an efficient and effective sampling design based on the mapped high-normality property. As discussed in Lines 260-266 left column, the n latents (corresponding to n IPC) probabilistically follows the distribution of the whole class C. Building on this, our group sampling method is simple and efficient to search the most representative subset. For computation overhead analysis, please refer to Q3-(1) of the response for xaqL. In contrast, the gradient-based optimization is computationally expensive and may compromise the probabilistic nature of the n latents.

Q5: The missing references

Regarding [c], we have already included the comparison in our paper under the name DWA in Table 1. As for [a] and [b], we provide the comparison on ImageNet-1K below, and will include the corresponding reference in the next version.

Resnet 18:

Cross Resnet101:

Q6: Time cost of a pre-trained diffusion model and DDIM inversion

It is common to use pre-trained models in dataset distillation. Diffusion-based methods (D4M, Minimax) rely on a pre-trained diffusion model, while other methods often use pre-trained teacher models to guide the distillation process. Thus, we do not count the training of pre-trained models towards our time cost.

The DDIM inversion is effective without high computation cost. (i) As discussed in Sec. 5.3, we only need to perform DDIM inversion once with a one-time cost, to generate multiple distilled datasets for different model architectures under various IPC settings. This is different from other methods which need to run their whole algorithms one more time if the architecture or IPC setting changes. (ii) The results of DDIM inversion are easy to store with little storage requirement, as shown in Appendix-Figure A3. (iii) Moreover, the computation cost of DDIM inversion is affordable. Even for ImageNet-1K, our method only requires approximately 4.5 hours on a single node with 8 A100 GPUs. In comparison to the SOTA diffusion-based method Minimax, which requires fine-tuning 50 separate pre-trained DiTs for ImageNet-1K, our approach is significantly more efficient. In addition, if more efficient denoisers (e.g., DPM-Solver families) are used, our paradigm can be further accelerated. This may be an orthogonal improvement direction and not the focus of our work.

Q7: Comparison with hard labels

We give the comparison results with Minimax diffusion, which is the SOTA method with hard labels under their main evaluation setting on ImageWoof. Our method with 224×224 resolution outperforms Minimax Diffusion with 256×256 resolution. Please refer to Q1 of the response for xaqL.