Skip the Steps: Data-Free Consistency Distillation for Diffusion-based Samplers

• Minor contribution.
  Thank you for your comment and for acknowledging the differences between our approach and consistency models. While it is true that our method shares some conceptual foundations with CMs, we believe that applying these ideas to enable single-step sampling for unnormalized distributions presents unique contributions.

  One of the key novelties of our work is adapting the consistency distillation framework to operate without access to a dataset, $\mathcal{D}$, unlike CMs, which rely on sampling from $\mathcal{D}$ as described in algorithms 2 and 3 of the CM paper (Song, 2023). The method we propose leverages only a pre-trained diffusion-based sampler to perform N-step simulations from a prior, where $N$ is uniformly sampled. This approach allows consistency distillation to be applied in a broader range of scenarios where CMs would be constrained by the need for pre-existing data.

• Insufficient experiments.
  Thank you for your observation. In the revised manuscript, we have added results for a 100-dimensional double-well example to evaluate the performance of our method in a more challenging, high-dimensional setting. This result is included as a new column in Table 1 of the revised manuscript. The added results show that, in the 100-dimensional double-well setting, the consistency sampler (CS) achieves performance comparable to that of the 200-step teacher models (DDS and DIS) while requiring significantly fewer steps (1 or 2 NFEs).

  Method	NFE	DW (d=100)
  DDS	200	10.9340
  DIS	200	10.9658
  DDS	2	30.5513
  DIS	2	20.7306
  CS DDS	2	9.4402
  CS DIS	2	10.5446
  DDS	1	48.8016
  DIS	1	34.0449
  CS DDS	1	9.3640
  CS DIS	1	12.8663

• Background.
  Thank you for your suggestion. In the revised manuscript, we have significantly reduced the length of the background section to focus on the most relevant details.

• Training computational costs.
  Thank you for pointing this out. We understand that simulating the given diffusion sampler during training could raise concerns about computational cost. However, the training cost of CS is less than that of the denoising diffusion sampler (DDS) and the time-reversed diffusion sampler (DIS). In DDS and DIS, the controlled process appears in the training objective (see equations (11) and (12)), necessitating the simulation of the entire trajectory and cannot resort to a Monte Carlo approximation, such as in the denoising score matching objective used in diffusion models [1]. Similarly, CS requires trajectory simulation during training; however, CS only integrates partial trajectories (as described in Algorithm 1) up to a random timestep. This saves approximately 50% of the training time, for a fixed number of training iterations.

  We have added a discussion in Section 5 of the revised manuscript that provides a comparison of the training costs for our method relative to DDS and DIS.

  1. Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.

• Meaning of $p_\text{target}$.
  This was a typo, and we removed it in the revised manuscript. Thank you for pointing it out.

• Data-free sampling.
  Thank you for your observation. To clarify, during training, consistency models (CMs) first sample $x \sim \mathcal{D}$, where $\mathcal{D}$ is a dataset of samples, and not from the prior distribution (as described in their algorithms 2 and 3). In contrast, our approach assumes that we are provided only with a pre-trained sampler and that no dataset $\mathcal{D}$ is available for training.

  While it is true that our method involves partially simulating the PF ODE trajectory using the pre-trained sampler, this does not equate to using data from a dataset. The training process leverages the pre-trained model directly, without needing to pre-collect and store a large dataset, as sampling from the prior is easy. This differs from CMs, where training relies on access to a dataset. We believe that this nuanced characteristic supports our claim of being data-free.

• Similarity to CM and comparison with CM.
  Consistency sampler (CS) and consistency model (CM) are designed to address different tasks. CS tackles the classical sampling problem, where the goal is to sample from a target density $p_\text{target}:= \rho / Z$, with an intractable normalizing constant $Z$. In this setting, we assume access to an unnormalized density $\rho$ but no data sampled from $p_\text{target}$.

  Applying CMs to this problem would first require converting the sampling task into a data generation task. This involves generating a dataset using a pre-trained sampler and then applying either consistency distillation or consistency training to learn the consistency model. In contrast, CS is specifically designed to avoid this intermediate step. By leveraging partial integration during training, CS bypasses the need to pre-collect and store a dataset, reducing both memory overhead and integration error by avoiding full trajectory integration.

• Benchmarks.
  Thank you for your thoughtful comment. To apply sampling methods like ours to high-dimensional data such as images (e.g., CIFAR or ImageNet), we would need access to an unnormalized probability distribution $\rho$ rather than training data. As mentioned previously, CS tackles the classical sampling problem, where the objective is to sample from a target density $\rho = p_\text{target}Z$. This setting differs from the data generation task addressed by diffusion models, which rely on access to training datasets.

  The added results of a 100-dimensional double-well example in the revised manuscript demonstrate that our approach can handle higher-dimensional settings effectively. This example provides insight into the scalability of CS to more complex and higher-dimensional tasks.

  Method	NFE	DW (d=100)
  DDS	200	10.9340
  DIS	200	10.9658
  DDS	2	30.5513
  DIS	2	20.7306
  CS DDS	2	9.4402
  CS DIS	2	10.5446
  DDS	1	48.8016
  DIS	1	34.0449
  CS DDS	1	9.3640
  CS DIS	1	12.8663

• Algorithm 1.
  Thank you for your observation. You are correct that the target density does not explicitly appear in Algorithm 1. In our approach, we assume a pre-trained diffusion-based sampler is given, such as DDS, which already includes information about the normalized distribution. This setting is practical, given the availability of pre-trained models on platforms like GitHub and HuggingFace. Our method can significantly accelerate their convergence (e.g., achieving results in just one step) while maintaining comparable sampling performance.

• Numerical examples are low-dimensional.
  Thank you for your observation. In the revised manuscript, we have added results for a 100-dimensional double-well example to evaluate the performance of our method in a more challenging, high-dimensional setting. This result is included as a new column in Table 1 of the revised manuscript. The added results show that, in the 100-dimensional double-well setting, the consistency sampler (CS) achieves performance comparable to that of the 200-step teacher models (DDS and DIS) while requiring significantly fewer steps (1 or 2 NFEs).

  Method	NFE	DW (d=100)
  DDS	200	10.9340
  DIS	200	10.9658
  DDS	2	30.5513
  DIS	2	20.7306
  CS DDS	2	9.4402
  CS DIS	2	10.5446
  DDS	1	48.8016
  DIS	1	34.0449
  CS DDS	1	9.3640
  CS DIS	1	12.8663

I have now read the other reviews. In light of this, I am keeping my score.

• Lipschitz assumption.
  Thank you for raising this question. The Lipschitz assumption in Theorem 1 is a standard regularity condition used to ensure that the consistency function behaves predictably with respect to small changes in its input. This assumption is also made for consistency models.

• Double-well dimension.
  Thank you for pointing this out. In the double-well example, the dimension was d=2. We have now clarified this in the revised manuscript. Additionally, we have included results for a higher-dimensional double-well example with d=100.

• Computational cost of training the consistency function.
  Thank you for raising this point. While it is true that we can generate samples directly using the pre-trained diffusion-based sampler, our work explores whether it is feasible to distill this model into a faster sampler. The training cost of our consistency sampler (CS) is a one-time expense, which amortizes the sampling cost at inference time. This trade-off is advantageous for applications where the sampler will be repeatedly used to generate new samples, like molecular design [1] and graph optimization [2].

  The training cost of CS is less than that of the denoising diffusion sampler (DDS) and the time-reversed diffusion sampler (DIS). In DDS and DIS, the controlled process appears in the training objective (see equations (11) and (12)), necessitating the simulation of the entire trajectory. Similarly, CS requires trajectory simulation during training; however, CS only integrates partial trajectories (as described in Algorithm 1) up to a random timestep, sampled uniformly. This saves approximately 50% of the training time for a fixed number of training iterations. We have added a discussion in Section 5 of the revised manuscript that provides a comparison of the training costs for our method relative to DDS and DIS.

  It is important to note that the speed comparison in Table 1 refers to the generative process and does not account for training times. DDS typically requires between 64 and 500 steps for generation [3], whereas CS achieves high-quality results in just 1 or 2 steps, with minimal quality trade-offs. This comparison method aligns with the distillation literature, where test-time efficiency improvements are emphasized (see, for example, Table 2 in [4]).

  1. Zhu, Y., Wu, J., Hu, C., Yan, J., Hsieh, C.-Y., Hou, T., and Wu, J. Sample-efficient multi-objective molecular optimization with GFlowNets. Advances in Neural Information Processing Systems, 2023.

  2. Zhang, D., Dai, H., Malkin, N., Courville, A., Bengio, Y., and Pan, L. Let the Flows Tell: Solving Graph Combinatorial Optimization Problems with GFlowNets. Proceedings of the Neural Information Processing Systems, 2023.

  3. Francisco Vargas, Will Sussman Grathwohl, and Arnaud Doucet. Denoising diffusion samplers. International Conference on Learning Representations, 2023.

  4. Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. International Conference on Learning Representations, 2022.

• Assumptions in Theorem 1.
  No, we do not need additional assumptions. The assumption that the loss is optimized to zero is crucial, and we have explicitly mentioned this in the paper on lines 335-336. A constant function would not satisfy this requirement due to the boundary condition $f(x_T^u, T) = x_T^u$ imposed by the parameterization in Eq. (6). This boundary condition ensures that the consistency sampler correctly maps the terminal state to itself, thereby disqualifying trivial solutions such as constant functions.

• Consistency function, mapping, and sampler.
  By “consistency mapping” and “consistency function,” we were referring to the function that maps any intermediate state of a trajectory directly to its terminal state. The consistency sampler is a model representation of this function. To enhance clarity, we have standardized the terminology throughout the revised manuscript: we no longer use the term “consistency mapping.” The terms “consistency function" and “consistency sampler" are defined in Section 4.1.

• Numerical examples are low-dimensional.
  Thank you for your observation. In the revised manuscript, we have added results for a 100-dimensional double-well example to evaluate the performance of our method in a more challenging, high-dimensional setting. This result is included as a new column in Table 1 of the revised manuscript. The added results show that, in the 100-dimensional double-well setting, the consistency sampler (CS) achieves performance comparable to that of the 200-step teacher models (DDS and DIS) while requiring significantly fewer steps (1 or 2).

• Typos.
  Thank you for pointing out typos. We fixed them in the revised version.