Elucidating the Preconditioning in Consistency Distillation

Thank you for your time and effort in reviewing our work! We appreciate your thoughtful and positive feedback on our work. We hope the responses below address your concerns.

Q1: How does performance and the actual values for the preconditioners change as we change the number of samples used to estimate Eqs. 15 and 17 (expressions for the preconditioners)? How small can we make the number of samples used while still retaining good performance? Can we further boost performance using more samples?

A: We did not extensively tune the number of samples due to the high cost of training experiments. Furthermore, even with the same number of samples, the trace estimator's variance and the specific dataset partition used can lead to differing estimations. Based on our experience, selecting a sample size between 1024 and 4096 produces preconditioners that appear visually similar and generally achieve speed-ups exceeding 1.5x. However, using fewer than 200 samples introduces estimation bias and degrades performance. We also experimented with larger sample sizes, up to 8192, but observed no additional improvements. This suggests that we have exploited the potential improvement space of the preconditioning, and optimizing the preconditioning alone may have upper bound and cannot bring further benefits.

Q2: Did you think about consistency training? These preconditioning parameters also show up in that case, but we don’t have access to the teacher, so it is unclear how to use the ideas in this work.

A: We believe consistency distillation is a more efficient and promising approach, as consistency training from scratch typically requires more iterations and results in sub-par performance. Intuitively, diffusion models can be stably and fastly trained to give the tangent of the ODE trajectory. It averages the high-variance tangents from random data-noise pairs and offers stable predictions. Even for consistency training, preparing a diffusion models in advance may also be beneficial. For example, in Appendix B.3 in the consistency model paper, they claim that they initialize the consistency model with the pretrained diffusion model when conducting continuous-time consistency training, and this initialization significantly stabilize the training.

Q3: While multi-step consistency models have been observed to underperform CTMs, it would be nice to have values in some of the results reported.

A: Thanks for your suggestion. We additionally tested multi-step CM and presents the results below.

As expected, CM cannot boost its performance with more sampling steps. It needs to perform alternative backward and forward jumps. As the forward jump is stochastic, it no longer ensures trajectory consistency and instead accumulates error.

Q4: What dataset is used for the GAN experiment?

A: CIFAR-10 is used as other datasets are in 64x64 resolution and demand higher computational resources. We have revised the paper to specify.

Thank you for your time and effort in reviewing our work! We appreciate your thoughtful and positive feedback on our work. We hope the responses below address your concerns.

Q1: The paper focuses on the case where $f=0, g=\sqrt{2t}$ because of the recent literature. Is the method still applicable for other choices of $f$ and $g$?

A: Sure! A more convenient way is to represent the forward process defined by the SDE as $q(x_t|x_0)=\mathcal{N}(\alpha_tx_0,\sigma_t^2I)$. The coefficients $\alpha_t,\sigma_t$ are determined by $f(t),g(t)$ in the forward SDE and satisfy $f(t)=\frac{d\log \alpha_t}{d t}$, $g^2(t)=\frac{d \sigma_t^2}{d t}-2\frac{d\log \alpha_t}{d t}\sigma_t^2$ [1]. The case $f=0, g=\sqrt{2t}$ is adopted in the recent literature as it corresponds to $\alpha_t=1,\sigma_t=t$ which is quite simple, and the corresponding diffusion ODE is $\frac{dx_t}{dt}=\frac{x_t-\hat x_0}{t}$, where $\hat x_0$ is the predicted $x_0$ by the denoiser function. For more general $f,g$, or equivalently $\alpha_t,\sigma_t$, we can apply some transformations to turn it into the simple case $\alpha_t=1,\sigma_t=t$.

Specifically, if we define $x_t'=\frac{x_t}{\alpha_t}$, then $x_t'$ satisties the forward process $q(x_t'|x_0')=\mathcal{N}(x_0',\frac{\sigma_t^2}{\alpha_t^2}I)$. If we additionally define a new time $t'=\frac{\sigma_t^2}{\alpha_t^2}$, the noise-to-signal ratio, then the forward process of $x_t'$ corresponds to $\alpha_t=1,\sigma_t=t'$. Intuitively, this creates a "wrapper" for the original process and turn them into the simple case. Therefore, the corresponding diffusion ODE can be written as $\frac{dx_t'}{dt'}=\frac{x_t'-\hat x_0}{t'}$, and we can still follow the procedure in the paper to derive preconditionings.

[1] Variational Diffusion Models

Q2: As far as I understand, the whole discussion depends on finding a good discretization of ODE (2). Both (9) and (13) use first order (Euler) methods. Can we get more insight if we try to use a better integrator?

A: That is an insightful understanding and question. We think in the design of preconditionings, we can only rely on first-order Euler methods. Better integrators such as high-order runge-kutta methods, require multiple evaluations of the ODE drift (which involves the network) to perform a single ODE step. However, in the case of preconditioning, which is a linear combination of $x_t$ and the network output, it will be much more expensive if we combine multiple network outputs, considering that they are involved in the gradient backpropagation during training.

Q3: Why is $q_T$ on line 118 a 0 mean Gaussian? Do we have some condition on $\mathbb{E}[q_0]$?

A: The zero-mean is only an approximation. There is no special restriction on the data distribution $q_0$. The intuition is, in practice, the data range is small (normalized to [-1,1]), while the final time $T$ is set to a very large value (like 80). This is often called a "variance-exploding" noise schedule. Therefore, compared to the large variance of the Gaussian distribution, the mean is relatively negligble. This can be understood from another perspective. The noisy data $x_t=x_0+t\epsilon,\epsilon\sim\mathcal{N}(0,I)$ has a very large scale when $t$ is large. For stability, before input into the network, it will be firstly normalized to something like $\frac{x_t}{\sqrt{1+t^2}}=\frac{1}{\sqrt{1+t^2}}x_0+\frac{t}{\sqrt{1+t^2}}\epsilon$. Therefore, as $t\rightarrow\infty$, the component of $x_0$ will tend to 0.

Q4: The $\lambda_t$ below equation (11) might be confused with $\lambda(t)$ in equations (5), (7).

A: Thank you for your suggestion. We have revised the notation of the weighting function from $\lambda(t)$ to $w(t)$.

Q5: $x$ and $x_t$ are used interchangeably in the RHS and LHS of the equations, better to be consistent. Example: line 182, line 276.

A: Thanks for spotting the typos. We have fixed them in the revised paper.

Q6: Is the code of the experiments released?

A: Due to the high cost of the training experiments and the need for proper permissions, we plan to release the code upon acceptance.

Thank you for your time and effort in reviewing our work! We appreciate your thoughtful and positive feedback on our work. We hope the responses below address your concerns.

W1: This paper does not provide whether BCM is better than CTM+ Analytic-Precond in terms of FID.

A: CTM, even combined with our Analytic-Precond, is slightly worse than BCM in terms of FID. The reason is that BCM adopts techniques from improved Consistency Training (iCT), such as better scheduler function and reweighting function. As our work mainly focuses on preconditioning instead of other techniques, we demonstrate in Figure 5 that BCM's preconditioning is not superior and cannot bring improvements to CTM, while ours can.

W2: Analytic-Precond does not perform better when GAN is incorporated into the CTM. Can the authors provide an explanation or intuition for this?

A: Sure! The analyses of preconditioning in our work, such as the consistency gap, are built on the investigations of how to learn better trajectory jumpers and maintain fidelity to the teacher ODE trajectory. However, the incorporation of the GAN loss is merely to enhance the FID at 1-step. As shown in Figure 6, in this scenario, the consistency function no longer faithfully adheres to the teacher ODE trajectory, and one-step generation is even better than two-step, deviating from our theoretical foundations.

Thank you for your time and effort in reviewing our work! We appreciate your thoughtful and positive feedback on our work. We hope the responses below address your concerns.

W1: For instance, the phrase "CMs aim to a consistency function" on line 134 might be better rephrased as "CMs aim to learn a consistency function"

A: Thank you for carefully reading our paper and pointing out the typo. We have fixed it in the revised paper.

Q1: Would it be possible for the authors to further expand on why such choice ensures the robustness of the resulting ODE again errors in $x_t$?

A: Sure! The reference you mentioned actually points to another paper that applies Rosenbrock-type exponential integrators to diffusion ODEs. We would like to give an illustrative example to explain its core idea and how it enhances the robustness against errors in $x_t$.

Suppose we want to solve an ODE from $t_n$ to $t_{n+1}$, where $t_{n+1}>t_n$:

The core idea of Rosenbrock-type exponential integrators is to separate as much linear component from $F$, as the linear part can be analytically absorbed into a "modulation" of the ODE. Specifically, let $F(x_t)=-l x_t+N(x_t)$, where $l>0$ (so that the ODE is not explosive in forward time), and the non-linear part $N$ satisfies that $\frac{dN(x_t)}{dx_t}\approx 0$ at $t=t_n$. Then by the chain rule, the original ODE can be turned into an ODE that describes the evolution of $e^{l t}x_t$ instead of $x_t$.

Denote $h=t_{n+1}-t_n$, the Euler discretization of the original ODE is

The Euler discretization of the modulated ODE is

Suppose $x_n'=x_n+e_n$ is the perturbed $x_n$ with error $e_n$, and $e_{n+1}=x_{n+1}'-x_{n+1}$ is the resulting error in $x_{n+1}$. As $\frac{dN(x)}{dx}\approx 0$ at $x=x_n$, we can omit $N(x_n')-N(x_n)$ for small $e_n$. Therefore, $|e_{n+1}|=|1-hl||e_n|$ for the original ODE, which may amplify the error when $h$ is large and $|1-hl|>1$. Instead, $|e_{n+1}|=e^{-l h}|e_n|<|e_n|$ after separating the linear part and modulate the ODE.