Adaptive Discretization for Consistency Models

Thank you very much for your encouraging feedback, your appreciation of our theoretical and practical contributions, and your deep understanding of this important problem. Below, we provide point-by-point responses to your concerns and questions.

Global Error: We first emphasize that ADCM focuses on the Consistency Training (CT) paradigm¹ due to its superior empirical performance²³⁴⁵, as noted in line 38. Under CT paradigm, ADCM has access to the data point $x_0$, which allows it to obtain an unbiased score estimation $s(x, t) = \frac{x - \alpha_t x_0}{\beta_t^2}$, as described in the original CM paper¹. Under this assumption，$x_t = \alpha_t x_0 + \beta_t z$ and $x_0$ lie on the same Probability Flow ODE (PF-ODE) trajectory for $\forall t \geq0$. For simplicity, we take the VE setting as an example. At this point, $x_t$ is given by $x_t = x_0 + tz$, and PF-ODE is given by

$\frac{dx(r)}{dr} = -r \cdot s(x(r),r) \quad \to \quad \frac{dx(r)}{dr} = \frac{x_0 - x(r)}{r}$.

We will prove $x_t = x_0 + tz$ and $x_0$ lie on the same PF-ODE trajectory from two perspectives.

1. We first consider the standard PF-ODE sampling process, i.e., taking $x(t) = x_t = x_0 + t z$ as the initial condition and solving the PF-ODE starting from time $t$.

In this case, the solution to the PF-ODE is given by $x(r) = x_0 + r z$. Substituting $r = 0$ yields the original data point $x_0$, confirming that $x_t$ and $x_0$ lie on the same PF-ODE trajectory.

2. To further address your concern, we consider the global error proposed by the reviewer. Under the CT paradigm, the ODE flow $g_t(x_0)$ mentioned by the reviewer corresponds to the solution $x(t)$ of the PF-ODE with initial condition $x(0) = x_0$, solved forward from $r = 0$ to time $t$.

In this case, the PF-ODE can only be solved under the assumption that $\lim_{r \to 0} \frac{x_0 - x(r)}{r} = z$, where $z$ is an arbitrary vector. Under this condition, the solution to the PF-ODE is $x(r) = x_0 + r z$. By taking $r = t$, we obtain $g_t(x_0) = x(t) = x_0 + t z$. Therefore, under the CT assumption, our definition of the global error is equivalent to that proposed by the reviewer.

The denoising error proposed by the reviewer does not involve any NN as it directly measures the distance between the noised data $x_t$ and the original data $x_0$. As such, its meaning is not entirely clear to us. We would greatly appreciate it if the reviewer could provide further clarification on the denoising error. We would be pleased to address any questions or concerns the reviewer may have.

Large Scale Results: To address your concern, we conduct experiments on ImageNet at $512 \times 512$ resolution. We choose EDM2⁶ as the base latent diffusion model which employ SD-VAE for image encoding and decoding. The detailed results are presented in the following table. We choose to compare our method with ECM² and sCT³, the most efficient prior CM, and our results are promising. We outperform ECM, with a similar training budget. While we do not yet match sCT’s performance, their method involves extremely higher training costs and specialized architecture. The results demonstrate that ADCM is capable of scaling to high-resolution images while maintaining high efficiency.

Footnotes

1. Song Y, et al. Consistency Models. ICML 2023. ↩ ↩²

2. Geng Z, Pokle A, Luo W, et al. Consistency models made easy. ICLR 2025. ↩ ↩²

3. Song Y, et al. Improved Techniques for Training Consistency Models. ICLR 2024. ↩ ↩²

4. Lu C et al. Simplifying, stabilizing and scaling continuous-time consistency models. ICLR 2025. ↩

5. Lee S, Xu Y, Geffner T, et al. Truncated consistency models. ICLR 2025. ↩

6. Karras T, Aittala M, Lehtinen J, et al. Analyzing and improving the training dynamics of diffusion models. CVPR 2024. ↩

Dear authors,

Thank you for the reply. Unfortunately, I must point out that you have an incorrect understanding of consistency models and PDF-ODE trajectories.

It is completely obvious that the PDF-ODE path cannot be $\gamma(t, x_0) = x_0 + t z$ sampled under the forward process since $z$ is sampled independently from $x_0$ and can have any value, while the PDF-ODE path $\gamma(t, x_0)$ is defined uniquely and has a well-defined terminal value \gamma(T, x_0) (the noise deterministically associated with the image x_0).

In the general case, the PDF-ODE path $\gamma(t, x_0)$ is not a straight line and it cannot be expressed as $\gamma(t, x_0) = x_0 + t z^*$ for any noise vectors $z^*$. It is true that $\frac{x_0 - x(t)}{\sigma^2(t)}$ is un unbiased estimator of the score, but this would give you an unbiased stochastic dynamics only if you re-sample $x_0$ for any given $x(t), t$ under the probability $p(x_0 \mid x(t),t)$. In any case, we are considering the deterministic dynamics here, which requires the exact score, not just an unbiased one-sample estimator of it.

In fact, the global denoising error that you are considering is identical to the consistency error if and only if $\gamma(t, x_0)$ is a straight line, which in a standard consistency model is true if and only if the dataset has a single datapoint. In this case, the mapping is not 1-to-1 and all arbitrary noise values map to the same x_0, making the linear equation correct. Needless to say, this case is not representative of real datasets and it is only sometimes valid approximately in the low noise regime.

Thank you for your positive evaluation and kind words regarding our method’s novelty, intuitive formulation, and strong experimental results. We also appreciate your recognition of the training efficiency improvements and address your concerns below.

Higher Resolution Generation Results: Following your suggestion, we conduct experiments on ImageNet at $512 \times 512$ resolution. We choose EDM2¹ as the base latent diffusion model, which employs SD-VAE for image encoding and decoding. The detailed results are presented in the following table. Training at this scale is time-consuming, which limits the extent of experiments we can perform during the rebuttal period. Moreover, we note that only sCT² reports results at this resolution. Comprehensive comparisons with other baselines are difficult, as training and re-implementing these methods demand significant time. Thus, we choose to compare our method with sCT and ECM, the most efficient prior CM. Our results are promising. We outperform ECM³, with a similar training budget. While we do not yet match sCT’s performance, their method involves extremely higher training costs and specialized architecture. The results demonstrate that ADCM is capable of scaling to high-resolution images while maintaining high efficiency.

$\lambda$ for Large Scale Runs: Due to the inherent instability of models in large-scale experiments, we recommend using a larger $\lambda$ to ensure training stability or just employ a larger $\lambda$ in the early training stage as a warm-up. This recommendation aligns with the design of our theoretical framework: a larger $\lambda$ places greater emphasis on global consistency, thereby enhancing stability.

Footnotes

1. Karras T, Aittala M, Lehtinen J, et al. Analyzing and improving the training dynamics of diffusion models. CVPR 2024. ↩

2. Lu C et al. Simplifying, stabilizing and scaling continuous-time consistency models. ICLR 2025. ↩

3. Geng Z, Pokle A, Luo W, et al. Consistency models made easy. ICLR 2025. ↩

Updated High-Resolution Experiments: We update the experimental results on high-resolution images with increased training budget and larger model, as shown in the table below. S stands for small with model capacity of 280.2M parameters and M stands for middle with model capacity of 497.8M parameters, following the notation used in sCT¹. It can be observed that our method scales well on large datasets. This further demonstrates the empirical effectiveness of our method. We are currently conducting experiments on larger scales. While these experiments are progressing well, full results are pending due to time and computational resource constraints. We will continue to update the results as they become available.

Footnotes

1. Lu C et al. Simplifying, stabilizing and scaling continuous-time consistency models. ICLR 2025. ↩

We thank the reviewer for your careful consideration. We greatly appreciate the positive comments and address your concerns below.

Optimization over $f_{\theta^-}(x_t-\Delta t)$: We kindly note that our objective is to enforce global consistency by optimizing $\Delta t$. Global consistency is reflected in the cumulative error between the practical training target $f_{\theta^-}(x_t - \Delta t)$ in Eq.(2) and the actual target $x_0$ (i.e., the data). This implies that, when $\Delta t = 0$, the practical target becomes $f_{\theta^-}(x_t)$, which often leads to a larger cumulative error. In this case, global consistency is at its weakest, resulting in unstable training. Therefore, we choose to optimize $f_{\theta^-}(x_t-\Delta t)$ instead of $f_{\theta^-}(x_t)$.

Stopgrad Definition: $\operatorname{stopgrad}(\theta)$ function is is a standard component in previous Consistency Model works¹²³⁴. To address your concern, we will clarify its definition and explain why $\theta^-$ can't be optimized after applying $\operatorname{stopgrad}$. It treats the given parameter $\theta$ as a constant during gradient computation. This means that while the forward pass still uses the actual value, the backward pass cuts off the gradient flow. As a result, we cannot optimize over $f_{\theta^-}(x_t)$ since it does not participate in gradient computation.

Computational Complexity of Algorithm 1: The computational overhead in Algorithm 1 compared to other methods comes from the computation of $\mathbb{T}$. For each value of $\Delta t$, we use a mini-batch to estimate the expectation and compute the Jacobian. The computational overhead introduced by the Jacobian calculations is negligible. As noted in line 224, the Jacobian can be efficiently computed using PyTorch’s built-in Jacobian-vector product (JVP).

Moreover, since we adopt the Gauss-Newton method, which incorporates second-order approximation information, the optimization typically converges faster than first-order methods⁵, further reducing the overall computational cost. Additionally, due to the infrequent updates of $\mathbb{T}$, the overhead of computing $\mathbb{T}$ is minimal relative to the total training time.

As shown in Figure 3a, the additional computational cost is typically less than 4% under the same number of training epochs, while significantly improving performance. As shown in Figure 3b, ADCM significantly accelerates the convergence of Consistency Models (CM), allowing it to achieve better generative performance with fewer training steps compared to other CMs.

Approximation Error: To solve the optimization problem in Eq.(8), we adopt the Gauss-Newton method, as it is well-suited for problems involving complex neural networks. Specifically, we approximate $f_{\theta^-}(x_t - \Delta t)$ using the first-order Taylor expansion of the neural network. It is important to note that this approximation error is acceptable, since, as stated in line 265, we do not require precise step sizes but only need to capture the trend of step size changes over time $t$. To further address your concern, we calculated the average approximation error incurred during the computation of $\mathbb{T}$ with different $\lambda$. The error is measured by $\frac{\|f_{\theta^-}(x_{t-\Delta t}) - (f_{\theta^-}(x_{t}) - v \Delta t) \|}{\| f_{\theta^-}(x_{t-\Delta t}) \|}$. The detailed results are presented in the following table. It can be observed that the approximation error is very small even with a large $\lambda$, demonstrating that the Gauss-Newton method provides a sufficiently accurate analytical solution in our setting.

Generation Results on CIFAR-10 and ImageNet: We kindly note that all baseline works conduct experiments exclusively on CIFAR-10 and ImageNet, which is consistent with our experimental setup. Therefore, our choice of datasets and resolutions aligns with established prior work and should not be considered a limitation.

Discussion on $\lambda \to \infty$: As noted in Remark 3.1, when $\lambda \to \infty$, this is equivalent to choosing the maximum optimization step $\Delta t = t - \epsilon$. In this case, the optimization objective of the Consistency Model in Eq.(2) degenerates into that of the Diffusion Model in Eq.(1), thereby losing the ability to perform single-step generation.

Discussion on bound $\delta$: In Eq.(6), we use a constant $\delta$ to bound the global consistency, rather than designing it as a time-dependent function $\delta(t)$. This is due to the complexity of neural networks, which makes it difficult to determine an appropriate $\delta(t)$ for every $t$. Therefore, we adopt a uniform constraint using a constant $\delta$ that satisfies $\delta \geq \max_t \delta(t)$. It is important to note that the choice of $\delta$ does not affect our subsequent conclusions, as we will relax the original optimization problem from Eq.(7) to Eq.(8).

Footnotes

1. Song Y, et al. Consistency Models. ICML 2023. ↩

2. Geng Z, Pokle A, Luo W, et al. Consistency models made easy. ICLR 2025. ↩

3. Lu C et al. Simplifying, stabilizing and scaling continuous-time consistency models. ICLR 2025. ↩

4. Frans K, Hafner D, Levine S, et al. One step diffusion via shortcut models. ICLR 2025. ↩

5. Gratton S, Lawless A S, Nichols N K. Approximate Gauss–Newton methods for nonlinear least squares problems.SIAM Journal on Optimization. ↩

Updated High-Resolution Experiments: We update the experimental results on high-resolution images with increased training budget and larger model, as shown in the table below. S stands for small with model capacity of 280.2M parameters and M stands for middle with model capacity of 497.8M parameters, following the notation used in sCT¹. It can be observed that our method scales well on large datasets. This further demonstrates the empirical effectiveness of our method. We are currently conducting experiments on larger scales. While these experiments are progressing well, full results are pending due to time and computational resource constraints. We will continue to update the results as they become available.

We appreciate your time and effort in evaluating our work and look forward to any feedback you may have.

Footnotes

1. Lu C et al. Simplifying, stabilizing and scaling continuous-time consistency models. ICLR 2025. ↩

We thank the reviewer for your careful consideration. We greatly appreciate the positive comments and address your concerns below.

Technical Significance: Our theoretical contributions lie in (1) Novel theoretical formulation: To the best of our knowledge, this is the first work to introduce global consistency into Consistency Training, in order to control accumulated errors across discretization intervals and improve the stability of training.

We respectfully disagree with the comment that "just presenting Eq.(8) directly without the wrapper of optimization" would suffice. The constrained optimization formulation is not merely a wrapper; it plays a central role in both the theoretical motivation and the practical design of our method. In our formulation (Eq.(7)), local consistency is cast as the primary training objective, while global consistency is introduced as a constraint. This separation is intentional and meaningful: local consistency drives accurate stepwise transitions (the core goal of CM training), whereas global consistency captures desirable trajectory-level behavior. Framing this as a constrained problem highlights their asymmetric roles and directly motivates the design of the relaxation in Eq.(8), where a small trade-off coefficient ensures that local consistency remains dominant. Without this optimization-based grounding, Eq.(8) would lack a clear theoretical justification, and the choice of trade-off weights would appear arbitrary.

Moreover, we kindly note that, as discussed in Remark 3.1, our formulation is not limited to deriving Eq.(8); rather, it provides a unified theoretical framework that encompasses previous heuristic discretization strategies as special cases (via different constraint formulations or trade-off weights). This unification not only clarifies existing methods but also enables principled extensions going forward.

(2) Closed-form and efficient solution: Another key aspect of our theoretical contribution lies in the solution of Eq.(8), which is nontrivial due to the inherent complexity and nonlinearity of the NN. We address this challenge by deriving a closed-form solution through a principled approach based on Gauss-Newton and first-order approximations, making the approach both efficient and theoretically grounded. As noted by other reviewers, this illustrates a "neat" example of the interplay between theoretical formulation and practical machine learning.

Higher Resolution Generation Results: We kindly note that most seminal baseline works ¹²³ conduct experiments exclusively on CIFAR-10 and ImageNet at $64 \times 64$ resolution, which is consistent with our experimental setup. Therefore, our choice of datasets and resolutions is not a limitation but aligns with prior art. Even on these lower-resolution datasets, previous methods exhibit training instability and rely on manually designed discretization schemes, incurring substantial training overhead. In contrast, our method effectively addresses these challenges, as demonstrated in our experiments, which provides strong support for our claims.

To further address your concern, we conduct additional experiments on ImageNet at $512 \times 512$ resolution. Training at this scale is time-consuming, which limits the extent of experiments we can perform during the rebuttal period. Moreover, we note that only sCT⁴ reports results at this resolution. Comprehensive comparisons with other baselines are difficult, as training and re-implementing these methods demand significant time. Thus, we choose to compare our method with sCT and ECM¹, the most efficient prior CM.

We choose EDM2 ⁵ as the base latent diffusion model which employs SD-VAE for image encoding and decoding. The detailed results are presented in the following table. Our results are promising. We outperform ECM, with a similar training budget. While we do not yet match sCT’s performance, their method involves extremely higher training costs and specialized architecture. The results demonstrate that ADCM is capable of scaling to high-resolution images while maintaining high efficiency.

Optimal Hyper-parameter $\lambda$: Following your suggestion, we examine the sensitivity of our proposed hyperparameter $\lambda$. Specifically, we investigate the difference between the optimal $\lambda$ under the Flow Matching setting and that under the VE setting. The detailed results are presented in the following table. Notably, the choice of $\lambda$ shows consistent behavior across settings: in both the VE and Flow Matching cases, setting $\lambda = 0.01$ achieves the optimal 1-Step FID. This consistent trend highlights the robustness of ADCM and substantially reduces the need for hyperparameter tuning.

Huber Loss: We adopt the pseudo-Huber loss, similar to standard practice in prior works ¹²³, as the pseudo-Huber metric is more robust to outliers ². The Pseudo-Huber loss, expressed as $d(x, y) = \sqrt { \| x-y \|_ 2^2 + c^ 2} - c$, smoothly bridges the gap between the L2 loss and the squared L2 loss. When $c=0$, the Pseudo-Huber loss degenerates to the standard L2 loss. When $c \to \infty$, it becomes equivalent to the squared L2 loss as $\lim_{c \to \infty} \sqrt{\|x-y\|_2^2 + c^2} - c = \frac{\|x-y\|_2^2}{2c}$.

To address this phenomenon, we conduct more ablation studies on the effect of c based on Table 7, and the results are presented in the table below. It can be observed that Pseudo-Huber smoothly interpolates between L2 and squared L2 through the control of the parameter c, thus achieving performance that surpasses both extremes. When c becomes too large, the Pseudo-Huber loss asymptotically approaches the squared L2 loss, leading to a decline in performance. These results clearly demonstrate the significance of choosing Pseudo-Huber as our distance metric.

Some Empirical Results deferred to appendix: We will bring key ablation results and empirical settings from the appendix back into the main text of the revised manuscript. Thank you for your valuable suggestion.

Footnotes

1. Geng Z, Pokle A, Luo W, et al. Consistency models made easy. ICLR 2025. ↩ ↩² ↩³

2. Song Y, et al. Improved Techniques for Training Consistency Models. ICLR 2024. ↩ ↩² ↩³

3. Lee S, Xu Y, Geffner T, et al. Truncated consistency models. ICLR 2025. ↩ ↩²

4. Lu C et al. Simplifying, stabilizing and scaling continuous-time consistency models. ICLR 2025. ↩

5. Karras T, Aittala M, Lehtinen J, et al. Analyzing and improving the training dynamics of diffusion models. CVPR 2024. ↩

Updated High-Resolution Experiments: We update the experimental results on high-resolution images with increased training budget and larger model, as shown in the table below. S stands for small with model capacity of 280.2M parameters and M stands for middle with model capacity of 497.8M parameters, following the notation used in sCT¹. It can be observed that our method scales well on large datasets. This further demonstrates the empirical effectiveness of our method. We are currently conducting experiments on larger scales. While these experiments are progressing well, full results are pending due to time and computational resource constraints. We will continue to update the results as they become available.

We appreciate your time and effort in evaluating our work and look forward to any feedback you may have.

Footnotes

1. Lu C et al. Simplifying, stabilizing and scaling continuous-time consistency models. ICLR 2025. ↩

Thank you very much for your thoughtful follow-up and for taking the time to re-evaluate our submission. We sincerely appreciate your acknowledgment of the clarification on the theoretical contribution and the empirical improvements.

We will incorporate the relevant clarifications and discussions into the final version of the paper. Regarding the point on asymmetric roles, we understand your perspective and agree that the asymmetry can be influenced by the choice of the Lagrange multiplier. We will make this clearer in the final version.

On the theoretical side, we would like to highlight that our modeling and optimization framework is grounded in theory while also incorporating practical considerations motivated by real-world scenarios. Standard discretization strategies are often handcrafted and overlook the underlying mathematical structure; we consider our work an important step forward that contributes meaningfully to the literature. By introducing global consistency as a constraint, we provide a closed-form solution that is both elegant and practical.

We’re grateful that you found the additional empirical evaluation helpful and truly appreciate your decision to raise the score.