Convergence of Consistency Model with Multistep Sampling under General Data Assumptions

Thank you for your feedback. We address your points in detail below:

1. non-uniform discretization: in this paper, we adopt a uniform discretization for clarity and ease of presentation. However, our results can be extended to the non-uniform discretization setting as well. Suppose $\tau_{0:M}$ is an arbitrary discretization of the interval $[0,T]$. In this scenario, it is reasonable to assume that the consistency loss scales with the length of the discretization interval:

Using the same argument as in the proof of Lemma 2, we can derive:

which is upper bounded by: $\sum_{s=0}^{i-1} (\tau_{s+1}-\tau_s)\epsilon = \tau_i \epsilon$. The rest of the proof remains unchanged. 2. regarding experiments on real-world data: in designing the experiments, we aimed to achieve two goals:

• evaluate the tightness of our upper bound: this requires access to the true data distribution, which is not available for real-world datasets, making such evaluation infeasible in those settings;
• demonstrate diminishing performance with multiple sampling steps: we already observe this phenomenon in real-world datasets, as shown in Table 1 and 2 of Luo et al. (2023), where the performance of the consistency model deteriorates with more sampling steps.
• Given these considerations, we prioritized simulation-based experiments over real-world datasets.

Thank you for your feedback. We address your points in detail below:

1. Interpretation of Theorem 2: trade-off means increasing the number of sampling steps does not necessarily lead to improved performance due to the influence of term (ii). This is in contrast to standard diffusion models, where, according to Theorem 2 of Chen et al. 2023b, the discretization error diminishes as the number of sampling steps increases, guaranteeing performance improvement with more steps.
   For Case Study 2, we now show that a uniform sampling schedule $t_j = (N-j+1) \Delta\tau$ with $N = \frac{t_1}{\Delta\tau}$ (more sampling steps) can yield a larger upper bound than the halving schedule defined in equation (13). Ignoring absolute constants, the upper bound in equation (12) becomes:

which is minimized when $t_1 = \frac{2^{1/3}R^{2/3}\Delta\tau}{\epsilon^{2/3}}$, yielding a minimum value of:

In contrast, the halving schedule achieves an upper bound of $\tilde O\left(R\sqrt{\frac{\epsilon_{\text{cm}}}{\Delta\tau}}\right)$, which is strictly smaller than that of the uniform schedule.
Practical evidence further supports this trade-off: - Our simulation results (Appendix G) show that both baseline methods experience degraded performance in the final sampling steps. - In Table 1 and 2 of (Luo et al. 2023), LCM with 4-step sampling achieves better FID scores than with 8 steps.

• Both theoretical analysis and empirical observations highlight the importance of stratigically designing the sampling schedule. Thus, we believe it is fair to emphasize the trade-off in choosing the number of sampling steps.

• clarification on the result in Case Study 1: Corollary 1 provides a universial upper bound on the $W_2$ distance for the VP process, without imposing constraints on $R$, $\Delta\tau$, and $\epsilon_{\text{cm}}$. Naturally, when the consistency model $\hat f$ is poorly estimated, the generated sample quality deteriorates. We believe the case you mentioned belongs to this category. If $\frac{\epsilon_{\text{cm}}}{\Delta\tau}\approx R$, it indicates that $\hat f$ is a poor approximation. Even when using the true marginal $P_T$ as input, Lemma 2 shows that $\hat f$ incurs an error of $O(R)$. In this scenario, the resulting upper bound is inevitably loose and uninformative.
• the significance of constant reduction in Case Study 1: while many theoretical results focus on asymptotic rates and ignore constants, constant reductions can have significant practical implications. For example, latent diffusion models (Rombach et al.) demonstrate improved performance (e.g., lower FID scores) even with fractional gains, enabling high-resolution image synthesis in practice.
• the rate improvement in Case Study 2: our analysis in Case Study 2 shows a clear rate improvement for the VE process when sampling with multiple steps. If we use only a single step ($N=1$), equation (12) simplifies to: $$\frac{R\sqrt{R}}{\sqrt{t_1}} + t_1 \frac{\epsilon_{\text{cm}}}{\Delta\tau},$$ with the minimum value being $R\left( \frac{\epsilon_{\text{cm}}}{\Delta\tau} \right)^{1/3}$. In contrast, using the specialized schedule from equation (13), we obtain a faster rate of $\tilde O(R\sqrt{\frac{\epsilon_{\text{cm}}}{\Delta\tau}})$.

3. Clarification on $\Delta\tau$: fix a $T$, decreasing $\Delta\tau$ results in a finer discretization $\mathcal{T}$ (line 147), thereby increasing the number of discretization points $M$. This adds more terms to the error decomposition (line 419-423) when evaluated at $\tau_M = T$.
   • Training steps vs sampling steps: for simplicity, our current formulation assumes sampling steps are chosen from the training steps. Under this assumption, a smaller $\Delta\tau$ allows a smaller $t_N$. However, our theoretical framework remains valid even when sampling steps are chosen arbitrarily. In that case, we need to refine Lemma 2 to upper bound $||\hat f(\cdot,t)-f(\cdot,t)||_2^2$ for all $t$. Because self-consistency is only enforced at the $\tau_i$'s during training, $\hat f(x,t)$ and $f(x,t)$ need to be Lipschitz in $t$.
   • Effect of $\Delta\tau$ on $\epsilon$: using a smaller $\Delta\tau$ may reduce the consistency error $\epsilon$ for two reasons: (1). by continuity of $\hat f$ and $\varphi$, equation (4) decreases as $|\tau_{i+1}-\tau_{i}|$ decreases; (2). As shown in Theorem 1 and 2 of Song et al. 2023, smaller $\Delta\tau$ improves the approximation for the consistency loss in both consistency distillation and consistency training. Hence, $\epsilon$ can be smaller.

Thank you for your feedback. Please see our detailed responses below:

1. Regarding high-dimension issue: even when accounting for the implicit dependency on the dimension, our upper bound remains at most polynomial in dimension and thus does not suffer from the curse of dimensionality. For example, in Corollary 1, our result shows that given an $\epsilon$-accurate consistency function estimate, the Wasserstein distance $W_2(\hat P, P_{\text{data}})$ depends only on consistency error $\epsilon$ (assuming $\Delta \tau = 1$ for simplicity) and the diameter of the distribution $R$. Considering the implicit dependency on $d$:
   • the diameter $R$ grows at most polynomially with $d$: if the ground-truth distribution $P_{\text{data}}$ has support on the $d$-dimensional hyper-cube $[0,1]^d$, then $R=\sqrt{d}$; if the ground truth distribution $P_{\text{data}}$ has support only on a low-dimensional manifold, e.g. $[0,1]\times \{0\} \times \cdots \times \{0\}$, then $R$ can be some constant.
   • the consistency error $\epsilon$ scales at most polynomially with $d$: by definition, $\epsilon$ arises from a $d$-dimensional prediction problem and is expected to scale at most polynomially in $d$. Given these factors, we believe our bounds remain both reasonable and meaningful in high-dimensional real-world scenarios.
2. Regarding experiments: Please refer to our Appendix G for simulations related to Case study 1. We show that for two heuristic sampling strategies, increasing the number of sampling steps yields diminishing improvements or even degraded performance. In contrast, our sampling strategy nearly matches the best performance of the heuristic strategies with only two sampling steps. Furthermore, our theoretical upper bound closely approximates the Wasserstein distance observed in practice during sampling. Additionally, Tables 1 and 2 in Luo et al. (2023) show that in LCM, 2-step sampling significantly outperforms 1-step sampling in terms of FID score, while 2-, 4-, and 8-step sampling exhibit comparable performance. Notably, 8-step sampling even yields worse FID scores than 4-step sampling. These empirical findings further support our theoretical results.
3. Regarding adaptive schedules:
   • In Case Study 1, we optimize the general upper bound in Theorem 2 by choosing $t_j$'s strategically and propose a two-step sampling strategy that adapts to both the problem parameters (the distribution diameter $R$) and the learned $\hat f$ (the consistency error $\epsilon_{\text{cm}}$);
   • For Case Study 2, here is a strategical derivation for a sampling schedule. In order to adjust the $t_j$'s to minimize the error upper bound, we first estimate the lower bound of equation (12). For an arbitrary sampling strategy $t_1 \ge t_2 \ge \ldots \ge t_N$, the summation inside term (i) can be lower bounded by:

A schedule defined by a geometric series approximates this lower bound well. Letting $t_j = t_N \rho^{N-j}$ with $\rho > 1$, we have $\sum_{j=2}^N \frac{t_{j-1}^2}{t_j^2} = \rho^2 \log_\rho\frac{t_1}{t_N} = \frac{\rho^2}{\log \rho}\log\frac{t_1}{t_N} \ge 2e\log\frac{t_1}{t_N}$, where equality holds when $\rho = \sqrt{e}$. Substituting this into equation (12), we obtain:

To minimize this expression, we choose $t_1 = \sqrt{\frac{1}{e}}\frac{R\Delta\tau}{\epsilon_{\text{cm}}}$. As $t_N \to 0$, the second term decreases linearly, while the first term increases slowly. Therefore, a reasonable choice is $t_N = \Delta\tau$, a small constant.

• In addition, for any new sampling strategy, one only needs to specify the noise schedule $\{\alpha_t,\sigma_t^2\}$ and the sampling steps $(t_1, t_2, \ldots, t_N)$ in Theorem 2. The majority of the proof remains unchanged.