Consistency Models Made Easy

While I believe this paper provides a valuable and actionable contribution, a few weaknesses, listed below in decreasing order of importance, prevent me from providing a positive recommendation.

Clarification of the novelty of some contributions

The idea of initializing consistency models with a pre-trained diffusion model has already been considered in the initial paper of Song et al. (2023, Appendix B.3), albeit in a restricted setting and without the experimental value highlighted in the present paper. Still, this should be discussed and may have consequences on the empirical study (see next weakness).

Section 3.1 remains vague whether the presented derivations are contributions of the present paper or reformulated background from consistency models. My understanding is that these derivations, including the final loss function, are all already included in the original paper of Song et al. (2023): the differential equation appears in another form in their Appendix Remark 4, and the loss function closely resembles the original loss function (by removing scheduling information). This should be clarified.

The choice of weighting function appears as a reformulation of Song & Dhariwal (2023), unless I misunderstood its description in the paper. The insight of the pseudo-Huber metric providing an adaptive weighting term is interesting, but I do not see how the proposed alternative (using the $\ell_2$ as metric and modulate it with the weighting that the pseudo-Huber metric would have brought) is any different. Could the authors clarify this?

Lacking experimental insights

While appealing, the presented experimental results lack additional insights and ablations to fully support the claims of the paper and highlight its added value w.r.t. prior work. Two main components are missing.

The first missing component is a full fledged ablation study. One is already included in Appendix Section B (Table 5). However, given this preliminary result showing that the simple diffusion pre-training (already considered in prior work, cf. previous weakness) brings most of the improvements, I think the paper would benefit from including in its main part an augmented ablation study in the same experimental setting as Section 4. This ablation study should then be discussed to assess the significance of each contribution in the new training recipe.

The second missing component is a study of the evolution of the methods' performance w.r.t. training time/iterations. As is, baseline and ablation results are presented at a fixed number of iterations. Plotting their performance w.r.t. the number of iterations would enable a more comprehensive efficiency comparison, as well as justify the choice of stopping time for baselines in e.g. Table 1.

Lack of moderation and rigor in some assertions

This is a less important issue that still deserves to be addressed. The following statements lack moderation and/or rigor.

• In Section 1, it is stated that "the speedup achieved by these sampling techniques usually comes at the expense of the quality of generated samples". This is partially incorrect as some works like the one of Karras et al. (2022) significantly reduced the number of required model evaluations while maintaining performance.
• It is incorrect that the proposed loss function of Eq. (12) "generalizes Song et al. (2023)’s discrete training schedule to continuous time". The proposed loss indeed relies on discretizing the aforementioned differential equation. Instead, this loss is exactly the consistency training loss of Song et al. (2023) untied from its specific discretization grid.
• Writing $\Delta t \to \mathrm{d}t$ instead of $0$ is confusing.
• In Section 4.1, the proposed model is said to "only [require] 1/1000 of [Score SDE's] inference cost". This is correct but misleading as state-of-the-art diffusion models no longer require thousands of model evaluations. This statement should be toned down.

Minor issues

• The diffusion ODE of Eq. (2), from Karras et al. (2022), requires that $\mathbf{f} = 0$ in Eq. (1).
• $f(\mathbf{x}_t, t)$ is said to be "a denoising function" after Eq. (3). This statement should be more detailed.
• To my knowledge, in Karras et al. (2022)'s timestep schedule, $\rho = 7$ instead of $0.7$ on line 134.
• To my understanding, a part of the numbers in Table 1 were not obtained by the authors but are reported from other papers. This should be clarified.
• $\epsilon$ in Eq. (19) should be $c^2$ to be consistent with the rest of the paper.

1. While the "curse of consistency" is indeed fascinating, discussing only the upper bound fails to capture the true nature of errors. An increase in the upper bound does not necessarily indicate a corresponding increase in error.
2. Since the primary advantage of your method lies in its training speed, I believe you may have overlooked an important scenario: "pretraining + iCT tuning," as illustrated in Figure 2.
3. The relationship between your primary observation and your main method is not clear. Specifically, I find it difficult to understand the necessity of proposing a "continuous-time training schedule" and a "weighting function" to address the "curse of consistency" problem. If none of your techniques are linked to the "curse of consistency," what is the rationale for mentioning it, and how does this discussion relate to the overall objectives of your paper? In fact, the "curse of consistency" is not even addressed in your abstract.
4. You should place the ablation studies of your techniques (Table 6) in your main paper instead of the appendix.
5. The "dropout" technique appears to be quite significant; could you clarify why it is placed in the appendix?

Novelty. Despite the strong results demonstrated in this work, I have questions about its novelty. The main methodological contribution is to initialize CM training (CT) with a pre-trained DM to enable faster convergence. In the setting of consistency distillation (CD) and DM distillation in general, it is already common practice to initialize the student from the weights of the teacher DM, effectively reducing distillation to a fine-tuning task to reduce computational requirements and facilitate convergence, similar to what’s being motivated here. The difference in this work compared to CD is to instead initialize a CM from a DM in the setting of CT, which alleviates DM error accumulation as in the case of CD. However, this seems like a rather simple extension of CMs and the connection between CMs and DMs does not seem particularly novel, so I would ask the authors to clarify their proposed novelty here.

Moreover, the authors formulate CMs in continuous-time which was already done in the seminal CM work by Song et al. in Appendix B.2, so it’s not clear what the difference is here if any. Along this vein, I would also ask the authors to clarify the purpose of Section 3.1 and why it’s necessary to formulate CMs in terms of the differential condition since it currently reads as a seemingly disconnected derivation and it’s not clear from reading the paper why this formulation is needed for the rest of their method instead of using the standard discrete/continuous CM formulation by Song et al. In addition, the particular weighting function derived in Section 3.1, $w(t,r)=\frac{1}{t-r}$, does not seem to be what the authors actually end up using in practice since, in Section 3.3, the authors go on to say “Instead, we consider decoupled weighting functions without relying on $p(r|t)$” in L311-L312. Can the authors please clarify these points and also update the manuscript accordingly.

Ablations. Given that the central contribution of ECT is initializing CMs from a DM, it would be important to see the performance of training ECT from scratch (i.e., w/o initializing from a DM) while keeping all other ECT design choices fixed, namely those outlined in Section 3.3 in terms of training schedule, loss function, and weighting function. Can the authors please provide this result in their rebuttal as it is a relevant ablation?

Easiness. The authors claim that their method makes CMs easy to train but, from a design perspective, various choices (e.g., Equation 15) do not necessarily seem more intuitive or “easier” compared to those in the original CT or in the follow-up iCT. I think it would be fair to say that ECT makes training CMs more accessible and more computationally efficient but it’s not clear that “easy” is the right characterization. Can the authors please clarify their position?

The main motivation of the paper is straightforward. It is hard for the reader to fullly trust their obverstaion that diffusion models can be viewed as a special case of CMs in practice. The data sets and metircs on the images generation are limited. More extensive experiments or analysis should be conducted to justify their claims.