Phased Consistency Models

Thank you for the in-depth review. We appreciate the chance to more comprehensively compare PCM with CTM.

Q1:

CIFAR-10

We additionally implement the core idea "phasing" technique into CIFAR (unconditional generation) and ImageNet (conditional generation). For CIFAR, we train our models with a batch-size 256 for 200, 000 iterations (800, 000 iterations used in official CM), without any additional GAN training. We listed the FID and inception score comparison in the following table.

In the above table, "CTM w/ GAN" and "CTM w/o GAN" are obtained from Table 3 of supplementary of CTM paper, tagged with "fair comparison" in the title. "CTM w/ GAN ++" is obtained from Table 1 of the paper, with more advanced techniques including classifier rejection sampling. We did not apply these techniques to our method. We can observe that the success of CTM greatly relies on GAN training. Yet, we show that our method, without any GAN training, can achieve comparable performance through multistep refinement.

ImageNet

For ImageNet, due to the limited time and GPU resources, we can only afford to run a tentative experiment. Specifically, we run a 4-phased PCM by adapting the official repo of CM with a normal L2 loss. For the last phase, we apply LPIPS loss. We first train the model for 100,000 iterations with a batch size of 64 and further train the model for 50, 000 iterations with a batch size of 1024. Note that the official CM trains the model with batch size 2048 for 800, 000 iterations. Therefore the current training budget for our method is 3.5% of the official CM. But importantly, we achieve even superior performance than the official CD. We believe this is clear evidence of the effectiveness of the "Phasing" ODE trajectory.

Q2 & Q5: The implementation details of CTM are the same as LCM and PCM, with the only difference on the loss design. We use the following table to show the difference.

where $s_m, m=0,1,\dots$ are pre-defined edge timesteps. Note that for both LCM and PCM, the target timesteps are fixed. While in CTM, the target timesteps are randomly chosen. Therefore it requires target timesteps embeddings. Besides, in the official CTM code base, the model can also optionally receive the target timestep embeddings. We also provide more theoretical proof in our submission, please see "III.2 Why CTM Needs Target Timestep Embeddings?". To put it straightforwardly, CTM does not require a target timestep embedding if and only if the ODE trajectory is first-order ( i.e., it is straight), which violates the truth since ODE trajectories are typically higher orders.

Q3: Training. No further difference. The LoRA has limited capacity, and therefore it is not enough for stable one-step generation.

Q4: Negative prompt. Please refer to Ln 289-296 ablation study of our submission and Figure 6. We show that our model has a higher CLIP score on positive prompts lower CLIP score on negative prompts compared with LCM.

Q6: Human metrics. Please refer to our rebuttal to Reviewer tuze.

Q7: Regarding recall, we have the following interesting findings. When computing the recall with coco, we find the introduced adversarial loss reduces the recall slightly. But when computing the recall with generation results of the teacher diffusion models, we find the adversarial loss increases the recall at a low-step regime, which we attributed to the quality improvements.

Besides, when comparing each baseline for fair, the principle design of CTM without GAN loss is even inferior to naive CM.

The core difference of CM, CTM and PCM are listed as the following table.

In our paper, we have provided thorough analysis and discussion with CTM. The random choice design of target timestep make the design of CTM :

• more complex (additionally target timestep embedding)
• need more capacity ($\mathcal O (N^2)$ learning objectives)
• and harder to train (inferior performance without GAN loss)

Please let us know which part do you think is not convincing enough, we are willing for further discussion!

Except for the validation on CIFAR and ImageNet. We hope to emphasize that we validate the design of PCM on much more baselines for all compared methods.

PCM (Ours): 1. CIFAR-10, 2. ImageNet, 3. SD-v15, 4. SDXL, 5.Text-to-Video

Instaflow: SD-v15

SDXL-Lightning: SDXL

CTM: CIFAR, ImageNet

CM: CIFAR, ImageNet

SDXL-Turbo: SDXL

LCM: SD v1-5, SDXL

AnimateLCM: Text-to-Video

We sincerely hope any further chance for discussion! Looking forward to your reply!

The main metrics listed in the Table 1 of original CTM paper is augmented with more advanced sampling techniques while all the compared methods do not use them. CTM applies a pretrained image classification model to score the generated samples, and only retain the sample with higher confidence and regenerate new samples if the confidence of the classification model is low. That means, It needs to generate much more samples than all the compared methods. It is noting that FID will generally become smaller when the number of samples become larger.

Thank you for your reply!

We understand your concern on the paper: Trajectory Consistency Distillation: Improved Latent Consistency Distillation by Semi-Linear Consistency Function with Trajectory Mapping (TCD) [1].

Parameterization

The parameterization of PCM was induced from the analysis form of the exact solution provided in the paper DPM-Solver [2]. All the characters used in our paper and first-order approximation proof follow DPM-Solver (including Eq. 2 - 3). We carefully checked the paper TCD and found it provided a similar discussion from DPM-Solver. It appeared on the arXiv earlier. We are willing to add citations of TCD in the relevant discussion. Both PCM and TCD cite DPM-Solver.

Additionally, we hope to clarify that's just a initially basic part for our further discussion. Our further discussion on the parameterization including the analysis of $\boldsymbol \epsilon_{\theta} (\mathbf x_t, t) = \frac{\int_{\lambda_t}^{\lambda_s}e^{-\lambda} \boldsymbol \epsilon_{\boldsymbol \phi}(\mathbf x_{t_{\lambda}(\lambda)}, t_{\lambda}(\lambda))\mathrm d \lambda}{\int_{\lambda_t}^{\lambda_s} e^{-\lambda} \mathrm d\lambda}$ and relevant proofs including, error bound, CFG, adversarial loss, have no intersections with TCD.

Training objective

The objective design of PCM is totally different from TCD.

For equation 29 in TCD, please note that the loss of TCD accepts an additional timestep $t_m$ for training, which is equivalent to the $s$ in the third row of following table. That is, they have to randomly choose the target timestep. While for PCM, we fixed the target timestep (i.e., $s:=s_{m}$) before training and set them to pre-defined. This is exactly the core difference between PCM and TCD, CTM [3]. Considering your concern, we are also willing to run additional comparison experiments with the official TCD and have a specific discussion on it in our paper.

We list the loss design comparison in the following table

From a more high-level perspective, as the sentence "enabling seamless transitions at any point along the trajectory governed by the PF ODE" stated in TCD, CTM and TCD all try to learn arbitrary pairs.

While our method PCM only learns transition within each sub-phase, which not only easy to train but also does not require any target timestep embeddings.

New Experiments

Our paper is originally for high-resolution text-conditioned image and video generation, where we try to tackle the limitations of LCM in text-to-image generation and text-to-video generation, as stated in our paper. We did not test on the CIFAR and ImageNet since they are for unconditional or class-conditional low-resolution generation. For your request, we have tried our best to run as many experiments as possible on the CIFAR and ImageNet. Following the official CM, the ImageNet experiments require training on 64 A100 GPUs for a week, which we can not afford in the limited rebuttal period.

[1] Trajectory consistency distillation

[2] Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps

[3] Consistency trajectory models: Learning probability flow ode trajectory of diffusion

I respectfully request that the other reviewers and the Area Chair take note of the potential plagiarism concerns regarding this paper.

I suspect that this paper may have plagiarized the work of [1] Zheng, Jianbin, et al. "Trajectory Consistency Distillation: Improved Latent Consistency Distillation by Semi-Linear Consistency Function with Trajectory Mapping." arXiv preprint arXiv:2402.19159 (2024). https://arxiv.org/abs/2402.19159v2

In accordance with Harvard’s Plagiarism Policy(https://usingsources.fas.harvard.edu/what-constitutes-plagiarism-0), this paper may be engaging in Inadequate Paraphrase and Uncited Paraphrasing.

The reasons for my concerns are as follows:

1. In the manuscript submitted by the author, from lines L134 to L161, the author asserts that their proposed parameterization is one of their core contributions. However, the equations presented, specifically Eq. 2 and 3, are identical to Eqs. 19 and 20 in [1]. Moreover, [1] was the first to apply Eq. 19 to the task of latent consistency distillation in text to image diffusion models. The description L134 to L161 also significantly overlaps with the parameterization discussion in Sec 4.2 of [1].
2. The objective function detailed in lines L62 to L178 of the manuscript are consistent with Eq. 29 in [1], and the content and explanation in Sec 4.2 “training” of [1] closely resemble the author’s manuscript.
3. The author frequently references [2] CTM, which conducted extensive experiments on CIFAR and ImageNet datasets during the experimental phase. However, the author’s manuscript does not mention these tasks in experiment at all. In their rebuttal, the author mentioned lacking sufficient computational resources to conduct experiments related to [2] CTM, which is particularly confusing to me. Furthermore, the tasks in the author’s manuscript are identical to those in [1].

And obviously, given the knowledge of [1], the innovation of this paper is significantly weakened both in task and experimental results.

I am not able to make a definitive judgment at this stage. However, based on the current evidence, I believe this paper may be suspected of plagiarism.

Here we paste the definition of Uncited Paraphrasing and Inadequate Paraphrase here:

1. Inadequate Paraphrase: When you paraphrase, your task is to distill the source's ideas in your own words. It's not enough to change a few words here and there and leave the rest; instead, you must completely restate the ideas in the passage in your own words. If your own language is too close to the original, then you are plagiarizing, even if you do provide a citation.

In order to make sure that you are using your own words, it's a good idea to put away the source material while you write your paraphrase of it. This way, you will force yourself to distill the point you think the author is making and articulate it in a new way. Once you have done this, you should look back at the original and make sure that you have represented the source’s ideas accurately and that you have not used the same words or sentence structure. If you do want to use some of the author's words for emphasis or clarity, you must put those words in quotation marks and provide a citation.

2. Uncited Paraphrasing: When you use your own language to describe someone else's idea, that idea still belongs to the author of the original material. Therefore, it's not enough to paraphrase the source material responsibly; you also need to cite the source, even if you have changed the wording significantly. As with quoting, when you paraphrase you are offering your reader a glimpse of someone else's work on your chosen topic, and you should also provide enough information for your reader to trace that work back to its original form. The rule of thumb here is simple: Whenever you use ideas that you did not think up yourself, you need to give credit to the source in which you found them, whether you quote directly from that material or provide a responsible paraphrase.

[1] Zheng, Jianbin, et al. "Trajectory consistency distillation: Improved Latent Consistency Distillation by Semi-Linear Consistency Function with Trajectory Mapping" arXiv preprint arXiv:2402.19159 (2024). https://arxiv.org/abs/2402.19159v2

[2] Kim, Dongjun, et al. "Consistency trajectory models: Learning probability flow ode trajectory of diffusion." arXiv preprint arXiv:2310.02279 (2023). https://arxiv.org/abs/2310.02279v3

Thank you for your response! Could you specify which parts are not resolved properly? We are willing for further reply and discussion. For the data, we already showed that our model without applying any adversarial loss consistently surpassing the CTM w/ GAN on CIFAR through multistep refinement. Additionally, we show that our model trained with only 3.5% training budget of the official CM achieved even superior performance.

Thank you for your recognition in our work and for providing constructive feedback on our paper.

Q1: Pre-defined number of sub-trajectories seems to fix the number of inference steps.

1. Note that inherently, each sub-trajectory of PCM can be perceived as a normal CM. Therefore, within each sub-trajectory, we can still support more step refinement just like normal CM. Specifically, our one-step model has the same flexibility as normal CM (CM is a subset of PCM). But for the best efficiency, we only apply one step for each sub-trajectory.
2. We show in the paper that except for one-step generation, all the other step models can be tuned through a small LoRA adapter with only 4% storage burden of original diffusion models. Therefore, users can switch the inference steps easily.

Q2: ReLU Function

This is a typo here, and it should be $\textrm{ReLU}(1 + D (\tilde{x}_s, c, s)) + \textrm{ReLU}(1 - D(\hat{x}_s, c, s))$, where $D$ is the discriminator, $c$ is the text conditions and $s$ is the timestep conditions. The $\textrm {ReLU}$ function is equivalent to normally applied hinge loss. We apologize if it caused any confusion.

Q3: Ablation study on the adversarial loss.

Please refer to Fig. 7 , Fig. 9, and Fig. 31 (In the supplementary) in our submission. In Fig. 7, we show that the adversarial loss greatly improves the FID at low-step regime and coverages the similar performance of PCM without applying the adversarial loss when the number of phases grows. In Fig.9, we show the visual ablation study on the discriminator choice and the comparison of normal GAN loss and our proposed adversarial loss. In Fig. 31, we show the side-by-side 2-step generation results with and without applying the adversarial loss. In Fig.31, we can observe a clear visual quality improvement. We also conduct a human preference metric comparison in the following table. The introduced adversarial loss consistency improves the human evaluation metrics consistently across different inference steps.

Q4: Hyper-parameters

For all training settings, we apply the loss as $\mathcal L_{\textrm{PCM}} + 0.1 \mathcal L_{\textrm {PCM}}^{adv}$. That is the $\lambda$ is set to $0.1$ in all training settings. For the $\lambda (t)$ in $\mathcal L_{\textrm{PCM}}$ (Equation 5), we use 1 for simplicity.

We are grateful for your decision to raise the score. Your recognition and feedback mean a lot to us.

Please feel free to reach out if there’s anything else we can assist with.

Best regards,

The Authors

Thank you for your very positive review, and constructive suggestions.

Q1: In terms of human evaluation metrics, we have re-evaluated the generation results of our method alongside all the comparative baselines mentioned in the paper. This re-evaluation was conducted over varying steps: 1, 2, 4, 8, and 16, using a range of well-established human evaluation metrics such as HPS-v2 [1], PickScore [2], and Laion Aesthetic Score [3]. Our method consistently achieved the best or at least comparable performance across all tested settings, notably outperforming the consistency model baseline LCM.

Experiments on SD v1-5

Experiments on SD XL

Q2: Regarding the training stability, we did not observe any training instability, even when using a very small batch size. For example, we trained the SDXL version of PCM with a batch size of 16 using fp16 mixed precision and achieved comparable performance to SDXL-Lightning one-step, which uses a batch size of 512 (32 times larger). We are willing to submit our training code for SD v1-5, SDXL, and SD3 if needed. All models showed clear improvements within 2,000 iterations and consistently improved with longer training times. We believe this stability is due to our specially designed adversarial loss rather than the standard GAN loss. However, as shown in our ablation study in Fig. 7, when replacing our proposed adversarial loss with normal GAN loss, we observe unstable training. We analyze the phenomenon from the perspective of distribution mismatch between pretraining datasets and distillation datasets in our paper (Theorem 5 & 6.).

[1] Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis.

[2] Pick-a-pic: An open dataset of user preferences for text-to-image generation. NeurIPS 2023.

[3] Laion-aesthetics.

2. Reviewer ukQK thinks we have the same training objective as TCD. Instead, our design is totally and inherently different from the training objective of TCD and CTM, which we have explained in our submission and the rebuttals.

Firstly, from a high-level perspective, we enforce the model to follow self-consistency within the pre-defined sub-phases (i.e. Our ODE is firstly split into multiphases along the time axis, and then we learn CM within the sub-trajectory). Therefore, our model enforces multiphase consistency. Instead, TCD just follows CTM and learns to follow trajectory consistency. That is, they enforce their model to learn how to move from arbitrary initial timesteps to arbitrary other timesteps.

We provide a not strict but very straightforward example to better illustrate the difference. Assume, the model has 1000 timesteps (0, 1, ..., 999) in total and we hope to train a 4-step model of PCM. Then we have

The learning objectives of PCM:

$f(x_{999}, 999) = x_{749}$

$f(x_{749}, 749) = x_{499}$

$f(x_{499}, 499) = x_{249}$

$f(x_{249}, 249) = x_0$

The learning objectives of TCD:

$f(x_{999}, 999, 749) = x_{749}$

$f(x_{999}, 999, 499) = x_{499}$

$f(x_{999}, 999, 249) = x_{249}$

$f(x_{999}, 999, 0) = x_0$

$f(x_{749}, 749, 499) = x_{499}$

$f(x_{749}, 749, 249) = x_{249}$

$f(x_{749}, 749, 0) = x_0$

$f(x_{499}, 499, 249) = x_{249}$

$f(x_{499}, 499, 0) = x_0$

$f(x_{249}, 249, 0) = x_0$

In the above example, PCM only maps one timestep to a predefined corresponding middle timestep (e.g., $999 \rightarrow 749$ ), which is pre-defined. That is, PCM is a one-to-one mapping. But for TCD, it has to accept an additional target timestep to indicate which timestep prediction it hopes to predict, and learn to predict all the other timesteps. That is, TCD is a one-to-N mapping. Therefore, PCM only has $O(N)$ training objectives but TCD has $\mathcal O(N + (N-1) + \cdots + 1) = \mathcal O(N^2)$. This is also the direct reason why it requires a target timestep embedding.

We list a more straightforward illustration of the differences of the loss functions,

For the accusation of Equation 29 in TCD, please note that the loss of TCD accepts an additional timestep $t_m$ for training, which is equivalent to the $s$ in the third row of the above table. That is, TCD needs to randomly choose the target timestep. While for PCM, we fixed the target timestep (i.e., $s:=s_{m}$) before training and set them to pre-defined. This is exactly the core difference between PCM and TCD, CTM.

Therefore, we argue that the training design of PCM and TCD, CTM are inherently different and the reviewer ukQK might misunderstand key aspects of our method.

3. In the following table, we summarize more differences between TCD and PCM.

In addition to the above accusation of plagiarism, we find the reviewer also made judgments on our method that we believed to be unfair.

4. The model of CTM requires a target timestep embedding. But the reviewer asked us the question Why does CTM need target timestep embeddings. It is clearly stated in the paper of CTM and also shown in the official code of CTM. Besides, we also set up an independent section to discuss why CTM needs target timestep embeddings (Ln 602 - 615, titled exactly "Why CTM Needs Target Timestep Embeddings?"). The comment might suggest that the reviewer may have a gap in understanding the related works and might not have fully grasped the details of our paper.

5. The motivation of our paper is to analyze the limitations of previous LCMs and generalize the design space to solve these limitations (as stated in the abstract, introduction, teaser figure). The LCM is only for high-resolution text-conditioned image and video generation. Therefore, we believe we have efficiently validate the effectiveness of PCM.

   For the reviewers' request to run CIFAR and ImageNet which are low-resolution unconditional generation and class-conditional generation tasks and have no direct relations to LCM, we have tried our best to run as many experiments as possible and provide clear evidence of the superiority of PCM compared to CM and CTM in the fair comparison setting. We have validated our methods on five baselines including CIFAR, ImageNet, SD v1-5, SDXL, and Text-to-Video. It is noting that all compared methods only validate their methods for at most two baselines, but the reviewer still claims that our method lacks validation.

We believe the contribution of our paper is underestimated. Please kindly take the time to look at our detailed rebuttal.

Thanks,

Paper 2513 Authors

[1] Consistency Models. ICML 2023 [2] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps. NeurIPS 2022

I disagree with the author's current response.

Given that the author was aware of [1], deliberately using the same reparameterization scheme without proper citation clearly constitutes plagiarism.

I am not questioning the innovation or contribution of this work. I am simply questioning why [1] was not mentioned at all in this manuscript submitted by authors, despite the striking similarity of both submitted manuscript and [1]. This seems to go beyond just a lack of citation.

Also I totally agree about The important conclusion of DPM-Solver (NeurIPS 2022) shall not be considered as contributions of TCD (arXiv 2024). But I also do believe TCD as the first work to apply a reparameterization method [2] for large-scale text-to-image distillation does exhibit a degree of innovation and contribution. (If using [2] for distillation doesn't qualify as contribution, then this reparameterization should have been placed in the related work section rather than the method section of the submitted manuscript.)

Furthermore, the author's explanation is intriguing. From Line 158 to 161, we do find some similar discussion on the boundary condition in the TCD paper. But please note that it is a natural conclusion from the exact solution formula proposed in DPM-Solver. I'm wondering if the author was aware of the TCD paper, does this not suggest the possibility of plagiarism, as I previously mentioned: Uncited Paraphrasing and Inadequate Paraphrase?

Lastly, I find it somewhat ironic that the author seems to suggest that if [1] cites [2], and their submitted work also cites [2], then their work can avoid citing [1], even though the author(s) know their work and [1] share substantial similarities in terms of tasks, comparisons, and experiments, while [2] is from an entirely different field compared to [1] and their work.

[1] Trajectory consistency distillation

[2] Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps