Hyper-SD: Trajectory Segmented Consistency Model for Efficient Image Synthesis

Thanks for your kind words about our method, experiments and writing. We would like to answer every question you have in the following.

W1:

We would like to clarify that our RLHF training is separated from TSCD and we apply RLHF through LoRA merging. So there would be no excessive memory usage in our distillation process. During RLHF, the convergence speed of RLHF LoRA is exceptionally rapid, requiring approximately 8000 iterations for around 96 A100 GPU hours. More importantly, the reward model is not involved in the evaluation process so no additional model needs to be decoded.

W2:

Thanks for your suggestion. We have read the provided paper carefully and will include it in our related work and discussion. The most essential difference is that we view feedback learning as a technique for trajectory-reformulating parallel to trajectory-preserving, and thus the training process is also separated. While in Reward Guided Latent Consistency Distillation (RG-LCM), the authors view it as some kind of guidance towards biasing the distillation process through a unified training objective.

Our advantage is that we do not need to modify the original model's scheduler (i.e. DDIM) and thus have greater generalizability, while in RG-LCM the model needs to directly predict x_0 and fit the LCM scheduler. Moreover, our approach supports one-step generation with better trajectory-preserving and trajectory-reformulating pieces.

Hi,

Please read other reviews and respond to the authors' rebuttal. You should edit your review accordingly.

Thanks.

Thanks for your kind words about our method and main results. We would like to answer every question you have in the following.

W0:

We fully understand your primary concerns about ablation studies. To further validate the effectiveness of our proposed method in multi-step or one-step generation, we would like to raise your attention to Table 1 in the global rebuttal pdf, where we conduct experiments on TCD+RLHF, and TCD+DMD.

1. From the results, performance across different training settings when replacing the TSCD with TCD has decreased significantly.
2. As for RLHF component, we can see that vanilla TCD demonstrates severe aesthetic dropping when no RLHF LoRA is applied. In contrast, TCD+RLHF performs better but still lags behind our full Hyper-SD thanks to the refined trajectory-preserving capability from TSCD.
3. For one-step generation, the conclusions are similar that only after our TSCD performs better trajectory-preserving, the score estimation will be more accurate and the power of DMD to match teacher distribution will be stronger.

W1:

Our GAN implementation is different from CTM that adopts StyleGAN-XL[1] discriminator for unconditional or classifier-guidance image synthesis. Since the prompting in text-to-image diffusion is quite important, we turn to a more advanced discriminator from SDXL-Lightning[2] that uses U-Net for backbone and student model for initialization. Therefore, the inputs to the discriminator would be noisy latent and timesteps together and it could better capture the subtle variation between the prediction and targets across timesteps.

To highlight the importance of introducing GAN loss, we conduct extra ablation studies in Table 3 of global rebuttal pdf. The results show that even if we also use adversarial loss for TCD, its performance is not as good as TSCD, which verifies our leadership in trajectory-preserving.

W2:

Yes it is. We apply RLHF through LoRA merging. It's exciting to see that simple merging of TSCD and RLHF LoRAs could bring significant benefits instead of distortion or collapse, which further demonstrates the importance of breaking down training objectives into trajectory-preserving and trajectory-reformulating pieces.

W3:

We have provided a more detailed comparison w/ or w/o RLHF (Table 1), w/ or w/o GAN (Table 3) in global rebuttal pdf. The results fully demonstrate the effectiveness of our proposed modules. For one, RLHF can still be effective even on TCD, and the results are even better on TSCD with better trajectory-preserving. For another, our GAN loss performs better against L2 only on TCD, and our TSCD benefit more from adversarial loss.

W4:

Not exactly. The training process of RLHF LoRA is with frozen TSCD LoRA applied. And we merge them into one after RLHF for convenience.

W5:

It is a continuously trained model based on the two-step TCSD LoRA since we set a progressive training strategy.

W6:

Yes it is. The result for DMD in Table 3 of manuscript is with RLHF. We're sorry to bring you this confusion. We have provided a clearer comparison for one-step generation w/o RLHF in Table 1 of global rebuttal pdf.

W7:

It's the same within a stage but different between stages. Like in the 4-segments stage, the Δt is 1000//16=62. While in the 2-segments stage, the Δt is 1000//8=125.

[1] Sauer, Axel, Katja Schwarz, and Andreas Geiger. "Stylegan-xl: Scaling stylegan to large diverse datasets." ACM SIGGRAPH 2022 conference proceedings. 2022. [2] Lin, Shanchuan, Anran Wang, and Xiao Yang. "Sdxl-lightning: Progressive adversarial diffusion distillation." arXiv preprint arXiv:2402.13929 (2024).

Hi,

Please read other reviews and respond to the authors' rebuttal. You should edit your review accordingly.

Thanks.

Thanks for your kind words about our writing, experiments and reward optimization approaches. We would like to answer every question you have in the following.

W1:

As for technical novelty, we would like to highlight and summarize as follows:

1. We're the first to split the acceleration objective of diffusion model into trajectory-preserving and trajectory-reformulating. While previous approaches all just focus on the specific aspect, we break the objective down into pieces and yield different technical solutions, bringing new insights and perspectives in this field.
2. As for trajectory-preserving, we're based on TCD[2] and hope to further detach the training objectives as neighboring timesteps and more distant timesteps to different training stages, which could reduce training difficulty and mitigate accumulation errors.
3. As for trajectory-reformulating, we propose to consider human feedback learning as an effective technique to bias output distribution toward human preference. It's exciting to see that simple merging of TSCD and RLHF LoRAs could bring significant benefits instead of distortion or collapse, which further demonstrates the importance of breaking down training objectives into pieces.

W2:

We agree that assessing per-prompt diversity is quite important and would like to raise your attention to Table 2 in the global rebuttal pdf, where we have picked 100 prompts and generated 4 images per prompt for each method shown. We extract the CLIP image embeddings per image and report their pair-wise average similarities for quantitative evaluation. The results show that our diversity without RLHF is quite similar compared to other acceleration approach. While biasing the output distribution toward human preference would inevitably compromise the diversity, our reward optimization doesn't sacrifice significantly and achieve an excellent trade-off compared to SDXL base model with a slight similarity increase (+0.0128). In Figure 2 of global rebuttal pdf, we also illustrate several examples for qualitative comparison.

W3:

We're sorry to bring you this confusion. The DMD result in Table 3 of manuscript is not vanilla but is with RLHF and without TSCD. To further validate the effectiveness of our proposed method in one-step generation, we have provided another ablation study in Table 1 of global rebuttal pdf. The results show that vanilla DMD doesn't yield competitive results compared to consistency distilled models in terms of CLIPScore, ImageReward and PickScore, while our TSCD performs to be better.

Thanks for your kind words about our writing, intuition and experiments. We would like to answer every question you have in the following.

W1:

Firstly, the randomness of t_end that broadened the boundary condition of the original consistency distillation in CM[1] (Theorem 1) has been proven to be valid and guaranteed to converge in TCD[2] (Theorem 4.1) or CTM[3] (Appendix B.1). And our time segments within each stage are fixed and identical, it just gets progressively smaller between stages.

Secondly, comparing with CTM, our multi-stage design choice is actually breaking down the training objective into several pieces. During the early training stage, the model focuses on learning the consistency of nearby neighboring timesteps. While in the later stages, the model can easily handle those with small intervals and extend its capability to the consistency of further timesteps or larger intervals based on that. In other words, there is overlap in the training objectives between stages, even when there are fewer segments in the later stages, we still have a probability that the target timestep with a small interval will be stepped on.

From the perspective of accumulation error, it comes from the difficulty of fitting the model from simple tasks (smaller intervals) to difficult tasks (larger intervals), and this is much smaller than the fitting error that comes from learning a mixed task (arbitrary intervals). Similar ideas could be evidenced by Progressive Distillation[4], where the promising distilled results indicate a smaller fitting error compared to the original non-distilled diffusion model. In our experiments, we also conduct extensive ablation and user studies to validate our superior performance against TCD across different settings and inference steps. For more ablation, we also conduct extra experiments in Table 1 of global rebuttal pdf which will be detailed later in W4 section.

W2:

Exactly, we would like to highlight that one of our main contributions is considering human feedback learning to be an effective technique to reformulate better trajectories, thus output distribution could be biased toward human preference rather than distorted badly. Since human preference is subjective and hard to be expressed through quantitative metrics, our assessment is also quite straightforward, that is, through extensive user studies as we show in the manuscript. To elaborate more on this, we would like to raise your attention to Figure 1 in the global rebuttal pdf, where we demonstrate the effectiveness of human feedback learning by different mixing ratios for TSCD and RLHF LoRAs.

W3:

As for technical contributions, we would like to highlight and summarize as follows:

1. We're the first to split the acceleration objective of diffusion model into trajectory-preserving and trajectory-reformulating. While previous approaches all just focus on the specific aspect, we break the objective down into pieces and yield different technical solutions, bringing new insights and perspectives in this field.
2. As for trajectory-preserving, we're based on TCD[2] and hope to further detach the training objectives as neighboring timesteps and more distant timesteps to different training stages, which could reduce training difficulty and mitigate accumulation errors.
3. As for trajectory-reformulating, we propose to consider human feedback learning as an effective technique to bias output distribution toward human preference. It's exciting to see that simple merging of TSCD and RLHF LoRAs could bring significant benefits instead of distortion or collapse, which further demonstrates the importance of breaking down training objectives into pieces.

W4:

We fully understand your concerns about ablation studies. To prove the effectiveness of TSCD against TCD, we would like to raise your attention to Table 1 in the global rebuttal pdf, where we conduct experiments on TCD+RLHF and TCD+DMD. The results show that our TSCD demonstrates superior performance consistently over different training settings, which prove the robustness of TSCD with reduced training difficulty and less accumulation errors.

[1] Song, Yang, et al. "Consistency Models." International Conference on Machine Learning. PMLR, 2023. [2] Zheng, Jianbin, et al. "Trajectory consistency distillation." arXiv preprint arXiv:2402.19159 (2024). [3] Kim, Dongjun, et al. "Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion." The Twelfth International Conference on Learning Representations. [4] Salimans, Tim, and Jonathan Ho. "Progressive Distillation for Fast Sampling of Diffusion Models." International Conference on Learning Representations.

Hi,

Please read other reviews and respond to the authors' rebuttal. You should edit your review accordingly.

Thanks.