T2V-Turbo: Breaking the Quality Bottleneck of Video Consistency Model with Mixed Reward Feedback

We thank the reviewer for the constructive feedback!

Regarding the technical contribution, the proposed method seems like a straightforward extension of the video consistency model. The idea of adding direct supervision on the clean samples in consistency distillation has also been explored in previous works, e.g., [1] Adversarial Diffusion Distillation (ADD).

We would like to clarify that ADD is not based on consistency distillation. Instead, ADD's distillation loss corresponds to score distillation sampling [2]. Moreover, adversarial training is prone to instability [3] and might require massive hyperparameter tuning [4]. In contrast, our methods optimize towards differentiable RMs enjoying a stable training process. To the best of our knowledge, we are the first to add direct supervision on the clean samples when distilling from a video diffusion generator. Additionally, we are the first to learn video generators from the feedback of video-text models.

[2] Poole et al. "Dreamfusion: Text-to-3d using 2d diffusion." ICLR 2023.

[3] Yue et al. "On the algorithmic stability of adversarial training." NeurIPS 2021

[4] Pang et al. "Bag of tricks for adversarial training." ICLR 2021

It's highly recommended to include the corresponding mp4 files

We appreciate the suggestions! We promise to include all corresponding mp4 files in our revised manuscript and create a website to better present the videos generated by our methods.

I wonder which reward model contributes more to quality improvement from Fig.4, row 1, and row 3?

For Fig. 4, the image-text RM HPSv2.1 contributes more to our T2V-Turbo's (row 3) quality improvement over the baseline VCM (row 1). In the attached PDF, we empirically show that incorporating feedback from the $\mathcal{R}\_\text{vid}$ improves the video quality.

This cannot be inferred from Tab.2 and it's recommended to present some qualitative results in the ablation study

First, we would like to clarify that Table 2 provides strong quantitative evidence for the effectiveness of each RM. For both VC2 and MS variants, leveraging feedback from $\mathcal{R}\_\text{img}$ alone (VCM + $\mathcal{R}\_\text{img}$) matches the Quality Score of our T2V-Turbo, indicating the similarly high visual quality of the generated videos. However, VCM + $\mathcal{R}\_\text{img}$ still falls behind our T2V-Turbo in terms of Semantic Score. Further incorporating feedback from $\mathcal{R}\_\text{vid}$ can bridge this gap, leading to better text-video alignment.

The attached PDF corroborates the results in Table 2 with video examples. Specifically, it compares videos generated by VCM (VC2) + $\mathcal{R}\_\text{img}$ and our T2V-Turbo (VC2). The results show that while the visual quality of both methods is generally similar, additional feedback from $\mathcal{R}_\text{vid}$ significantly enhances the text-to-video alignment in T2V-Turbo.

The paper lacks in-depth discussions on the choice and influence of the image/video RMs. For instance, would it be better to CLIP/GAN-discriminator compared to the HPS?

We thank the reviewer for the insightful question! Firstly, we would like to remind the reviewer that we have experimented with additional image-text RMs, including PickScore and ImageReward, as detailed in Appendix E. We qualitatively show that incorporating reward feedback from any of these image-text RMs leads to quality improvement over the baseline VCM (VC2). It is worth noting that HPSv2.1 and PickScore are fine-tuned from CLIP with human preference data. Therefore, learning from CLIP might not lead to better performance compared to learning from image-text RMs.

We did not experiment with a GAN-discriminator, as training a GAN-discriminator can suffer from instability and may require substantial hyperparameter tuning. However, given the success of ADD, we acknowledge the potential benefit of learning from a GAN-discriminator. Consequently, we consider this a promising direction for future work.

Lastly, we promise to include more in-depth discussions in our revised manuscript on the choice and influence of image/video RMs.

Why do MS and VC2 perform differently with InternVidS2?

Firstly, we would like to emphasize that when training T2V-Turbo (MS) with InternVid2 S2, T2V-Turbo (MS) still shows improvement over both VCM (MS) and VCM (MS) + $\mathcal{R}\_\text{img}$. However, we acknowledge that ViCLIP is more effective in enhancing T2V-Turbo (MS) compared to InternVid2 S2, as demonstrated in Tables 2 and 3. In contrast, ViCLIP and InternVid2 S2 work similarly well for T2V-Turbo (VC2).

We conjecture that this difference might be due to the quality disparity between the teacher models ModelScopeT2V and VideoCrafter2. For example, ModelScopeT2V suffers from low resolution, and the generated videos contain watermarks, which may limit VCM (MS)'s ability to learn from InternVid2 S2. Nonetheless, further tuning of the $\beta_\text{vid}$ parameter might help improve the performance of T2V-Turbo (MS) with InternVid2 S2.

Why use different video RMs for ModelScope and VC2?

Our main purpose is to examine our methods with a diverse set of $\mathcal{R}\_\text{vid}$. As we did not have access to a video-text RM trained to reflect human preference on video, we instead chose to experiment with video foundation models, including ViCLIP and InternVid2 S2. Note that we perform an ablation study and report results with both ViCLIP and InternVid2 S2 in Table 3.

What leads to different hyperparameter choices in L164-165?

Compared to VideoCrafter2, ModelScopeT2V has not been trained on high-quality video or image datasets. Additionally, ModelScopeT2V suffers from low resolution, and the generated videos contain watermarks. Thus, ModelScope suffers from lower video quality. As a result, T2V-Turbo (MS) requires larger weighting parameters $\beta_\text{img}$ and $\beta_\text{vid}$ to get stronger external supervision from the RMs to improve its generation quality.

Dear Reviewer Kq5y,

Your feedback has been invaluable in helping us clarify, improve, and refine our work. We have diligently addressed your comments in our response, made every effort to dispel misunderstandings, and provided various video examples for qualitative comparisons to demonstrate the effectiveness of our method.

We kindly ask you to revisit our paper in light of our response and consider whether the changes and clarifications we have provided might warrant a reconsideration of your rating.

Best regards,

The Authors

Dear Reviewer Kq5y,

We would like to kindly invite you to review the results we have submitted during the rebuttal period. We have thoroughly addressed your concerns and included videos corresponding to the static frames in our paper. Thanks to the support of our Area Chair, we are able to share an anonymous website containing all these videos. We hope that you will reconsider your ratings in light of these new results : )

Best regards,

The Authors

I appreciate the authors' detailed rebuttal and the additional results. They address some of my concerns.

However, my major concern about the technical contribution still exists. Especially when comparing Fig. 2 in this paper and Fig. 2 in VideoLCM, the major technical contribution is adding a reward loss on the clean samples, which seems straightforward without too many challenges. Therefore, I tend to maintain my original score.

We are grateful for the reviewer's positive feedback on our work. Please find our detailed response below.

I would really like the authors to point out if I have missed out details in their contributions or any other novel aspects of their work.

We appreciate the opportunity to clarify our contributions. We would like to emphasize the importance of our mixture of RMs design, which enables us to achieve significant empirical results even without access to a $\mathcal{R}_\text{vid}$ trained to reflect human preference.

Specifically, our method leverages feedback from both an image-text RM $\mathcal{R}\_\text{img}$ and a video foundation model $\mathcal{R}\_\text{vid}$, such as ViCLIP and InternVid2 S2. This combination allows our T2V-Turbo to break the quality bottlenecks in the video consistency model, resulting in both fast and high-quality video generation. Notably, the 4-step generations from our T2V-Turbo achieve the SOTA performance on VBench, surpassing proprietary models, including Gen-2 and Pika.

Additionally, our ablation study in Table 2 empirically provides valuable scientific insights into the effectiveness of different RMs. While leveraging feedback from $\mathcal{R}\_\text{img}$ alone (VCM + $\mathcal{R}\_\text{img}$) is sufficient to match the visual quality (Quality Score) of our T2V-Turbo, the additional feedback from $\mathcal{R}\_\text{vid}$ further enhances text-video alignment, resulting in higher Semantic Score. Qualitative evidence is provided in the attached PDF. To the best of our knowledge, we are the first to improve video generation using the feedback from video-text RMs. We believe that future advancements in video-text RMs will further enhance the performance of our methods.

Our joint training pipeline is computationally efficient. For example, InstructVideo requires over 40 hours of training to align a pretrained T2V model. In contrast, our method reduces training time to less than 10 hours, achieving a fourfold increase in efficiency in wall-clock training time.

Lastly, our method is broadly applicable to a diverse set of prompts. We have empirically shown that our approach outperforms existing methods on comprehensive benchmarks, including VBench and EvalCrafter. In contrast, previous methods, such as InstructVideo, conduct experiments on a limited set of user prompts, most of which are related to animals.

I don't find too many qualitative examples, and I couldn't find more videos in the supplementary (apart from the code). It would definitely be nicer if there were more examples that were provided.

Thank you for your suggestions! We have now included more videos in the attached PDF. We promise to include more videos in our revised manuscript and create a website to better present the videos generated by our methods.

We thank the reviewer for the constructive comments!

The paper seems to simple combine the video consistency model (VCM) and the differentiable reward models

We emphasize that our method is NOT a simple combination of VCM and differentiable RM. Previous works focus on aligning a pretrained DM to the preference given by RMs. In contrast to previous methods, e.g., InstructVideo, aligning a pretrained T2V model by backpropagating gradients through the memory-intensive iterative sampling process, our method cleverly leverages the single-step generation that naturally arises from consistency distillation. By optimizing the rewards of the single-step generation, our method avoids the highly memory-intensive issues associated with passing gradients through an iterative sampling process.

Additionally, our joint optimization technique is notably computationally efficient. Previous approaches, such as InstructVideo, require over 40 hours of training to align a pretrained T2V model. In contrast, our method reduces training time to less than 10 hours, achieving a fourfold increase in efficiency in wall-clock training time.

Empirically, we have shown that our approach outperforms existing methods on comprehensive benchmarks, including VBench and EvalCrafter. In contrast, previous reward learning methods for video generation, such as InstructVideo, conducted experiments on a limited set of user prompts, most of which were related to animals.

the authors utilize a mixture of reward models, where a video-text reward model is additionally added to encourage the diffusion model to better model the temporal dynamics. However, to me, this contribution is quite incremental.

We would like to clarify the contribution of our mixture of RMs design, which enables us to break the quality bottleneck in VCM without access to a $\mathcal{R}\_\text{vid}$ trained to mirror human preference. By learning from an image-text RM $\mathcal{R}\_\text{img}$ and a video foundation model $\mathcal{R}\_\text{vid}$, such as ViCLIP and InternVid2 S2, we achieve both fast and high-quality video generation. Notably, the 4-step generations from our T2V-Turbo achieve the SOTA performance on VBench, surpassing proprietary models, including Gen-2 and Pika.

Additionally, our ablation study in Table 2 further provides valuable scientific insights into the effectiveness of different RMs. While leveraging feedback from $\mathcal{R}\_\text{img}$ alone (VCM + $\mathcal{R}\_\text{img}$) is sufficient to match the visual quality (Quality Score) of our T2V-Turbo, the additional feedback from $\mathcal{R}\_\text{vid}$ further enhances text-video alignment, resulting in higher Semantic Score. Qualitative evidence is provided in the attached PDF. To the best of our knowledge, we are the first to improve video generation using feedback from video-text RMs. We believe that future advancements in video-text RMs will further enhance the performance of our methods.

I think it is better to show what the text prompt is in Fig. 1 to help reader better understand the difference between different models.

We thank the reviewer for the suggestion. We will include the corresponding prompts in our revised manuscript. The prompt for the top two and bottom two rows are

1. With the style of low-poly game art, A majestic, white horse gallops gracefully across a moonlit beach.
2. Kung Fu Panda posing in cyberpunk, neonpunk style

For the quantitative comparison, it can be seen that the proposed method actually doesn't perform the best for most of the evaluation metrics. I am not sure if comparing total score (i.e., a weighted sum of Quality Score and Semantic Score) only is enough to show the superiority of proposed method.

First, we emphasize that our automatic evaluation results in Table 1 are corroborated by the human evaluations in Fig 3, where the 4-step generation from both T2V-Turbo (VC2) and T2V-Turbo (MS) are preferred over the 50-step generations from their teacher VideoCrafter2 and ModelScopeT2V.

Second, we highlight that the Total Score, Quality Score, and Semantic Score are sufficient proxies for human preference. Consider the comparison between VideoCrafter2 and our T2V-Turbo (VC2). Figure 3 indicates that human annotators favor the 4-step generation from our T2V-Turbo (VC2) in terms of Visual Quality, Text-Video Alignment, and General Preference. These preferences align with the higher Quality Score, Semantic Score, and Total Score of our T2V-Turbo (VC2), thereby validating these metrics' effectiveness in reflecting our method's superiority.

Lastly, we address why our T2V-Turbo does not perform the best across all evaluation metrics yet still achieves the highest overall scores. We extracted the performance of VideoCrafter2 and our T2V-Turbo (VC2) from Table 1 of our paper. Although VideoCrafter2 scores slightly higher on 5 out of 7 dimensions constituting the Quality Score and 4 out of 9 dimensions constituting the Semantic Score, the differences are not significant. In contrast, our T2V-Turbo (VC2) significantly outperforms VideoCrafter2 in metrics such as Dynamic Degree, Image Quality, and Multiple Objects, contributing to its overall superiority.

Dear Reviewer MMwk,

Thanks for your response.

Firstly, we would like to emphasize that the technical simplicity of our method should not be seen as a drawback. On the contrary, being technically simple yet highly effective is a distinct advantage of our approach. In this paper, we address the core challenges of video generation: 1) improving generation quality, 2) reducing inference time, and 3) alleviating the intensive memory and computational cost when aligning a video generator.

Current state-of-the-art proprietary video generation systems, such as Gen-2 and Pika, require several minutes to produce a short video clip. In contrast, our T2V-Turbo can generate high-quality videos within 5 seconds, making it significantly more suitable for real-time applications. T2V-Turbo achieves both fast and high-quality video generation. As validated on VBench, its 4-step generation process surpasses the performance of proprietary systems like Gen-2 and Pika, which rely on extensive resources. To the best of our knowledge, we are the first to simultaneously address these two contradictory aspects—speed and quality—within the same video-generation framework.

Secondly, we would like to reiterate the significance of our mixture of RMs design, which allows us to break the quality bottleneck in VCM without access to a $\mathcal{R}\_\text{vid}$ trained to mirror human preference. Our ablation study in Table 2 empirically provides valuable scientific insights into the effectiveness of different RMs.

While leveraging feedback from $\mathcal{R}\_\text{img}$ alone (VCM + $\mathcal{R}\_\text{img}$) is sufficient to match the visual quality (Quality Score) of our T2V-Turbo, the additional feedback from $\mathcal{R}\_\text{vid}$ further enhances text-video alignment, resulting in higher Semantic Score. We provide qualitative evidence in the attached PDF. To the best of our knowledge, we are the first to improve video generation with feedback from video-text RMs. We believe that future advancements in video-text RMs will further improve the performance of our methods.

Thank you again for your time and effort in providing valuable feedback on our work. We hope our clarifications will lead to a more favorable evaluation of our contribution.

Best,

The Authors

We thank the reviewer for the positive feedback on our work! Please find our detailed response below.

Q1: The proposed approach is simply a mixture of previous works, i.e., consistency distillation and reward feedback learning

We emphasize that our method is NOT simply combining the video consistency model and reward feedback learning. Previous methods, such as InstructVideo [1], require backpropagating gradients through an iterative sampling process, which can lead to substantial memory costs. In contrast, our method cleverly leverages the single-step generation arising from consistency distillation. By optimizing the rewards of the single-step generation, our method avoids the highly memory-intensive issues associated with passing gradients through an iterative sampling process.

Additionally, we would like to emphasize the contribution of our mixture of RMs design, which enables us to break the quality bottleneck in VCM without access to a $\mathcal{R}\_\text{vid}$ trained to mirror human preference. By learning from an image-text RM $\mathcal{R}\_\text{img}$ and a video foundation model $\mathcal{R}\_\text{vid}$, such as ViCLIP and InternVid2 S2, we achieve both fast and high-quality video generation. Notably, the 4-step generations from our T2V-Turbo achieve the SOTA performance on VBench, surpassing proprietary models, including Gen-2 and Pika. To the best of our knowledge, we are the first to improve video generation using feedback from video-text RMs. We believe that future advancements in video-text RMs will further enhance the performance of our methods.

[1] Yuan et al., InstructVideo: Instructing Video Diffusion Models with Human Feedback. CVPR 2024

Q2: However, it is a more right approach to first design a good reward model for video generation than developing a video generation model to align with such reward models.

We fully recognize the importance of designing a good video-text RM $\mathcal{R}\_\text{vid}$. However, creating an effective $\mathcal{R}\_\text{vid}$ might require long-term efforts. A video is more complex than an image due to its additional temporal dimension. We envision that it requires multiple iterations to derive an effective $\mathcal{R}\_\text{vid}$. Specifically, one iteration involves 1) Training $\mathcal{R}\_\text{vid}$ by collecting new preference data from a video generator and 2) Training the video generator to align with $\mathcal{R}\_\text{vid}$. Our method provides an efficient way to accomplish the second step.

On the other hand, even without a $\mathcal{R}_\text{vid}$ trained to reflect human preferences on video-text pairs, we highlight that our method can still enhance the generation quality of a T2V model by aligning it with video-text foundation models, such as ViCLIP and InternVid2 S2.

Q3: Could the author provide more details about how the reward models should be designed in order to obtain better video generation or evaluating video generation models?

We thank the reviewer for the question. In addition to our response to Q2, we conjecture a $\mathcal{R}_\text{vid}$ effective for training video generators might need to be finetuned from existing video foundation models, such as ViCLIP and InternVid2 S2. This approach mirrors the success seen with image-text RMs, including HPSv2, ImageReward, PickScore, and AestheticScore, which have proven effective for training image generators. These models are finetuned from image-text foundation models, such as CLIP and BLIP, using human preference data.

Moreover, learning a multi-dimensional reward to reflect fine-grained human preferences might also be helpful. For example, each dimension of the reward vector could be trained to reflect visual quality, transition dynamics, text-to-video alignment, etc.

Q4: Does the joint training of feedback learning and consistency distillation is better than sequential fine-tuning?

First, sequential fine-tuning is computationally more expensive than joint training. According to the InstructVideo paper, their reward feedback learning phase costs more than 40 hours. If we add the distillation time cost further, the total time cost for sequential fine-tuning can easily exceed 50 hours. Conversely, our joint training only requires less than 10 hours, representing a fivefold reduction in terms of training wall-clock time.

Second, as mentioned in Sec. 1 of our paper, finetuning a diffusion T2V model towards a differentiable RM requires backpropagating gradients through the diffusion model's iterative sampling process. Therefore, calculating the full reward gradient is prohibitively expensive, resulting in substantial memory costs. Conversely, our method leverages the single-step generation that arises naturally from computing the CD loss, effectively bypassing the memory constraints.

Q5: It seems like only consistency distillation for VC2 degrades the performance (Table 1)

In terms of the performance degradation of distilling VC2, we conjecture it can be alleviated by performing full model training, or further optimizing the hyperparameters related to learning the LoRA weights.

We thank the reviewer for the positive feedback on our work! Please find our detailed feedback below.

The method mainly combines the consistency distillation in the Video Consistency Model (VCM) with multiple reward models. Although it has achieved good results in solving the existing problems of the T2V model, this combination is relatively conventional and may be somewhat lacking in originality.

We would like to highlight that our method cleverly leverages the single-step generation arising from consistency distillation. By optimizing the rewards of the single-step generation, our method avoids the highly memory-intensive issues associated with passing gradients through an iterative sampling process. In contrast, previous methods, such as InstructVideo [1], require backpropagating gradients through diffusion model's iterative sampling process, resulting in substantial memory costs.

Additionally, we emphasize the importance of our mixture of RMs design, which enables us to break the quality bottleneck in VCM without access to a $\mathcal{R}\_\text{vid}$ trained to mirror human preference. By learning from an image-text RM $\mathcal{R}\_\text{img}$ and a video foundation model $\mathcal{R}\_\text{vid}$, such as ViCLIP and InternVid2 S2, we achieve both fast and high-quality video generation. Notably, the 4-step generations from our T2V-Turbo achieve the SOTA performance on VBench, surpassing proprietary models, including Gen-2 and Pika. To the best of our knowledge, we are the first to improve video generation using feedback from video-text RMs. We believe that future advancements in video-text RMs will further enhance the performance of our methods.

The citation format of this paper should adhere to the NeurIPS standard, the authors misuse \citet throughout the paper, especially in Sections 2 and 5, making it hard to read smoothly.

We thank the reviewer for pointing out the issues! We will make sure to fix the citation issues in our revised manuscript!

why choose ViCLIP as the video RM for MS rather than using a stronger InternVid2 S2?

Our main purpose is to examine our methods with a diverse set of $\mathcal{R}\_\text{vid}$. As we did not have access to a $\mathcal{R}\_\text{vid}$ trained to reflect human preference on video, we instead chose to experiment with video foundation models, including ViCLIP and InternVid2 S2. Notably, we conduct a comprehensive ablation study in Table 3, presenting results for our T2V-Turbo (VC2) and T2V-Turbo (MS) when setting $\mathcal{R}\_\text{vid}$ to both ViCLIP and InternVid2 S2. This approach allows for a thorough assessment of our methods across different $\mathcal{R}\_\text{vid}$.