T2V-Turbo-v2: Enhancing Video Model Post-Training through Data, Reward, and Conditional Guidance Design

We thank the reviewer for the constructive feedback. Please find our detailed response below.

The integration of MotionClone into T2V-Turbo is an interesting direction. However, while the contribution highlights the potential for diverse forms of energy functions, this study primarily utilizes the motion representation from the MotionClone work without substantial modifications to the energy function's format.

We would like to clarify some misunderstandings in the reviewer’s assessment of our work:

1. Objective of the Study: The primary goal of our work is not to design a new energy function for motion guidance. Instead, we aim to empirically demonstrate that augmenting the teacher ODE solver with the energy function's gradients of a conditional guidance strategy can distill a more capable student video generator while significantly reducing inference costs. In this context, we showcase the potential of our approach by integrating MotionClone’s energy function into the teacher ODE solver. In other words, our method is not confined to MotionClone’s motion guidance.

2. Challenges in Scaling MotionClone for Training: Adapting MotionClone’s motion guidance techniques for training is significantly non-trivial for several reasons:

   • Dependency on Reference Videos: MotionClone relies on access to reference videos with high-quality motion. However, identifying suitable reference videos for general text prompts is challenging, which limits its effectiveness and applicability for generic video generation tasks.
   • High Computational Cost: Computing the gradient of the energy function during inference incurs substantial memory and latency overhead. For instance, generating a single video with MotionClone can take approximately seven minutes and 30 GB GPU memory.

   To address these challenges, we leverage the critical insight that each training video inherently serves as an ideal reference video for its corresponding training prompt. Additionally, we design a separate preprocessing phase to precompute the motion guidance before the actual training phase. As a result, this preprocessing phase eliminates the need for computationally expensive gradient calculations during training.

The other enhancements are relatively minor, such as the reward model used, which only adds a CLIP compared to T2V-Turbo.

We would like to clarify the contributions of our work and address the concerns regarding the significance of our enhancements.

1. Coupled Effects of Training Data and Reward Models:

   The choices of training data and reward models are arguably the most critical components in the post-training phase of a generative AI model. In our work, we conduct a rigorous and thorough empirical investigation into how these factors impact the performance of T2V models. A key finding of our study is that their effects are not orthogonal—the interaction between training datasets and RMs plays a pivotal role in shaping the final performance. Specifically, our Section 4.2, 4.3, and Appendix C.2 empirically demonstrate that curating training datasets for different learning objectives is crucial for achieving optimal results. To the best of our knowledge, this is the first work to systematically study how the selection of training data and RMs affects the post-training performance of a T2V model. Therefore, we firmly believe our findings provide invaluable insights for advancing post-training research in video generation models.

2. Different Conclusion Regarding the Use of Reward Models:

   Our findings regarding the effects of different RMs diverge significantly from the conclusions of the T2V-Turbo paper. T2V-Turbo suggests that feedback from a single image-text RM (HPSv2.1) is sufficient to achieve substantial performance gains. In contrast, our work reveals that relying solely on HPSv2.1 results in only minimal enhancements to video quality. Instead, we show that incorporating feedback from a more diverse set of RMs is essential to achieve meaningful performance improvements.

   The reasons behind these differing conclusions are centered on the datasets used for different learning objectives, which are summarized below:

   • Our T2V-Turbo-v2 leverages a dataset that combines videos from VG and WV. Specifically, we minimize the consistency distillation loss on the entire dataset but optimize reward objectives only on the short-captioned WV data.
   • T2V-Turbo was trained exclusively on WV data with short video captions.

   These experimental differences underscore a critical insight: the impact of reward feedback is highly dependent on the dataset composition and design. Our results highlight the importance of curating datasets and carefully selecting RM sets to achieve optimal performance in video generation tasks.

Dear Reviewer XVsE,

We greatly appreciate your insightful feedback, which has significantly contributed to the clarity and enhancement of our work. We have carefully addressed your comments in our response, clarified potential misunderstandings, and included the new human evaluation results you requested.

We kindly invite you to revisit our paper in light of these updates and clarifications. We would greatly appreciate it if you could consider whether these changes warrant a reevaluation of your rating.

Best regards,

The Authors

This method is overly engineering-focused and lacks novelty, as the motion guidance and consistency distillation techniques involved are already established, making it appear less innovative.

We respectfully disagree with the reviewer's argument on our novelty. In terms of guidance strategy, the primary goal of our work is not to design a new energy function for motion guidance. Instead, we aim to empirically demonstrate that augmenting the teacher ODE solver with the energy function's gradients of a conditional guidance strategy can distill a more capable student video generator while significantly reducing inference costs. In this context, we showcase the potential of our approach by integrating MotionClone’s energy function into the teacher ODE solver.

Additionally, adapting MotionClone’s motion guidance techniques for training is significantly non-trivial for several reasons:

• Dependency on Reference Videos: MotionClone relies on access to reference videos with high-quality motion. However, identifying suitable reference videos for general text prompts is challenging, which limits its effectiveness and applicability for generic video generation tasks.
• High Computational Cost: Computing the gradient of the energy function during inference incurs substantial memory and latency overhead. For instance, generating a single video with MotionClone can take approximately seven minutes and 30 GB GPU memory.

To address these challenges, we leverage the critical insight that each training video inherently serves as an ideal reference video for its corresponding training prompt. Additionally, we design a separate preprocessing phase to precompute the motion guidance before the actual training phase. As a result, this preprocessing phase eliminates the need for computationally expensive gradient calculations during training.

As demonstrated in Tables 1 and 4, augmenting the teacher ODE solver with motion guidance leads to significant performance gains and improved motion quality across different evaluation metrics.

Additionally, while it conducts extensive ablation experiments on motion guidance and reward models, this does not constitute a significant contribution of the paper. I am unclear about the paper's contributions; is it providing more interesting insights? It would be helpful if the authors could briefly summarize this in their response.

We firmly believe that our findings provide invaluable insights for advancing post-training research in video generation models. The choices of training data and reward models are arguably the most critical components in the post-training phase of a generative AI model. In our work, we conduct a rigorous and thorough empirical investigation into how these factors impact the performance of T2V models. A key finding of our study is that their effects are not orthogonal—the interaction between training datasets and RMs plays a pivotal role in shaping the final performance. Specifically, our Section 4.2, 4.3, and Appendix C.2 empirically demonstrate that curating training datasets for different learning objectives is crucial for achieving optimal results. To the best of our knowledge, this is the first work to systematically study how the selection of training data and RMs affects the post-training performance of a T2V model.

Regarding the contribution summary of the paper (L117), it seems to emphasize the advantages of existing work and the potential of extracting motion priors. And, I do not see any strong insights that stand out; motionclone has already demonstrated this fairly clearly. If the authors mean that motion priors are particularly useful during T2V training, they should provide more experiments. For example, training SVD and VideoCrafter2 shows that the insights presented in T2V-Turbo are quite limited.

In our paper, we perform rigorous and thorough ablation studies to investigate the impacts of two critical components—training data and reward models—and demonstrate how different design choices affect model performance during the post-training phase. Our findings reveal the surprising coupled effects of training data and RMs, offering strong insights that can guide future research on post-training strategies for video generation models.

Additionally, MotionClone only demonstrated its effectiveness during inference. Our work scales its application to the training phase by addressing two significant challenges: (1) its reliance on reference videos, and (2) the high computational cost of gradient calculations. Notably, our method is not confined to MotionClone’s motion guidance. It supports the integration of other energy functions into the teacher ODE solver, highlighting the vast design space for conditional guidance strategies. In this work, we use MotionClone’s energy function as a concrete example to showcase the potential of this approach, but its utility extends far beyond this specific application.

Dear Reviewer dkgW,

We greatly appreciate your insightful feedback, which has significantly contributed to the clarity and enhancement of our work. We have carefully addressed your comments in our response, clarified potential misunderstandings, and explained the technical contributions of our paper. Additionally, we also included new experimental results to corroborate our findings.

We kindly invite you to revisit our paper in light of these updates and clarifications. We would greatly appreciate it if you could consider whether these changes warrant a reevaluation of your rating.

Best regards,

The Authors

We thank the reviewer for their positive feedback on our work. Please find our detailed responses below.

The model appears optimized for single-caption, short-context prompts, its ability to generate longer or more complex video context may be limited.

We would like to clarify the reviewer's misunderstanding. The limitation in generating longer or more complex video contexts stems from the text encoder used by the teacher diffusion model, not from our method. In this paper, we adopt VideoCrafter2 as our teacher model, which employs the CLIP text encoder with a context length of 77. This constraint is inherent to VideoCrafter2.

However, this limitation can be easily overcome by using a more advanced teacher model, such as CogVideoX-5B or Mochi-1, both of which utilize the T5-XXL text encoder, supporting longer context lengths. Therefore, the limitation noted by the reviewer reflects a restriction of the teacher model (VideoCrafter2) rather than our proposed approach.

When generating the dataset for the motion guidance, it may require considerable computation resources

We acknowledge that generating the dataset for motion guidance requires additional computational resources. Our preprocessing takes about 400 GPU hours, which is indeed manageable as it can be completed in approximately 2 days on a server with 8 NVIDIA A100 GPU (each GPU is of 40 GB memory). Importantly, this additional preprocessing effort yields significant benefits in terms of motion quality and inference acceleration.

In comparison, the original MotionClone approach requires approximately 6 minutes to generate a single video, with 3 minutes spent on DDIM inversion from a reference video. Moreover, the peak GPU memory consumption can reach up to 35 GB due to the gradient calculations involved. In contrast, our model eliminates the need for identifying appropriate reference videos, conducting DDIM inversion, or calculating gradients during inference without adding any inference overhead compared to inference from a VCM. For example, our T2V-Turbo-v2 only takes 5 seconds to generate a video in 16 steps using BFloat16.

The efficiency and performance gains of our approach are clearly reflected in its results. Our model achieves the #1 ranking on VBench, surpassing numerous proprietary video generation systems, including Gen-3, Kling, and MiniMax. This demonstrates that the preprocessing trade-off is well-justified, offering both superior motion quality and significant reductions in inference time.

What are the evaluation datasets when comparing the T2V-Turbo-V2 with the SOTA methods (Table 1)?

The results in Table 1 are obtained by comparing our T2V-Turbo-V2 with the SOTA methods using the VBench datasets, which contain 946 unique prompts. We carefully follow VBench’s evaluation protocols by generating 5 videos for each prompt, as discussed in Lines 301 - 310.

I want to know the metrics of using OV + VG and OV + VG + WV (in Table 2 setting)

We thank the reviewer for the question. Below, we include the results for OV + VG and OV + VG + WV in Table 2 setting.

As shown in the table, VCM achieves a high Quality Score on the OV + VG dataset, similar to training on pure OV data, but adding the lower-quality WV data slightly decreases this score. Conversely, incorporating reward feedback in T2V-Turbo-v2 significantly improves the Semantic Score for the OV + VG + WV dataset, while the gains for OV + VG remain comparatively moderate. These findings align with our discussion in the main text: RMs with short context lengths operate optimally on datasets with shorter captions, highlighting the importance of aligning dataset characteristics with RM capabilities. Furthermore, the results justify our decision to exclude OV data when training the main T2V-Turbo-v2 models, as incorporating OV into VG + WV datasets negatively impacts model performance.

why the author does not use the OV dataset?

We make our decision based on the performance when training on OV + WV. Specifically, VCM's Total Score when training on OV + WV (73.30) is worse than when training on OV (78.52) or WV (76.15). Similarly, our method's Total Score when training on OV + WV (81.00) barely improves the performance of training OV (80.97) and is worse than when training on WV (81.34) data. This phenomenon suggests a big domain gap between the OV and WV datasets. Thus, we do not use the OV dataset to train the main version of our method.

Dear Reviewer y5Jq,

We greatly appreciate your insightful feedback, which has significantly contributed to the clarity and enhancement of our work. We have carefully addressed your comments in our response, clarified potential misunderstandings, and included the results for OV + VG and OV + VG + WV in the same settings as Table 2 to explain why we do not use the OV datasets to train our main models.

We kindly invite you to revisit our paper in light of these updates and clarifications. We hope our responses address your concerns thoroughly and provide additional support for advocating the acceptance of our paper, potentially leading to an improved rating.

Best regards,

The Authors

The early work Energy-guided stochastic differential equations[1] first presents a framework that utilizes an energy function to guide the generation process for a diffusion model. Please cite this paper.

Can you please include the citation for the paper you mentioned? We are happy to cite the paper in our manuscript.

In Figure 2, does the DDIM inversion require k forward passes for each training step? If so, does this introduce excessive computational cost?

As our paper already mentions, the inverse DDIM used to obtain motion prior can be done in a separate preprocessing phase and thus will NOT introduce any additional computational cost for training.

Please provide .mp4 files for visual comparisons

Thank you for the suggestions. We have included the original video files in the supplemental material. On the other hand, you can click to play the videos in our manuscript if you open it using Adobe Acrobat Reader.

Dear Reviewer N6o1,

We greatly appreciate your time and feedback on our work. We have carefully addressed your comments and clarified potential misunderstandings. Additionally, we also included new experimental results to corroborate our findings.

We kindly invite you to revisit our paper in light of these updates and clarifications. We would greatly appreciate it if you could consider whether our responses warrant a reevaluation of your rating.

Moreover, please take time to include the citation for the paper you mentioned. We are happy to cite the paper in our manuscript.

Best regards,

The Authors

Dear Reviewer N6o1,

Thank you again for serving as a reviewer! Your feedback has been valuable in helping us clarify and improve our work. We have worked diligently to address your comments and included experimental results.

We would like to invite you to continue the discussion with us. We hope that our responses can successfully address your concerns so that you might consider a reevaluation of your rating.

Thanks and best regards,

The Authors

We thank the reviewer for their feedback on our work. Please find our detailed response below.

The method combines existing techniques such as consistency distillation and motion guidance, so its novelty is somewhat limited.

We respectfully disagree with the reviewer's argument on our novelty. In terms of guidance strategy, the primary goal of our work is not to design a new energy function for motion guidance. Instead, we aim to empirically demonstrate that augmenting the teacher ODE solver with the energy function's gradients of a conditional guidance strategy can distill a more capable student video generator while significantly reducing inference costs. In this context, we showcase the potential of our approach by integrating MotionClone’s energy function into the teacher ODE solver.

Additionally, adapting MotionClone’s motion guidance techniques for training is significantly non-trivial for several reasons:

• Dependency on Reference Videos: MotionClone relies on access to reference videos with high-quality motion. However, identifying suitable reference videos for general text prompts is challenging, which limits its effectiveness and applicability for generic video generation tasks.
• High Computational Cost: Computing the gradient of the energy function during inference incurs substantial memory and latency overhead. For instance, generating a single video with MotionClone can take approximately seven minutes and 30 GB GPU memory.

To address these challenges, we leverage the critical insight that each training video inherently serves as an ideal reference video for its corresponding training prompt. Additionally, we design a separate preprocessing phase to precompute the motion guidance before the actual training phase. As a result, this preprocessing phase eliminates the need for computationally expensive gradient calculations during training.

As demonstrated in Tables 1 and 4, augmenting the teacher ODE solver with motion guidance leads to significant performance gains and improved motion quality across different evaluation metrics.

VideoCrafter uses a 2D+1D decoupled spatial-temporal approach, whereas most recent advanced methods employ full 3D attention. How would motion guidance be applied when using 3D attention?

We would like to address the reviewer’s understanding of our contributions. The applicability and generalizability of MotionClone’s technique are beyond the scope of our paper. The primary goal of our work is not to design a new energy function for motion guidance.

Instead, we aim to empirically demonstrate that augmenting the teacher ODE solver with the energy function gradients of a conditional guidance strategy can distill a more capable student video generator while significantly reducing inference costs. To illustrate this, we integrate MotionClone’s energy function into the teacher ODE solver, highlighting the potential of this approach.

Nonetheless, MotionClone’s motion guidance can also be applied to advanced methods, such as Open-Sora, Open-Sora-Plan, and Latte [1]. As shown in Figure 3 of [2], these models employ a similar DiT-based architecture featuring two types of Transformer blocks: spatial and temporal. Since the temporal Transformer blocks process information across temporal dimensions, MotionClone’s success can be replicated within these frameworks with minimal adaptation. For methods utilizing full 3D attention, e.g., CogVideoX, we can always reshape the attention matrix to obtain temporal attention across different video frames. And thus, full 3D attention should not be a barrier to leverage MotionClone’s techniques.

[1] Ma et al., Latte: Latent Diffusion Transformer for Video Generation.

[2] Zhao et al., Real-Time Video Generation with Pyramid Attention Broadcast.

What is the peak training cost, such as peak GPU memory, compared to training a single model?

The peak training cost of our method is identical to that of training a single model. As highlighted in our paper, the motion prior is extracted during a separate preprocessing phase and does not contribute to the peak GPU memory usage during training.

does it perform on less powerful video diffusion models like Zeroscope—can it still achieve results comparable to VideoCrafter?

T2V-Turbo’s success has already been demonstrated with two different teacher models: VideoCrafter2 and ModelScope. Regarding the generalizability of MotionClone, the original MotionClone paper conducted its experiments using AnimateDiff [3] as the base model, and our work successfully extended this technique to VideoCrafter2. Therefore, there is no inherent barrier to applying this approach to less powerful models like Zeroscope, and we expect it our T2V-Turbo-v2 achieve similar successful results when employing Zeroscope as the teacher model.

Dear Reviewer PqeK,

We greatly appreciate your time and feedback on our work. We have carefully addressed your comments and clarified potential misunderstandings. Additionally, we also included new experimental results to corroborate our findings.

We kindly invite you to revisit our paper in light of these updates and clarifications. We would greatly appreciate it if you could consider whether our responses warrant a reevaluation of your rating.

Best regards,

The Authors