Enhancing Motion in Text-to-Video Generation with Decomposed Encoding and Conditioning

Thank you for your detailed feedback and insightful questions. Below are our responses to the concerns raised:

Q1: The comparison with other approaches is not 100% fair since additional parameters (a CLIP encoder) are trained.

Response: We appreciate your concern about the fairness of comparisons due to additional parameters being trained on the text encoder side (a CLIP encoder). We conducted an experiment to rule out the effect of the additional parameters in the text encoder. Specially, we trained the baseline ModelScopeT2V model with a new CLIP text encoder while keeping its original training loss. We then compared its performance to the model without the additional CLIP encoder parameters (ModelScopeT2V fine-tuned column in the table). From the experimental results, we observed that the performance of the baseline model with the additional text encoder parameters is similar to that of directly fine-tuning the ModelScopeT2V without additional parameters. This indicates that without appropriate supervision, the impact of the additional parameters in the text encoder has only a marginal influence on the overall performance.

Q2: The definition of the 3D U-Net in your paper states: "the temporal transformer consists of temporal self-attention and feed-forward layers." Some papers also use cross-attention in temporal self-attention. Is this a mistake, and which specific papers are you referring to as the "vanilla" 3D U-Net?

Response: Thank you for pointing out the ambiguity in our definition of the 3D U-Net. In the methodology section, when we refer to the 3D U-Net, we are specifically referencing the ModelScopeT2V model. This model employs spatial transformers for spatial conditioning and temporal transformers with temporal self-attention for smoothing individual frames along the temporal dimension. We will clarify this in the revised manuscript to avoid further confusion.

Q3: How your temporal attention is performed? Do you perform the attention on a single patch or full attention across all patches of all other frames?

Response: Thank you for raising your question about the attention mechanism in our model. The temporal cross-attention in our model operates on a single patch across the frame dimension rather than full attention across all patches of all frames.

Our approach involves decomposing the full 3D cross-attention into two separate components: a 2D spatial cross-attention (content conditioning) and a 1D temporal cross-attention (motion conditioning). This design choice enables us to: 1) Preserve the text-to-video generation capability (generate 2D content for each individual frame separately) of the base model. 2) Allow the text instructions to control the temporal dynamics of the video effectively. By separating the spatial and temporal attention mechanisms, we achieve a balance between computational efficiency and performance. This approach ensures fair comparisons and meaningful ablations of the introduced losses, demonstrating the effectiveness of our conditioning mechanism.

Thank you for your detailed feedback. Below are our responses to the concerns raised:

Q1: The related work section contains too little content.

Response: We acknowledge that the related work section is limited due to page constraints. In the revised paper, we will expand this section and include additional relevant studies (references omitted here due to limits):

T2V Generation: T2V has advanced significantly, leveraging T2I success. The first T2V model, VDM, introduced a space-time factorized U-Net for temporal modeling, training on both images and videos. For high-definition videos, models like ImagenVideo, Make-A-Video, LaVie, and Show-1 use cascades of diffusion models with spatial and temporal super-resolution. MagicVideo, Video LDM, and LVDM apply latent diffusion for video, working in a compressed latent space. VideoFusion separates video noise into base and residual components. ModelScopeT2V, the first open-source T2V diffusion model, uses 1D convolutions and attention to approximate 3D operations. Stable Video Diffusion (SVD) divides the process into T2I pre-training, video pre-training, and fine-tuning.

Q2: There is a lack of contrast experiments with other methods that rely on external references.

Response: Thank you for raising your concern. Here we list the key differences between our method and other methods that rely on external references to address your concern.

1. Different Goal: Our focus is to enhance motion synthesis in general text-to-video generation without relying on external references, which are often infeasible in real-world scenarios. Methods using external references, like video customization or controllable video generation, aim to enable more control to generate videos with specific object appearances, motions, and other details, which is not our goal.

2. Different Information Sources: Our model takes only sparse textual descriptions. It cannot directly take external references without additional training and adaptation, which is beyond the scope of our paper. Methods using external references have access to more detailed visual signals. Comparing our approach with such methods is unfair due to the disparity in information density.

Q3: There is a lack of experiments with other base model to show the generalization of the proposed method. Additionally, the performance of the baseline model is poor.

Response: We agree that demonstrating the generalization of our method on other base models would be beneficial. To this end, we have applied DEMO to ZeroScope [1] and included the experimental results in the table below. The results are consistent with those obtained using ModelScopeT2V, showing significant improvements in motion quality without loss of visual quality.

Additionally, ModelScopeT2V is the only fully open-source text-to-video model (dataset, model, and code). While its performance may be lower than some closed-source models, it performs reasonably well especially in motion quality. Below are its performance rankings across various benchmarks against both closed-source and open-source models:

Q4: The provided videos are less impressive and the separate files make it difficult to compare those methods.

Response: Please refer to Q1 in Global Response.

Q5: Why DEMO can improve the individual frame quality (FID) since DEMO is designed to enhance the motion generation?

Response: Thank you for raising this concern. The confusion arises from the fact that individual frame quality (FID) involves both the realism of the frame itself and the diversity of frames. The FID [6] calculate the Fréchet Distance between generated frames and real frames as follows:

$

d(P_R, P_G) = \| \mu_R - \mu_G \|^2 + \text{Tr}(\Sigma_R + \Sigma_G - 2(\Sigma_R \Sigma_G)^{\frac{1}{2}})

$

where $\mu$, and $\Sigma$ are mean, covariance of real ($R$) and generated ($G$) frames. The mean term measures frame realism by comparing the average feature representation of generated and real frames. The covariance term assesses frame diversity by capturing how well generated frames replicate the variability and feature relationships of real frames.

Our base model uses content conditioning for each individual frame and temporal self-attention for smoothing those frames, which improves motion smoothness but limits motion dynamics and frame diversity. In contrast, DEMO introduces a separate motion conditioning module, enhancing motion dynamics and generating more diverse frames. This increased diversity leads to a better FID score, thus improving individual frame quality.

[1] Zeroscope. https://huggingface.co/cerspense/zeroscope_v1_320s. 2023.

[2] Evaluation of Text-to-Video Generation Models: A Dynamics Perspective. 2024.

[3] Fetv: A benchmark for fine-grained evaluation of open-domain text-to-video generation. 2024.

[4] T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation. 2024.

[5] Vbench: Comprehensive benchmark suite for video generative models. 2024.

[6] Gans trained by a two time-scale update rule converge to a local nash equilibrium. 2017.

We appreciate your valuable feedback. Below are our responses to the concerns raised:

Q1: In the pilot study, employing only one kind of sentence template may not be sufficient to draw the conclusion that the text encoder is less sensitive to the motion instructions in the text prompt.

Response: We thank the reviewer for highlighting this concern. Due to page limitations, we included the details of our pilot study in the appendix. As stated in the appendix (lines 527 to 530), "It is noteworthy that we did not observe significant differences when using different templates or different sets of words within each POS. The results were consistent across different setups, and we selected these prompts to try to make these prompts meaningful." We will revise the caption for our pilot study (Fig.1) to include this information and to address this concern as follows:

"We generated a set of prompts (262144 in total) following a fixed template, grouping them according to the different parts of speech (POS). These grouped texts are then passed into the CLIP text encoder, and we calculate the sensitivity as the average sentence distance within each group. As shown on the left-hand side, compared to POS representing content, CLIP is less sensitive to POS representing motion. (Results are consistent across different templates and different sets of words within each POS. Further details can be found in the appendix.)"

Q2: In Fig.3, it is hard to tell which rows are generated by the base model and which rows are generated by the fine-tuned model.

Response: Thank you for pointing out the potential confusion caused by the arrangement of video frames in Fig.3. This complexity arises because our quantitative evaluation involves multiple dimensions. We need to compare our model with other models, show the differences within each video, and evaluate multiple text inputs.

In Fig.3, we present video frames generated by four different models: LaVie, VideoCrafter2, ModelScopeT2V (base model), and DEMO (fine-tuned model), across three different text prompts. As mentioned in the caption of Fig.3, "Each video is generated with 16 frames. We display frames 1, 2, 4, 6, 8, 10, 12, 14, 15, and 16, arranged in two rows from left to right. Full videos are available in the supplementary materials."

To improve clarity, we plan to display these videos in a GIF format in our revised paper. This will help illustrate the differences more effectively. Please refer to Fig.1 in the PDF of Global Response.

Q3: It is hard to evaluate that our DEMO significantly outperforms the base model with the text prompt "Slow motion flower petals fall from a blossom, landing softly on the ground (the first example in Fig.3)" as the frames look almost the same.

Response: Thank you for pointing out the potential confusion in our presentation of video frames. The motion between different frames, especially those involving many objects such as flower petals, can be difficult to observe with static frames alone. We have provided video files in the supplementary materials, which clearly show significant differences between these two videos.

Additionally, we plan to revise Fig.3 in our paper to better highlight these differences. Please refer to Q1 and Fig.1 in the Global Response.

Q4: For qualitative evaluations, the improvements in generated motions in most shown cases are limited.

Response: Thanks for raising your concern. This is due to the fact that the improvements of motion are hardly to observe by only showing sequence of static frames. We have provided video files in the supplementary materials to better observe these improvements. Additionally, we plan to revise Fig.3 in our paper to better highlight these differences. Please refer to Q1 and Fig.1 in the Global Response.

Q1: We strongly recommend to incorporate a user study in the research to obtain statistical qualitative results.

Response: Thank you for your excellent suggestion. We conducted a user study to compare our method with other video generation models. We selected 50 prompts from EvalCrafter [1], covering diverse scenes, styles, and objects. For each model comparison, 15 annotators were asked to select their preferred video from three options: method 1, method 2, and comparable results. They evaluated the videos based on visual quality, motion quality, and text-video alignment. The results are shown as follows:

Specifically, 74% of user feedback indicated that DEMO has better motion quality compared to the baseline ModelScopeT2V. Additionally, 62% and 66% of the user feedback agreed that DEMO has better text-video alignment and visual quality, respectively. These findings align with our quantitative results, demonstrating that DEMO significantly improves motion quality without compromising visual quality and text-video alignment.

We also performed a user study to test the effect of our video-motion supervision term, $\mathcal{L}_{\text{video-motion}}$, which is designed to guide the motion conditioning module to generate better motion dynamics. We achieved win rates of 58%, 56%, and 72% in terms of text-video alignment, visual quality, and motion quality, respectively, compared to DEMO without video-motion supervision. Additionally, we observed that both ModelScopeT2V and our improved version, DEMO, have lower visual quality compared to LaVie and VideoCrafter2. We attribute this to the fact that the training dataset for ModelScopeT2V and DEMO (specifically WebVid10M) is low quality visually. In contrast, LaVie uses the high-quality video dataset Vimeo-25M [2], and VideoCrafter2 adopts the high-quality image dataset JDB [3] for better visual quality.

We appreciate the recommendation to incorporate this into the revised paper, and we will include a summary of these findings to enhance the robustness and reliability of our qualitative results.

Q2: How to calculate the optical flow between attention maps is not clearly described. Furthermore, considering that attention maps in diffusion models are often noisy, it is crucial to address the reliability and potential limitations associated with this computation.

Response: Thank you for pointing out the lack of clarity in our explanation. In our approach, we utilize RAFT [4] to calculate the optical flow, as detailed in Table 6 of the appendix. Basically, we treat cross-attention maps as kind of videos but in a different space compared with real world videos. We then directly apply RAFT to extract the optical flows. We recognize that attention maps in diffusion models can be noisy, which poses challenges for reliable optical flow computation. To address this, we tried a simple thresholding strategy that zeroes out regions with little activation. This approach resulted in slight improvements in the stability and accuracy of the optical flow calculations. Nonetheless, we acknowledge that this is a preliminary solution and that more sophisticated strategies could yield better results.

In the revised paper, we will provide a more detailed explanation of our methodology for calculating optical flow. This will include a step-by-step description of how we preprocess the attention maps, apply the RAFT model, and handle the inherent noise in the attention maps. Additionally, we will discuss the potential limitations of our current approach and outline possible directions for future work to enhance the robustness and reliability of optical flow calculations in noisy conditions.

[1] Teed, Z., & Deng, J. (2020). Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16 (pp. 402-419). Springer International Publishing.

[2] Wang, Y., Chen, X., Ma, X., Zhou, S., Huang, Z., Wang, Y., ... & Liu, Z. (2023). Lavie: High-quality video generation with cascaded latent diffusion models. arXiv preprint arXiv:2309.15103.

[3] Sun, K., Pan, J., Ge, Y., Li, H., Duan, H., Wu, X., ... & Li, H. (2024). Journeydb: A benchmark for generative image understanding. Advances in Neural Information Processing Systems, 36.

[4] Teed, Z., & Deng, J. (2020). Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16 (pp. 402-419). Springer International Publishing.

Dear Authors,

Thank you for your clarifications and conducted user study. My major concerns were addressed, so I raise my score to 7: Accept.

However, the fact that directly applying RAFT to extract the optical flows from attention maps works that well is surprising to me. Intuitively, it should be an out of domain input for RAFT. Looking forward for your detailed explanation of this part of pipeline in the camera-ready version of the paper.

Best regards,

Reviewer WCt2