VideoTetris: Towards Compositional Text-to-Video Generation

We sincerely thank you for your time and efforts in reviewing our paper and your valuable feedback. We are glad to see that the paper is easy to follow, the approach is novel and the experiment are comprehensive. Please see below for our responses to your comments.

Q1: Concerns about the computational cost of LLMs, cross attention, and ControlNet.

A1：

1. Using LLMs for prompt enhancement is a widely adopted technique. Similar works like VideoDirectorGPT [1], Vlogger [2], and industrial models such as DALL-E 3 [3] use LLMs for prompt enhancement or decomposition, proving its effectiveness. Our model uses LLMs only once for spatiotemporal decomposition, aligning with most advanced or commercial models.
2. The computational complexity of the cross-attention mechanism is $O(N\times L1\times L2\times d)$, where $N$ is the number of sub-prompts and $L1, L2$ are the sequence lengths of Q and K/V. This complexity is negligible compared to the base model (UNet) and doesn't significantly increase computational cost. Additionally, our cross-attention calculations are batch-level, allowing multiple sub-prompts to be processed simultaneously, leveraging GPU parallelism. In practical tests using VideoCrafter2 as the backbone, a single inference on an A800 takes approximately 57 seconds. Adding one sub-prompt increases the total time by around 0.9 seconds. This additional cost is minimal and acceptable.
3. ControlNet is employed to balance video quality, generating long videos with superior visual effects, dynamics, and flexibility. Please kindly note that our Spatio-Temporal Compositional Diffusion method can be applied to any video generation framework. We plan to explore more cost-effective autoregressive methods in the future to further improve efficiency and quality.

Q2: Investigating more robust consistency regularization methods that can handle dynamic scenes with fast movements or significant changes in lighting

A2: Thank you for your insightful comment. We clarify as below:

1. Suitability for Autoregressive Long Video Generation: Given that our research focuses on autoregressive long video generation, our approach necessitates smooth and gradual movements without rapid acceleration or drastic changes. This allows our autoregressive framework to generate coherent videos effectively. Hence, the assumption that object features remain consistent across frames is appropriate for this specific scenario.
2. Dynamic Scene Improving: We have extensively explored dynamic scene handling. Our proposed Dynamic-Aware Video Data Processing method elevates video data to a highly dynamic and trainable level. The Reference Frame Attention module ensures consistency across multiple frames, significantly improving the dynamics of our videos compared to previous models. For detailed comparisons, please refer to Section 4.4 and the uploaded PDF, where we showcase long video comparisons.
3. Future Work for Extremely Dynamic Scenes: For scenarios involving extremely dynamic scenes with fast movements or significant changes in lighting, we recommend training separate models tailored to different motion intensities, such as LoRA, motion modules, or specific base models. Alternatively, encoding the magnitude of motion and incorporating it as a condition during training could facilitate the generation of highly dynamic videos. In such cases, setting up multi-frame attention mechanisms like the Reference Frame Attention module, along with additional ID and motion magnitude conditions, could prove beneficial. We plan to explore these possibilities in our future work.

Q3: The generated videos exhibit relatively subtle variations in content, which could result in static or less dynamic scenes. This may limit their practical applicability.

A3: Thank you for your observation, we explain below:

1. Base Model Limitations for Short Video Generation: For short video generation, we used VideoCrafter2 as the base model without retraining it. The subtle variations in content are primarily due to the inherent limitations of VideoCrafter2, which was not predominantly trained for dynamic scenes. Therefore, some subtle changes are expected.
2. Enhanced Dynamics in Long Video Generation: It is important to note that for long video generation, we trained our own model using a unique Dynamic-Aware Video Data Processing method. This significantly enhances the dynamics of the generated videos. For instance, in Section 4.4 and the uploaded PDF, there is a detailed comparison showcasing the dynamic behavior of a squirrel. The squirrel dynamically changes its position, eats a hazelnut, and interacts with another squirrel throughout a 30-second video. These processes are both highly dynamic and natural, demonstrating the model's capability to produce vibrant and engaging scenes.
3. Future Enhancements and Retraining: With our training process, if we were to retrain the base model for short video generation, we could undoubtedly enhance the dynamic range of the videos, thereby improving their practical applicability. Due to resource constraints, we used a pre-trained model for our experiments. However, we are committed to exploring further possibilities. With ample resources, we plan to retrain the base model to enhance video dynamics. Increasing dynamics is one of our primary goals, and we will continue to work towards achieving it.

[1] Lin, Han, et al. "Videodirectorgpt: Consistent multi-scene video generation via llm-guided planning." arXiv preprint arXiv:2309.15091 (2023).

[2] Zhuang, Shaobin, et al. "Vlogger: Make your dream a vlog." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[3] Betker, J., et al. "Improving image generation with better captions. "Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2023.

Dear Reviewer 9Z5k,

We greatly appreciate the time and effort you have invested in reviewing our paper. Your thoughtful questions and insightful feedback have been invaluable. In response to your queries regarding computational cost and motion dynamics, we have prepared detailed answers.

As the discussion period is set to conclude in two days, we would like to ask if you had a chance to check our response to your rebuttal questions. Should you require any further clarification or improvements, please know that we are fully committed to addressing them promptly!

Thank you once again for your invaluable contribution to our research.

Warm regards, The Authors

We sincerely thank you for your time and efforts in reviewing our paper and your valuable feedback. We are glad to see that the paper is well written the motivations are clearly explained, and the experiments are sufficiently thorough. Please see below for our responses to your comments.

Q1: Although Spatio-Temporal Compositional Diffusion directly adjusts cross-attention, it still segments videos into different shots, each with a specific layout, suggesting it's essentially a layout-based method. This approach contradicts the motivation outlined in section 3.1.

A1: Our approach fundamentally differs from layout-based methods in that we adopt an initial region-based method. Layout-based methods strictly confine each object's generation within its corresponding mask, resulting in limited expressiveness and fidelity. In contrast, we provide both local and global semantic information within specific regions at the outset. This allows us to fuse local and global semantics, enabling the content within each region to adjust its position and size based on global information, significantly enhancing the interactivity among objects and resulting in a richer expressive capacity. Our initial positions and sizes of objects can be adaptively adjusted during the generation process; we do not restrict each object to its planned subregion. This region-based division is more flexible and has a richer expressive capacity.

We have included a comparison with LVD [1], a layout-based method, in our uploaded PDF. The quantitative results reported therein highlight our method's advantages over layout-based methods. Our approach significantly outperforms in terms of attribute handling, numeracy, naturalness, and harmony of the generated frames.

Q2:The paper presents limited technical innovations, focusing more on engineering implementation. In Spatio-Temporal Compositional Diffusion, it uses LLMs to pre-process prompts and region masks, which are then sequentially applied during generation. Meanwhile, Dynamic-Aware Video Data Processing enhances data preprocessing for higher quality.

A2: To clarify, our core innovation is the Spatio-Temporal Compositional Diffusion. In real-world video generation tasks, compositional generation is a common and practical scenario. Our method is the first to define and effectively address this task, extending beyond single video generation to progressive long video generation. This marks a significant advancement over previous long video generation tasks, involving dynamic changes in objects, which is unprecedented.

Our Dynamic-Aware Video Data Processing and Reference Frame Attention processes further enhance video visual quality by improving both video dynamics and continuity. These processes support our Spatio-Temporal Compositional Diffusion, making it more suitable for industrial production. The task definition, method, and actual results of this combination represent a novel and significant innovation in the field.

Q3: How is continuity ensured between different shots?

A3: As mentioned in Section 3.3, we proposed Reference Frame Attention to enhance continuity between different shots. Innovatively, we use VAE instead of CLIP to encode reference frames from the previous video clip, capturing characters and background information. This encoded data is input into cross-attention, interacting with the initial latent of the current denoising step, ensuring continuity across different shots.

Additionally, our trained ControlNet-like branch uses the first 8 frames of a video as a condition to predict the entire 16-frame sequence. Through Dynamic-Aware Video Data Processing, this model ensures smooth autoregressive generation, maintaining consistent base information and natural motion transitions across shots.

[1] Lian, Long, et al. "Llm-grounded video diffusion models." arXiv preprint arXiv:2309.17444 (2023).

Dear Reviewer Qhp4,

We greatly appreciate the time and effort you have invested in reviewing our paper. Your thoughtful questions and insightful feedback have been invaluable. In response to your queries regarding our method's innovation and continuity ensuring, we have prepared detailed answers.

As the discussion period is set to conclude in two days, we would like to ask if you had a chance to check our response to your rebuttal questions. Should you require any further clarification or improvements, please know that we are fully committed to addressing them promptly!

Thank you once again for your invaluable contribution to our research.

Warm regards, The Authors

We sincerely thank you for your time and efforts in reviewing our paper and your valuable feedback. We are grateful for your acknowledgment of the clarity, methodological strengths, and resource efficiency of our paper. Please see below for our responses.

Q1: Novelty Clarification & Caparison with [1, 2, 3]

A1: Thank you for your suggestion. Our method fundamentally differs from these three methods in the following ways:

1. Interactivity, Flexibility and Capacity: While [1, 2, 3] generate masks for each entity using a standard cross-attention mechanism, our approach innovatively generates linguistic descriptions for each region. It composes cross-attention for sub-prompts in parallel and fuses them from the initial denoising step. Unlike conventional techniques that strictly confine object generation within corresponding masks—thereby limiting expressiveness and fidelity—our method offers greater flexibility. By providing coarse-grained initial positions and sizes, each object can adjust its position and size based on global information, significantly enhancing the interactivity among objects and resulting in a richer expressive capacity.
2. Compositional Generation: Our method for the first time focuses on compositional generation tasks. In contrast, LVD primarily aims to generate LLM-grounded smooth movements of objects within a box. We empirically find that this backward guidance based methods like [1] and [2] have limitations in compositional generation. As demonstrated in our uploaded PDF, when dealing with multiple objects' attributes or numeracy tasks, the paradigms in [1] and [2] result in inferior outcomes.
3. Training-Free Spatio-Temporal Compositional Diffusion: Compared to VideoDirectorGPT [3], our method is training-free for short video generation. Given the rapid advancements in T2V models, training-based methods struggle to keep pace. Our efficient, training-free spatio-temporal comspositional diffusion delivers comparable or superior visual quality, and can generalize to existing video diffusion backbones seamlessly.

Q2:Concerns about Contribution and Novelty of Dynamic-Aware Video Data Processing Pipeline

A2: Our Dynamic-Aware Video Data Processing pipeline comprises two main components: Video and Text, which extends beyond merely filtering videos. We first select videos with moderate motion using an optical flow score, then optimize dynamic semantic information for these videos by generating captions with multiple Multimodal Large Language Models (MLLMs) and summarizing them with a local LLM. This results in prompts that capture complex cross-modal semantic consistency between video contents and text captions.

In contrast, Stable Video Diffusion (SVD) only filters videos and uses traditional annotation methods, which struggle to maintain cross-modal semantic consistency. Our approach achieves this consistency by using MLLMs to match high-quality dynamic texts with corresponding videos.

Q3: More Samples and missing User Study

A3: Thanks for your constructive suggestion. We provided six examples for short video generation in Appendix A.6, Figure 8, and included additional short and long video samples in the uploaded PDF. For further evaluation, we conducted a user study comparing our method with other video models. Using GPT-4, we generated 100 short prompts and 20 long story prompts, totaling 120 video samples across diverse scenes, styles, and objects. Users compared model pairs by selecting their preferred video from three options: method 1, method 2, and comparable results. The user study results are also reported in our uploaded PDF.

These supplementary materials and the user study demonstrate that our model has significant advantages in qualitative experiments, proving its effectiveness. And we will include these materials and the user study in our final version.

Q4: Comparision with LVD

A4: We have now included a comparison with LVD in our uploaded PDF. The results indicate that while LVD is capable of maintaining the natural motion trajectories of objects, it exhibits more errors in compositional tasks. Additionally, we tested LVD using our metrics and reported the results as in the table.

LVD outperforms modelscope and animatediff but still lags behind VideoTetris, highlighting our method's advantages in compositional tasks. In the final version of our paper, we will include LVD’s results in all comparative experiments and quantitative results to provide a more relevant baseline.

Q5: Ablation studies about comparisons with method [1,2,3]

A5: Thank you for your insightful comment. We have conducted the ablation studies accordingly. First, we evaluated the decomposition methods from [1] and [3], generated final long videos, and reported all results in the uploaded table. These methods only isolate specific tokens from the original prompt, struggling with complex attributes or multiple identical objects. This leads to difficulties in understanding numeracy and attribute binding, causing performance degradation. In contrast, our method extracts subobjects and uses global information for recaptioning, resulting in richer, more natural, detailed, and semantically accurate frames.

Next, we have conducted ablation studies about the composition method from [2] and reported the results in the uploaded table. The backward guidance approach in [2] tends to restrict objects within specified boxes, offering low flexibility, interactivity and poor responsiveness to multiple attribute bindings or repeated objects. In contrast, our model's local-global information fusion ensures that the final generated images are more harmonious and visually appealing, performing better in compositional generation.

In the final version of our paper, we will include all of the above ablation studies to provide a more comprehensive analysis.

We sincerely appreciate the time and effort you have dedicated to reviewing our paper and providing valuable feedback. We are pleased that the spatial-temporal decomposing method, visual quality, and the adequacy of our quantitative and ablation experiments were well-received. Below, we address your comments.

Q1: Concerns about LLM's decomposing capabilities. My major concern is that the model heavily relies on the performance of LLM. In the proposed method, LLM acts as a director to split an input story prompt into spatiotemporal prompts. I am wondering whether LLM is robust enough to generate temporal-consistent prompts with spatial-consistent composition. Q2: Following the concern above, the supplementary only provides one example of long video generation. The authors should more detailed and more samples. For example, provide 100 story prompts and the corresponding frame-wise prompts with object composition. For what we have now, it is hard to fairly justify the model performance.

A1&A2: Thank you for raising this concern regarding the robustness of the LLM in our proposed method. We have addressed these concerns in several ways:

1. Robustness and Capability of GPT-4: As detailed in Appendix A.1, we employed GPT-4, a large-scale multimodal model. During its pre-training phase, it leveraged a vast amount of vision-language pairs, which inherently equips it with the capability to establish connections between visual and linguistic elements, thereby enabling accurate spatiotemporal decomposition.
2. In-Context Learning (ICL): We leveraged in-context learning (ICL) to enhance the model's performance, as outlined in Appendix A.1. By providing the LLM with pre-designed examples, we simplified the learning task, allowing the model to combine its visual knowledge with the ICL examples to produce consistent outputs. This approach is well within the capabilities of GPT-4 and strengthens its ability to handle complex spatiotemporal prompts.
3. Validation Through Similar Works: Similar works, such as VideoDirectorGPT [1], VideoDrafter [2], Vlogger [3], and LVD [4], have adopted the same approach, using LLMs for spatiotemporal planning. Their experimental results have validated the efficacy of this method. Our own quantitative and qualitative experiments, as shown in Section 4, further validate the efficacy of using GPT-4 for this purpose. The consistency and reliability of the outputs in our experiments affirm the model's capability to manage the tasks at hand.

Q3: Another concern is the autoregressive strategy of long video synthesis. While the long video shows good consistency of motion and subject identity, I found the visual quality gradually decreases in the video. For example, in the sample "fig5-ours.mp4", the contour of leaves is visible at the beginning; however, it gets more and more blurry until the end of the video. Is this also happens on other long synthesis videos?

A3: The sample video "fig5-ours.mp4" you referred to is an isolated case. In our selection process, we did not cherry-pick the results. To address your concern, we have included an additional long video in the uploaded PDF, where the quality remains consistent. Compared to the baseline StreamingT2V and FreeNoise, our videos exhibit more accurate and vibrant colors. Due to rebuttal length constraints, we cannot include more video samples here, but we will open-source all code for independent testing. Our method shows significant color enhancements over the baseline StreamingT2V.

Q4: Does the proposed framework have the scalability limitations in terms of the length and complexity of the generated videos?

A4: As illustrated in Figure 2, our framework leverages a ControlNet branch for autoregressive generation, similar to StreamingT2V. Taking a T2V UNet that generates 16 frames as a base model, it uses the first 8 frames of a video as the condition to predict the entire 16-frame video. In the real-world long video generation process, this module continuously uses the last 8 frames of the previous segment as the condition to predict the subsequent frames in an autoregressive manner. Consequently, this framework naturally supports the generation of infinitely long videos without scalability limitations. We have provided videos with hundreds of frames in the uploaded PDF, which demonstrate stable visual quality throughout, thereby validating the effectiveness of this autoregressive framework.

Q4: How does the framework perform in real-world scenarios where the input text prompts may be less structured and more diverse?

A4: As shown in Appendix A.1, Table 5, in real-world scenarios, notably, our framework incorporates a recaptioning step after spatiotemporal decomposition. Inspired by approaches like DALL-E 3 and RPG, and various industry practices, this step optimizes irregular prompts into more detailed and coherent descriptions, aligning with the training patterns to generate more visually appealing content. Therefore, for less structured and more diverse input text prompts, the robust understanding capabilities of LLMs ensure that the quality of the generated videos does not significantly decline.

[1] Lin, Han, et al. "Videodirectorgpt: Consistent multi-scene video generation via llm-guided planning." arXiv preprint arXiv:2309.15091 (2023).

[2] Long, Fuchen, et al. "Videodrafter: Content-consistent multi-scene video generation with llm." arXiv preprint arXiv:2401.01256 (2024).

[3] Zhuang, Shaobin, et al. "Vlogger: Make your dream a vlog." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[4] Lian, Long, et al. "Llm-grounded video diffusion models." arXiv preprint arXiv:2309.17444 (2023).

Dear Reviewer Qhp4,

We greatly appreciate the time and effort you have invested in reviewing our paper. Your thoughtful questions and insightful feedback have been invaluable. In response to your queries regarding LLM capabilities, motion dynamics, and model scalability, we have prepared detailed answers.

As the discussion period is set to conclude in two days, we would like to ask if you had a chance to check our response to your rebuttal questions. Should you require any further clarification or improvements, please know that we are fully committed to addressing them promptly!

Thank you once again for your invaluable contribution to our research.

Warm regards, The Authors

Thanks authors for the detailed response. While some of my concerns are addressed, one of my main concerns is the visual quality of the long video generation. The supplementary only shows one example and it is hard to judge the quality based on the sampled video frames provided in the pdf which the authors uploaded during the rebuttal period. So, I will maintain my initial decision as a borderline accept.