Compositional Video Generation as Flow Equalization

We truly thank the review for all the questions. They are very important to improve the practical application of our Vico to more video diffusion model.

Note: Kindly reminder that, according to this year's ICLR policy, papers published (peer-reviewed) after July 1, 2024, are treated as concurrent work and authors are not required to compare their own work to that paper. References [5] and [4] were accepted to COLM24 and NeurIPS24, respectively, with announcements made after July 1. Additionally, Cogvideox [2] was released in August and T2V-CompBench [1] was released in July, and [6] and [7] are not research papers. We kindly ask the reviewer to consider these circumstances.

>>> Q1 Recent Baseline Models

>>> A1 We truly thank the review for the suggestions. As suggested, we built Vico on top of Open-sora [1] and CogVideoX [2], and discussed in Appendix F.

• Open-sora Adaptation: Open-sora [1] was relatively straightforward to adapt, as it uses STDiT, which separates spatial and temporal attention.

• CogVideoX Adaptation: Interestingly, our solution also applies well to CogVideoX. It uses a more complex 3D MM-DiT architecture. This model flattens all text and video tokens, processing them in a single joint attention layer. Since there is no explicit cross-attention mechanism, traditional cross-attention control methods don’t apply. However, our graphical abstraction approach is highly effective in this scenario because, fundamentally, it is still attention between tokens. Specifically, we redefine our graph construction rule as follows:

  E_{vt,l} & E_{vv,l} \end{bmatrix}$$ $$\mathbf{W} = \begin{bmatrix} \mathbf{W}_1 & \mathbf{0} & \dots & \mathbf{0} \\\\ \mathbf{0} &\mathbf{W}_2 & \dots & \mathbf{0} \\\\ \mathbf{0} &\mathbf{0} & \vdots & \mathbf{0} \\\\ \mathbf{0} &\mathbf{0} & \vdots & \mathbf{W}_L \end{bmatrix}$$ Here, $\mathbf{W}\_l$ is the adjacency matrix at each layer, with $E_{tt,l}$, $E_{tv,l}$, $E_{vt,l}$, and $E_{vv,l}$ representing weights for text-to-text, text-to-video, video-to-text, and video-to-video connections, respectively. Each $E$ is still subject to our computation defined from `Line 236`. Stacking these matrices across layers gives the final capacity matrix $\mathbf{W}$.

Results: We tested Vico with new baselines on VBench, measuring the Multiple Object Composition score. Due to the high memory demands for DiT, we used an 80GB A100 GPU for inference.

The results show that Vico substantially enhances performance, proving the effectiveness of our approach across various architectures.

We have included the results in the revise paper Appendix F.

>>> Q2 Recent work on compositional video generation

>>> A2 We thank the R-Y7gd for bring those nice reference. We have add them to our reference and discuss with their difference. We differ from them in two main perspectives,

• Training-free. Methods like [3][5] require curated datasets with bounding boxes and prompts, while [4] relies on long-video datasets with enhanced motion dynamics. In contrast, Vico is fully training-free, using only test-time optimization without additional training data.
• Additional components: [3][5] depend on external LLMs for spatial planning, and [4] introduces a Reference Frame Attention module and optionally uses LLMs for prompt decomposition. Vico requires no additional modules or external support.

Comparing with new papers: As suggested, we compare our VC2+Vico model with those new baselines on T2V-CompBench. The results is shown below. Even without reliance on external datas, models and training, we can gets comparable performance, and even better on action binding and object interactions.

We have also included the new results in the main paper Table 2.

>>> Q3 Equal influence justification

>>> A3 Thanks for the nice question. The reviewer is correct that not all tokens should be equally weighted.

Our approach does not assume that all tokens need balancing; rather, we focus on giving equal weight to specific important tokens. As mentioned in Line 417, we only equalize nouns and verbs for equalization.

We have also revise Section 3.1 to highlight this point.

>>> Q4 Vico on 3D attention

>>> A4 This is an excellent question. Vico does not strictly require cross-attention. In fact, it can be seamlessly applied to recent diffusion transformer models with 3D attention. The only adjustment needed is in constructing the capacity matrix.

As implemented in A1, we have successfully apply Vico to 3D attention.

>>> Q5 'Bird/Cat' Example

>>> A5 Sorry for the confusion. In fact, the example can be found in Figure 1 with prompt A bird and a cat.

In Line 51, we use the phrase a bird looks like a cat to explain how Videocrafterv2 mistakenly places a cat’s face on the bird’s body. We did not mean to imply there was an actual prompt a bird looks like a cat.

We have revised the text for better clarity.

>>> Q6 Vico on 3D attention

>>> A6 Yes. Please see A1 and A4.

>>> Q7 Computation when applying Vico to 3D attention

>>> A7 The computation is large, still manageable.

• Forward Consumption: Assume each MM-DiT layer has $n$ tokens (including all text and video tokens), in the end of the day, computing our approximated ST-Flow for $L$ layers is $Ln^3$.

  For CogVideoX-2B, generating a 512x512 video with 49 frames involves 13,312 video tokens and fewer than 226 text tokens, totaling roughly 13,000 tokens. Using FP16 precision, this adds approximately 10GB of GPU memory to the original model's inference.

• Backward Consumption: Since we only need to back-propagate to the input, we set all network parameters to require_grad=False.

Altogether, we need ~76GB of GPU memory for CogVideoX-2B inference using FP16, which fits within a single A100 GPU. The running time is ~200s for each prompt.

>>> Q8 Vico on Long Prompt

>>> A8 The current token length is primarily limited by the CLIP text encoder used in the diffusion model, which typically defaults to 77 tokens.

As suggested, we test on longer prompts, and show the results in Appendix E.2. The findings show that Vico can still generate coherent videos even with longer input texts.

>>> Q9 Results on T2V CompBench

>>> A9 We appreciate the suggestion for new brilliant benchmark and we have add the reference in the paper.

As suggested, we evaluated our method on this new benchmark and compared it with several recent baselines. As shown below, combining VideoCrafterv2 with Vico delivers competitive performance on this benchmark.

We truly thank the reviewer for the thoughtful questions and feedback.

>>> Q1 Qualitative Experiments on CogVideoX

>>> A1 Thank R-Y7gd for the suggestion. As requested, we selected representative samples generated by CogVideoX-2B and CogVideoX-2B+Vico from the Multiple Object subset of Vbench. The results are included in the revised Appendix F, Figure 7.

Our observations indicate that compositional errors persist even in advanced video diffusion models like CogVideoX. For instance, it blends a boat and an airplane into a single object, such as a seaplane, or generates only a pizza while neglecting a tie. In comparison, Vico resolves these conflicts and ensure all concepts are represented accurately and fairly.

>>> Q2 How to integrate Vico framework in training?

>>> A2 We sincerely thank the reviewer for this excellent question; this is something we are actively working on.

Compositional Regularization: One approach we are considering is incorporating the Vico objective function as a regularization loss during training. Given a noisy latent $\mathbf{x}_t$ at step $t$ and the text $y$, we can compute its attribution score $A$ during a training iteration. The diffusion model can then be optimized simultaneously for denoising and balancing token influence.

Synthetic Videos: Another potential approach is to use Vico to generate videos containing compositional concepts. These synthetic samples can then be used to retrain the diffusion model, improving its ability to handle complex compositions.

In general, test-time optimization is indeed lengthy and costly. Moving it to the training phase is indeed a promising direction to improve efficiency.

---

Once again, thank the reviewer for the thoughtful feedback and for recognizing the potential of our work.

We greatly appreciate Reviewer kAsb's thoughtful suggestions and have thoroughly incorporated them into the revised manuscript.

>>> Q1 Not all token are equal

>>> A1 We are sorry for any confusion this may have caused. The reviewer is right; some tokens indeed carry less information.

Vico does not assume that all tokens require balancing. Instead, our approach focuses on assigning equal weight to specific important tokens. As mentioned in Line 417, we only equalize nouns and verbs.

Based on your suggestion, we have revised the manuscript Section 3.1 to emphasize this point more clearly.

>>> Q2 Resource Consumption

>>> A2 We truly thank the reviewer for this question. We have incorpate the running speed and complexisty in Appendix G. Regarding memory consumption, for generating a 16-frame video at $320\times 512$ resolution with VideoCrafterv2, the results are as follows:

The GPU memory usage increases by $\times 2.4$. We are actively working on optimizing and reducing this.

>>> Q3 Extend Image Methods to Vidoe

>>> A3 Excellent question. Yes, image-based methods can be extended to compositional video generation tasks. In fact, we have implemented Token Re-weight and Attend-and-Excite and compared their performance with Vico, as shown in Figure 4 and Table 1 of the paper. However, their results do not match Vico's performance.

The main reason for the gap is the difference in model architectures and the need to handle temporal dynamics. This is explained in Lines 200-215, where we show why we need new design for video tasks.

>>> Q4 Compositional Motion Scenarios

>>> A4 Yes, Vico can handle compositional motion scenarios.

Quantitative Results: We introduced the Motion Composition metric in Line 371. Column 4 of Table 1 shows our results. Vico improves motion composition by handling different movements for multiple subjects.

New Cases: We have added new examples of this scenario in Appendix E.1. These samples highlight Vico's improved performance in complex motion cases.

We sincerely thank Reviewer kAsb for the insightful questions.

>>> Q1 Test time of Vico

>>> A1 Originally presented in Appendix G, Table 9, we report test time based on a 50-step DPM denoising process using the VideoCrafterv2 model at a resolution of 512 × 320 for 16 frames, running on a single A6000 GPU.

The time is also reported below. Specifically, the baseline VideoCrafterv2 completed the task in 23 seconds, while our Vico took 45 seconds. Despite its additional complexity, Vico’s efficient design keeps the inference time within a reasonable range.

>>> Q2 Selection of key words

>>> A2 Thank the reviewer for raising this concern. We totally understand the doubts about selecting only nouns and verbs, which may seem overly simplistic. However, this approach aligns with our core evaluation metrics in the paper.

Our primary metrics—Spatial Relations, Multiple Objects, and Motion Composition—ultimately assess the accuracy of nouns and verbs in the generated videos. For this reason, we rely on this method at this stage.

Extensions and Improvements

While this approach works well for now, we acknowledge its limitations and are open to enhancements. For instance, advanced keyword extraction techniques like prompting LLM could be employed to enhance keyword selection. We provide an example on how to use this.

User: Extract the key action or concept from the sentence for use in a text-to-video generative model. Provide only the answer. Example sentence: 'On a certain day, a boy is running.'

ChatGPT: a boy is running.

Such techniques can be incorporated into our framework to enhance its robustness and flexibility when selecting keywords.

---

Should you have any further questions or concerns, please let us know and we will try our best to clarify.

Again, thank your for all the suggestions and questions—they've really helped us to improve our paper!

We sincerely appreciate Reviewer JUy3's suggestions. We have carefully incorporated them into the revised manuscript.

>>> Q1 Readability

>>> A1 Thank R-JUy3 for the suggestion. We sincerely apologize for any issues with the writing. The paper adapts many mathematical definitions to a new problem, which may have caused confusion. We have revised the paper to address this and improve clarity.

>>> Q2 Capacity matrix W

>>> A2 Thanks for the question.

• Row and Column: As shown in Line 250, our $\mathbf{W}$ is written as a form of block matrix. Each $E_i$ and $E_{t,i}$ itself is a smaller matrix, with each element in them representing the weight $e$ in the edge.
• $t$ in first row: Only the first row has t because that, the text tokens are positioned only in the first row. The rest of rows are video tokens. In this sense, the matrix $E_{t,l}$ represents the influence from text tokens to video tokens at each cross-attention layer. $E_{l}$ stands for the self-attention connections at the $l$-th layer, which only connect between video tokens.

We have included this explanation in the revised version to clarify the structure of $\mathbf{W}$ in Line 253-257.

>>> Q3 $e_{i,j}$ when $i=j$ and $e_{i,i}$

>>> A3 Sorry for the confusion. Yes, $e_{i,j}$ when $i=j$ and $e_{i,i}$ are the same; they represent the same value using different notations.

>>> Q4 $\mathbf{x}_t$ and $A_i$

>>> A3 Thank the reviewer for the question. $\mathbf{x}_t$ stands for the noisy latent at timestep $t$, which is defined in Equation 1.

$A_i$ is the importance for each token. We make it a general definition because there could be other measurement of $A_i$. In this paper, we define it as the flow value from each token to the final video, which is in Sec 3.2 and Sec 3.3. A simple intuition is that, $A_i$ is the accumulated weight of all cross-attention and self-attention score through all layers, from text token $v_i$.

We have added new explaination in the Sec 3.1.

>>> Q4 Token Re-weighting

>>> A3 Thanks for the nice question. Token Re-weighting method manually adjusts the weights of certain tokens to control their influence.

Specifically, a CLIP text encoder embeds the input text into a sequence of tokens $s = \{v_1, \dots, v_K\}$. Token Re-weighting multiplies a scalar $\alpha$ with specific tokens, for example, modifying the first token to $s' = \{\alpha v_1, \dots, v_K\}$. The updated sequence is then used as a new conditioning input for the diffusion model. As mentioned in Line 354, it is implemented by the compel package.

As suggested, we add the explaination in the revise Appendix I.

>>> Q5 Factors for compositional issues

>>> A5 Great question. Actually, fair attribution is not the only reason for compositional failures.

Specifically, fair attribution is key for first-order composition, which means that all concepts are represented in the generated output.

However, for second-order or high-order composition, other types of errors may still occur. For instance, even when all concepts are fairly presented, attributes might be incorrectly assigned or mis-bidded [A] with objects. A common example would be generating "a red dog and a yellow cat" from a prompt of "a red cat and a yellow dog".

Our primary focus is on solving simpler cases first. We are working to extend our approach to handle more complex problems.

[A] Linguistic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignment, NeurIPS 2023

We truly thank the R-6LSE for the nice comments.

>>> Q1 User Study

>>> A1 Great suggestion. As suggested, we conducted a new user study using 50 video clips generated from VideoCrafterv2 with prompts from VBench. We generated attribution heatmaps using Cross-Attention, Attention Rollout, and our ST-Flow. Additionally, we increased the number of participants to 43, who rated each method from 1 to 5 on Temporal Consistency and Reasonability.

The results is shown below as the mean$\pm$std. It illustrates that, our ST-Flow consistently outperformed the other methods.

We have included this results in the manuscript Table 3.