VideoDirectorGPT: Consistent Multi-Scene Video Generation via LLM-Guided Planning

W1 Did not outperform other baselines in CLIPSIM:

We would like to point out that our focus is on enabling the generation of consistent multi-scene videos, with LLM-based planning and layout guidance, while maintaining visual quality (FID/FVD) and video-text alignment (CLIPSIM). We demonstrate the usefulness of our method of layout control and multi-scene consistency in Tables 1 and 3. Moreover, as mentioned in Sec 5.1, while Make-A-Video and VideoLDM achieve better CLIPSIM than our VideoDirectorGPT, they are trained with larger video data or with higher resolution than our backbone (ModelScopeT2V). As our framework can flexibly use other video backbone models, we believe that our framework can achieve even better CLIPSim once we have access to those stronger backbone models.

W2 Missing UCF-101:

Since ModelScopeT2V doesn’t report their evaluation results on UCF-101 dataset in their technical report, we use their publicly released checkpoint and run evaluation on the same 2048 randomly sampled test prompts used in our VideoDirectorGPT evaluation. Compared with ModelScopeT2V, we achieve a significant improvement in FVD (ModelScopeT2V 1093 vs. Ours 748) and competitive performance in the Inception Score (ModelScopeT2V 19.49 vs. Ours 19.42).

W3 Comparison to multi-scene models:

Our model is not directly comparable with other long-video generation models based on the following two reasons. Firstly, the main focus of our model is multi-scene video generation with both layout and consistency control, which is not covered by any of the previous long video generation works. Secondly, most works on long video generation (e.g., NUWA-XL and Phenaki) are not open-sourced, therefore it’s hard for us to make a fair comparison. Specifically, NUWA-XL is trained on cartoon characters, and it’s hard for our VideoDirectorGPT to generate these cartoon characters without fine-tuning.

W4 The qualitative comparison only presents results between Layout2Vid and ModelScopeT2V:

The code and checkpoint of models such as NUWA, VideoLDM, MagicVideo, and Make-A-Video are not publicly accessible. Since ModelScopeT2V is the only open-source strong video backbone model that we could access, we built our Layout2Vid with ModelScopeT2V. We would be happy to experiment with our frameworks with other video backbone models when we can access them in the future.

Q1 Human Evaluation Setup:

As mentioned in Sec. 4, the human evaluation presented in the paper used three crowd-sourced workers from Amazon Mechanical Turk to evaluate each video. The workers were not given any information about which model generated each video (randomly shuffled) to ensure there was no bias towards either model. While we took the agreement of three workers per video, we had 19 unique workers to complete the human evaluation.

We also extended the human evaluation to the agreement of 5 workers per video and a total of 28 unique workers. The results are shown below and VideoDirectorGPT still has a much higher human preference than ModelScopeT2V.

Q3 Open-sourced LLM Ablation:

We conducted an ablation study of using the open-sourced LLama2-13B-Chat model as well as GPT3.5-Turbo in the video planning stage on MSR-VTT dataset. Here we use LLama2 in a zero-shot evaluation setting without fine-tuning on layout-annotated videos. As we can see from the following table, LLama2 achieves worse scores on all three metrics (FID, FVD, and CLIPSIM), which shows that GPT-3.5/4 can generate more accurate layouts. Another point to notice is that GPT-3.5-turbo and GPT-4 achieve very close performance on MSR-VTT. One explanation for this is that MSR-VTT prompts don’t need strong layout control.

To test this, we conduct another comparison of GPT-3.5-Turbo and GPT-4 on the ActionBench-Direction dataset we proposed in our paper. With $\alpha$=0.2, GPT-4 achieves around 10% higher accuracy than GPT-3.5-turbo in our movement direction metric. When we use LLM-dynamic-$\alpha$, such a gap increases to more than 20%. This indicates that (1) on datasets that need strong layout control, GPT-4 has a better performance compared with GPT-3.5-Turbo, and (2) GPT-4 is better at estimating the importance of spatial layouts from input prompts compared with GPT-3.5-Turbo.

W1 How does unclip prior affect video quality and image-text alignment?:

We’d like to first briefly remind the reason for using unCLIP prior. During training, we can obtain image embedding of entity representation by bounding box crop of the entities in the target image. During inference, since we don’t have access to the target image to crop from, we employ the unCLIP prior to obtain CLIP image embeddings from the entity text descriptions (see appendix C2).

In Figure 6, we show that the image embedding from unCLIP prior can effectively represent its original text description (‘white cat’). In our single- and multi-scene video generation results (Tables 2 and 3), we have shown qualitatively that our VideoDirectorGPT achieves comparable or better scores in video quality (FID/FVD) and video-text alignment (CLIPSIM).

W2 What if the representations are not shared:

We believe this is a misunderstanding of our method. If an entity (for example, a cat) should maintain visual consistency across two different scenes, then we use the same CLIP image+text embeddings to represent the cat when constructing grounding tokens. Instead, if the cats in scene1 and scene2 are different, then we use different CLIP image+text embeddings to represent them respectively.

W4 Only finetuning gated-self attention layer:

Thanks for pointing this out. In the Guided 2D Attention module, only the Gated Self-Attention layers are trainable, while all other layers (self-attention and cross-attention layers) are kept frozen.

W5 Does the baseline ModelScopeT2V use the same dataset as the papers uses for fine-tuning?:

We would like to remind the reviewer that all the experiments on single- and multi-scene video generation are based on zero-shot. Both our VideoDirectorGPT and the benchmark models (including ModelScopeT2V) are not specifically fine-tuned for the benchmark datasets.

In addition, our video generation module, Layout2Vid, is built upon the ModelScopeT2V backbone with additional Gated Self-Attention layers. Only these gated self-attention layers are trained, while all parameters in the original ModelScopeT2V backbone are kept frozen. The dataset we used for training is the same as the one used in GLIGEN, which contains only image-level annotations without any video-level data (see Sec. 3.2) .

W.A-2 / Q.1 Pororo-SV dataset details:

As mentioned in the main paper section 4 and appendix D.3, we modified the text descriptions of the Pororo-SV to create Coref-SV. We replace character names with basic animals and then duplicate instances of character names with a co-reference. For example, if the description is “Pororo got out of bed. Pororo went to the kitchen to get a snack.” then the Coref-SV version of the prompt would be “Dog got out of bed. He went to the kitchen to get a snack.” We do not use an LLM to complete this. We do not modify the videos of Pororo-SV either. The purpose of doing this is to measure whether or not a model can maintain the visual consistency of a character across scenes without explicitly being told that they are the same object. This exemplifies the benefit of using an LLM planner to create “video plans” as the LLM is able to understand this coreference and map it to the same consistency grouping across scenes, whereas existing models like ModelscopeT2V would get confused.

W.A-3 Ablation of using LLM-expanded prompts in ModelScopeT2V:

Following your suggestion, we experiment with generating videos with ModelScopeT2V using the same scene descriptions from our video plans (produced by GPT-4), on the HiREST dataset. As we can see from the following table, ModelScopeT2V with LLM-expanded prompts (2nd row) achieves better performance than the original ModelsScopeT2V (1st row), showing the effectiveness of our video plans. However, it still performs worse than our VideoDirectorGPT (3rd row), also showing the effectiveness of our Layout2Vid.

W.A-4 CLIP Variance based consistency metric:

As per your suggestion, we compute variance of the CLIP features across all scenes. Results are shown in the table below (lower is better). As you can see, our model still outperforms ModelscopeT2V, even when we give it the advantage of the GT co-references.

W.A.5 Not enough participants for human evaluation:

As mentioned in Sec. 4, the human evaluation presented in the paper used three crowd-sourced workers from Amazon Mechanical Turk to evaluate each video. The workers were not given any information about which model generated each video (randomly shuffled) to ensure there was no bias towards either model. While we took the agreement of three workers per video, we had 19 unique workers to complete the human evaluation.

To address the concern that there were not enough participants, we extended the human evaluation to the agreement of 5 workers per video and a total of 28 unique workers. The results are shown below and VideoDirectorGPT still has a much higher human preference than ModelScopeT2V.

W.B-1 Objects not exactly following bounding boxes:

In our video generation module Layout2Vid, the strength of layout control is determined by the hyper-parameter $\alpha$, which represents the number of denoising steps we activate the Gated Self-Attention layers. High $\alpha$ value yields stronger layout control, with the cost of reducing visual quality, which is also observed in other T2I generation works [1, 2, 3]. We found that an $\alpha$ value between 0.1-0.3 usually gives the best performance, and we set it as 0.1 (which is equivalent to 5 denoising steps) as the default value (as shown in our ablation study of $alpha$ hyper-parameter in Sec. 5.3 of our paper). Therefore, the bounding boxes in our frameworks are more about providing layouts and motion guidance, rather than strict restriction of object locations.

[1] Li et al., GLIGEN: Open-Set Grounded Text-to-Image Generation (CVPR 2023)

[2] Cho et al., Visual Programming for Text-to-Image Generation and Evaluation (NeurIPS 2023)

[3] Lian et al., LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models