MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequences

Thank you for your valuable feedback and suggestions. To clarify our approach, we first provide a brief summary of our video generation process:

1. The autoregressive model predicts keyframe tokens.
2. The DAE decoder decodes the keyframe tokens into keyframe images.
3. Leveraging existing image-to-video methods to generate video clips for each keyframe, resulting in a long video.

W1&W2. Questrions about DAE.

Thank you for pointing out the confusion. DAE refers to the Image Renderer Training in Fig. 2. Our DAE training is conducted in three stages:

1. In the first stage, we train on natural images (768x768) using Eq. 1 as the loss function.
2. In the second stage, we train on movie images (896x512) using Eq. 1 as the loss function.
3. In the third stage, we further incorporate face embedding and description embedding and train the model on movie images (896x512). The training loss for this stage becomes : $\mathcal{L}\_{DAE} = \mathbb{E}\_{\mathbf{x\_0}, \epsilon \sim \mathcal{N}(0, \mathbf{I})} \left\| \epsilon - \mathcal{D} \left( \mathbf{z\_t}, t, \mathcal{E}(\mathbf{x\_0}), \mathbf{f}, \mathbf{d} \right) \right\|\_2^2,$ where $\mathbf{f}$ and $\mathbf{d}$ is the face embedding and description embedding, respectively.

We apologize for not clarifying this point. We will refine the figure and the explanation in the revised paper. Further details can be found in Appendix A.

W3. Random masking.

The random masking strategy is used only during training. It is applied in both the DAE and autoregressive model training. In the third stage of DAE training, we input concatenated image tokens, face embeddings, and description embeddings. With a probability of 0.15, we randomly replace each token with a zero token. During autoregressive model training, we randomly prevent the model from attending to masked tokens with a probability of 0.15 by applying attention mask. This strategy helps the model learn to infer missing content from the available information, improving its performance and reducing overfitting. During inference, random masking is not used.

W4. High dropout rate.

All parameters in the VLM are updated. We didn't try only tuning part of the model. We conducted detailed experiments and found that a high dropout rate is indeed beneficial in our setting.

W5. Questions about image-to-video model.

• i2v (image-to-video) is not our primary focus; the emphasis is on ultra-long generation. i2v is just a tool we use. The core of i2v lies in generating high-quality short videos from images, whereas the core of our work is to generate coherent and consistent keyframes, and then use i2v to produce long videos. We choose to use SVD as our i2v method by default, as it is widely adopted in many existing works. However, it tends to produce results with limited motion. Therefore, this is an issue with SVD itself, rather than a limitation of our approach.

• Our main contribution is in exploring ultra-long video generation. We have emphasized in the paper that our method can be combined with any image-to-video model, and we demonstrate in the supplementary video that using other, more advanced i2v models can produce higher-quality, action-rich results. Therefore, we respectfully believe that the performance of our method would not be affected by any particular i2v approach. On the contrary, the performance of our method is determined by the upper bound of the entire i2v field.

W6. The frame numbers of full-length video.

• Currently, our model supports generating up to 128 keyframes at once. The total number of frames in the full-length video depends on the specific image-to-video method used. When employing the SVD-based approach outlined in the paper, the maximum number of video frames is 6400.

• However, if the storylines can be seamlessly connected between each long video, the long videos we generate can also be combined to form even longer coherent videos. Therefore, to some extent, the number of frames in our full-length video could be unlimited.

W7. Gaps that sometimes occur and solution.

• First, the target ID in Fig. 8 is not meant to generate an exact match with the target ID, but rather to ensure that the generated character can be recognized as the same person as the target ID. We believe we have successfully achieved this goal.

• In our opinion, the gap that sometimes occurs is due to our data not reaching the model's capacity. We are confident that scaling up the model with more data will address this issue.

W8. Training and inference cost.

• Training cost has been discussed in Appendix A. For DAE, it takes 3 weeks with 6 NVIDIA A800 GPUs. For Autoregressive model, it takes 3 days with 4 NVIDIA H100 GPUs.
• For inference, both the 24GB and 80GB GPUs are capable of running the model. With the 80GB GPU, generating 128 keyframes takes approximately 3 minutes. This is because, during inference, all keyframe tokens are first generated, and then they are batched together for rendering into images. On the 24GB GPU, inference takes about 6 to 7 minutes, as it requires some off-the-shelf memory-efficient libraries, and during image rendering, smaller batch sizes are needed due to memory constraints.

W9. Metrics on spatial-temporal quality.

We have considered using metrics such as KVD, FVD to evaluate the spatio-temporal quality of videos. However, as discussed earlier regarding I2V in W5, we believe that video-related metrics primarily assess the image-to-video model's ability to generate short videos, rather than evaluating the core contribution of our work: ultra-long video generation. By taking both Table 1 and Table 2 into account, we believe that they already provide a comprehensive evaluation on long video level. Therefore, we respectfully believe that spatio-temporal metrics do not provide significant help in evaluating our core contribution.

We sincerely thank you for your time and thoughtful feedback. We are happy to provide any further clarifications or engage in additional discussions to address any remaining concerns!

We sincerely appreciate your concerns and are delighted to have the opportunity to engage in this discussion!

---

1. Regarding Fig. 8 and the associated issues

We believe there might be some misunderstanding here.

• The results in Fig. 8 are the [predicted keyframes] at intervals of a few minutes from the autoregressive model, rather than a few seconds of continuous video generation.
• From our experiments, we found that the data we collected has not fully reached the capacity of the autoregressive model. Therefore, we believe scaling up the autoregressive model can address this issue. This problem is unrelated to the i2v model.

Of course, we have tried using different i2v models to generate results (including CogVideoX). We compared the video results generated by various methods and ultimately chose to showcase LUMA (see supplementary materials) because it produces the best results. Replacing the original SVD with LUMA leads to significant improvements. Similarly, replacing SVD with CogVideoX-5B also improves the results. This highlights the advantage we have repeatedly emphasized: the quality of our videos depends on the state-of-the-art methods in the i2v field.

However, the problem you mentioned in Fig. 8 stems from the limitations of the autoregressive model used for generating keyframes, which still has potential for improvement because the data we collected does not fully reach the model's capacity. It is independent of the i2v methods. Thus, using different i2v methods cannot solve this problem. And we are already focused on gathering more refined data to enhance the performance.

---

2. Regarding the evaluation with spatial-temporal metrics

We understand your concern, but we believe there may be some misunderstanding here as well.

As we have repeatedly emphasized, our method can be combined with any existing i2v methods. Therefore, when evaluated with spatial-temporal metrics, it is actually the i2v methods being assessed. For instance:

• If we use SVD, then VBench evaluates SVD.
• If we use CogVideoX-5B, then VBench evaluates CogVideoX-5B.
• If we use Gen2 or PIKA, then VBench evaluates Gen2 or PIKA.

This is why we believe these spatial-temporal metrics are more focused on evaluating the i2v methods themselves, rather than our key contributions.

As for consistency, we have already provided evaluations of short-term and long-term consistency in Table 1 and explained in detail how these metrics are computed in Appendix B. We believe this adequately evaluates consistency.

Finally, if you still believe that evaluating our method using metrics like VBench is necessary, we are more than happy to make every effort to provide such results. :)

---

Thank you again for your feedback! We consider this discussion very enjoyable and valuable! Please do not hesitate to share further concerns, and we would be very delighted to address them!

Dear Reviewer,

As soon as we received your previous comment, we immediately began using VBench to evaluate our method.

Although our perspectives may differ, we deeply respect your emphasis on the importance of metrics like those in VBench. We have made every effort to further evaluate our method using the i2v-related metrics from VBench. :)

If you have any additional concerns, please do not hesitate to let us know! :)

We sincerely thank you for your thoughtful feedback and valuable suggestions!

W1. Typo of kd means and variances.

Thank you for pointing out the typo. The GMM parameters should be kd means and kd variances. We apologize for any confusion this may have caused.

W2. Experiments with Eq.2 only.

• We conduct additional experiments where only Eq. 2 is used as the objective function. The results indicate that relying solely on Eq. 2 causes the model to generate characters positioned near the image edges and, in some cases, even fail to generate the characters, which results in a significant decline in quality. While previous works have not specifically discussed the necessity of incorporating both L1 and L2, our experiments demonstrate that these two terms complement Eq. 2, resulting in higher-quality results. In the appendix E.3 of the revised paper, we provide visualizations of the experimental results along with metrics for a detailed evaluation.

W3. Confusion in Fig 2.

We sincerely appreciate your valuable suggestions and feedback. You are correct that the multimodal script in the top-left corner is used to generate a single keyframe. We will include clearer illustrations in the revised paper to enhance understanding.

Q1. Is covariance matrix diagonal?

Yes, the covariance matrix is diagonal.

Q2. What is the episode's visual tokens?

We apologize for the confusion. In this section, we randomly select 10 keyframes from the current video segment, encode them into visual tokens, and position them at the beginning of the input sequence. The term "episode's visual tokens" refers to these encoded keyframe tokens.

Q3. Reference images in Fig 8, and how do these images help?

In Fig. 8, we select 10 keyframes containing characters as references. These references improve the results because they serve as the reference example for in-context learning. More specifically, by providing some examples as context, the model can leverage these reference contexts to produce higher-quality outputs.

Once again, we thank you for your constructive feedback! Please let us know if you have further suggestions or concerns!

Thank you for your valuable feedback and suggestions!

ID-consistency Evaluation

We have already provided ID-related evaluation in the paper, as seen in Table 1, with the ST and LT metrics. These are used to evaluate consistency in short-term and long-term, respectively. Detailed explanation for these two metrics is provided in Appendix B. Please let us know if you have any further questions.

Boundaries of Our Method

In Appendix F, we analyze the boundaries of the proposed method. Here, we summarize them as follows:

• When too many characters appear simultaneously, the model may sometimes confuse character IDs. (Figure 22 of the original paper, or Figure 23 of the revised paper)
• Since we perform sentence-level tokenization to ensure the resulting length, the model struggles to understand highly fine-grained and complex text prompts.
• Our method does not control object-level IDs. This can be addressed by further constructing object-level IDs.

Technical Novelty and Contributions

Our method is not a trivial combination of existing approaches. Our motivation and intuition behind the design are fundamentally different from those of existing methods. We believe our technical novelty and contributions are as follows:

• The problem we address is new. We solve an under-explored problem: unified long video generation and long story generation. To the best of our knowledge, few efforts have tackled this. Existing approaches are unable to generate videos of the length we achieve, and most are constrained by single scenes and lack controllability. Additionally, their results demonstrate significant drift over extended periods.
• The motivation behind our autoregression + diffusion approach is completely different from prior works using autoregressive + diffusion models.
  • Previous video generation methods mostly rely on diffusion models. While these methods achieve good results on short videos of a few seconds, they face fundamental challenges when generating long videos, especially those in the tens of minutes. Scaling up video diffusion model requires enormous computational resources and data, which is nearly impossible for ultra-long video tasks.
  • In the image domain, previous works about autoregression + diffusion mainly focus on the instruction-following ability. However, our focus is on the autoregressive model's ability to model long sequences. Our innovative framework inherits the advantages of both diffusion and autoregressive models, achieving state-of-the-art long video generation for videos of tens of minutes.
• Our approach unifies story generation and video generation in a hierarchical way.
• Our method is orthogonal to existing video generation works. It can be integrated with various strong image-to-video models to produce higher-quality results.
• Ensuring Length: We design compact yet representative compressed tokens to ensure both quality and context length. We further implement sentence-level tokenization, ensuring the generated results achieve the desired length despite the limitations of maximum sequence length.
• Preserving ID: We implement an ID-preserving strategy to ensure the consistency of multiple characters.
• Ensuring Controllability: We propose a multimodal script and innovatively implement sentence-level compression for text descriptions, ensuring both quality and extended context length. And we propose the corresponding data pipeline.
• Supervision: Traditional autoregressive models are supervised with cross-entropy loss, but this is not feasible for our task, as we need to predict continuous tokens rather than discrete ones. Additionally, we observe poor performance when supervised solely with L1 and L2 loss. To address this, we supervise continuous real-valued token prediction, which effectively improves the model's performance.
• We further explore the potential of our method, including few-shot in-context learning and implicit modeling of shot types.
• New Metrics: We propose new ST and LT metrics to evaluate ID consistency from both short-term and long-term perspectives.

In conclusion, our approach is grounded in a thorough and careful analysis of the strengths and weaknesses of autoregression and diffusion in video generation, rather than simply combining existing methods. The task we address is both novel and of significant practical relevance. Furthermore, we have explored various aspects of our method, such as shot types, which further demonstrates its value and potential.

Once again, we are grateful for your thoughtful feedback! We hope these clarifications effectively address your concerns and we welcome any further discussion. Thank you for your time and attention.

Thanks for your kind review on our paper!

Non-human Character Results

Thank you for your valuable feedback. As discussed in our Future Work, our ability to maintain ID consistency for non-human characters is limited. For example, when generating a dog, the results often differ from the specific target dog we aimed for. Therefore, our method currently excels when the main character is human.

However, we have included this limitation in our future work. We believe that constructing and leveraging non-human IDs will effectively address this issue.

Adaptation Cost and Cherry-pick

During implementation, we observed that the LLM adapts well to sentence CLIP embedding inputs with minimal adaptation cost. We believe this is because:

1. Both the CLIP text encoder and LLaMA handle rich semantic features. CLIP sentence embeddings are highly semantic, and understanding these semantic embeddings is easy for LLaMA because of its strong language capabilities.
2. The structured multi-modal scripts further assist the model in effectively understanding the sentence embeddings.

Our approach achieves a high success rate with minimal need for cherry-picking. Most of results are generated directly without selection.

We hope these responses address your concerns. Please feel free to let us know if you have any further questions!

Dear reviewers,

We sincerely appreciate your valuable efforts in reviewing our paper. As the Author-Reviewer discussion period is nearing its conclusion, we would like to kindly inquire if the reviewers have any additional questions or concerns regarding our previous responses. Please feel free to reach out if there are any remaining points that need clarification.

Thank you!