FreeLong: Training-Free Long Video Generation with SpectralBlend Temporal Attention

My first concern is that 128 frames itself cannot be considered long video generation. Therefore, it would be more convincing to see how effective the authors' proposed method is on longer videos, such as generating 1-minute videos like Sora.

Thank you for your feedback regarding the length of the generated videos. Following other works such as [1,2,3], which consider videos longer than 10 seconds as "long," we chose 128 frames to speed up our experiments. We understand that 128 frames may not fully represent long video generation compared to examples like Sora. It is important to note that our FreeLong method is not limited to this length. To address your concern, we have conducted further experiments evaluating FreeLong on generating longer videos of 256, 384, and 512 frames (approximately 1 minute at 8 fps). These results, shown in Figure 1 of the submitted one-page PDF, indicate that FreeLong achieves robust longer video generation across these lengths.

[1] StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text

[2] A Survey on Long Video Generation: Challenges, Methods, and Prospects

[3] FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling

The proposed method includes several hyperparameters, such as alpha and tau. Are these parameters significant in influencing the final results?

We appreciate your question regarding the significance of hyperparameters such as $\alpha$ and $\tau$. We set $\alpha$ to match the pre-trained video length. To explore the impact of $\tau$, we conducted quantitative ablation studies, as shown in the table below:

Bigger $\tau$ means more steps to blend global and local features, the table shows that our method keeps enhancing the consistency when the steps increase and saturated at step 40.

Because the authors tested only 25 prompts, it's unclear whether these hyperparameters remain suitable across a broader range of scenarios.

Regarding the number of test prompts, as described in Lines 178-179, we randomly selected 25 prompts from each of the 8 test categories in VBench, resulting in a total of 200 prompts to ensure a fair comparison. This approach ensures a fair and comprehensive comparison across all categories. All test prompts are illustrated in the supplementary materials.

Although adding local features to long-range features can mitigate some loss of spatial details, for longer video generation with significant content variation, it remains uncertain whether these local features adequately supplement spatial details.

Thank you for pointing out the concern regarding the adequacy of spatial detail supplementation. We use the MUSIQ model, which evaluates over-exposure, noise, and blur of image, to validate image quality of the generated longer videos. The results are as follows:

Our method achieves comparable image quality to 16-frame videos, and significantly outperforms other methods. More qualitative examples on longer videos and additional base models are provided in Figure 1,2 in the one-page pdf.

Figure 2's right panel demonstrates strong diagonal correlations in VideoCrafter at 64-frame and 128-frame intervals. How does this correlate with the findings proposed by the authors?

Thank you for your valuable question. VideoCrafter2 is pre-trained on varied video lengths and is relatively robust to longer video generation than LaVie. To better understand the temporal relationships in temporal attention, we set the diagonal elements, such as (0,0), (1,1) to 0, which makes visualization of relationships between different frames more effective. As shown in Figure 4 of one-page pdf, the attention map reveals less structured attention patterns. This approach helps explain the long-range temporal attention of VideoCrafter2.

how to demonstrate that the results obtained reflect the algorithm's performance in long video generation quality.

Thank you for your valuable suggestions. VBench is a comprehensive evaluation benchmark for video diffusion models, with evaluation tools not constrained by video length, such as CLIP similarity, DINO similarity, and Temporal Flickering. Since no long video benchmark was available before the paper submission, we used this benchmark. Per your suggestion, we compared our method on the VBench-Long [4], designed for Sora-like video evaluation, as shown in the table below. It is evident that our method achieves consistent performance improvement compared to other methods.

[4] https://github.com/Vchitect/VBench/tree/master/vbench2_beta_long

Thank you for your valuable feedback. We have carefully considered your comments and suggestions to improve our paper. Below are our responses to each of your concerns:

The related work section is not reader-friendly for those unfamiliar with the field

We apologize for the lack of clarity in the related work section. We understand that it may be challenging for readers unfamiliar with the field. To address this, we will revise this section to provide a more comprehensive overview of the research landscape. We will include a structured summary that highlights the key developments and contributions in video diffusion models, making it more accessible and informative for all readers.

Lack of comparisons and baselines

We appreciate your insightful feedback regarding the need for more comparisons and baselines. Initially, we chose LaVie and VideoCrafter2 due to their state-of-the-art performance in video generation. However, we recognize the importance of demonstrating the generalizability of our proposed method, FreeLong, across different base models.

In response, we have extended our experiments to include additional models, such as OpenSORA, AnimateDiff, and ModelScope. Figure 2 in the one-page pdf presents these results, showing that FreeLong performs well across these diverse models. This broader evaluation demonstrates that FreeLong is not limited to a specific positional encoding and can be effectively applied to various video diffusion models.

The visual result quality is limited and does not include more motions

We understand your concern about the visual result quality and the limited motion content in our generated videos. For single-prompt inputs, we focus on maintaining consistency a cross long sequences, while for multi-prompt inputs, we aim to facilitate more complex and accurate motions.

As illustrated in Figure 3 of the one-page pdf, our method achieves more natural variations compared to FreeNoise. FreeNoise often generates repetitive content due to repeated noise initialization to maintain temporal consistency. In contrast, FreeLong utilizes global temporal attention to explicitly maintain temporal consistency without sacrificing motion variety.

FreeLong still do not have rich details

Thank you for pointing out the need for richer high-frequency details, particularly in the background. FreeLong aims to improve both consistency and fidelity in long video generation. To validate spatial details of generated videos, we used the MUSIQ model from VBench to evaluate image quality~(e.g. over-exposure, noise, blur). The results are shown in the table below:

Our method achieves comparable image quality to 16-frame videos, demonstrating improved spatial detail. More qualitative examples on longer videos and additional base models are provided in Figure 1,2 in the one-page pdf.

Memory Cost

We have calculated the computational cost, including inference time and memory usage, as shown in the table below:

Compared to the direct application method, our FreeLong method slightly increases both memory cost and inference time to achieve consistent and high-fidelity long video generation. FreeLong can generate 128-frame videos using a single NVIDIA RTX 4090 GPU, ensuring feasibility for practical use. When compared to FreeNoise, FreeLong reduces the inference time per step from 2.6s to 2.2s while increasing memory usage. This highlights FreeLong's efficiency in generating longer videos, as it effectively balances inference time with memory cost. By optimizing these trade-offs, FreeLong demonstrates its capability to generate longer videos efficiently.

Thank you for your thorough review and constructive feedback. We greatly appreciate your thoughtful comments and are pleased that our response has addressed most of your concerns. We will highlight the expansion on other models in future versions. Regarding the demonstration of FreeLong's effectiveness in generating complex motions over longer sequences:

Due to the 50 MB limit of the one-page PDF, we could only provide a few images for each example. While these images demonstrate the generalizability of our approach, they may not fully capture the complexity of the motions FreeLong can produce. Videos offer more coherence and motion variation than images. We would like to highlight that, compared to previous methods like FreeNoise, FreeLong achieves consistent long video generation with more natural motions, as shown in Figure 3 of the one-page PDF.

In future versions, we will include more detailed examples to showcase FreeLong's capability to handle complex motions in longer sequences effectively.

Thank you once again for your valuable insights, which have been instrumental in enhancing our work.

what is the computational cost compared to previous method (like freenoise) ?

Based on your feedback, we conducted a comparison of inference times between our method, FreeLong, and other methods, including FreeNoise, the sliding window method, and direct application. The results are summarized in the table below:

Our method, FreeLong, utilizes both global and local attention streams, performing a single-pass temporal attention forward operation. In contrast, previous methods like FreeNoise rely on sliding window temporal attentions, which require multiple rounds of temporal attention and thus increase inference time. Our results demonstrate that FreeLong achieves faster inference time.

Can freelong solve the repetitive generation problem?

Regarding the repetitive generation problem observed in method FreeNoise, it is primarily due to their reliance on repetitive noise initialization. Our proposed method, FreeLong, overcomes this issue by explicitly capturing global consistency during the temporal attention process. This enables FreeLong to generate consistent longer videos without depending on repetitive initial noises.

By effectively capturing global consistency, FreeLong produces more diverse and coherent video sequences, which significantly reduces the repetitive generation problem. Figure 3 in the submitted one-page PDF provides a comparison of the results between FreeLong and FreeNoise, demonstrating our approach's effectiveness in reducing repetitive generation while maintaining high-quality video generation results.

Inference time comparisons

Thank you for your question. Following your advice, we conducted a comparison of inference times. As shown in the table below, our method achieves faster inference speeds than multi-pass methods, such as FreeNoise.

User study

Thank you for your valuable suggestions. We conducted a user study to evaluate temporal consistency, fidelity, and overall rankings. Ten participants from academia and industry took part in this study. The results are as follows:

Our method, FreeLong, outperformed other methods across all evaluated criteria.

Longer Videos

We have provided longer videos, including results with 256, 384, and 512 frames, in the one-page PDF. These results demonstrate the generalization ability of FreeLong to handle longer videos.