FreqPrior: Improving Video Diffusion Models with Frequency Filtering Gaussian Noise

We thank the reviewer for the constructive feedback. We will address the remaining questions below.

---

Results on different iterations

What would the results be if both FreeInit and FreqPrior were implemented with 4 extra iterations? Would FreqPrior still outperform FreeInit?

We have conducted experiments in which both FreeInit and our method are implemented with 4 extra iterations. The reaults are presented in the following tables.

As shown in the tables, our method consistently outperforms FreeInit with the setting of 4 extra iterations, highlighting the superiority of our approach. In our paper, we opted for 2 extra iterations to balance computational time with performance improvements, as we found that this setting provides a good trade-off.

---

Experiments on more recent diffusion models

If high-quality T2V models are available, making low-frequency matching unnecessary, would this method still be effective?

We conducted experiments on OpenSora, sampling videos with 16 frames for evaluation. The results are presented in the table below:

As shown in the table, both FreeInit and our method improve performance, demonstrating the effectiveness of low-frequency matching.

Thanks again for reviewer's time and efforts for reviewing our paper and providing insightful comment. Our responses to the reviewer’s concerns are below:

why the performance of FreeInit decreases while the performance of the proposed method improves?

Due to the variance decay issue in FreeInit, excessive iterations can degrade imaging details and motion dynamics, negatively affecting overall quality. While FreeInit enhances low-frequency information with each iteration, the issue of variance decay persists. With four iterations (i.e., adding two additional iterations), the negative impact of variance decay outweighs the benefits of enhanced low-frequency information, leading to a slight decrease in scores.

In contrast, our method effectively addresses the variance decay issue through our novel frequency filtering approach. As shown in Table 1, the covariance error of noise prior refined by our method is less than $10^{-16}$, making it negligible. The addition of two extra iterations further enhances the low-frequency information, improving the consistency of the generated video and resulting in a increase in scores.

why the performance gain seems relatively incremental for the high-quality T2V model?

The incremental performance gain of our method (with high-quality T2V model OpenSora) over FreeInit could be partly ascribed to the difference of the network structure:

(i) The T2V model OpenSora is based on DiT, which pathifies the latent into a sequence before passing it through the network. In contrast, UNet does not pathify the latent in this manner. This pathification can make it more challenging for the model to effectively capture different frequency information.

(ii) Additionally, UNet could be more sensitive to varying frequency information. As FreeU [1] highlights: 'The main backbone of the U-Net primarily contributes to denoising, whereas the skip connections introduce high-frequency features into the decoder module.' DiT-based OpenSora, on the other hand, may be less sensitive to high-frequency information.

Above two factors could explain why the performance gain in OpenSora appears to be incremental compared to FreeInit.

Hope our responses clarify above thoughtful questions, and it is very much appreciated if the reviewer can kindly check our responses and provide feedback with further questions/concerns (if any). We would be more than happy to address them. Thank you!

[1] Si, Chenyang, et al. "FreeU: Free lunch in diffusion u-net." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

We thank the reviewer for the constructive feedback. We will address the remaining questions below.

---

Improvements over FreeInit are marginal

According to Table 2, the quantitative improvements over FreeIniit is also marginal.

On the total score, our method achieves improvements over FreeInit by 0.68, 0.70, and 0.48 on AnimateDiff, ModelScope, and VideoCrafter, respectively, resulting in an average improvement of 0.62 on VBench—a significant gain. The improvements of our method over FreeInit are even more pronounced in the semantic aspects. Furthermore, FreeInit achieves a total score of 77.43, which is worse than the Gaussian noise baseline (77.54), whereas our method surpasses both.

Considering both the metrics and inference time (save 23% compared to FreeInit), our method demonstrates significant improvements over FreeInit.

---

Experiments on more recent diffusion models

What is the performance on more recent ODE-based diffusion models?

We conducted experiments on OpenSora, sampling videos with 16 frames for evaluation. The results are presented in the table below:

Our method achieves the highest scores across all metrics, highlighting its effectiveness.

---

Evidence for over smoothed results

The paper argues that the variance-decreasing problem in FreeInit causes it to generate over-smoothed results. But the provided evidence is weak.

In addition to the example in Figure 1, further qualitative results in Figures 4 and Figure 8 demonstrate that FreeInit tends to generate over-smoothed outputs. A representative case is shown in the top-right corner of Figure 8, where FreeInit tends to 'simplify' video frames, lacking complex image details. This issue arises due to the variance decay problem, which is caused by a lack of high-frequency information in FreeInit.

Moreover, beyond the lack of imaging details, FreeInit also tends to generate videos with reduced motion dynamics. The results of motion dynamics are presented in the following table.

As shown in the table, FreeInit reduces motion dynamics. This supports our statement, as a lack of motion dynamics can be interpreted as over-smoothing in the temporal dimension.

Dear Reviewer QxjB,

We sincerely appreciate the reviewer's time for reviewing, and we really want to have a further discussion with the reviewer to see if our response solves the concerns. We have addressed all the thoughtful questions raised by the reviewer (the performance of our method) and we hope that our work's impact and results are better highlighted with our responses. It would be great if the reviewer can kindly check our responses and provide feedback with further questions/concerns (if any). We would be more than happy to address them. Thank you!

Best wishes,

Authors

We thank the reviewer for the constructive feedback. We will address the remaining questions below.

---

VBench

My first question is the experimental analysis, why only Vbench is provided?

We chose VBench for evaluation due to its distinct advantages. Specifically, there are several key reasons:

1. VBench divides the evaluation into two main components: video quality and video-condition consistency, offering a comprehensive and hierarchical evaluation framework.
2. VBench desinged compact yet representative prompts in terms of both evaluation and content categories.
3. VBench has conducted experiments, validating its evaluation results align with human perception.

---

Generation prompts & Standard Deviation

does not provide the detailed prompts used for video generation. Have you measured the standard deviation for your generated videos with different seeds? It contains lots of randomness in video generation.

As mentioned above, VBench includes a carefully designed prompt suite, which comprises 946 prompts in total.

For evaluation, VBench requires generating 5 different videos per prompt. As noted in our paper, we generated 4730 videos ($946\times5=4730$) for each method, with the random seed initialized to 42.

Generating multiple videos per prompt helps mitigate randomness and reduce the standard deviation, resulting in more reliable evaluation results.

---

More comparisons

It is suggested to include more comparisons in terms of FID, FVD, etc. Whether the conclusion will stand under these metrics.

We have conducted more comparisons with metrics IS and FVD on UCF101 dataset. The results of Inception Score are shown in the following table.

A higher IS value means better generation quality. Our method performs the best across these three base models.

The results of Fréchet Video Distance are shown in the following table.

A lower FVD value indicates better performance. Our method performs well on ModelScope and VideoCrafter; however, it does not enhance the generation quality on AnimateDiff, nor does FreeInit. FVD is calculated by comparing the distribution of generated videos to that of the ground truth videos. However, for a single prompt, the generated video can vary significantly from the ground truth, even if both are aligned with the text prompt. FreeInit and our method may alter the video content (as illustrated in the last row of Figure 4 in our paper), leading to differing values. In contrast, FVD is more suitable for evaluating image-conditioned video generation and less appropriate for text-only conditioned video generation.

---

User study

Since the video quality measurement for AIGCs is not absolutely reliable, providing a user study for video generation is essential.

To address this concern, we conducted a user study by randomly selecting 36 different cases generated using VBench prompts. Each base model (AnimateDiff, ModelScope, and VideoCrafter) contributed 12 cases, with each case including 3 videos generated by Gaussian noise, FreeInit, and our method. We collected feedbacks from 25 participants, who were asked to vote for each case based on two dimensions: video quality and text-video alignment.

As shown in the table, our method outperforms both baseline method and FreeInit across three different base models in terms of both video quality and text-video alignment.

Dear Reviewer PCyZ,

We sincerely appreciate the reviewer's time for reviewing, and we really want to have a further discussion with the reviewer to see if our response solves the concerns. We have addressed all the thoughtful questions raised by the reviewer (user study, details of equation(7), more evaluation results, and the explanation of worse motion dynamics existed in FreeInit) and we hope that our work's impact and results are better highlighted with our responses. It would be great if the reviewer can kindly check our responses and provide feedback with further questions/concerns (if any). We would be more than happy to address them. Thank you!

Best wishes,

Authors

We thank the reviewer for the constructive feedback. We will address the remaining questions below.

---

Paper title

The title should explicitly mention "Video Diffusion Models" to clarify that the method is specifically designed for video generation.

Thank the reviewer for the valuable suggestion. We will revise the title to explicitly include 'Video Diffusion Models', ensuring it clearly reflects the focus on video generation in our method.

---

GPU memory usage

The paper should provide detailed measurements of GPU memory usage before and after applying the proposed FreqPrior method, particularly focusing on peak memory consumption.

We have measured the peak GPU memory usage before and after applying our proposed FreqPrior. We conducted the experiments on VideoCrafter. The results of peak GPU memory consumption are provided in the following table.

As the resolution increases, FreqPrior does lead to a slight increase in peak GPU memory usage. However, the additional memory consumption is minimal. Theoretically, the computational complexity of FFT is $O(n\log n)$, whereas that of attention is typically at least $O(n^2)$. Consequently, the majority of peak memory usage stems from the diffusion model and the inference period itself.

In summary, while FreqPrior slightly increases peak memory usage, the increase is negligible ---- less than 1% of the baseline methods' peak memory usage. Therefore, we conclude that GPU memory usage is not a concern with our proposed method.

---

Impacts of different Classifier-Free Guidance strengths

The paper should explore the impact of different Classifier-Free Guidance (CFG) strengths when using FreqPrior.

We conducted additional experiments to evaluate the impact of Classifier-Free Guidance (CFG) strength on AnimateDiff using FreqPrior. The results indicate that the total score is not significantly sensitive to changes in CFG strength. Specifically, across the range of 6.0 to 10.5, the total score fluctuates slightly between 78.01 and 78.20.