[W1] Scaling-up and FIFO-Diffusion

Thank you for your insightful comments. We agree that scaling up in deep learning is crucial, particularly for video generation. However, we argue that FIFO-Diffusion offers benefits even for scaled-up models as an universal inference technique, and may aid in scaling up dataset sizes.

First, FIFO-Diffusion is an inference method applicable to a wide range of video diffusion models regardless of architecture (e.g., DiT or UNet), as shown in Figure 6. This implies that any future scaled-up model with longer and advanced video generation capabilities would also benefit from our technique by generating scalable lengths of videos than their original capacity.

Furthermore, this expandability might reduce the burden of preparing datasets. With the scalability of FIFO-Diffusion during inference, those who want to generate extremely long videos can utilize much shorter video datasets. Given that the number of sufficiently long videos is much smaller than shorter ones, FIFO-Diffusion provides a valuable alternative for preparing highly long video datasets.

[W2, Q1] Controllability and Random Motion

We believe the results shown in Figure 6 (c) and Appendix E address these concerns. The multi-prompt results demonstrate the controllability and capability of our technique for generating long-term ordered motion using varying text prompts. Our method effectively maintains visual consistency by keeping sufficient forward frames in the queue, enabling seamless motion transitions across prompt changes. This approach adds the desired level of control and order to the generated videos, making them more suitable for practical applications.

Dear uUJo,

Thank you for your time and consideration on our paper. We are writing to follow up on the rebuttal we submitted for our paper in response to your initial review. If you have any additional comments or questions regarding our rebuttal, we would be grateful to hear them. Please let us know if there is anything else we can provide to assist in your review.

Warm regards,

Authors

We appreciate your comments. However, we are concerned about the rationale for lowering the score, which seems to be based on superficial and uninformed criteria: "limited novelty" and "less satisfactory visual results."

In our initial rebuttal, we addressed the concerns and questions regarding scalability and controllability, arguing that our technique is highly relevant to scalability and is controllable with multiple text prompts. However, it appears these points were not considered in the final assessment.

Furthermore, while we understand that different reviewers may have different backgrounds and perspectives, we cannot find any detailed explanations for “limited novelty” and “less satisfactory visual results”. This assessment conflicts with another reviewer’s evaluation, which includes a high score (8, strong accept) from reviewer bkyd, “The quality of the generated videos is good.” from reviewer Hj7n, and “The contributions of this work are innovative” from reviewer Tbwe. We believe that the assessment based on subjective criteria, such as visual quality, does not fully capture our contributions.

We kindly request a reconsideration of the assessment using objective and descriptive criteria.

[W1] Prior work: Relevance to Tedi and Rolling Diffusion Model

Thank you for bringing this to our attention. Since we became aware of Tedi [R5] and Rolling Diffusion Model [R1] after the ICML conference, which was subsequent to the submission, we gladly plan to cite [R5] and [R1], acknowledging its relevance to our work.

Both [R5] and [R1] aim to train diffusion models from scratch, while our focus is on achieving this without additional training. To this end, we exclusively introduce latent partitioning and lookahead denoising in our paper to realize a training-free approach. Particularly in the text-to-video (T2V) generation domain, we believe that training-free approaches are extremely valuable. T2V models require substantial computational resources (GPUs and storage) for preparing text-video pair datasets and large-scale training. By avoiding additional training, our method provides a more accessible and efficient solution for generating high-quality videos.

Additionally, we have demonstrated differences between [R1] and our work in terms of sliding window methodology and the target task in the third and fourth paragraphs of the global rebuttal. We kindly ask you to refer to the global rebuttal for further details.

[W2] Limitations: Beyond Gap Bridging

The benefits of both latent partitioning and lookahead denoising extend beyond addressing the training-inference gap. Lookahead denoising leverages the advantage of diagonal denoising, where noisier frames benefit from clearer frames. This benefit persists regardless of the gap due to the inherent similarity of neighboring video frames. As shown in Table 2, lookahead denoising enhances the baseline model's performance, suggesting its usefulness even without the gap.

While latent partitioning aims to reduce the training-inference gap, it also offers two additional significant advantages: facilitating parallelized inference on multiple GPUs and reducing discretization error, as mentioned in L170-176. Although the first advantage may diminish without the gap, the remaining two benefits, particularly parallelizable inference, are crucial in practice for reducing inference time, as evidenced in Table 1.

[W3] Evaluation: Quantitative Results

Thank you for your valuable comments. We believe that measuring both quantitative motion magnitude and FVD scores supports our claim in L454: “FVD is more favorable to algorithms that generate nearly static videos due to the feature similarity to the reference frames.”

In response, we measure optical flow magnitudes (average of optical flow magnitudes) to compare the amount of motion between 512 videos generated by FIFO-Diffusion and FreeNoise. We provide the result plot in the PDF file of our global comment. The plot (Fig. A) illustrates that over 65% of videos generated by FreeNoise are located in the first bin, indicating significantly less motion compared to FIFO-Diffusion. In contrast, our method generates videos with a broader range of motion.

Additionally, following the recommendation from Reviewer bkyd, we evaluate long video FVD (i.e. $FVD_{128}$) and IS on the UCF-101 dataset. We kindly encourage you to refer to response [W1] to Reviewer bkyd for detailed results.

Regarding the growing trend in the FVD, we argue that this does not imply instability in FIFO-Diffusion as we have discussed in L449-451. Since FIFO-Diffusion starts the process from a queue with corrupted latents, generated by the baseline model (the reference set for the FVD scores), it is natural for the FVD to increase rapidly in the first 16 windows.

[Q1] Latent partitioning: Window Size and Performance

We appreciate your insightful comments. We argue that the improved quality primarily stems from addressing the training/inference gap, while it might also be positively influenced by the increased window size. Although latent partitioning increases both the number of partitions ($n$) and the queue length ($nf$), the denoising process for each partition is conducted independently. This means that each frame only attends to temporal consistency within its partition at each step. Instead, increasing model capacity ($f$) would more directly enhance temporal consistency within the queue.

However, it is a valid point that the model might benefit from increasing $n$. Since the first-in, first-out mechanism mixes the partitions at every iteration, all frames in the queue implicitly interact with each other throughout the denoising process. This interaction could indeed contribute to maintaining overall consistency in the queue, as you suggested.

[Q2] Lookahead denoising: About Equation 7

We apologize if Equation 7 was unclear. We use these representations for coherence with the model output, $\epsilon_\theta$. Similar to training loss in general diffusion models, we directly add noise sampled from a normal distribution into the latents and set this added noise as the ground truth for the score. Thank you for pointing this out, we will revise the equation in the future version.

[R1] Ruhe et al., Rolling Diffusion Models, ICML, 2024

[R5] Zhang et al., Tedi: Temporally-entangled diffusion for long-term motion synthesis, arXiv, 2023.

Dear Reviewer Hj7n,

We are glad that our rebuttal has addressed your concerns. We appreciate for your thoughtful consideration and reviews.

Sincerely, Authors

[W1] Relevance to Rolling Diffusion Models

Thank you for bringing this to our attention. We gladly plan to cite Rolling Diffusion Models [R1], acknowledging its relevance to our work. Please understand that we became aware of Rolling Diffusion Models [R1] after the ICML conference, which was subsequent to the submission. We kindly ask you to refer to the global rebuttal.

[W2] Latent Partitioning and Lookahead Denoising

Thank you for your reviews, but we found it challenging to fully understand your concerns regarding the second weakness. Could you please provide more details or examples to help us address this effectively?

From our understanding, you may feel our focus on latent partitioning and lookahead denoising is disproportionate to their importance. We would like to clarify that these two methods are crucial for generating high-quality long videos, especially in a training-free manner. While diagonal denoising introduces the concept of long video generation, latent partitioning and lookahead denoising address its limitations and leverage its advantages. The seamless integration of these methods significantly enhances the quality of video generation.

[W3] Relevance to MultiDiffusion

First, please let us clarify that lookahead denoising is not relevant to “maintaining consistency across video partitions” as you summarized in the paper overview and the third weakness. Lookahead denoising aims to make all frames take advantage of earlier, clearer frames, leveraging characteristics of diagonal denoising. Our first-in, first-out strategy naturally mixes up the partitions, thereby inherently maintaining consistency between them.

While lookahead denoising might seem similar to strategies employed in MultiDiffusion [R4] due to the overlap, the two methods are fundamentally different in purpose and implementation. The overlap in [R4] is a strategy to maintain consistency of the independent outputs, resulting in the optimization process described in equation (3) of [R4]. In contrast, our method does not require joint alignment of the outputs. The overlap is simply a resultant phenomenon and does not necessitate any optimization process.

Although the suggested methods are not directly relevant, both papers address similar challenges by proposing methods to handle out-of-range input sizes within their respective domains. With this in mind, we plan to cite [R4] in the future version. Thank you for bringing this to our attention.

[W4] Experimental Explanation of Diagonal Denoising in Training-free Context

We would like to clarify that diagonal denoising is not primarily designed for "training-free" algorithms but rather for generating long videos with fixed memory allocation. The key components that facilitate the training-free context in our techniques are latent partitioning and lookahead denoising, which seamlessly integrate with diagonal denoising.

We have demonstrated the experimental performance of diagonal denoising alone in our ablation study, both quantitatively (Table 2 in Section 5.2) and qualitatively (Figures 20 and 21 in Appendix H). The results indicate that, in training-free contexts, diagonal denoising by itself predicts less accurate noise (Table 2) and generates less favorable videos (Figures 20 and 21) compared to the complete version of FIFO-Diffusion.

We did not focus solely on diagonal denoising in other experiments because we believe that the success of FIFO-Diffusion in a training-free context is due to the integration of latent partitioning and lookahead denoising.

[W5] Table 1 with quality metric

We have provided qualitative and quantitative comparisons between different FIFO settings in our ablation studies (Section 5.4 and Appendix H), particularly in L289-291. We also encourage you to visit our project page (Section F. Ablation Study) to view the generated video samples. Table 1 is intended to demonstrate the fixed memory allocation and parallelizability of our method. For your reference, we report $\text{FVD}_{128}$ and IS on UCF-101 dataset in Table A, following the recommendation from Reviewer bkyd.

Table A $FVD_{128}$ and IS on the UCF-101 dataset. We sample $256\times 256$ resolution 2048 videos utilizing the model from [R3] pretrained on the UCF-101 dataset, sampling by FIFO-Diffusion with and without lookahead denoising. LD denotes the inclusion of lookahead denoising.

[Q1] Initialization of FIFO-Diffusion

We initiate the FIFO-Diffusion process with corrupted latents generated by the baseline model, as discussed in L129-130 and L166-169 of our paper. To summarize, since latent partitioning requires $nf$ latents and the baseline model generates $f$ initial latents, we repeat the first frame of the initial latents to construct $nf$ latents.

FIFO-Diffusion can be initiated even without prior video latent. You can start directly with $nf$ random noises and use a varying timestep schedule. Specifically, for $n=1$, the timestep at the first iteration would be $\tau = [\tau_f, \ldots, \tau_f]$, as all latents are random noise. Subsequently, the timestep will be updated at each iteration, such as $[\tau_{f-1}, \ldots, \tau_{f-1}, \tau_f]$, then $[\tau_{f-2}, \ldots, \tau_{f-2}, \tau_{f-1}, \tau_f]$, and so on. You can decode the dequeued latent into a frame after the $f^\text{th}$ iteration. This strategy can be similarly expanded for $n>1$ by substituting $f$ with $nf$.

[R1] Ruhe et al., Rolling Diffusion Models, ICML, 2024.

[R3] Xin et al., Latte: Latent Diffusion Transformer for Video Generation, arXiv, 2024.

[R4] Omer et al., MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation, ICML, 2024.

We appreciate your dedicating effort to review and reply for our rebuttal.

However, we respectfully argue that the suggested framework to extend denoising steps does not mitigate the training-inference gap. To describe the reason, we will follow your suggestion step by step, sliding a denoising window every $n \geq 2$ denoising steps. We define the time step schedule as $0<\tau_0<\tau_1<\ldots<\tau_{nf}=T$.

1. Frame number $1 \sim f$ with diffusion time steps $[\tau_{nf-f+1}, \tau_{nf-f+2}, \ldots, \tau_{nf}]$ will be denoised $n$ times without sliding, reaching time steps of $[\tau_{nf-f+1-n}, \tau_{nf-f+2-n}, \ldots, \tau_{nf-n}]$.

2. Now we should slide the window, but we encounter two challenges. First, the foremost frame cannot be dequeued as it is not fully denoised. Second, adding a new random noise with the noise level of $\tau_{nf}$ results in an additional training-inference gap. Specifically, the queue now contains frames with time steps of $[\tau_{nf-f+2-n}, \tau_{nf-f+3-n}, \ldots, \tau_{nf-n},\tau_{nf}]$, where the last two frames have a time-step difference by $n$ while the rest has a gap of $1$. Moreover, the total noise level difference increases from $f-1$ to $f+n-2 > f-1$.

3. Assume that we repeat step 2 ignoring the aforementioned problems. Then, we will eventually have frames with time steps of $[\tau_{f}, \tau_{2f}, \ldots, \tau_{nf}]$ as a model input. The total time-step difference is now $(n-1)f$. In contrast, the time-step difference of latent partitioning is $f-1$ since frames with time steps of $[\tau_{kf+1}, \tau_{kf+2}, \ldots, \tau_{kf+f}]$ are model input.

Consequently, while your suggestion will increase denoising time steps, it cannot tackle the training-inference gap, which is latent partitioning’s motivation. Moreover, although your suggestion and latent partitioning require the same amount of computation, only our method allows parallelized inference on multiple GPUs.

Meanwhile, lookahead denoising is an independent component with latent partitioning, which can be applied solely to diagonal denoising. It clearly leverages the advantage of diagonal denoising that noisier frames are benefitted by clearer frames. We have demonstrated its effectiveness in ablation study (Section 5.4) both qualitatively and quantitatively. Moreover, we would like to clarify that all components in our paper have their own roles, and their necessities are well supported by theoretical and empirical analysis.

Finally, we want to clarify that we would rather consider our paper as concurrent work with [R1]. We have also submitted this paper in ICML 2024 with submission number 9782. Moreover, while there are some similarities, the two papers also have many differences as we described in the global response.

Dear reviewer Tbwe,

Thank you for your response to the rebuttal. We appreciate the time and effort you have put into your evaluation.

We understand that you had concerns regarding our initial rebuttal, and we have carefully addressed your points in our responses. We believe that this latest rebuttal discusses the issues you raised and provides further clarification on the matters you pointed out. We kindly ask you to review our updated rebuttal and share your thoughts.

Sincerely,

Authors

[W1] Long-video FVD Evaluation

We appreciate your invaluable recommendation. Including long-video evaluations on specific datasets would definitely strengthen our paper by evaluating general performance of our techniques for video generation.

In response, we have evaluated the class-conditional long video generation performance on the UCF-101 dataset by computing the Frechet Video Distance (FVD) and Inception Score (IS). Specifically, we computed $FVD_{128}$ and IS following the details and implementation outlined in [R2], utilizing the model from [R3] pretrained on the UCF-101 dataset. The results are shown in Table A.

Table A $FVD_{128}$ and IS on the UCF-101 dataset. We sample 128-frame, $256\times 256$ resolution 2048 videos utilizing the model from [R3] pretrained on the UCF-101 dataset, sampling by FIFO-Diffusion. We also compute the IS of [R3] with DDIM 64 steps. Note that we cannot compute $FVD_{128}$ of [R3], as [R3] generates 16-frame videos.

[W2] Readability

Thank you for your valuable suggestion, and we apologize for any confusion. We will add a brief explanation in the introduction to clarify the training-inference gap with diagonal denoising. This should help readers understand the rationale behind our approach earlier and avoid any misunderstanding. We appreciate your feedback and will make these adjustments to enhance readability.

[Q1] Relevance to Rolling Diffusion Models

Thank you for bringing this to our attention. We gladly plan to cite Rolling Diffusion Models [R1], acknowledging its relevance to our work, that [R1] and diagonal denoising process latents with different noise levels in a sliding window manner. Furthermore, we have demonstrated differences between the two approaches in terms of training burden, sliding window methodology, and the target task in the global rebuttal. We kindly ask you to refer to the official comments for further details.

[R1] Ruhe et al., Rolling Diffusion Models, ICML, 2024.

[R2] Ivan et al., StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2, CVPR, 2022.

[R3] Xin et al., Latte: Latent Diffusion Transformer for Video Generation, arXiv, 2024.

Dear reviewer bkyd,

We want to express our sincere gratitude for your positive feedback on our paper. Your thoughtful comments and insights have been incredibly valuable to us.

As our paper is currently under discussion, your continued support would be greatly appreciated. Your perspective is instrumental in highlighting the strengths of our work and advancing the discussion in a constructive direction.

Thank you once again for your support and consideration.

Sincerely,

Authors