The Blessing of Smooth Initialization for Video Diffusion Models

• While the authors present theoretical analyses of their proposed methods, the theorems as stated lack clarity and rigor with clear and correct notations and the proof processes are also not clear with some errors, leading to questions about their accuracy and validity.

  • Theorem 3.1 is not properly stated with clear defined notations. For example, $\mathcal{S}$ is a variable related with H? $\dot{\epsilon}$ is a derivative with which variable? The relation betwen $H, k, T$, etc. It is not easy to read. Besides, in Appendix C.2, the process of how to simplify the continued product $\prod$ in (12) to keep only the main terms is not so clear. Why can just neglect the cross terms？
  • In Theorem 3.2, what is the meaning of dx/dt? It refers to the diffusion ODE functions? Seems not. Besides, in its proof, the eq. 22 integral with respect to t, where is the t? If x is related to t, then why $q_1(x_t)$and $q_{1-\eta}(x_t)$? These equations are so confusing and it even contains the basic mathematical errors.
  • Same unclear notations in the remaining theorems.

• Overhead problem. In the appendix, the authors states its overhead on the inference is less substantial compared to FREEINIT and UNICTRL but didn't provide the quantitative comparison with other methods and baseline. Besides, it should be a tradeoff between the inference overhead and quality improved. Showing this tradeoff through qualitative is important for a training-free method to validate its usuability in real applications.

• The main concern for this paper is that smooth and fast-smooth techniques are mainly for better estimation of $f_s$ where $f_s$ is an approximation DDIM-Inversion(DDIM$(\cdot))$. However, DDIM-Inversion(DDIM$(\cdot))$ alone does not guarantee that the intended future semantic information is correctly introduced into the initial noise. While DDIM-Inversion(DDIM(·)) can inject semantic information into the initial noise, it is unclear how it ensures that this information is meaningful or beneficial.

• Analysis Considering the Properties of Video Diffusion Models: While the proposed framework is grounded in theoretical motivations, these observations are not explicitly tailored to video diffusion models. Although this generalizability may be advantageous, more discussion is needed to explain how the proposed initialization specifically enhances sampling in the video context. For instance, FreeInit emphasizes temporal consistency in videos, noting that low-frequency components of the initial noise dictate the spatio-temporal layout of generated videos, and aims to refine these components.

• Gap Between Theory and Practice: Related to the above point, there are likely differences in applying SmoothInit to image versus video diffusion models. DDIM inversion often performs more worse in In video diffusion models then image diffusion models, potentially violating the linear approximation assumptions of the score function. Additionally, CFG scales are typically larger in video diffusion models, leading to more pronounced off-manifold noise artifacts, as shown in CFG++ [1]. Furthermore, reducing the number of inference steps is more complex for video generation compared to image generation [2], which may be linked to the failure of DDIM inversion and the curved trajectory issue. Thus, while SmoothInit in an ideal continuous setting may nudge the initial noise toward regions containing semantically rich information, in video diffusion models with time-discretized setting this process might introduce additional discrepancies, exacerbating off-manifold noise artifacts and deviating from theoretical predictions.

• Qualitative Results: The quality of the provided videos appears suboptimal, suffering from reduced motion dynamics (e.g., ModelScope S30 6), color saturation, and flickering artifacts (e.g., AnimateDiff F30 3). More comprehensive video visualizations comparing the proposed method with baselines like FreeInit would be helpful. Publishing these visualizations on an anonymous project page would further enhance accessibility and comparison.

• Quantitative Results: While the forward-inversion process improves results to some extent, its performance is only quantiatively comparable to FreeInit without smoothing. With smoothing, it outperforms other baselines, but the approach shares similarities with FreeInit, which diminishes its novelty. Additionally, it may still face the trade-off between motion dynamics and temporal consistency, a limitation also observed in FreeInit.

References

• [1] Chung, Hyungjin, et al. "CFG++: Manifold-constrained Classifier Free Guidance for Diffusion Models." arXiv preprint arXiv:2406.08070 (2024).
• [2] Polyak, Adam, et al. "Movie Gen: A Cast of Media Foundation Models." arXiv preprint arXiv:2410.13720 (2024).

There are also various weaknesses.

• The paper relies on dense math in several places, which is sometimes hard to follow. For instance, the smoothing theory will not be known to most readers and I think most readers, myself included, will not be able to intuitively understand highly technical theorems like Theorem 3.3. I would suggest the authors to explain and present the intuitions here (as well as in the smoothing paragraph on page 3) better. The concrete value of these theorems remains somewhat unclear. This should be explained better and supported with more intuitions and maybe examples.
• Somewhat related, I believe the authors should better explain some obvious questions around the proposed method. Why is it that generation is so sensitive to the initial noise in video models, but less so in image models? What is special about video generation? Algorithmically, it's the same process. This should be discussed.
• When optimizing the noise with the proposed methods, how does the latent evolve? Do noise values for different frames become more similar? Does this make the initialization maybe better? This should be analyzed and discussed, too (and could give a hint why the situation is different for video models, compared to image model). The paper essentially proposes a new method that seems to empirically work but does not sufficiently analyze the why and how.
• The paper should also better put the work in context with respect to related work, in particular FreeInit (I am aware of section D in the appendix, but this is just a high-level summary of related work). FreeInit refines the spatial-temporal low-frequency components of the initial latent. The proposed methods here instead rely on a smoothing framework. It is known that smoothing also corresponds to a removal of low frequency information. Hence, is there maybe a deeper relation here between the methods? Does the smoothing in this paper somehow adjust low frequency components of the initial noise, too? I am just speculating here, but I think these relations and questions should could be discussed and maybe analyzed more.
• Experimentally, some of the results are somewhat incremental and some results are so close together that it remains unclear how statistically significant they are. This could be addressed by estimating errors and also reporting error bars.
• How are the y-axes of Figure 6, bottom, created? Are these L2 distances, all independently normalized to be in [0,1] for each plot? It is not clear how comparable these plots are. Related, in section 4.4, the authors write ''Specifically, at the endpoint of the sampling process (timestep = 1 in Fig.6), the video generated by Fast-SmoothInit tends to be more uniformly distributed within a sphere, whereas SmoothInit, Forward-Inversion, and Origin tend to be distributed along a spherical shell''. This is not clear, unless I am misunderstanding something. Also for SmoothInit, trajectories spread almost the entire [0,1] interval at t = 1. This should be discussed better.

Conclusion: While the paper proposes a novel method and shows favourable empirical results, there remain many questions. Therefore, I don't think the paper meets the bar for acceptance in its current form, but I would be open to raising my score if my concerns and questions can be addressed.