Optimal Noise Pursuit for Augmenting Text-to-Video Generation

• Obtaining a reference video through video retrieval and generating videos using the inversion noise from the reference video could result in a lack of diversity in video generation. Even though the diversity of individual frame content can be enhanced through LLMs, the crucial diversity of motion in text-to-video is constrained by inversion noise.
• Temporal coherence is a crucial aspect of video generation. However, the evaluation metrics such as FID, CLIP_FID, and CLIP-SMI are not related to the temporal coherence of videos, and the supplementary materials also do not provide the generated videos. Authors are suggested to provide the retrieved reference videos and their corresponding generated videos, along with an evaluation of their temporal coherence.
• The employed text-to-video model has been trained on WebVid-10M, and reference videos are retrieved from WebVid-10M. What happens if the retrieved video is not from WebVid-10M? Because the model has been trained on WebVid-10M, reference videos from WebVid-10M naturally invert noise that conforms to the data distribution and domain. This gives the model an inherent advantage in terms of image generation quality, and as a result, the comparison may not be entirely fair.
• Based on my understanding, the method proposed in the paper focuses on improving the quality of single-frame generation and does not have additional designs specifically aimed at enhancing temporal coherence. Therefore, it would be more appropriate to compare it with methods used to improve generation quality in text-to-image diffusion models.
• The paper lacks substantial technological contributions and seems to be primarily an integration of existing methods.
  • The inversion of a reference video is used by video editing models, such as [tune a video]， [video p2p]，[Fate Zero], and so on.
  • The weighted combination of random noise and reference noise is used in video generation techniques, as seen in various methods such as [Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models] and [Free-Bloom].
  • Utilizing LLMs to enhance the content of video generation is not a new approach and has already been explored by previous works such as [MovieFactory] and [DirecT2V].

1. As also mentioned by the authors, the computational overhead and storage needs currently limits the method’s future potential to be adopted to applications.
2. One thing I was wondering is that - - Currently the reference text-video pool consists of only real-world videos. But one may want to generate “unreal” videos using prompts like “airplane flaps its wings like a bird”. How can the retrieval process searches reasonable matches in these cases? How the performance will change in such situations?
3. Will the ONA module still be effective if the retrieval recall rate is under certain threshold? The authors may study how the performance will change if the retrieval module randomly select reference video.
4. The novelty of SPR module is limited. As also presented in related works, previous works also points out the importance of prompt rewriting.

The paper introduces two distinct methods—Optimal Noise Approximator (ONA) and Semantic-Preserving Rewriter (SPR). The two ideas, although contributing to the same overall objective, appear to be laid out in parallel, without a significant integration or interconnection between them. They are presented as separate entities.

The proposed improvements are applied to only two video generation methods, one of which is a closed-source method that the authors have in-house and we don't have many details about it. The paper would benefit for applying the proposed method to additional text-to-video generation methods to strengthen the contribution that the proposed methods are model-agnostic.

User studies assessing the quality of the generated videos could supplement automated metrics.

The idea of using a LLM for improving the text prompt it's not entirely novel and have been used for various other works, especially in the text-to-image generation [1,2].

Method is only compared against the baselines: one in-house baseline and ModelScope/VideoFussion. Qualitative or quantitative comparison with other video generation methods [3,4 etc] is lacking.

[1] Lian, Long, et al. "LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models." arXiv preprint arXiv:2305.13655 (2023).

[2] https://cdn.openai.com/papers/DALL_E_3_System_Card.pdf

[3] Wu, Jay Zhangjie, et al. "Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[4] Singer, Uriel, et al. "Make-a-video: Text-to-video generation without text-video data." arXiv preprint arXiv:2209.14792 (2022).

1. The authors claimed that the unsatisfiring quality of generated results could be random input noises. However, I didn't find enough analysis and discussion in the paper to prove such claims. Why can random noise not always generate high-quality results? It is because the learned T2V model is not good enough?
2. How to maintain the creativity of the original text-to-video model if we retrieve a noise? Will such operation reduce the diversity and generalizability of the original model?
3. I didn't find any videos in the supplementary material. It is hard to tell wether the generated quality especially temporal consistency is improved from still images. I suggest the authors provide some demo videos.
4. Using retrieval* to improve the quality of input noise is not novel, nor using LLM to improve the input prompt. I hope the authors could clarify this paper's contributions

• Blattmann et al., Semi-Parametric Neural Image Synthesis, NeurIPS 2022