ControlVideo: Conditional Control for Text-driven Video Editing and Beyond

As this paper follows the framework of ControlNet that introduces the idea of providing a visual condition (e.g., edge map) to guide editing, merely adding a condition in the video editing system (This idea is already presented in the paper ControlNet) does not give much insight. I am sorry to think that there is no depth to the challenge that this paper is facing.

The current presented studies in this paper seem like applications of ControlNet in the video domain. if the proposed method is the first work that performs an application for video editing, then it must be published and shared in our societies, but this is not the case.

Authors may claim the contribution about proposed attention, but to the best of my knowledge, Prompt-to-Prompt proposes a more general form of attention blending, it seems like following that work. To be specific, The h = h_{u} + \lambda h_{c} seems like one of the attention blending methods, where prompt-to-prompt has already presented these types of blending with a more generalized version. (attention replace, attention refine, attention re-weighting)

Unfortunately, the further proposed long video editing is also not related to the focus of the proposed method. (I mean that is a good issue but not convincing for this paper. I am hoping to meet deeper studies on it as a single paper to tackle that issue.)

What readers may want to read in this paper should be examples as given below: (1) What was challenging to adding visual condition in video diffusion model and how to solve (2) What is the proper condition for applying video editing system. (3) The correlation between visual condition and fidelity.

• Limited novelty: The paper heavily relies on previous research for almost every key component of the entire editing pipeline. Building upon previous research is intrinsic to the nature of research, but the "addition" should be both clear and significant. Unfortunately, in this paper's case, I find the contributions to be marginal. The concepts of conditioning on previous signals, addressing inconsistencies in conditioning signals, and employing temporal attentions have all been previously explored, and this paper does not provide any substantial updates in these areas. Instead, it largely borrows these ideas from the domain of image generation and applies them to video, which is not a novel approach. For example, Blattmann et al., in "Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models," employed a very similar approach (i.e. temporal attention) to enhance temporal consistency on top of Stable Diffusion.

• Missing implementation details: The paper lacks essential implementation details, which could hinder its reproducibility and diminish its impact on future research. Important details, such as the size of additional parameters in LoRA, specifics of temporal attentions, fine-tuning procedures, including optimizers and data, are absent. This omission also raises concerns about the reliability of the comparisons in Figure 5, as the models may differ significantly in terms of size.

• Lack of videos: Surprisingly, for a paper focusing on videos, there is a notable absence of video content. I strongly encourage the authors to present their results, along with comparisons to other models, on an anonymous website for the benefit of both reviewers and readers.

1. The novelty seems limited. The use of the first frame for key-frame attention has been previously applied in Text2Video-Zero[1], while Make-A-Video[2] has explored temporal attention. The integration of them is a little bit trivial.
2. For long video editing, some existing methods should be compared, such as the hierarchical sampler in NÜWA-XL[3].
3. The results of Video-P2P in Figures 4 and 5 appear wired, especially for text alignment. How about the distribution of the test data, e.g., proportions of object editing, background editing, and style editing?

[1] Khachatryan, Levon, et al. "Text2video-zero: Text-to-image diffusion models are zero-shot video generators." arXiv preprint arXiv:2303.13439 (2023).
[2] Singer, Uriel, et al. "Make-a-video: Text-to-video generation without text-video data." arXiv preprint arXiv:2209.14792 (2022).
[3] Yin, Shengming, et al. "NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation." arXiv preprint arXiv:2303.12346 (2023).

1. I think the author should polish the Related Work section (e.g., “Diffusion Models for Text-driven Video Editing.”) and the experimental section. Although the authors mention Gen-1, one key emphasis of Gen-1 is utilizing additional controls, which is omitted in the section. Moreover, there are other works targeting similar goals like VideoComposer, which shares a lot of similar functionalities and should be compared with in the experiments. The authors are encouraged to enrich this part in order to objectively formulate their contributions.

2. Concerning temporal attention and key-frame attention, the authors’ contributions are rather limited although many empirical studies have been conducted to finalize the best design choice.

3. From my perspective, long video editing is a significant part of this part. However, in the experimental section, the authors did not provide much visualization and in-depth analysis for this. I suggest putting more emphasis on this part.