ControlVideo: Training-free Controllable Text-to-video Generation

Dear reviewer, thanks for your insightful comments. We address the comments below and will include them in revision.

[W1]: The proposed components are more-or-less explored in recent works.

Our key insights lie in enhancing temporal consistency and continuity of videos, including three folds:

(i) First of all, we introduce interleaved-frame smoother to alternately stabilize the video latents during sampling, and show that adding random noise produces more realistic videos ([DDPM results], refer to [Q3 of Reviewer XMqd] for details)

(ii) The hierarchical sampler achieves efficient long-video generation in commodity GPUs (See [long-video results]).

(iii) We also empirically demonstrate the superior consistency and quality of fully cross-frame interaction (See [cross-frame attention ablations]).

[W2]: More robust metric for temporal consistency.

Following [1], we employ the optical-flow based warping error to robustly evaluate temporal continuity, where the optical flows comes from source videos. The following table reports the warping error comparisons of different methods. From the table, ControlVideo achieves a significantly lower warping error than other baselines, demonstrating its superiority in enhancing continuity. In addition, one can see that Canny-based models achieve lower warping error than depth-based ones, as Canny edges have more similar structure with optical flow.

[W3 & Q1]: The potential of training-free framework in video generation.

Albeit training on large-corpus videos can help improve the video consistency, there are three factors limiting further research: (i) Extremely high requirements for massive computational resources, which usually are unavailable for researchers from universities. (ii) Absence of sufficient video-caption datasets. Despite requiring modeling more complex video distributions, existing public video datasets only have limited number of videos compared to image datasets, e.g., WebVid-10M vs. LAION-5B. (iii) Lower frame quality than image generative models. Existing video datasets usually contain low-quality factors (e.g., blur and watermarks), and training on them may lead to visibly lower frame quality than the image counterparts.

Due to limited computation resources, we investigate the effectiveness of proposed modules under the "training-free" setting, and demonstrate superior performance than some finetuning based methods, i.e., Tune-A-Video and Follow-Your-Pose. Since "training-free" ControlVideo has shown promisingly temporal consistency and continuity, it is natural to take it as a well-initialized model for the following finetuning. Compared to training from the scratch, it has two potential advantages: (i) Reduce the requirements for massive computation resources and data. The well-initialized model only needs to learn fine-grained motion pattern from videos. (ii) Alleviate low-quality effects from training videos. Requiring less finetuning in video dataset, ControlVideo may better preserve frame quality from the image counterparts.

Dear reviewer, thanks for your insightful comments. We hope our below replies could address the comments, and will include them in revision.

[W1]: Average innovation.

Albeit applying ControlNet to video generation is straightforward, our key insights lie in improving the temporal consistency and continuity, including three folds:

(i) First of all, we introduce interleaved-frame smoother to alternately stabilize the video latents during sampling, and show that adding random noise produces more realistic videos ([DDPM results], refer to [Q3 of Reviewer XMqd] for details)

(ii) The hierarchical sampler achieves efficient long-video generation in commodity GPUs (See [long-video results]).

(iii) We also empirically demonstrate the superior consistency and quality of fully cross-frame interaction (See [cross-frame attention ablations]).

[W2 & Q2]: Comparisons with more baselines.

Apart from original 30 examples, we have added 38 new examples to [qualitative results], including another two baselines of video editing (FateZero[1] and Vid2Vid-Zero[2]). From [qualitative results], ControlVideo generates videos with better consistency and continuity than other approaches. In contrast, FateZero may fail to align with text prompts, e.g., Iron man in [human-pose results], while Vid2Vid-zero produces videos with more visible flickers.

[W3 & Q1]: Failure cases analysis.

We provide two additional failure cases in [failure-case results]. As one can see, when input text prompts (e.g., rabbit) seriously conflict with input motion (e.g., cow), the synthesized videos usually tend to align with input motion, ignoring the implicit structure in text prompts. To increase the ratio of text prompts over structure, we decrease the scale of ControlNet $\lambda$ to 0.3 ($\lambda=1$ by default). In [failure-case results], it can be seen that $\lambda=0.3$ achieves a better trade-off between two input conditions than $\lambda=1$. In the future, we will explore how to adaptively modify input motions according to text prompts, so that users can create more vivid videos.

[1] Chenyang Qi, et al. FateZero: Fusing Attentions for Zero-shot Text-based Video Editing. In ICCV 2023.

[2] Wen Wang, et al. Zero-Shot Video Editing Using Off-The-Shelf Image Diffusion Models. Arxiv 2023.

Dear reviewer, thanks for your insightful comments.We provide our replies to the comments below, and hope they could address your concerns.

[W1]: Temporal consistency to be improved and unchanged background.

(i) Temporal inconsistency. For some videos containing flickers, there are two potential reasons: (a) The input motion contains visible flickers and may result in the flickers in synthesized videos. (b) Sometimes, the scene correspondence in motions cannot be well maintained through cross-frame attention only, leading to incoherence in appearance. For (a), we can leverage video-based annotator[1] or smoothing filters[2] to obtain more stable motion sequence. For (b), one may first estimate the correspondence of motions with[3,4], and then recalibrate the attention weights based on the correspondence.

(ii) Unchanged background. In such cases, the background part of input motion is unchanged, and its synthesized background will also be static via DDIM sampling. As shown in [DDPM results], when ControlVideo employs non-deterministic DDPM sampling, it produces more vivid videos with lively background. Kindly refer to [Q3 of Reviewer XMqd] for more details.

[W2]: Consider all frames as a large image and select some key frames

(i) For original self attention in Stable Diffusion (SD), both its queries and keys come from the same image. Similarly, fully cross-frame attention computes queries and keys with the same frames, and may better inherit high-quality and consistent generation than other cross-frame mechanisms. (ii) We conduct ablation experiments on different number $k$ of key frames, and the results are given in the following table and [keyframes results]. With the increasing of $k$, ControlVideo obtains better performance in temporal consistency, i.e., higher frame consistency and lower warping error. Also, larger $k$ visibly reduces frame incoherence in appearance, e.g., the orientation of elephant.

[W3]: Apply smoother at different timesteps.

The following table and [smoother-timesteps results] show ablation experiments in different timesteps (refer to Appendix C for more details). As one can observe, applying the smoother at timesteps {30,31} effectively deflickers the video while ensuring its frame quality. In contrast, using the smoother at timesteps {48,49} or timesteps {0,1} lead to slight flickering or distortion, degrading the effectiveness of smoother.

[W4]: More computational cost of smoother.

From above table and [cross-frame attention and smoother], the proposed smoother greatly improves temporal continuity of synthesized videos, and the extra 0.5 min can be seen as acceptable.

[Q1]: Why does the hierarchical sampler improve model efficiency? The hierarchical sampler is designed to improve efficiency of fully cross-frame attention, especially in producing long videos. Given $N$ frames, $K$ key frames and $T$ tokens for each frame, the complexity for fully attention is $O(T^2 \cdot N^2)$, while that of hierarchical sampler is $O(T^2 \cdot (K^2 + \frac{N^2}{K}))=O(T^2 \cdot (N + N^{\frac{3}{2}}))$ ($K=\sqrt{N}$ by default). When producing a video of 100 frames, the fully attention requires about 50 mins while the hierarchical sampler only takes about 10 mins.

Dear reviewer, thanks for your insightful comments. We provide our replies to the comments below, and hope they could address your concerns.

[W1]: Increasing computational time.

The following table and [cross-frame attention and smoother] show ablation experiments of the fully cross-frame attention and interleaved-frame smoother. As one can see, the proposed two modules significantly improve both video consistency and continuity, with acceptable increase in computational time.

[W2 & Q2]: Flickering background issue.

For some cases with flickering background, there are two potential reasons: (a) The input motion contains visible flickers and may bring flickering controls to synthesized background. (b) Sometimes, the scene correspondence in motions cannot be well maintained through cross-frame attention only, producing videos with incoherent appearance. For (a), we can leverage video-based annotator[3] or smoothing filters [4] to obtain more stable motion sequence. For (b), we first estimate the correspondence of motions with [5,6], and then recalibrate the attention weights based on the correspondence.

[W3]: Quantitative comparisons with Text2Video-Zero in pose conditions.

In terms of pose condition, we compare our ControlVideo with Text2Video-Zero in the test set of Fashion-Text2Video dataset[1], which contains 380 video-prompt pairs in total. From the following table, ControlVideo is superior to Text2Video-Zero by frame consistency, while with marginal gains in prompt consistency.

[Q1]: Additional results for long video generation.

In addition to the original two long videos, we have added another three ones to [long-video results]. Benefiting from our hierarchical sampler, it only takes $\sim$ 10 minutes to generate a video with 100 frames in one NVIDIA RTX 2080Ti.

[Q3]: Non-deterministic DDPM-style sampler.

Yes, our ControlVideo can also employ a non-deterministic DDPM-style sampler during inference. Following Eq.12 in DDIM[2], one can predict $z_{t-1}$ from $z_t$ via (i.e., line 10 of Alg.1 in paper):

$z\_{t-1} = \sqrt{\alpha\_{t-1}} \tilde{z}\_{t\rightarrow 0} + \sqrt{1 - \alpha\_{t-1} - \sigma\_t^2} \cdot \epsilon\_\theta(z_t, t, c, \tau) + \sigma\_t \epsilon_t$.

where $\epsilon\_t$ is random Gaussian noise and $\sigma\_t = \lambda \cdot \sqrt{(1 - \alpha\_{t-1})/(1 - \alpha\_{t})}\sqrt{1 - \alpha\_{t}/\alpha\_{t-1}}$ controls the level of random noise.

[DDPM results] presents the generated videos of ControlVideo at different noise levels. Notably, as the noise level increases, ControlVideo generates more photo-realistic videos with dynamic details, e.g., ripples in the water.

[1] Yuming Jiang, et al. Text2Performer: Text-Driven Human Video Generation. In ICCV 2023.

[2] Jiaming Song, et al. Denoising Diffusion Implicit Models. In ICLR 2021.

[3] Xuan Luo, et al. Consistent Video Depth Estimation. In SIGGRAPH 2020.

[4] Press W.H, et al. Savitzky-Golay smoothing filters. In Computers in Physics.

[5] David Lowe, et al. Distinctive image features from scale-invariant keypoints. In IJCV 2004.

[6] Luming Tang, et al. Emergent Correspondence from Image Diffusion. In NeurIPS 2023.