Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition

We deeply appreciate your insightful comments and efforts in reviewing our manuscript. We mark our major revision in the revised manuscript with “blue.” We respond to each of your comments one-by-one in what follows.

[W1] Experiment comparison setting: Adjust frame sampling stride with LVDM.

As mentioned in Appendix B, we followed the setup of VideoLDM [1] to show our method can generate videos with smooth motions. Nevertheless, we agree that adjusting FPS with LVDM and ModelScope will provide a more fair comparison, and we re-trained CMD with an adjusted frame sampling rate that you suggested and reported the FVD score in the Table below. As shown in this table, our method still shows a better FVD than baselines.

[1] Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models, CVPR 2023

[W2] Quantitative evaluation of autoencoder.

Thanks for your constructive comments. Besides the quantitative evaluation of our autoencoder using reconstruction FVD in Table 4(b), we additionally evaluate our autoencoder with the metrics you suggested, SSIM and PSNR, and report them in the Table below. While our autoencoder (CMD) has a 9.14$\times$ smaller dimension in the latent space of encoding videos, it still shows reasonable reconstruction quality.

[W3] Training cost in Appendix seems not considerably lower than LVDM and ModelScope.

There are two factors that make a direct comparison in terms of the total training GPU hours difficult. First, to our best knowledge, we could not find the exact total GPU hours for training LVDM and ModelScope. If the reviewer can provide a reference on that, we will be happy to perform a comparison and add the comparison and incorporate that in the manuscript. Second, the training cost of CMD in Appendix is measured with video clips of resolution 512$\times$1024 with a length 16, which is significantly higher-dimensional than the data preprocessed in LVDM and ModelScope (which is 256$\times$256 with a length 16). Hence, if the reported training cost of CMD is similar to (or slightly lower than) LVDM or ModelScope, it implies CMD can be trained much more efficiently than baselines.

Given the above reasons, we instead measured the training cost of our method and baselines in the same normalized setup (i.e., batch size 1 and a single forward pass with the same resolution). As shown in Figure 5, CMD has much lower training cost than other baselines in our evaluation setting.

[W4] Visual definition problem: Better high-resolution visualization?

To address your concern, we visualized each video with a single text prompt and saved them to different video files (in total 28 videos). We attached them to the supplementary material.

[C1] There are other video diffusion models that utilizes pretrained text-to-image diffusion models.

As you mentioned, several works use pretrained text-to-image for text-to-video generation. However, they need to add some temporal layers inside the text-to-image models to deal with a sequence of image-wise latent vectors, which is both memory and computation inefficient. Our approach is different: CMD directly fine-tunes the text-to-image model without adding any additional parameters inside the text-to-image model, enabling efficient training and inference. Nevertheless, we also agree that the current abstract might mislead readers and thus we revised it accordingly.

[C2] Add a number to equations.

Thanks for your constructive comments. We now added equation numbers in the revised manuscript.

[C3] Training time for the content frame diffusion generation is now shown.

Thank you for pointing this out. We now added training time for content frame diffusion in Appendix B.3.

We deeply appreciate your insightful comments and efforts in reviewing our manuscript. We mark our major revision in the revised manuscript with “blue.” We respond to each of your comments one-by-one in what follows.

[W1] Constrained by computational resources, the auto-encoder can only handle a limited number of frames at one time.

As you mentioned, there exists a limit length to an autoencoder that can encode videos as succinct latent vectors at once. Nevertheless, we believe that it does help in exploring generations of longer videos. Specifically, for longer videos, one can consider dividing it into multiple video clips of a fixed length, encode each clip as latent vectors using our autoencoder, and then consider modeling these multiple latent vectors. Such a strategy has also been widely adopted in latent video generative modeling literature [1,2]. While we did not try this due to the lack of resources, we strongly believe our method can be similarly extended for longer videos.

[1] Villegas et al., Phenaki: Variable Length Video Generation From Open Domain Textual Description, ICLR 2022.
[2] Yu et al., Video Probabilistic Diffusion Models in Projected Latent Space, CVPR 2023.

[W2] Content code as a weight sum of the original frame may limit the representing ability.

As you stated, content frames alone as a weighted sum of the original frame might limit representation capability. However, at the same time, this also constrains the space of content code to easily exploit pretrained image diffusion models. Also, recall that we also included "motion latents" in the autoencoder design, which can compensate for the representing ability of content frames. Because of this, our encoding scheme with this content code design can encode video with a high quality even if the scene change is dramatic, as empirically shown in Table 4a and 4b. Exploring a better design of content code and motion latent vectors should be an interesting future work.

[Q1] What is the resolution of the generated videos in Table 1, 2, and 3?

All of the video frames in Figure 1 and 2 have 512$\times$1024 resolution.

We deeply appreciate your insightful comments and efforts in reviewing our manuscript. We mark our major revision in the revised manuscript with “blue.” We respond to each of your comments one-by-one in what follows.

[W1] Performance of CMD when the number of frames increases?

We fully agree that it would be an exciting future direction to extend our framework for longer video generation, and we strongly believe CMD has a strong potential for such a goal. First, we provided the performance of an autoencoder with a longer length of 24 in Table 4. It still shows a reasonable reconstruction performance, and thus, we think our CMD can be extended for longer video clips. For extremely long videos (e.g., a length of thousands), our method can be extended as well by following the strategy in PVDM [1] that divides a long video into multiple short video clips, encodes each video clip into succinct latents, and then perform clip-by-clip generation conditioned on a previous clip.

[1] Yu et al., Video Probabilistic Diffusion Models in Projected Latent Space, CVPR 2023.

[W2, Q1] Confidence interval of the method.

Thank you for your constructive comments. We added a standard deviation of CMD in Table 1 in the revised manuscript. We also added a standard deviation of baselines if it is reported from references. For Table 2 and 3, it is difficult to add confidence intervals of baselines, because most baselines have not reported them in their quantitative evaluation. Nevertheless, we fully agree that at least providing confidence intervals of CMD in Table 2 and 3 will help future readers and research, and we will add them to the final manuscript. Please understand it was difficult to add in the revision due to the limited resources we have.

[Q2] More intuition behind 2D-projection-based encoding for the motion latents.

Our motivation is based on recent single-video encoding [2] and video dataset encoding [1] via 2D projections. This is possible as videos are temporally coherent 3D arrays with common contents (e.g., background or moving objects) and thus can be parameterized succinctly by optimizing 2D planes across each axis to capture them. Accordingly, we use triplane latent space to encode video datasets using projections along with a popular video encoder/decoder. This is also related to the recent 3D generation method [3] that encodes 3D voxels succinctly with 2D triplanes.

[2] Kim et al., Scalable Neural Video Representations with Learnable Positional Features, NeuriPS 2022.
[3] Chan et al., EG3D: Efficient 3D Geometry-aware Generative Adversarial Networks, CVPR 2022.

[Q3] Single bar for the CMD with three stacked parts for a fair efficiency comparison (Figure 5)?

We did not report the stacked values for Figure 5, since training of three components do not need to be done simultaneously. Specifically, we do not need to allocate memory for three models at once since each model can be trained separately, so we just provided the component-wise memory and computation consumption.

Nevertheless, we agree that for computation efficiency (FLOPs and sec/step) in training, stacking all components of CMD can help readers understand the computation efficiency of entire pipelines in handling a single video batch. Our method has 1.25 TFLOPs and 0.360 sec/step, which is still much better than 6.28 TFLOPs and 0.456 sec/step of LVDM.

[Q4] How were the baselines chosen in Figure 5?

As mentioned in Section 4.3, we chose text-to-video diffusion models as baselines if their implementation and checkpoints are publicly available.

[Q5] Plan for releasing an implementation.

We are planning to release the implementation.

[C1] Editorial comment.

Thanks for your careful reading. We fixed the typo in the revised manuscript.

We deeply appreciate your insightful comments and efforts in reviewing our manuscript. We mark our major revision in the revised manuscript with “blue.” We respond to each of your comments one-by-one in what follows.

[W1] Idea of decomposing video generation into two factors has been proposed in previous methods

We agree that previous works have shared a similar idea of decomposing videos into two factors. However, the key difference between CMD and them is in the purpose of decomposition. In this work, we explored a new video decomposition that (a) exploits pretrained image diffusion models for video generation (b) in computation- and memory-efficient manner. To achieve this, we introduced an encoding scheme in which one of the factors (content frame) lies in the image space (for (a)), and the remaining part (motion latent vectors) becomes a low-dimensional vector, and thus we train an additional lightweight diffusion model to predict these motion latents vectors (for (b)).

Unlike our method, existing approaches have not considered (a) and (b) simultaneously. Specifically, [1] exploits pretrained image diffusion models, but it deals with videos as high-dimensional cubic tensors directly. Moreover, [2, 3] do not use pretrained image diffusion models, and [3] uses flow for motion encoding, which is a high-dimensional cubic tensor as well. Nevertheless, we think adding a discussion with relevant work [1, 2, 3] that you mentioned will give better insight to future readers, and we added a related discussion in Appendix A in the revised manuscript.

[1] Guo et al.,. AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning
[2] Jiang et al., Text2Performer: Text-Driven Human Video Generation
[3] Ni et al., Conditional Image-to-Video Generation with Latent Flow Diffusion Models.

[W2] How to ensure the model learns meaningful motion representation?

This is a good point. In our video autoencoder, the content frame is a static image without the time dimension, so without meaningful motion representation learned by the motion latent, it cannot reconstruct the video. In other words, it is the video reconstruction and content-motion decomposition in our video autoencoder that ensures the motion latent captures the meaningful motion representation. Empirically, we also found our motion latent vectors indeed contain meaningful motion information. For instance, when generating time-lapse videos of the sky, our model can generate videos with diverse cloud-moving directions at a given fixed content frame. We included this result in Appendix F in the revised manuscript.

[W3] Generated frames being blurry due to the design of the content frame

Thanks for pointing this out. In principle, since our design includes not only a content frame but also a “motion latent vector”, the learned motion latent will compensate for the representing power of the content frame, and reconstruct each frame with a high quality. This has been empirically supported by the high-quality encoding performance in Table 4a and 4b. In practice, we think the appearance of blurriness in generated videos may be attributed to a relatively not large dataset and a relatively small model size. Hence, we believe visual quality can be much better if we use a larger text-video dataset, as validated in the concurrent work [4]. Moreover, recall that our model includes a “single” diffusion model that has a relatively small model size (1.7B including the autoencoder) in total without any cascaded modules, unlike the existing video diffusion models; we also believe training larger models will greatly improve the frame quality.

[4] Wang et al., VideoFactory: Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generation, arXiv 2023

We are truly grateful for taking your time to provide additional recommendations and acknowledge our efforts.

[W2] We use an extremely low-dimensional motion vector as the constraint. For instance, in our video experiments to encode 512$\times$1024 resolution RGB frames with a length of 16, the dimensionality of the motion latent vector is 24,576, 0.1% of the original dimensionality). Such a low dimensionality gives a strong bottleneck to encode the entire video without using content frames; hence, the autoencoder should utilize rich spatial information included in the content frame to achieve high-quality encoding. Also, in Appendix F, we have shown that fixing the content frame maintains the overall contents of the generated videos even if it is combined with different motion latent vectors. This result also empirically verifies that our autoencoder does use the information in the content frame, not just solely relying on the motion latent vector to encode the videos.

[W3] To further address your concern, we provide some qualitative results in Appendix H and the supplementary material (left are real videos, right are reconstructions) to support our hypothesis further. The results demonstrate the high quality reconstructions from our autoencoder without blurry artifacts. Moreover, we think low reconstruction FVD (lower is better) that we provided in Table 4a and 4b implies high-quality video encoding hardly suffer from blurry artifact, because with such artifact, it is unlikely to achieve FVDs at our level asd due to the high distribution shift. Note that this metric has been widely used in previous latent video generation works [1, 2] to measure the reconstruction quality of videos.

[1] Yu et al., MAGVIT: Masked Generative Video Transformer, CVPR 2023.
[2] Yu et al., Video Probabilistic Diffusion Models in Projected Latent Space, CVPR 2023.

Meanwhile, using the average value of images seems not a good idea. For example in the uploaded example_videos.mp4, the panda case has significant movement however it brings some obvious artifacts around the panda. It may be better to represent content with some position insensitive embedded latent. That may also help to address the concern of Reviewer BXkg.