CV-VAE: A Compatible Video VAE for Latent Generative Video Models

W1: Flickering problems of CV-VAE

Our optimization loss is a trade-off between compatibility and better reconstruction quality. Therefore, there is still a domain gap between CV-VAE and diffusion models, which leads to color shifts or flickering problems. This offset can be alleviated by further fine-tuning.

W2: Missing details about training

Thank you for your comments. Actually, we provided the training details in the manuscript. (1) In the "Training Details" paragraph of Section 4.1, we introduced the training datasets, resolution, batch size, learning rate, and other augmentations. We also mentioned joint training with two image datasets of different resolutions (256x256, 512x512) and two video datasets of different resolutions (9x256x256, 17x192x192), with batch sizes of 8, 4, 1, and 1 for the four settings, respectively. (2) We will add some details about the video fps. During the training process, we randomly used a value between 1 and 4 as the frame stride for sampling.

Q2: Comparison between CV-VAE and interpolation model

Thank you for your suggestion. As SVD only released versions for 14 and 25 frames, direct inference of 97 frames would cause the video to collapse. Therefore, we provided comparative results for the frame interpolation model in attached file (Table 3). We use RIFE [4] as the frame interpolation model for comparison, as it is popular and has 4.2K stars on GitHub. Our model outperforms RIFE in 2 out of 3 metrics, further validating the effectiveness of our method.

Q3: Decoder-based regularization during training with images

During the training process, we use both images and videos for joint training, so decoder-based regularization is also employed when taking images as inputs.

Q4: Comparison between SVD and SVD + CV-VAE on text-to-video generation

Thank you for your suggestion. Regrettably, SVD only released the image-to-video version. As an alternative, we compared the performance of VideoCrafter2 and VideoCrafter2+CV-VAE on text-to-video in Figure 10. Additionally, we provide the quantitative metrics and extra visual results for text-to-video in the attached file (Table 4, Figure 2), which further validates the compatibility of CV-VAE with various video generation models.

Q5: Can the method be supported by theoretical proof?

Currently, the effectiveness of this method is proven through empirical evidence, and latent regularization can also be seen as a form of knowledge distillation [1], which is applied in different fields [2][3]. Providing a detailed theoretical proof is quite challenging and can be a direction for future exploration.

[1] Distilling the knowledge in a neural network, arXiv 2015.

[2] Adversarial diffusion distillation, arXiv 2023.

[3] MiniLLM: Knowledge distillation of large language models, ICLR 2024.

[4] Real-Time Intermediate Flow Estimation for Video Frame Interpolation, ECCV 2022.

W1： More examples to explain the model is not just learning simple replication

Thank you for your suggestion. (1) Due to the size limitation of the paper (50MB), we did not include the text-to-video results as videos in the pdf. The visual results were provied in the supplementary material. (2) We alslo provide more visual results of VideoCrafter2 + CV-VAE and quantitative metrics in the attached file (Table 4, Figure 2). It can be observed that the motion between frames is smooth, and it is not simply a duplication.

W2: Which pretrained 2D VAE is aligned with "Ours" in Table 1?

Thank you for your comments. In Table 1, both of our models (2D+3D, 3D) are aligned and compatible with VAE-SD2.1. Our models are also aligned with a lot of open-source diffusion models in the community that use VAE-SD2.1 as the auto-encoder.

Q1: Would training the VAE with bf16 cause a performance collapse?

Yes, training the model with either bf16 or fp16 would result in numerical collapse (NaN), which might be due to the instability caused by GAN loss. Therefore, CV-VAE is trained with fp32.

Q2: Jointing training between images and videos

We use four settings for joint training: 256 × 256 images, 512x512 images, 9 × 256 × 256 video clips, and 17 × 192 × 192 video clips. In each iteration, a batch from one of the four settings is generated with different batch sizes (8, 2, 1, 1), allowing samples of different resolutions to be trained simultaneously.

Q3： Performing SVD on high resolution and high frame rate

Since CV-VAE is composed of convolutional networks and we train it on different resolutions and frame rates, CV-VAE can adapt to higher resolutions and different frame rates with a negligible performance degradation.

W1: Comparison between recent video autoencoders

(1) There might be a missunderstanding. We have compared with the latest Video VAE in Table 1: VAE-OSP, which was released in May 2024. (2) MAGVIT2 [2] and CMD [3] are SOTA methods, but they have not released their weights. Following the suggestion of the reviewer to compare with more recent Video VAEs, we add new quantitative and qualitative comparisons of Open-MAGVIT2 [4] (released in July 2024) and PVDM[1] in the attached file (Table 2, Figure 1). Open-MAGVIT2 is an opensource project to replicate MAGVIT2.

[1] Video Probabilistic Diffusion Models in Projected Latent Space, CVPR 2023.

[2] Language Model Beats Diffusion: Tokenizer is key to visual generation, ICLR 2024.

[3] Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition, ICLR 2024.

[4] Open-MAGVIT2: Democratizing Autoregressive Visual Generation, 2024.

W2: Lack of novelty

We would like to claim that the major contribution of this work is the framework (latent regularization with 2D decoder) to obtain a video VAE from a pretrained image VAE, which is compatible with existing image and video diffusion models, rather than a novel network component or a loss. The compatibility greatly saves additional efforts of training diffusion models to adapt the VAE and being compatible with a wide range of community models. For example, Open-Sora-Plan, based on the incompatible Video VAE, spent an additional ~7138 GPU hours to obtain the image diffusion model.

W3: Training Time

Training the CV-VAE took approximately 800 A100 GPU hours. We will clarify this in the final version.

Q1: Results of Text-to-video diffusion model

Thanks for your concern. (1) We have provided the visual results of CV-VAE + VideoCrafter2 (text-to-video) in Figure 10 and the supplementary material. (2) We also provide the qualitative results and more visualizations in the attached file (Table 4, Figure 2).

Q2： Release of code and weights

We will make the code and weights publicly available to promote community development. Following the rules of response, we have submitted a preview version of the open-source code to the Area Chair.

Thanks for the further clarification - can authors can provide rFID and rFVD (reconstruction FID and FVD) of the proposed methods and other baselines in MSR-VTT? It seems MAGVIT-2 results show too high LPIPS and low SSIM scores considering that Open-MAGVIT2 shows quite a good performance on ImageNet and the original MAGVIT2 paper shows surprising low LPIPS on UCF-101.

The comparison results of FID and FVD metrics on MSR-VTT are as follows. we use the first frame of MSR-VTT to calculate FID.

Due to quantization error, discrete VAEs usually lose more information than continuous VAEs. For example, MAGVIT2 encodes an image with resolution of $256\times 256$ into integers ranging from 0 to 262144 with a size of $32\times 32$, while continuous VAEs (SD-VAE2.1, VAE-OSP, CV-VAE) encode an image with resolution of $256\times 256$ into floating-point vectors with a size of $32\times32\times z$ where $z$ represents channels of latent and are equal to 4 for the continuous VAEs in Table 1. Therefore, it is not surprising that continuous VAEs can achieve better reconstruction results than discrete VAEs.

Moreover, reconstruction quality is not our primary goal in designing CV-VAE, since the reconstruction quality can be easily improved by increasing the size of $z$ without changing the model structure and size [1][2]. For example, in Stable Diffusion 3[2], the PSNR of VAE with $z = 4$ is 25.12dB, while the PSNR of VAE with $z=16$ is 28.62dB. Our primary goal is to design a compatible Video VAE based on the existing 2D VAE, so we must keep the size of $z$ the same as the 2D VAE ($z=4$ for VAE-SD2.1).

[1] Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystack, arXiv 2023.

[2] Scaling Rectified Flow Transformers for High-Resolution Image Synthesis, ICML 2024.

Thank you for your comments. We clarify that CV-VAE and MAGVIT are completely different works: (1) MAGVIT2 represents images or videos as discrete tokens, which are used in autoregression models to generate images or videos; Our CV-VAE represents images or videos as continuous latents, which are used in diffusion models to generate images or videos. (2) Our main goal and contribution is to design a Video VAE that is compatible with 2D VAEs, while MAGVIT2 is not compatible with any other VAEs. here are the responses:

R1: The term "Comp." has the same meaning as in Table 1 of the main paper, indicating compatibility. This also represents the core contribution of our work, which is to design a Video VAE that is compatible with other 2D VAEs, Image Diffusion Models, and Video Diffusion Models.

R2: We use the 8x downsampling version from MAGVIT2 for comparison, which is also the same as the downsampling factor in our CV-VAE.

W1: Fairness of training and evaluation data

Thank you for your comments on the fairness. (1) Unified training data. Since the VAE models in Table 1 are trained on different datasets and some of them did not release their training and dataset details, such as VAE-OSP, it is difficult to train all models on the same dataset. (2) Unified testing data. We appreciate you pointing out that evaluating on the in-domain validation set is unfair. Therefore, we conduct additional evaluations of all VAE models on the publicly available MSR-VTT validation set that is not mentioned as the training set of those models. The evaluation results are presented in the attached file (Table 1). Our method performs better than other video VAE models.

W2: Evaluation on different diffusion models

(1) We would like to first clarify that we have performed compatibility validation on multiple diffusion models, including quantitative evaluations on SD2.1 (text-to-image), SVD (image-to-video) (Table 1, 2), and visual evaluations on SD2.1, SVD, and Videocrafter2 (text-to-video) (Figure 5, 6, 10). (2) To further validate our compatibility with text-to-video models, we have additionally conducted quantitative evaluations on Videocrafter2+CV-VAE and also provided visual results in the attached file (Table 4, Figure 2).

Q1: Clarification of "mapping function"

Thank you for your comments. We will revise "mapping function" to "sampling function" to clarify the expression. We observed that the "1st Frame" results in poorer reconstruction of subsequent frames due to the lack of constraints on other frames, "Average" leads to more motion blur in the reconstructed frames, and "Slice" causes frames without 2D Decoder constraints to be more prone to artifacts. "Random" is proposed based on "Slice" to cover the content in the whole sequence and uses the randomly sampled frames instead of the average frame in a slice to avoid motion blur.