UniVAE: A Unified Frame-Enriched Video VAE for Latent Video Diffusion Models

Q1: Clarification on the novelty of the proposed UniVAE.

A1: This is an original effort to unify video reconsturction and frame interpolation in one VAE framework for latent video generative models, not on the spatial and temporal compression. There are two key innovations compared to existing video VAEs.

(1) Multi-scale convolution kernels for temporal downsampling. We analyze the issues in existing video VAEs, and propose multi-scale temporal convolution kernels for temporal downsampling, which enhance the VAE's ability to capture dynamic patterns in videos, as described in Sec.3.2. It brings better video reconstruction and generation results, as shown in Tab.1 (main paper) and Tab.2 (main paper).

(2) Frame interpolation within the UniVAE decoder. To the best of our knowledge, UniVAE is the first attempt to explore the potential of video VAEs to generate richer video content beyond reconstruction. We design the latent-guided refinement training strategy to enable UniVAE generate additional intermediate frames, which further allows latent video diffusion models to generate high fps videos with improved efficiency.

Q2: Interpolation Decoder Training: The interpolation decoder is trained on its own after freezing the basic VAE encoder and decoder, making it more like an independent interpolation model. It would be more meaningful if the basic VAE encoder and decoder and the interpolation decoder are trained together to improve the representation ability of latent space.

A2: (1) We do not agree the training of the refinement decoder is the same as training a separate interpolation model. Similar with Imagen [1] that utilizes super-resolution diffusion models to upsample the generated image, the refinement decoder is a component of the UniVAE. There are two key difference between the refinement decoder and frame interpolation method. (a) The frame interpolation methods typically can only generate the intermediate frame $x_{new}$ based on the given frames $[x_1, x_2]$. In contrast, our refinement decoder can directly reconstruct the video and the generated additional frame $[x_1, x_{new}, x_2]$ in a single pass. (b) The frame interpolation methods are typically independent and do not rely on video generation techniques. While, our refinement decoder generate video and additional frames based on the latent features extracted by the VAE encoder, which is integrated with our UniVAE.

(2) If we train the UniVAE encoder $\mathcal{E}$, regular decoder $\mathcal{D}_1$, and refinement decoder $\mathcal{D}_2$ jointly, it may obscure the overall training objective of UniVAE. The optimization objective for video reconstruction in $\mathcal{D}_1$ and additional frame generation $\mathcal{D}_2$ may conflict, leading to unstable training.

[1] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479–36494, 2022.

Q1: The writing of the paper left me somewhat confused. In lines 79-80, what does "Our observations indicate..." refer to? How did the authors come to the subsequent conclusion that "If the encoder can effectively model temporal variations, the decoder could theoretically synthesize frames, leading to high fps videos without the for separate interpolation models"?

A1: We apologize for the confusing expression in lines 79-80.

(1) "Our observations" refer to the experiment results in Sec.4.2 and Sec.4.3. As we can see, our UniVAE not only fulfills the typical VAE role of video encoding and decoding in latent video diffusion models (Tab.1 (main paper) and Tab.2 (main paper)), but also extends to generate additional intermediate frames, as shown in Fig.5 (main paper). This is what we refer to in lines 79-80 as "VAEs can assume multiple roles in current video generation stream."

(2) If the encoder of the VAE effectively captures the temporal dynamics of a video, the latent features $\mathbf{Z}$ will contain rich information about the sequence changes, such as object motion trajectories. This allows the decoder to leverage these dynamic patterns in $\mathbf{Z}$ to not only reconstruct the original video frames but also generate intermediate frames between them, leading to high fps videos without separate interpolation models. As we can see in Fig.5 (main paper), the refinement decoder can leverage the rich temporal information in $\mathbf{Z}$ to predict the motion trajectories and generate precise intermediate frames.

Q2: The paper emphasizes that both the Encoder and Decoder in UniVAE are causal. What specifically does "causal" mean in this context? What would be the results if either or both were non-causal?

A2: The term ``causal" indicates that the VAE processes the relationships between frames in a causal manner when encoding and decoding video. Specifically, when handling frame $f_i$, it only considers the current frame $f_i$ and the preceding frames $[f_1, f_2, ..., f_{i-1}]$, without referencing subsequent frames, which means the output for each frame only depends on the previous frames. This design enables VAE to encode both video and image [1]. The causal setup is widely applied in decoder-only architectures of large language models (LLMs) and video VAEs [2][3][4].

[1] Lijun Yu, Jos´e Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats diffusion–tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737, 2023.

[2] Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, March 2024. URL https://github.com/hpcaitech/Open-Sora.

[3] Sijie Zhao, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Muyao Niu, Xiaoyu Li, Wenbo Hu, and Ying Shan. Cv-vae: A compatible video vae for latent generative video models. arXiv preprint arXiv:2405.20279, 2024.

[4] Liuhan Chen, Zongjian Li, Bin Lin, Bin Zhu, Qian Wang, Shenghai Yuan, Xing Zhou, Xinghua Cheng, and Li Yuan. Od-vae: An omni-dimensional video compressor for improving latent video diffusion model. arXiv preprint arXiv:2409.01199, 2024.

Q3: Could the latent-guided refinement training scheme proposed in this paper also be used to enhance the performance of other pre-trained VAE?

A3: Yes. The latent-guided refinement training scheme can be utilized in other pre-trained video VAE. Here, we retrain OD-VAE with our proposed latent-guided refinement training scheme, and show the qualitative comparison in Fig.8 (response letter). As we an see, our latent-guided can also be utilized to enhance the output of OD-VAE. While, our UniVAE still obtains better results.

Q1: The effectiveness of the refinement decoder is not sufficiently demonstrated, as it relies solely on the qualitative analysis presented in Figure 5. However, I have observed a color bleeding phenomenon in the interpolated images. While this is a good idea, I would appreciate more discussion on this aspect.

A1: (1) As the original work to leverage VAE for generating additional frames, beyond existing video VAE, we focus on exploring the possibility and potential of using VAE decoder for frame interpolation. Thus, unlike current frame interpolation techniques, we do not design specific modules (like motion prediction module) in the refinement decoder, which may lead to detailed issues such as color bleeding in the generated intermediate frames. However, as shown in Fig.5 (main paper), even without any specially designed motion estimation modules, our UniVAE can still accurately predict the motion trajectories in videos, and generate precise intermediate frames.

(2) To further prove the effectiveness of our UniVAE for frame interpolation, we compare our UniVAE with existing separate interpolation method FILM [1]. FILM is an independent frame interpolation method, which designs flow estimation modules to compute flows based on feature pyramids. For FILM, we first send the input video into the UniVAE and get the ordinary output from the regular decoder $\mathcal{D}_1$, and then use pre-trained FILM for frame interpolation. We denote it as ``VAE + FILM''.

(a) Performance comparison: The input videos are pre-processed to a length of 65 frames with a resolution of $512 \times 512$. We show the qualitative results in Fig.5 (response letter). As we can see, our UniVAE achieve comparable interpolation results with the specifically designed interpolation method FILM.

(b) Time consumption: We further provide the time consumption for the two pipelines, i,e,, UniVAE and VAE + FILM. As described in (1), for each 65-frame video with a resolution of $512 \times 512$, we apply the both interpolation methods to extend it to 129 frames, respectively. We show the time consumption of these two methods in Tab.1 below. As we can see, video reconstruction is faster than video interpolation in UniVAE. However, the subsequent separate interpolation will cost more time than UniVAE. As a result, our UniVAE is more efficient than "VAE + FILM", which further prove the potential of UniVAE in supporting latent video diffusion models.

The experiment results in Fig.5 (response letter) and Tab.1 below show that even without dedicated interpolation modules (such as flow estimation modules in FILM), our UniVAE still achieve comparable performance to exiting interpolation methods while reducing complexity of pipeline and improving efficiency. This is crucial for further advancing the latent video diffusion models.

Table 1: Time consumption comparison between UniVAE and "VAE + FILM".

[1] Fitsum Reda, Janne Kontkanen, Eric Tabellion, Deqing Sun, Caroline Pantofaru, and Brian Curless. Film: Frame interpolation for large motion. In European Conference on Computer Vision, pp. 250–266. Springer, 2022.

Q2: Given that most current VAEs utilize tiling to reduce inference memory, I wonder if the Multi-Scale Spatial-Temporal downsampling method imposes any limitations within tiling. Can it still ensure satisfactory performance in such scenarios?

A2: Our spatial-temporal downsampling imposes no limitations within tiling. In fact, following the previous methods [2], we also employ tiling technique when evaluating the performance of our UniVAE, and show the results in Tab.1 (main paper), Tab.2 (main paper), and Fig.6 (main paper).

[2] Liuhan Chen, Zongjian Li, Bin Lin, Bin Zhu, Qian Wang, Shenghai Yuan, Xing Zhou, Xinghua Cheng, and Li Yuan. Od-vae: An omni-dimensional video compressor for improving latent video diffusion model. arXiv preprint arXiv:2409.01199, 2024.

Q3: Have the authors attempted to use latent videos generated by diffusion models (e.g., Latte) as input to the refinement decoder? This approach may pose greater challenges compared to directly encoding videos and performing interpolation, but it could also provide stronger evidence for the significance of the proposed method.

A3: Thanks for your good suggestion. Since the regular decoder $\mathcal{D}_1$ and the refinement decoder $\mathcal{D}_2$ share the latent representation $\mathbf{Z}$, the Latte equipped with our UniVAE can directly leverage the refinement decoder to output videos with richer content. We show some qualitative results in Fig.6 (response letter).

1. Clarification on our motivation and technical contribution.

We sincerely appreciate your insightful comments and will revise our wording and discussion accordingly. In our humble opinion, joint spatiotemporal modeling using multi-scale convolution in video VAE is a reasonable and intriguing approach for both diffusion and interpolation, which may be beneficial to share with the field timely.

Q1: The introduction does not convincingly explain why we need the proposed multi-kernel convolution approach.

A1: Thanks. The challenges of handling video data largely come from their multi-resolution nature in both spatial and temporal domain, which has been the research theme for multidecade. The multi-scale convolution with a large receptive field in VAE offers one way to enhance compression and reconstruction quality of long-duration and high frame rate videos, especifically using joint spatiotemporal modeling at the feature level in one framework for both diffusion and interpolation tasks. We would not claim multi-scale convolution is the only way, yet we claim it deserves the research efforts for its potential in VAE and video regeneration.

From the perspective of experiments results:

(1) Single-scale convolution kernel results in video reconstruction performance degradation. As illustrated in Fig.6 (main paper), both OS-VAE and OD-VAE which use single-scale convolution kernel exhibit performance degradation when reconstructing long videos, due to the limited ability of single-scale convolution kernel to capture changes occurring over varying time scales in videos.

(2) Single-scale convolution cannot support our refinement decoder design. Compared with single-scale convolution, multi-scale convolution kernel can better extract temporal patterns at various time scales, which can provide the refinement decoder more informative features for additional intermediate frame generation. Our multi-scale convolution kernels are indispensable for the subsequent refinement decoder. To demonstrate the advantage of multi-scale convolution kernels for additional frame generation, we train a new refinement decoder based on pre-trained OD-VAE that only uses single-scale convolution kernel when temporal downsampling. The comparison in Fig.2 (response letter) shows UniVAE with multi-scale convolution kernels can generate better intermediate frames compared with OD-VAE with single-scale convolution.

From the perspective motivation:

We aims to improve VAE performance by enhancing the encoder's ability to capture temporal clues for spatio-temporal modeling. Our multi-scale temporal convolution kernels endows the UniVAE ability to perceive and capture the dynamics patterns across different time scales in videos, which can result in better video reconstruction and generation results compared with existing video VAE, as shown in Tab.1 (main paper) and Tab.2 (main paper).

Q2: Why does temporal pooling operation limit the ability of video VAEs (Line 51)?

A2: The temporal pooling operation is a rudimentary approach to temporal dimensionality reduction, as evidenced by various improved methods [1] developed for different tasks [2][3] over time. Fixed pooling operations may miss visual or motion details and lack of learnability. Instead, 3D convolutions generally are robust to capture more appearance details and varying motion patterns [4]. To further support this argument, we replace the temporal downsampling operator in our UniVAE with average pooling, and shows the comparison results in Tab.1 below. As we can see, our UniVAE achieves better performance than VAE with temporal pooling operator.

Table 1. Performance comparison between different VAEs on video reconstruction.

[1] Avraham Ruderman, Neil C Rabinowitz, Ari S Morcos, and Daniel Zoran. Pooling is neither necessary nor sufficient for appropriate deformation stability in cnns. arXiv preprint arXiv:1804.04438, 2018.

[2] Eric Kauderer-Abrams. Quantifying translation-invariance in convolutional neural networks. arXiv preprint arXiv:1801.01450, 2017.

[3] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431–3440, 2015.

[4] Lijun Yu, Jos´e Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats diffusion–tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737, 2023.