High-Quality Joint Image and Video Tokenization with Causal VAE

Thank you for your constructive review and insightful suggestions. We have thoughtfully considered your feedback and revised our paper accordingly, as outlined below.

• Clarification of the Ablation Study:

  Following the reviewer’s suggestion, we have included detailed ablation studies in the revised manuscript, analyzing the contribution of each individual component by incrementally incorporating them into a base model. The results are summarized in Table 8 in the appendix, with a related discussion provided in Appendix A.7.

  We have also included qualitative visualizations for the ablation experiments in the supplemental video.

• Inclusion of Visualization Analysis:

  We have added qualitative examples of videos generated by Open-Sora using Open-Sora’s own video VAE in comparison with using our video VAE in the supplemental video to strengthen the qualitative analysis of our work. Please refer to the updated supplemental video.

• Comparison with SVD-VAE:

  Thank you for the suggestion. We have added SVD-VAE in Table 1 in our revised manuscript. We would like to clarify that SVD-VAE is not temporally causal and cannot be used for image autoencoding. Therefore, it was not included in Table 2.

Dear Reviewer MHdT,

Thank you for your valuable feedback. We have revised our manuscript to address your concerns and clarify any potential misunderstandings, as outlined below.

Clarification

1. We would like to clarify that Section 3.1 (Baseline Methods) specifies that our models use a latent channel size of 4. For additional details, please refer to this section.

2. We would also like to clarify that the compression ratio difference between an $8 \times 8 \times 8$ model and a $4 \times 4 \times 4$ model is only $8$ times, not $64$ times as suggested by Reviewer MHdT.

   Similarly, the compression ratio difference between a $16 \times 8 \times 8$ model and a $4 \times 4 \times 4$ model is $16$ times, not $128$ times as suggested by Reviewer MHdT.

Response to questions

1. Has the author explored the number of latent channels? If so, I would like to see those findings and understand why they were not presented in the paper.

   • We explored latent channel sizes of $4$, $8$, and $16$. Our experimental observations indicate that, for the same spatio-temporal compression rate, higher latent channel sizes consistently improve autoencoding performance ($16 > 8 > 4$), which aligns with the findings in CogVideoX [1].

     We have updated Table 1 in the revised manuscript to include a column explicitly showing the latent channel size of our approach and competing baselines. Additionally, we have added baselines of our model with latent channel sizes of 8 and 16 for comparison. Please refer to Table 1 in the updated manuscript.

   • We focused on a latent channel size of $4$ mainly for fair comparisons with existing approaches, such as Open-Sora and Open-Sora-Plan, and to align with generative frameworks that use the same size. Additionally, we fixed the latent channel size to better highlight the contributions of our proposed modules across different spatio-temporal compression rates.

2. Why does our $8 \times 8 \times 8$ model give a superior performance compared to VideoGPT [2] ($4 \times 4 \times 4$)?

   • We would like to highlight that VideoGPT uses discrete quantization, while our approach employs continuous Gaussian sampling. As shown in previous works, such as LDM [3], continuous tokenization better preserves the semantic details of the input visual data compared to discrete tokenization with a predefined codebook, resulting in improved autoencoding performance. For instance, the experiments in Table 8 (page 21) of the LDM [3] paper demonstrate that the continuous model consistently outperforms the discrete one at the same or lower spatio-temporal compression rates. Our results are consistent with this observation.

   • It is also important to note that the encoder in VideoGPT employs a 3D convolutional layer with a kernel size of $4 \times 4 \times 4$ to directly downsample the input video to the target latent size. In contrast, our approach utilizes hierarchical downsampling through the proposed FILM encoder and spatio-temporal downsampling modules, facilitating more effective encoding. This likely contributed to the superior performance of our approach, even with $8 \times$ more overall compression.

3. Why is the LPIPS score of our $16 \times 8 \times 8$ slightly better than the LPIPS score of VideoGPT [2] ($4 \times 4 \times 4$)?

   • As noted in [2], the training setup of VideoGPT does not include a perceptual (LPIPS) loss, which likely contributes to its subpar performance on the LPIPS metric. In contrast, our training setup incorporates a perceptual loss, which helps explain the slightly better performance of our $16 \times 8 \times 8$ model on the LPIPS metric.
   • It is also worth noting that other baselines with higher compression rates, such as Open-Sora ($4 \times 8 \times 8$) and Open-Sora-Plan ($4 \times 8 \times 8$), which incorporate perceptual training, notably outperform VideoGPT ($4 \times 4 \times 4$) on the LPIPS metric, as shown in Table 1.

References

[1] CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer

[2] VideoGPT: Video Generation using VQ-VAE and Transformers,

[3] High-Resolution Image Synthesis with Latent Diffusion Models

Thank you for your valuable review and constructive suggestions. We have carefully addressed your concerns and revised our paper accordingly, as outlined below.

Response to "Weaknesses"

• (1) Title could be misleading:

  We appreciate the reviewer’s concern. We would like to clarify that our paper consistently specifies that the term “compression” is used in the context of generation. Additionally, in Related Works section (refer to Appendix B), we provide an in-depth discussion on both general video compression and video compression for generation. It is also worth noting that “video compression” is frequently used in recent literature as a synonym for “video autoencoding” [1,2,3,4]. To prevent any potential misunderstandings, we updated our paper’s title, replacing “compression” with “autoencoding”. Please check the revised manuscript.

• (2) Regarding novelty:

  As acknowledged by the other reviewers, we would like to emphasize that the primary contribution of our work is identifying three key issues in existing video VAEs, as outlined in the Introduction section (Lines 63–96), and proposing an effective method to address them. The effectiveness of our approach has been validated through extensive ablation studies, as shown in Table 4.

• (3) Necessity of causality:

  Although our work primarily focuses on video autoencoding for generative modeling, the causality formulation is inspired by recent related works [1,2,3,4] and it enables our approach to be flexibly applied to both image and video autoencoding, as demonstrated in Table 1 and Table 2. The ability to jointly encode images and videos has been shown in previous works [1,2] to be crucial for improving video generation quality. Additionally, it enables other applications such as image-to-video generation and variable-length video generation [1,2,3].

Response to "Questions"

• (1) About title of the paper:

  Please refer to our response to question (1) under Weaknesses

• (2) Comparison with VideoLDM on large motions:

  We would like to clarify that VideoLDM ($1 \times 8 \times 8$) employs a temporal reasoning layer to align latents from individual frames but does not perform any temporal compression. Consequently, it is expected to perform well on the temporal smoothness (TS) metric, particularly for large-motion (low frame-rate) datasets such as DAVIS. In comparison, our model ($4 \times 8 \times 8$) achieves $4 \times$ greater temporal compression while still achieving a strong performance on the TS metric.

  For a fairer comparison, we have included the results of our $1 \times 8 \times 8$ model in Table 1 in the revised manuscript. As can be inferred from the table, our ($1 \times 8 \times 8$) model significantly outperforms VideoLDM ($1 \times 8 \times 8$) on temporal smoothness (TS) metric across both DAVID and Xiph-2K datasets.

• (3) Computational complexity and video-based metrics:

  We have included a comparison of computational complexity, including model size and inference speed, between the proposed model and competing approaches in Table 7 in the appendix. A related discussion has also been added to Appendix A.5.

  We have also included two additional video-based metrics in Table 1 in the updated manuscript: reconstruction FVD (rFVD) and STREAM [5], which evaluate the spatio-temporal perceptual quality and fidelity of the decoded videos. As shown in Table 1, our approach consistently outperforms competing methods on the newly added video-based metrics, aligning with the superior performance observed using other metrics.

  P.S. We attempted to include the VMAF metric in our comparisons but could not find an open-source PyTorch-based implementation. The official codebase [6] posed challenges due to its restrictive requirements, such as specific video formats and resolutions. Consequently, we opted for more commonly used and easily reproducible metrics.

• (4) Regarding GAN loss:

  We would like to point out that GAN loss is a standard loss in training image/video autoencoders and all the competing approaches listed in Table 1 are trained with GAN loss included.

  To show the contribution of GAN loss in our proposed model, we report the performance of our proposed method with and without GAN training in Table 9 in the appendix. Our experimental analysis shows that GAN training primarily enhances the perceptual quality of the decoded videos, as evidenced by a slight degradation in metrics like LPIPS and rFVD when the model is trained without GAN loss.

  This discussion has been added to Appendix A.8 of the revised manuscript.

I thank the authors for taking much effort in addressing my comments. Most of them have been addressed properly. Some further questions are as follows:

(2) Comparison with VideoLDM on large motions It appears that temporal compression has a significant impact on generated video quality particularly for videos with large motion. This leads me to the question whether temporal compression is impractical for real-world applications, especially when one like to generate videos with complex motion. Under what circumstances should this temporal compression be used? What are the benefits of performing temporal compression?

(3) Computational complexity and video-based metrics In Table 7 of the appendix, how come the two variants of the proposed method with different Ch. sizes have the same model size.

(7) About proposed VAE I mean that at inference time, you may as well draw a Gaussian sample and do the video generation. In this case, you do not train another diffusion model to learn the prior distribution for the pre-trained VAE. I am curious how the video quality would be if you used the Gaussian sample to generate videos.

Dear Reviewer tD7c,

Thank you for your response. We are pleased to hear that most of your comments have been addressed. Below, we carefully address the follow-up questions you raised.

• (2) Regarding temporal compression:

  We agree with the reviewer that temporal compression can affect the quality of the decoded video, particularly when the video has a low frame rate or involves highly complex motion. This reflects the general trade-off between spatio-temporal compression and the quality of the decoded video in generative modeling.

  However, most user-generated videos in internet traffic—commonly used to train video generation models—do not exhibit highly complex motions, tend to have relatively high frame rates, and contain significant temporal redundancy. For example, the recently proposed OpenVid-1M dataset [1] contains videos with frame rates ranging from 30-60 fps. Additionally, a recent study [2] shows that most video datasets used for generative training exhibit relatively small dynamics. In such cases, temporal compression can still be effectively leveraged.

  More importantly, temporal compression improves the computational efficiency of long video generation during both training and inference. For instance, when latents are encoded with a $1 \times 8 \times 8$ model, each generated latent corresponds to a single frame. However, encoding with an $8 \times 8 \times 8$ model results in decoding 8 frames from each generated latent, allowing for the generation of much longer videos with the same computational cost.

• (3) Regarding the model size of the two variants:

  We would like to clarify that the two model variants do not have exactly the same size. The model with a latent channel size of 4 has a total of 243,384,571 parameters, while the model with a latent channel size of 16 has a total of 243,385,555 parameters. The sizes listed in Table 7 of the appendix appear identical due to rounding to the nearest $N \times 10^6$, where N=243.4. We will clarify this in the final version of our paper.

  We would like to emphasize that the encoder-decoder architecture of both models is identical. The only difference is the output channel size of the Causal3DConv layer before Gaussian sampling and the input channel size of the Causal3DConv layer after Gaussian sampling in the Middle Block. Please refer to Figure 1 for further details.

• (7) Regarding the proposed VAE:

  Thank you for providing a clarification to the question.

  In response, we would like to highlight a subtle distinction between VAEs used for direct generation [3, 4] and those designed to create a meaningful latent space for other generative models [5, 6]. Specifically, VAEs for compression often employ a smaller KL divergence term.

  In a Variational Autoencoder (VAE), the loss function consists of two key components: the reconstruction loss, which ensures accurate reconstruction of the input, and the KL divergence loss, which regularizes the latent space to align with a standard Gaussian distribution. In line with the approach described in the seminal work LDM [5], we train our video VAE with a small KL divergence coefficient of $1e-6$. This low weight significantly reduces the impact of the KL divergence term, causing the latent space to deviate notably from a standard Gaussian distribution. As a result, sampling from Gaussian distribution is unable to generate meaningful videos as the VAE is dominated more by the reconstruction term and behaves more like an autoencoder.

References

[1] OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation

[2] Evaluation of Text-to-Video Generation Models: A Dynamics Perspective

[3] NVAE: A Deep Hierarchical Variational Autoencoder

[4] An Introduction to Variational Autoencoders

[5] High-Resolution Image Synthesis with Latent Diffusion Models

[6] Sora: Video generation models as world simulators

Thank you for your insightful review and valuable feedback. We have thoroughly considered your suggestions and made the necessary revisions to our paper, as outlined below.

• 1. Missed comparisons:

  Thank you for the suggestion. We have updated our manuscript by including both CogVideoX VAE and CV-VAE in our experimental comparisons. Please refer to Table 1 and Table 2 in the revised manuscript.

• 2. Video quality metrics:

  We have also added rFVD results in Table 1. Please refer to the revised manuscript.

• 3. Video generation results:

  We would like to highlight that utilizing SkyTimelapse and UCF101 for generation experiments is a standard practice commonly followed in related works.

  We appreciate the reviewer’s suggestion to conduct large-scale text-to-video generation experiments. However, these experiments are beyond the scope of our work due to current resource constraints and are left for future exploration. To support such efforts, we will open-source our code and pretrained checkpoints.

• 4. Evaluation resolution:

  The resolution used in Table 1 is $480$p. For Table 2, we used $256 \times 256$ resolution. We have added this clarification in the updated manuscript (Lines 305-307).

• 5. Analysis on different resolutions:

  Following the reviewer's suggestion, we have conducted a performance analysis of our approach and competing baselines across various video resolutions: $128 \times 128$, $256 \times 256$, $480$p, and $720$p. The results are presented in Table 7 in the appendix, with a related discussion included in Appendix A.4.

• 6. Inference time:

  We have included the inference time comparison of our approach and baseline models in Table 7 in the appendix, along with a related discussion in Appendix A.5.

• 7. Method’s limitations:

  While our work advances the state-of-the-art in video autoencoding, particularly at higher spatio-temporal compression rates, it does have a few limitations. Our experimental analysis reveals that temporal jittering can occur between decoded frames at very high compression rates, such as $16 \times 16 \times 16$. Qualitative examples of such cases are provided in the supplementary video. While this phenomenon is expected due to the inherent challenges of extreme compression, we argue that our low-resolution training also contributes to this limitation.

  For an input clip of size $17 \times 128 \times 128 \times 3$ and a given compression rate of $16 \times 16 \times 16$, the encoded latent will have a dimension of $2 \times 8 \times 8 \times 4$. Due to its significantly smaller size, the encoded latent may not capture all the information needed for the decoder to reconstruct the input video. While the straightforward solution is to use higher resolution clips during training, this was computationally infeasible for our approach due to the significant cost of the quadratic scale attention in the spatio-temporal attention blocks (STAttnBlock). For instance, using a $256 \times 256$ resolution increases the token count by a factor of 4. Removing the attention blocks from our model is not a viable solution either, as our ablation experiments in Table 4(d) have shown them to be a crucial component of the proposed video VAE.

  Therefore, a promising direction for future work to address this issue would be to explore more computationally efficient spatio-temporal attention mechanisms, such as Mamba [1], for higher-resolution training. This could, in turn, enable more robust higher spatio-temporal compression while mitigating the associated computational challenges.

  This discussion has been added to Section 5 of the revised manuscript.

References

[1] Mamba: Linear-Time Sequence Modeling with Selective State Spaces

Thank you for your thoughtful review and constructive comments. We have carefully addressed your suggestions and revised our paper accordingly, as detailed below.

• Cite and compare with HVDM (ECCV'24):

  Thank you for bringing HVDM (ECCV'24) [1] to our attention. We have updated our manuscript to include a citation and incorporate it into our experimental comparisons (Table 1). Since pretrained checkpoints for HVDM were not made publicly available, we trained their model from scratch adhering to the training code provided in their official repository. As highlighted in Table 1 of the updated manuscript, our $1 \times 8 \times 8$ and $4 \times 8 \times 8$ models significantly outperform HVDM ($1 \times 8 \times 8$), while our $8 \times 8 \times 8$ model demonstrates highly competitive performance. Additionally, qualitative comparisons of our approach with HVDM have been added to the Figure 8 in the appendix.

• Clarifications on the usage of "temporally agnostic'':

  Following the reviewer’s suggestion, we have toned down the claim and replaced “temporally agnostic” with “better temporal adaptability” throughout our paper. Please refer to Lines 87-89, Lines 188-189, and Lines 477-479 in the revised manuscript.

• Evaluation using STREAM metrics:

  Thank you for the suggestion. We have included the metrics from STREAM [2] in our experimental evaluation. Specifically, we report the average of the spatio-temporal fidelity (STREAM-F) and temporal flow (STREAM-T) scores for our models and competing approaches. The updated results are presented in Table 1 of the revised manuscript. As shown in the table, our approach achieves superior spatio-temporal fidelity and temporal flow compared to competing methods, aligning with the performance trends observed using other metrics in Table 1.

• Possibility of overfitting on UCF101 and SkyTimelapse:

  We appreciate the reviewer’s concern and clarify that our pretrained video VAE remains frozen during the generation experiments, ensuring it cannot overfit to SkyTimelapse or UCF-101, as these datasets were not part of its training. However, we acknowledge the reviewer’s point that training with Open-Sora’s generative framework may still be susceptible to overfitting.

  To ensure a fair comparison, we have demonstrated in Table 3 that our video VAE significantly outperforms Open-Sora’s own VAE in video generation experiments. Additionally, to address this concern further, we have included diverse video samples generated by Open-Sora using both our video VAE and Open-Sora’s own VAE in the supplemental video. We have also added more generated samples in Figure 4.

• Adding baselines for an apples-to-apples comparison:

  We have added the performance of our $1 \times 8 \times 8$ and $4 \times 4 \times 4$ models in Table 1 of the revised manuscript. As anticipated, reducing spatio-temporal compression significantly improves autoencoding performance.

• Discussion on computation resources and convergence speed:

  Our experiments identified three key factors affecting computational resource usage and convergence speed: spatio-temporal attention blocks, training video resolution, and batch size. Our default configuration uses a video resolution of $17 \times 128 \times 128 \times 3$, spatio-temporal attention blocks in both the encoder and decoder networks, and a batch size of 48. On 48 A100 (40 GB) GPUs, this setup converges in about 96 hours. Removing spatio-temporal attention layers reduces convergence time to 40 hours, but as shown in the ablation study (Table 4d), it significantly degrades performance, underscoring the trade-off between efficiency and model quality. The quadratic scaling of attention in these blocks is the main factor driving the longer training times. Increasing resolution to $17 \times 256 \times 256 \times 3$ was infeasible on 40 GB GPUs due to the $4 \times$ increase in token count, which greatly increases the computational demands of spatio-temporal attention. Smaller batch sizes, such as 8 on 8 A100 GPUs, extended training time to 144 hours to match the performance of the default setting.

  We have included this discussion in the new Appendix A.6 and provided a clear reference to it in the main paper (Line 293)

• Elaborate more on the GAN architecture:

  We have expanded on the details of the 3D convolution-based discriminator used in our work. Please refer to Lines 278-281 in the updated manuscript for further information.

• Missing references directing to the appendix:

  Thanks for pointing this out. We have added clear references to the appendix in the revised manuscript. Please refer to Lines 124-128

• Regarding concatenation of FILM’s outputs:

  We experimented with different methods of combining FILM's output, including normalizing before concatenation and passing each output through a 3D Conv layer. However, we did not observe any significant performance improvement with these approaches. Therefore, we chose direct concatenation for simplicity. Our intuition is that any potential statistical differences in the concatenated latent are addressed during training through the Middle Block layers before Gaussian sampling (refer to Figure 1).

• Reason for using $1+T$ over $T$:

  The main reason for using $1+T$ instead of $T$ is to maintain a temporally causal formulation throughout the encoder and decoder, particularly during downsampling and upsampling, enabling the joint encoding of images and videos within a single model. This approach is commonly adopted in recent works [3, 4].

• "3 key contributions" -> "three key contributions":

  Thank you for pointing this out. We have corrected it in the revised manuscript. Please refer to Line 533.

References

[1] HVDM: Hybrid Video Diffusion Models with 2D Triplane and 3D Wavelet Representation

[2] STREAM: Spatio-TempoRal Evaluation and Analysis Metric for Video Generative Models

[3] Phenaki: Variable Length Video Generation from Open Domain Textual Descriptions

[4] Magvit-v2: Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation

Dear Reviewers,

Thanks for your contributions in reviewing this paper.

As the author-reviewer discussion deadline is approaching, please could you take a look at the authors' rebuttal and see if it addressed your concerns or if you have any further questions. Please feel free to start a discussion.

Thanks,

AC