VideoDiT: Bridging Image Diffusion Transformers for Streamlined Video Generation

1. The method illustration is quite unclear to me, where I'm confused by many parts of the approach. For example, how do we choose the key frame? Is the key frame always the first frame? What if there're several scene changes, so that the encoded image latent is largely different from the video latent. Why is the compression ratio 4? More questions are elaborated in the next section. Please provide a more detailed explanation or diagram of the key frame selection process and how it handles different scenarios like scene changes. Additionally, please clarify on how the compression ratio is determined and applied consistently across different video lengths.
2. I would suggest the authors to also include the inference speed (wall-clock time) and memory cost on encoding-decoding a video. Since the proposed framework actually has more FLOPs - one image VAE + one video VAE - I think the inference speed might be slow and memory cost would be higher compared to baselines. Please provide a table comparing inference speed and memory usage for the proposed method versus baselines on standardized video lengths and resolutions.
3. I would suggest the authors to highlight the compression ratio for baselines and the proposed video VAE. Since it's unclear to me whether all methods have the same compression ratio, hence whether this is a fair comparison. Please add a column in the comparison tables that explicitly states the compression ratio for each method, or to clarify in the text how you ensured comparable compression ratios across methods.
4. Novelty. The DP-VAE part is generally OK to me, conditioned on the authors have clarified my unclear points. However, the extension from image DiT to video DiT has no clear novelty to me. I think it's already a quite common practice for all the recent video DiT generators. They would even incorporate better position embedding extension like 2D RoPE -> 3D RoPE, so the section 3.2 is actually of no novelty to me. When changing from image to video under the DiT architecture, except for the position embedding change, it's always the full attention and I see nothing new.
5. The generation results still have motion blur and not that high-quality to me.
6. I'm wondering whether this method performs well on higher compression ratio. Since 4x8x8 compression ratio is still too low compared to current SOTA video generators (e.g., Meta MovieGen uses 8x8x8), the sequence length would be unaffordable long. Please include an experiment or discussion on how your method performs with higher compression ratios, specifically comparing to the 8x8x8 ratio.

The proposed DP-VAE offers an interesting solution on how to adapt the 2D VAE to the 3D VAE. However, the contributions are limited, and many details need to be clarified.

Regarding the DP-VAE part, it has many details that need to be addressed, which is also the biggest concern so far. First, the paper does not clarify how the keyframes were selected. Merely it mentions the Groups of Pictures (GoPs) from the video compression. However, it will be necessary to specify how the key video frames were selected in this task. For instance, what are the algorithms or criteria used to perform the keyframe selection? This selection significantly affects the DP-VAE design.

Secondly, the architecture presented in Figure 1, as well as the descriptions presented, contains some confusion. It has been mentioned that the 3D VAE has been applied to the entire video segment x in Section 3.1. But in the description of some other places (see Abstract description), the 3D VAE was employed to deal with the non-keyframe compression.

Thirdly, the DP-VAE was trained on a self-collected video dataset, which raises concerns for fair comparisons and reproducibility. Could you discuss how to ensure the fair comparisons given the use of a custom dataset? More details regarding the self-collected dataset and its usage purpose would be appreciated.

Finally, the results presented in Table 3 offer some contradicting observations regarding the effectiveness of 2D VAE and regularization, which are not very well supported with explanations. The results show that the proposed method only offers a trade-off, other than providing the best performance. The paper mentions, "Incorporating a 2D VAE without regularization yields the best reconstruction performance". But this training will result in original weight disruption, leading to suboptimal performance. Please also provide more details in this aspect.

The regularization term of DP-VAE puts constraints on the key-frame reconstruction and the 3D latent variables. This is a commonly used regularization in the VAE design but applied to different sets of inputs here. Therefore, the contribution is not significant.

The paper also explores the design of transferring 2D attention to 3D attention. It claims the direct copy of 2D attention for 3D attention serves as an effective initialization strategy without introducing additional parameters in the method section. However, in the experimental section, the paper refers the a trainable copy of the original spatial weights dedicated to global attention as the “2D+3D” attention, which differs from the 2D→3D. The two schemes need to be further clarified with more details.

Other parts such as Joint Image-Video Training have been explored in the existing video generation works such as VideoCrafter, Lavie and Open-Sora.

The paper shows the results in Table 2 and claims that the proposed approach uses a small amount of video training data, but still achieves competitive performances. While this claim makes sense, the reviewers would also be interested to know if the same set of training data was used, will the performance further surpass the existing ones? The paper also uses self-collected videos, it could be a concern for fair comparisons.

In the light of recent development in the community, it is clear that the amount of resources that can be used for training of the video model, is proportional to the final quality of these models. Having said this, there is an avenue for two types of papers: (1) papers that try to push generation quality to the limit at a large scale, and (2) papers that present some novel insight or technique on a small scale and thoroughly evaluate it.

This particular paper is in-between and neither demonstrates a good quality of generations, nor presents any novel insight or technique. Moreover it does not provide thoughtful evaluation of the proposed techniques.

In more details:

• The quality of the results provided in supplementary material is not very good. It can be even said that the model did not learn anything, since the motion is meaningless and the quality of individual images is just inherited from the pre-trained SD3 model. So if we evaluate this model on (1), models like [1, 2, 3] which are also diffusion transformers demonstrate much better results.

• The modification proposed by this paper was already proposed in other papers. Initializing the auto-encoder from image one was done in [5]. Video DiT was used in [2, 4], the modification to Video DiT provided in this paper is minimal.

• The evaluation of VAE does not include the account number of channels used. DP-VAE has 16 channels, while others have 4. It is clear that VAE with more channels will have better reconstruction. So a fair comparison will be against VAE of [1] which also has 16 channels.

• The main model is evaluated at zero-shot UCF-101, which is highly dependent on the choice of the training videos and overall is not a reliable benchmark. The better approach would be to use some video evaluation benchmark such as VBench[6] or VideoScore[7], or do user study.

[1] CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer

[2] Photorealistic Video Generation with Diffusion Models

[3] Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesis

[4] Latte: Latent Diffusion Transformer for Video Generation

[5] Cv-vae: A compatible video vae for latent generative video models.

[6] VBench : Comprehensive Benchmark Suite for Video Generative Models

[7] VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generation

1. Video reconstruction results: The reconstruction performance in Table 1 is reported on a small subset of 50 videos randomly selected from the Pexels dataset. This benchmark for evaluating video reconstruction performance is novel and, as such, has not been previously reported in the literature. However, it contains too few samples to be considered robust. A more meaningful evaluation benchmark should include at least 1,000 videos. It is suggested to evaluate models on a larger dataset, for example WebVid eval [4] set as in CV-VAE paper [1].

2. Missed VAE comparisons: Table 1 does not include comparisons with CogVideoX VAE [2] and Open-Sora VAE [3], which are significant baselines in this field.

3. Video quality metrics: Table 1 reports only frame-wise video quality metrics and neglects any temporal consistency metrics. It is recommended to measure the Fréchet Video Distance (FVD) between original and reconstructed videos (rFVD) to provide a more comprehensive evaluation.

4. Video generation results: The results in Table 2 make it difficult to assess the potential of the proposed video generation method. While VideoDiT shows comparable performance when trained on a smaller dataset, it remains unclear if the model would outperform competitors when trained on larger datasets. For the purity of the experiment it is recommended to train VideoDiT on the full WebVid-10m [4] dataset and compare its performance to baselines trained on the same data.

5. Zero-shot T2V benchmark: The evaluation benchmark used in Table 2 is known to have a poor correlation with human perception [6]. It is recommended to use more sophisticated benchmarks for evaluation, such as Vbench [7], to provide a more comprehensive assessment of the model's performance.

6. Missed T2V comparisons: Table 2 does not include comparisons with CogVideoX [2], Open-Sora [3], and Open-Sora-Plan [3], which are significant baselines of Video DiT architectures.