ARLON: Boosting Diffusion Transformers with Autoregressive Models for Long Video Generation

W3. Although the authors introduce noise-resilient strategies for the DiT model training to mitigate the error issue from AR inference, I am concerned that these strategies cannot truly simulate the error of AR inference, which may limit the model performance.

Response: We thank the reviewer for the insightful comment!

We would like to clarify that

1. There are two noise-resilient strategies are utilized in our work, coarser visual latent tokens and uncertainty sampling. We acknowledge that the approach of coarser visual latent tokens does not fully simulate the errors inherent in AR inference, as it is a holistic simulation, whereas the errors introduced by AR inference occur at the token level. However, the uncertainty sampling method introduces noise at the token level, which is a simulation of the errors introduced by AR inference to some extent.
2. In our preliminary experiments, we employed a more direct simulation strategy, which involved randomly replacing 30-50% of the AR codes with incorrect ones during training. However, this approach yielded unsatisfactory results. We attribute this to the excessive errors introduced by such a simulation method, which confounded the DiT model's ability to discern the underlying relationship between the AR codes and the corresponding videos. In comparison, uncertainty sampling serves as a notably more effective approach.
3. In fact, we do not necessarily need to fully simulate the errors introduced during AR inference. Instead, we aim to enable the DiT model to follow the information provided by AR while simultaneously being tolerant of the errors introduced during the AR inference phase, as we discussed in Section 2.3 ("To tolerate the errors inevitably introduced during AR inference, we implement two noise-resilient training strategies: coarser visual latent tokens and uncertainty sampling."). Specifically, for utilizing the coarser visual latent tokens, the coarser the AR code, the less information it contains, and the more the DiT model needs to generate. Even if the AR model provides an incorrect code, the DiT model does not blindly follow it but rather attempts to generate videos that align with ground truth videos. On the other hand, introducing noise through the uncertainty sampling approach introduces noise directly at the token level, offering a more straightforward method to improve the DiT model's robustness against errors.

In addition, we have revised the content of Section 4.3 to avoid any confusion for the readers ("which could simulate the errors" -> "which could make the DiT model tolerate the errors"). The specific changes made are highlighted in red. We once again express our gratitude to the reviewer for the constructive questions, which have helped us improve the quality of the paper.

Q1. In Figure 1, the previously generated video seems to be used as a reference for the subsequent generation, but this is not illustrated in Figure 2. Does the DiT generate videos in an autoregressive way?

Response: We thank the reviewer for the question!

Yes, the DiT model in ARLON generates videos in an autoregressive approach. The reason why it is not illustrated in Figure 2, is that Figure 2 is the overview of the training stage of ARLON, while the autoregressive way is used in training.

As described in Section 2.3 "Training Strategy", to enable this autoregressive approach, we randomly unmask certain frames, keeping them noise-free to serve as conditioning frames. This allows the diffusion model to consider preceding frames as conditions during inference.

Q2. In Table 1 and Table 2, do the higher scores of metrics indicate better performance?

Response: We thank the reviewer for the question!

Yes, in Table 1 and Table 2, the higher scores of metrics indicate better performance.

To improve the readability of Table 1 and Table 2, we have added the sentence "the higher scores of metrics indicate better performance." to the table captions.

Q3. For long video generation in section 4.2 as well as the demo videos on the webpage, the authors only compare their proposed method with open-source text-to-long video generation models. Why not compare with the commercial closed-source text-to-video generation models like in Table 1?

Response: We thank the reviewer for the question!

Commercial closed-source text-to-video models, like Kling, typically produce short clips limited to around 4-10 seconds, lacking continuous scenes over 30 seconds. They often rely on multiple prompts, resulting in stitched rather than seamless videos. Therefore, we focus on comparing open-source text-to-long video generation models.

We would like to express our gratitude to the reviewer for highlighting the innovative combination of AR and DiT for long video generation, the noise-resilient strategies during DiT training, the clear presentation of the paper, and the improvements in dynamic degree and consistency of long video generation with progressive text prompts. We would also like to thank the reviewer for the constructive suggestions and comments which will be responded to one-by-one below.

W1. From Table 1, the proposed method lags behind compared methods in many metrics other than dynamic degree, such as Imaging Quality and Subject Consistency. In Table 2, the dynamic degree of the proposed method (50.42) is significantly lower than that of the StreamingT2V(85.64).

Response: We thank the reviewer for the insightful comments!

Table 2 (Table 1 in the original manuscript) presents the metrics for short video generation. Due to the presence of many closed-source algorithms and the inconsistency in model sizes and training datasets, it is challenging for any single algorithm to excel across all metrics. Our approach is based on OpenSora, and it significantly outperforms the baseline OpenSora-V1.2 on eight out of eleven metrics selected from VBench, with notable improvements in dynamic range and aesthetic quality. Moreover, it delivers competitive performance on the remaining three metrics, while also accelerating the video generation process. Notably, our method focuses on long video generation, where it excels in maintaining both consistency and dynamic range over extended sequences.

Regarding long video generation of Table 1, (Table 2 in the original manuscript), while StreamingT2V achieves a high dynamic degree, this is primarily due to its chaotic scene transitions and frequent object movements, which often lead to instability. As shown in Figures 4 (Figure 5 in the original manuscript) and 14, StreamingT2V demonstrates substantial changes in object motion and background over time, with sudden object switches, abrupt scene changes, and screen disruptions. Consequently, it exhibits a high degree of dynamism but suffers from low consistency in both subject and background. In contrast, our approach ensures consistent backgrounds and subjects while achieving high dynamic scores in long video generation, maintaining both coherence and dynamism over extended video sequences.

W2. From the demo videos on the webpage, there is some room for improvement for the proposed method compared to others. For example, the result of "A teddy bear is swimming in the ocean." lacks of subject consistency, and its motion is not realistic, which may be consistent with the quantitative results in Table 1 and Table 2.

Response: We thank the reviewer for the insightful comment!

We would like to clarify that the demo of "A teddy bear is swimming in the ocean." is for long video generation. This aligns with the quantitative results presented in Table 1 (Table 2 in the original manuscript), where our model, ARLON, demonstrates the highest levels of subject consistency and background consistency, along with superior dynamics, when compared to other open-source models.

However, we acknowledge that there is indeed potential for enhancement in our models. We are confident that integrating a more robust and advanced DiT model could yield even superior results. Moving forward, we intend to persistently update and investigate improvements to our methodology.

Dear Reviewer 45zF,

We sincerely appreciate your thoughtful feedback, which we have carefully addressed in our response. We hope our responses effectively address your questions. If you have any further inquiries or suggestions, we would be delighted to engage in further discussion before the conclusion of the discussion period. Your insights have been invaluable in enhancing our work, and we look forward to your thoughts on the revised version.

Best regards

Thanks for the authors' reply, my raised concerns have been mostly addressed. After I further read the author's responses to the questions I raised, as well as the communication between the author and other reviewers, I am more confident in the technical novelty and value of this paper. I tend to slightly raise my score.

We would like to express our gratitude to the reviewer for acknowledging the motivation and contributions of our ARLON, the promising qualitative and quantitative results, and the quality of our paper's writing. We would also like to thank the reviewer for the constructive suggestions and comments which will be responded to one-by-one below.

W1. No ablation on the impact of model structure and training data size.

Response: We thank the reviewer for the insightful comments!

We would like to clarify that the analysis of the model structure has already been provided in the second point of Section 4.3. Specifically, we compare the performance of the semantic injection module across various structural configurations, including ControlNet, MLP adapter and adaptive norm.

Additionally, we would like to present an analysis of the impact of varying training data sizes.

Firstly, when comparing our model's performance on the OpenVid 1M dataset with that on OpenVid-HQ (which contains higher quality videos, totaling 0.4M), we observed a marked improvement in our model's performance on OpenVid-HQ. This indicates that the quality of the data plays a crucial role in the task of video generation.

Furthermore, when we combined the OpenVid-HQ and Mixkit (the quality of videos is also high) datasets as our training set (approximately 0.7M), improvements in both quality and dynamic degree are obtained. This suggests that in the context of video generation, prioritizing high-quality videos while also utilizing a larger dataset can effectively enhance the overall quality of generated videos.

We have incorporated these results into the Appendix, which can be found in Table 5. We greatly appreciate your valuable feedback and suggestions.

W2. No discussion on failure cases and limitations.

Response: We thank the reviewer for the insightful comments!

We have included the failure cases in the Appendix and provided explanations for why these cases failed, along with an analysis of potential future directions for exploration. The details can be found in Appendix Section A4 and Figure 16 of the revised manuscript.

In addition, Limitations and Broader Impact are also provided in the Appendix, which can also be found in the response to Ethics Concerns.

W3. There might be some missing references such as nuwa-XL and Phenaki. GAN-based long video generation might also be related.

Response: We thank the reviewer for the insightful comments!

We have revised the related work section to include these references including NUWA-XL, Phenaki and GAN-based long video generation works, and discuss their relevance to our approach.

Please find them in Section 3.2 of the updated manuscript (Highlighted in blue).

Q1. How many seconds can the models generate for the longest videos and how is the performance? For How many seconds do the longest videos that your method generates can last? In my understanding, this is the key advantage of the hierarchical generation framework.

Response: We thank the reviewer for the insightful questions.

For our model, ARLON, as depicted in Figure 1 of the paper, all coarse AR codes are generated in a single inference pass, providing coarse spatial and long-range temporal information. This effectively guides the DiT model to autoregressively produce high-quality videos with rich dynamic motion. In comparison to previous methods, each segment generated by ARLON exhibits notable consistency, leading to an outstanding performance in terms of both the quality and consistency of the final long video. we agree that this is the one of key advantages of ARLON. The longest duration of our training video clips for AR model is about 60 seconds, therefore the quality of generated videos by ARLON can last at least 60 seconds.

Although employing autoregressive approaches for long video generation using DiT models—where successive video segments are generated based on the last frames of the previous segment— theoretically allows for the generation of videos of almost infinite length. However, the inherent computational constraints limit the length of these conditioned segments, thereby restricting the historical context available for the generation of each new segment. Moreover, when identical text prompts are used, the generated short video segments frequently exhibit identical content, heightening the risk of repetition throughout the entire long video. Additionally, the process of autoregressive generation inevitably leads to the accumulation of errors. Given these three critical points, the quality of videos generated using solely autoregressive DiT models is notably poor.

Q2. The main limitation of this work seems to be the huge computational cost of training, but the related information (type and number of GPU, training time) is not provided. It would be nice to know this information.

Response: We thank the reviewer for the insightful comments!

We would like to clarify that the computational cost of training our model is not huge compared to the baseline OpenSora-V1.2. This is because we initialized the parameters of the DiT model (1.2 B) from OpenSora-V1.2 and subsequently froze them. Consequently, only an additional 192 M + 92 M + 30 M parameters require training (for more details, please refer to the Summary response).

For your interest in the computational aspects of our work, we would like to provide the details. All experiments were conducted on NVIDIA A100 40G GPUs. Specifically, the AR model requires one day of training on 8 NVIDIA A100 40G GPUs, while the DIT model takes two days of training on 32 NVIDIA A100 40G GPUs (For reference, the training of OpenSora-v1.1 requires approximately 9 days on 64 H800 GPUs). Therefore, our method demonstrates a lower computational cost. We hope this information addresses your concerns.

Ethics Concerns: Text-to-video models can generate harmful and troublesome content is a broad concern, the discussion on this problem is needed.

Response: We thank the reviewer for the insightful comments!

We would like to provide the Limitations and Broader Impact.

Limitations

Although ARLON achieves state-of-the-art performance in long video generation, it also exhibits some specific constraints. First, ARLON is built upon OpenSora-V1.2, which potentially caps the upper limit of video quality. Nonetheless, this limitation can be mitigated by substituting the DiT model with more advanced alternatives, such as CogVideoX-5B or MovieGen. Second, if we aim to train ARLON at 2K resolution, the sequence length of AR codes will become excessively long, making both training and inference impractical. Viable solutions involve employing a higher compression ratio in VQ-VAE, or selectively retaining essential information while disregarding irrelevant details. Additionally, for the AR model, parallel prediction emerges as an alternative approach. Our future research endeavors will delve into addressing these issues.

Broader Impact

Synthetic video generation is a powerful technology that can be misused to create fake videos or videos containing harmful and troublesome content, hence it is important to limit and safely deploy these models. From a safety perspective, we emphasize that the training data of ARLON are all open-sourced, and we do not add any new restrictions nor relax any existing ones to OpenSora-V1.2. If you suspect that ARLON is being used in a manner that is abusive or illegal or infringes on your rights or the rights of other people, you can report it to us.

We also add the Limitations and Broader Impact into the Appendix, please find them in A.4 and A.5.

Dear Reviewer Rwnx,

We sincerely appreciate the valuable feedback provided you have provided. We have taken great care to address all the concerns in detail. As we near the conclusion of the discussion phase, we genuinely value your insights and would be grateful for any further feedback you may have. We hope that our responses meet your expectations and remain open to addressing any additional questions you might have. Thank you once again for your time and consideration.

Best regards,

Authors

We would like to express our gratitude to the reviewer for highlighting the novelty and the acceleration capacity of our ARLON. We would also like to thank the reviewer for the constructive suggestions and comments which will be responded to one-by-one below.

W1. It's better to use bold fonts and underscores in Table 1. According to the listed numbers, the proposed method doesn't reach the top 3 in many columns. The reproduced notation is missing.

Response: We thank the reviewer for the insightful comments!

We will use bold fonts and underscores for the best numbers in Table 2 (Table 1 in the original manuscript). All the compared results in Table 2 (Table 1 in the original manuscript) are numbers reported in their respective papers, and we will remove the sentence 'The reproduced notation is missing.'. We also would like to clarify the results compared with other methods as follows:

1. The performance of ARLON is also outstanding in the field of short video generation. Our ARLON is built upon OpenSora-V1.2, making it a fair comparison to OpenSora-V1.2. As evidenced by Table 2 (Table 1 in the original manuscript), it is observed that ARLON significantly outperforms the baseline OpenSora-V1.2 on eight out of eleven metrics selected from VBench, with notable improvements in dynamic range and aesthetic quality. Moreover, it delivers competitive performance on the remaining three metrics, while also accelerating the video generation process.
2. Due to the presence of many closed-source algorithms and the inconsistency in sizes of model parameters and training datasets, it is challenging for any single model to excel across all metrics.
3. ARLON focuses on long video generation, where it excels in maintaining both consistency and dynamic range over extended sequences as highlighted in Table 1 (Table 2 in the original manuscript). In contrast, Table 2 (Table 1 in the original manuscript) provides the metrics for short video generation.
4. To better present results and emphasize our strengths, we have divided Table 2 (Table 1 in the original manuscript) into two sections. The lower section presents the results of ARLON and OpenSora-V1.2, with our superior results highlighted with specific improvements. The upper section, meanwhile, displays the results of other text-to-video models.

Q1. What is the motivation of using a latent VQ-VAE nested inside a pretrained 3D Autoencoder, instead of training a single-stage pixel-to-latent tokenizer? And what is the additional computational cost or gain in comparison?

Response: Thank you for the insightful question.

The motivation for using a latent VQ-VAE nested inside a pretrained 3D Autoencoder, rather than training a single-stage pixel-to-latent tokenizer, is based on several key considerations:

1. Leveraging the Benefits of Pretrained Models: Initiating with a large-scale data pretrained model is an efficient and effective strategy. OpenSora-v1.2, a text-to-video DiT-based model based on a 3D VAE, has been trained on an extensive corpus of video data. It stands out as one of the most effective open-source models, boasting a significant following and serving as our baseline model. Following OpenSora-v1.2's setup, we leverage its pretrained 3D VAE.
2. Ensuring a Consistent Latent Space between AR and Diffusion Models: Constrained with the latent space of the pre-trained 3D VAE and the OpenSora-v1.2, we need to convert the sequence of features generated by the 3D VAE encoder to discrete token sequence for AR model training, as well as a reverse conversion for inference. Aiming this, we introduce a nested VQ-VAE to align the semantic space of the AR and diffusion models.
3. Balancing Information Density and Learning Complexity: The VQ-VAE, when configured with an appropriate compression ratio, efficiently condenses the input latent space into a compact and highly quantized set of visual tokens, while retaining the essential information. This allows the AR model to focus on predicting coarse information rather than grappling with fine-grained details, thereby enhancing learning efficiency.

In addition, we would like to clarify the additional computation cost and gains:

Although the introduced VQ-VAE may incur some additional computational overhead, we believe this cost can be justified.

1. The quantization process of VQ-VAE substantially reduces the amount of information that the subsequent AR model needs to process, thereby alleviating the overall computational burden. Furthermore, the pretrained 3D VAE can accelerate the training process and decrease the number of iterations required, ultimately conserving computational resources in the long term. Consequently, despite a small potential short-term rise in computational cost, we are confident that this design choice is advantageous in terms of overall efficiency and performance.
2. Furthermore, since the latent VQ-VAE operates within the latent space, its parameters are significantly smaller than the 3D VAE.

We have incorporated these results into the Appendix, which can be found in A.2. We greatly appreciate your valuable feedback and suggestions.

Q2. The proposed coarse latent token with different compression ratio, while how is this ablated in Table 3 or any other ablation studies? (Does the 4×8×8 row refer to both the same scale training and the 4×16×16 rows refer to different scale training?)

Response: We thank the reviewer for the insightful question!

We are sorry that the column name may cause some confusion. The "4×8×8" row indicates that both the AR model and the semantic injection module employ a latent VQ-VAE with a 4×8×8 compression ratio during training. In contrast, "4×16×16" indicates that while the AR model uses the 4×8×8 compression ratio latent VQ-VAE, the semantic injection module is trained with a 4×16×16 scale. In this configuration, the DiT module is provided with a coarser latent representation, which makes the DiT model tolerate the errors introduced in the AR inference, thereby improving its robustness, and maintaining the consistency and qualities of the generated videos.

To further improve the readability of Table 3, we have changed the column name "Compress Ratio" to "Compress Ratio in DiT", and added the description "The compression ratio of the latent VQ-VAE for AR model is 4x8x8" into the table caption.

Dear Reviewer gQNT,

We sincerely thank you for your thoughtful feedback and for taking the time to review our response and update your evaluation. Your insights and suggestions have been invaluable in enhancing our manuscript, and we deeply appreciate your engagement. We are also grateful for your recognition of ARLON's novelty and effectiveness, as well as your increased rating.

Best regards,

Authors

Q3: The hierarchical generation of cascaded AR-diffusion is a novel framework. Is there any related work with similar prototype or inspiring you? I see that Sec. 3.2 has been enriched while it would help if more discussion could be provided from this perspective (e.g. how are the sliding window or diffusion-over-diffusion etc. methods relevant and different from your proposed one; why does your proposed frame condition connection outperform previous key frame injection approach; etc).

Response: We thank the reviewer for the insightful comment!

We were inspired by advancements in the speech generation domain, with a notable example being VALL-E [1]. This work incorporates an autoregressive model to generate the first layer codes of Encodec [2], followed by a non-autoregressive model to predict the subsequent layers of codes. For ARLON, it first generates long-term, coarse-grained discrete visual units (AR codes) autoregressively using a decoder-only Transformer. These discrete AR codes are then segmented and sequentially fed into the DiT model to autoregressively generate high-quality video segments.

At the same time, we would like to illustrate the difference between these two models, even ignoring the difference in modality. The information percentage of AR codes varies significantly between ARLON and VALL-E. In ARLON, AR codes are characterized by a coarse granularity, primarily due to the highly compact nature of the latent VQ-VAE. Conversely, in VALL-E, the first layer codes are densely packed with the majority of the information, a result of the Residual-VQ mechanism employed by Encodec. Therefore, within the ARLON framework, the DiT model plays a more central role in video generation. In contrast, within the VALL-E system, the AR model is given priority.

Moreover, We would like to elaborate on the relevance and differences of our proposed method compared to existing techniques. Firstly, while sliding window methods ensure consistency between adjacent segments, they struggle to capture long-range dependencies in videos. In contrast, keyframe injection methods often require maintaining a similar appearance to the keyframes; however, this does not guarantee continuity in the action scenes, as relying solely on a single image for constraints can be limiting. As for diffusion-over-diffusion methods, they typically generate keyframes first and then stitch them together, which can lead to a loss of continuity in actions over time, making them more suitable for generating cartoon-style sequences. Our proposed method effectively integrates an autoregressive model for long-term coherence (AR) with a diffusion-based DiT model for short-term continuity (DiT), overcoming the limitations of existing techniques such as sliding window and diffusion-over-diffusion methods. This approach ensures video integrity and detail coherence over extended periods without repetition.
We have detailed the relevance and differences of our proposed method compared to existing techniques in the related work.

[1]. Wang C, Chen S, Wu Y, et al. Neural codec language models are zero-shot text to speech synthesizers[J]. arXiv preprint arXiv:2301.02111, 2023.

[2]. Défossez A, Copet J, Synnaeve G, et al. High fidelity neural audio compression[J]. arXiv preprint arXiv:2210.13438, 2022.

W3. There is no analysis of ARLON's memory footprint during training and inference, which would clarify its computational efficiency relative to models like OpenSora-V1.2.

Response: We thank the reviewer for the insightful comments!

We would like to provide the detailed memory requirements for both training and inference of ARLON:

Caption: The batch size for training both the AR and DiT models is set to 2x68-frame 512x512 video clips, and the inference is evaluated by generating a 68-frame 512x512 video. Both the training and inference processes are executed on a single 40G A100 GPU.

From the Table, it is evident that our method does not require significant additional memory and computational resources compared to the baseline model. Specifically, during the training and inference phases, there are 2.2% and 4.8% relative increases respectively (the AR and DiT models can be trained independently).

Conversely, by leveraging the AR code as an efficient initialization, our model is capable of generating high-quality videos with significantly fewer steps. Consequently, our inference time and total FLOPs are superior to those of the baseline, thereby significantly accelerating the denoising process. Specifically, our model achieves a 48% relative improvement in inference speed and a 64% relative reduction in computational FLOPs. For further results of long video (578-frame) generation, please refer to the general response titled "Model details".

We have incorporated these details into the Appendix, which can be found in Table 4. We greatly appreciate your valuable feedback and suggestions.

We would like to thank the reviewer for highlighting the novelty of our proposed methods and the capability to generate long videos. We would also like to thank the reviewer for the constructive suggestions and comments which will be responded to one-by-one below.

W1. In Table 2, why does StreamingT2V have a higher Dynamic Degree score (85.64) compared to ARLON (50.42)?

Response: We appreciate the reviewer's insightful question.

As illustrated in Figures 4 (Figure 5 in the original manuscript) and 14, and further exemplified in the "Long Video Results" section of the demo page, the videos generated by StreamingT2V exhibit notable fluctuations in object motion and background over time. These include instances of sudden object transitions, abrupt scene alterations, and screen disruptions. These characteristics collectively contribute to a high Dynamic Degree score for StreamingT2V. It should be noted that this dynamism comes at the expense of subject consistency, background consistency, and aesthetic quality. Specifically, StreamingT2V scores lower in these aspects, resulting in videos that are, subjectively, less enjoyable to watch.

We want to highlight that ARLON's primary strength lies in its ability to generate long videos with remarkable consistency and quality. By leveraging the AR model, ARLON achieves top scores in subject consistency, background consistency, and aesthetic quality, while also delivering impressive dynamic scores in long video generation, as shown in Table 1 (Table 2 in the original manuscript).

W2. The paper lacks a detailed comparison of the number of parameters in ARLON versus baseline models.

Response: We thank the reviewer for the insightful comments!

We would like to give the number of parameters for each component in ARLON compared to OpenSora-V1.2.

As illustrated in the Table, the increase in the number of parameters (192M + 92M + 30M) of our method is minimal compared to the 1.2B parameters of the baseline, OpenSora-V1.2. This is primarily due to our adoption of an efficiency adapter approach during training, which enables us to introduce fewer parameters while preserving performance. Additionally, the latent VQ-VAE operates within a compressed latent space, and the AR model is trained to generate highly quantized tokens, both of which significantly contribute to the minimized parameter requirements.

We have incorporated these details into the Appendix, which can be found in Table 4. We greatly appreciate your valuable feedback and suggestions.

Dear Reviewer N9At,

Thank you for your thorough review of our paper. In our previous response and the revised manuscript, we conducted additional experiments and provided detailed explanations addressing your questions and concerns. As we near the end of the author-reviewer discussion phase, we kindly ask you to review our revised manuscript and our responses, and consider reevaluating our work if we have satisfactorily addressed all your concerns. If you have any further questions, please feel free to reach out, and we would be delighted to offer any additional clarification.

Best regards,

Authors

we wanted to reach out to confirm that all your concerns have been adequately addressed. Should you have any further questions or require additional clarifications, please do not hesitate to discuss with us at your earliest convenience.