Autoregressive Video Generation without Vector Quantization

[Q1]. The paper mentions temporal encoder, spatial encoder, and decoder with 16 layers each on line 260, but the full article does not explain what the encoder and decoder in spatial layers are.

[A1]. Thank you for your question and valuable feedback. As discussed in Section 3.3 (Line 199), our spatial layers adopt the encoder-decoder architecture of MAR [1], similar to MAE [2]. Specifically, the encoder processes the visible patches for reconstruction. The decoder further processes visible and masked patches for generation. We have revised this part in Sec 4.1.

[Q2]. In Figure 1, when Spatial Layers performs mask modeling on S2, it can directly obtain the complete embedding output by S2 after being encoded by Temporal Layers. Isn't this a kind of information leakage? However, in the actual inference generation At the time of S2, there was no known S2. We only had S1 before we get S2.

[A2]. Thanks for your valuable feedback. We acknowledge the ambiguities in Figure 1. To clarify, the input S2′ of the spatial layers originates from the <BOV> token and the S1-attended outputs of the temporal layers, not the <BOV> token and S2-attended outputs. Thus, there is no information leakage. We have corrected this in Figure 1 and Section 3.3.

To eliminate any potential confusion, we outline the inference process of NOVA in more detail below:

• Initially, the output temporal-layer features of the BOV tokens (serving as anchor features) are scaled by 1 and shifted by 0 to act as the first-frame condition in the spatial layers.
• The first-frame temporal features are projected to scale and shift tensors, which are subsequently used to apply element-wise affine transformations to the anchor features. This process generates the second-frame condition for the spatial layers.
• During training, this autoregressive process is parallelized using a frame-wise causal mask in the temporal layers.

[Q3]. in the evaluation on T2I-CompBench mentioned by the author in lines 292-295 of the article, there are only 300 evaluation prompts for each category, and the others are training prompts. And according to the setting of the original T2I-CompBench paper, each evaluation prompt should generate 10 images.

[A3]. Thank you for pointing out typos. We have thoroughly reviewed our evaluation code and confirmed that all results in the paper were obtained using standard settings: For each sub-category in the testing set, we generate 10 images per prompt for 300 prompts, resulting in a total of 18,000 images generated in a zero-shot manner. We have revised in Sec 4.1.

[Q4]. In the comparison of Table 2, there is no comparison with the most advanced diffusion model. At least it should be compared with SD3 and DALL-E3.

[A4]. Thanks for your valuable feedback. Given that SD3 and DALL·E3 were not assessed in T2Ibench, we opted to use Geneval[1] for additional comparisons. By further training NOVA on larger text-to-image datasets, we achieved significantly superior results compared to commercial models like DALL-E 3 and SD3, showcasing its impressive scaling properties as follows:

[1] Ghosh, Dhruba, Hannaneh Hajishirzi, and Ludwig Schmidt. "Geneval: An object-focused framework for evaluating text-to-image alignment." Advances in Neural Information Processing Systems 36 (2024).

[Q5]. For the curve in Figure 8(a), loss is optimized best under the image-to-video (w/o scale&shift) setting. Doesn’t this mean that the scale&shift operation is useless? Then why do we need to add this operation?

[A5]. Thanks for your comment. As discussed in Lines 428–471, a lower training loss without the scale-shift layer does not necessarily indicate better generative performance. We infer that this behavior is caused by overfitting redundant information among video tokens. This is clearly demonstrated in Figure 9, which shows that as the inner rank of the MLP in the scale-shift layers increases, NOVA's visual performance declines. Moreover, removing the scale-shift layer altogether results in the worst visual outcomes. By introducing the scale-shift layer and reducing its inner rank, we increase the learning difficulty, which helps NOVA focus more effectively on learning structural and motion priors, as illustrated in Figure 9.

[Q6]. In order to ensure fairness when comparing with other methods in Table 2, you should submit the comparison results under the same settings. This is also to ensure that the settings of other methods can be unified when comparing in the future. And you should emphasize whether you conducted the evaluation under zero-shot or the result after fine-tuning on the training set.

[A6]. Thank you for your suggestion. All results of NOVA presented in this paper were evaluated in a zero-shot setting. To showcase the generalization and scalability of our model, we trained it on a larger dataset and tested it on multiple benchmarks, as outlined in Q4. We will release the code and model weights to ensure transparency and reproducibility.

Dear Reviewer xK2A,

Thank you once again for your valuable suggestions and comments. With the discussion deadline approaching, we are more than happy to provide any further clarification you may need.

In our previous response, we took great care in addressing your feedback and have made the following updates and additions:

• Provided further explanations of our architecture.
• Offered a detailed description of Fig.1.
• Provided a revised T2I-CompBench setting.
• Included a comparison with more SOTA models on GenEval.
• Provided further explanations of Figure 8(a) and the role of the Scaling and Shift layer.
• Provided further explanations of evaluation settings.

We trust that the new experiments and explanations have clarified the contributions and enhanced the quality of our submission. If you require any further clarifications or additional analyses, please feel free to reach out.

Thank you for your time and valuable feedback!

Best regards,
The Authors

Dear Reviewer xK2A,

Thank you once again for your valuable suggestions and feedback.

As the discussion deadline approaches, please feel free to reach out if you need any further clarifications or additional experiments.

[Q1]. seems like an extension of MAR on video generation task.

[A1]. Thanks for your feedback.

(1) Currently, MAR is limited to class-to-image generation and has not been tested on the more complex task of text-to-image generation. A natural approach to extend MAR to text-to-video generation would involve generating multiple images simultaneously and applying autoregression to a single large image. However, our experiments reveal significant challenges with this method, including issues with mask scheduling, autoregressive setup, computational efficiency, generation quality, and, most importantly, scalability across diverse contexts.

(2) While inspired by MAR, notably, it is non-trivial at all from MAR to NOVA. Comparing to MAR, NOVA not only addressed the issues stated above, but also presented an efficient and state-of-the-art framework for video generation. NOVA is a brand-new autoregressive generation framework that first predicts temporal frames sequentially and then processes spatial sets within each frame. NOVA is also the first to integrate a non-quantized autoregressive model for video generation. This approach improves the model's ability to naturally handle a wide range of contextual conditions, including text-to-image, image-to-video, and video extrapolation, extending beyond the capabilities of existing methods. Moreover, NOVA offers potential compatibility with kv-cache acceleration, ensuring both high efficiency and superior generation quality. We believe that NOVA will bolster community confidence in this emerging technique and inspire further advancements in the field.

[Q2]. In Table 3, I don't see improvements in the proposed on the basis of previous diffusion-based methods.

[A2]. Thanks for your valuable comment.

(1) We are actively working on experiments involving larger models on extensive video datasets. At present, we have invested considerable effort in training larger text-to-image models using extensive datasets, serving as a stronger foundation for video generation. By further training NOVA on larger text-to-image datasets, we achieved significantly superior results compared to commercial models like DALL-E 3 and SD3, showcasing its impressive scaling properties as follows:

(2) However, these experiments on text-to-video generation are resource-intensive and time-consuming, so obtaining final results will require some time. If we complete them within the review period, we will promptly share the findings.

[Q3]. Are AR-based methods really needed for video generation tasks?

[A3]. The temporal autoregressive characteristic enables NOVA to handle a wide range of contextual conditions in one single model, including text-to-image(Figure 3), image-to-video(Figure 4), video extrapolation(Figure 5) and others(Figure 6). Meanwhile, masked spatial-wise autoregression allows for bidirectional modeling combined with per-token denoising process, demonstrating high efficiency(12s vs 50s+ of diffusion models in Table 3) and superior generation quality (75.84 vs 75.66 of opensora 1.1 in Table 3). We believe that NOVA demonstrates strong model capabilities and scaling efficiency, paving a promising route of autoregressive branch for video generation tasks.

Dear Reviewer yFXn,

Thank you once again for your valuable suggestions and comments. With the discussion deadline approaching, we are more than happy to provide any further clarification you may need.

In our previous response, we took great care in addressing your feedback and have made the following updates and additions:

1. Provided further explanations of our novelty and contributions to the community.
2. Conducted additional experiments to examine the improvements on the basis of previous diffusion-based methods.
3. Provided an explanation of the role of AR based methods in video generation task.

We trust that the new experiments and explanations have clarified the contributions and enhanced the quality of our submission. If you require any further clarifications or additional analyses, please feel free to reach out.

Thank you for your time and valuable feedback!

Best regards,
The Authors

Dear Reviewer yFXn,

In response to your inquiry about the improvements made to our model based on previous diffusion-based methods, we have gathered an additional 7 million high-quality video samples and retrained our model. This has led to a significant performance improvement, from 75.84 to 80.12, as shown in the attached table. Our updated results are competitive with the latest state-of-the-art T2V models, such as OpenSoraPlan V1.3 (77.23), LTX-Video (80.00), Mochi-1 (80.13) and CogVideoX-2B (80.91). These findings further highlight the scalability and robustness of our approach when applied to larger and more complex datasets.

Since the deadline of discussion is approaching, we are happy to provide any additional clarification that you may need.

[Q4]. The ablation study on frame-by-frame autoregressive modeling lacks clarity

[A4]. Thanks for your comment. For better convenience and clearer presentation, we have refined figure 7 in the revised version. In addition, we have added a comparison of quantitative results in Section E of the supplementary material. Our findings are summarized as follows:

(1) Efficient Motion Modeling : We observed that the total score marginally lower than compared to NOVA(75.38 vs 75.84), especially in terms of the dynamic degree metric, which showed a more pronounced decline(11.38 vs. 23.27). We hypothesize that while bidirectional attention enhances model capacity, it demands more extensive data and longer training times to capture subtle motion changes compared to causal models.

(2) Efficient Video Inference : Thanks to the autoregressive generation of the kv-cache technology and frame-by-frame processing, NOVA's inference time is much faster compared to methods without TAM, with a greater speed advantage for longer videos.

[Q5]. However, these claims are not sufficiently supported by the experimental results.

[A5]. Thank you for your question and valuable feedback.

claim 1 : NOVA combines the advantages of temporal autoregressive modeling with spatial bidirectional modeling. The temporal autoregressive characteristic enables NOVA to handle a wide range of contextual conditions in one single model, including text-to-image(Figure 3), image-to-video(Figure 4), video extrapolation(Figure 5) and others(Figure 6). Meanwhile, masked spatial-wise autoregression allows for bidirectional modeling combined with per-token denoising process, demonstrating high efficiency(12s vs 50s+ of diffusion models in Table 3) and superior generation quality (0.72 vs 0.68 of SD3 in Supp. Table 2). We believe that these results can well validate that NOVA combines the superiority of both AR and diffusion approaches.

claim 2 : NOVA demonstrates data and parameter efficiency in practice. With fewer computational resources and model capacity, NOVA achieves notable results in text-to-image generation (0.4955 vs 0.4815 of PixArt-α in Table 2) and comparable performance in text-to-video domain (75.84 vs 75.66 of opensora 1.1 in Table 3). In the supplementary material, we present additional text-to-image experiments using more data and larger model capacities. As shown in Supp. Table 2, NOVA with 1.4B parameters achieves state-of-the-art (SOTA) results (0.72 vs. 0.68 for the strongest SD3), highlighting its promising potential for text-to-video generation. If additional experiments are completed within the review period, we will promptly share the findings. These results underscore NOVA’s strong model capabilities and scaling efficiency, demonstrating notable data and parameter efficiency.

[Q6]. To make the Scaling and Shift Layer easier to understand, the authors could mention its similarity to FiLM[ or AdaIN.

[A6]. Thank you for your suggestions. We have supplemented the description of Scaling and Shift Layer in the revised appendix for better understanding.

[Q7]. Typo: Line 296 - each with

[A6]. Thanks for your reminder, we have corrected it in the revised

[Q1]. feels like a straightforward extension of MAR

[A1]. Thanks for your feedback.

(1) Currently, MAR is limited to class-to-image generation and has not been tested on the more complex task of text-to-image generation. A natural approach to extend MAR to text-to-video generation would involve generating multiple images simultaneously and applying autoregression to a single large image. However, our experiments reveal significant challenges with this method, including issues with mask scheduling, autoregressive setup, computational efficiency, generation quality, and, most importantly, scalability across diverse contexts.

(2) While inspired by MAR, notably, it is non-trivial at all from MAR to NOVA. Comparing to MAR, NOVA not only addressed the issues stated above, but also presented an efficient and state-of-the-art framework for video generation. NOVA is a brand-new autoregressive generation framework that first predicts temporal frames sequentially and then processes spatial sets within each frame. NOVA is also the first to integrate a non-quantized autoregressive model for video generation. This approach improves the model's ability to naturally handle a wide range of contextual conditions, including text-to-image, image-to-video, and video extrapolation, extending beyond the capabilities of existing methods. Moreover, NOVA offers potential compatibility with kv-cache acceleration, ensuring both high efficiency and superior generation quality. We believe that NOVA will bolster community confidence in this emerging technique and inspire further advancements in the field.

[Q2]. testing about scalability

[A2]. Thanks for your valuable comment.

(1) We are actively working on experiments involving larger models on extensive video datasets. At present, we have invested considerable effort in training larger text-to-image models using extensive datasets, serving as a stronger foundation for video generation. By further training NOVA on larger text-to-image datasets, we achieved significantly superior results compared to commercial models like DALL-E 3 and SD3, showcasing its impressive scaling properties as follows:

(2) However, these experiments on text-to-video generation are resource-intensive and time-consuming, so obtaining final results will require some time. If we complete them within the review period, we will promptly share the findings.

[Q3]. whether extrapolation is truly unique to autoregressive (AR) models.

[A3]. Thanks for your comment. We acknowledge that several training-free methods [1] enable text-to-video diffusion models to extrapolate beyond their training lengths. However, these approaches often struggle with temporal inconsistencies [2]. A more practical alternative is to train (or fine-tune) a separate video-to-video diffusion model specifically for video extrapolation. In contrast, NOVA's autoregressive design allows it to naturally generalize across extended video durations and, more importantly, handle a diverse range of zero-shot tasks within a single unified model. Its temporal autoregressive framework enables seamless execution of various tasks, such as text-to-image, text+image-to-video, and text+video-to-video generation, offering significantly more flexibility compared to diffusion-based methods.

[1] Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., & Fleet, D. J. (2022). Video Diffusion Models. http://arxiv.org/abs/2204.03458

[2] Luo, Z., Chen, D., Zhang, Y., Huang, Y., Wang, L., Shen, Y., Zhao, D., Zhou, J., & Tan, T. (2023). VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation. 10209–10218. http://arxiv.org/abs/2303.08320

Dear Reviewer nKGc,

Thank you once again for your valuable suggestions and comments. With the discussion deadline approaching, we are more than happy to provide any further clarification you may need.

In our previous response, we took great care in addressing your feedback and have made the following updates and additions:

1. Provided further explanations of our novelty and contributions to the community.
2. Included a comparison of the scaling behavior of NOVA.
3. Provided an explanation of the relationship between extrapolation and autoregressive (AR) Models.
4. Conducted additional experiments to examine the effect of frame-by-frame autoregressive modeling.
5. Provided detailed explanations of our two claims.
6. Provided a detailed description of the Scaling and Shift Layer.
7. Fixed some typos.

We trust that the new experiments and explanations have clarified the contributions and enhanced the quality of our submission. If you require any further clarifications or additional analyses, please feel free to reach out.

Thank you for your time and valuable feedback!

Best regards, The Authors

First, I’d like to thank the authors for sincerely addressing my comments and conducting the experiments.

After reviewing the authors' responses, my concerns regarding the novelty, scalability, and the effectiveness of TAM have been resolved. While it would have been nice to see scalability experiments for NOVA on video data in addition to images, I believe the NOVA's temporal autoregressive modeling and the expansion of a Transformer without vector quantization to large-scale text-to-image and text-to-video data have significant value. Therefore, I will raise my score from 5 to 6.

This paper is already valuable as it is, and while I understand the cost challenges of conducting video experiments, I believe it would become an even stronger paper if scalability experiments on videos were added in the future.

[Q1]. inconsistency between training and inference

[A1]. Thanks for your valuable feedback. We acknowledge the ambiguities in Figure 1. To clarify, the input S2′ of the spatial layers originates from the <BOV> token and the S1-attended outputs of the temporal layers, not the <BOV> token and S2-attended outputs. Thus, there is no discrepancy between the training and inference phases of NOVA.

[Q2]. A step-by-step explanation of how frames are generated during inference.

[A2]. Specifically, the temporal-layer outputs of <BOV> tokens (which also serve as anchor features) are scaled by 1 and shifted by 0 to act as the first-frame condition in the spatial layers. The first-frame temporal features are further projected into scale and shift tensors, which are applied via element-wise affine transformations to the anchor features, forming the second-frame condition in the spatial layers. During training, this autoregressive process is parallelized using a frame-wise causal mask in the temporal layers. We recognize that the misrepresentation in Figure 1 regarding the connection between temporal and spatial layers contributed to this misunderstanding. We have corrected this in Figure 1 and Section 3.3.

[Q3]. The exact number of frames used during training is not clear.

[A3]. NOVA processes 29 frames during both training and testing processes, corresponding to approximately 2.417 seconds at a frame rate of 12 frames per second. To ensure clarity, we have refined the related descriptions to avoid any ambiguity.

[Q4]. Is the context truncated when the length of video extrapolation exceeds 29 frames?

[A4]. Thank you for your comment.

(1) In our approach, we perform video extrapolation by inferring frames within each training-length clip, which is consistent with the training process. For example, to generate the 30th frame, we condition on the text, flow, BOV tokens, and the 29th frame, instead of relying on all preceding frames.

[text, flow, bov, x_1, ..., x_28] -> x_29 # training length is reached
[text, flow, bov, x_29] -> x_30 # infer video extrapolation frames within a new training length
...
[text, flow, bov, x_29, ..., x_56] -> x_57 # a new training length is reached
...
[text, flow, bov, x_57] -> x_58 # infer video extrapolation frames within a next training length

This approach utilizes an anchor feature set shared across all generated frames, ensuring coherence in extrapolated video frames. We found that this straightforward strategy effectively maintains temporal consistency while significantly reducing the computational cost of extrapolation.

(2) We also explore inferring video extrapolation frames within a fixed-length video clip. However, the use of 1D temporal positional embeddings requires recalculating the mixed features of positional and previous frame features during each extrapolation, which affects computational efficiency. In the future, we plan to introduce 1D-ROPE[1] alternatives to improve compatibility with this approach and enable a more flexible video extrapolation process.

[1] Su, Jianlin, et al. "Roformer: Enhanced transformer with rotary position embedding." Neurocomputing 568 (2024): 127063.

[Q5]. How are the 1D sine-cosine temporal positional embeddings applied for frames beyond the training length?

[A5]. Thank you for your comment. As described earlier, we perform video extrapolation by predicting frames within each training-length clip. In this process, a fixed number of frames is generated from scratch for each segment, mirroring the training setup. This approach ensures consistency with the training phase while allowing efficient reuse of features from previous frames through the KV-cache. In the future, we plan to explore alternatives to 1D-ROPE to enhance compatibility with fixed-length window strategies, paving the way for a more flexible and efficient video extrapolation process.

[Q6]. What's the time cost for each stage of temporal and spatial autoregression?

[A6]. We report inference times on a single NVIDIA A100 GPU (40GB) with a batch size of 24. The input video resolution is 29 × 768 × 480. Temporal layers account for only 0.03 seconds, while spatial layers take 11.97 seconds, highlighting the exceptional efficiency of the temporal layers. Notably, NOVA demonstrates high efficiency in text-to-video generation and offers potential for further acceleration. Techniques such as speculative decoding [1] present promising avenues for future optimization, which we plan to explore.

[1] Wang, Zili, et al. "Continuous Speculative Decoding for Autoregressive Image Generation." arXiv preprint arXiv:2411.11925 (2024).

[Q7]. What is the limit of the model's video extrapolation capability?

[A7]. Thank you for your valuable feedback. Through visualization experiments, we empirically found that the model can extrapolate up to 3$\times$training length. We conducted quantitative experiments using LPIPS and PSNR metrics, and discovered that per-frame PSNR values decrease, while LPIPS scores increase over time. Nevertheless, the generated frames show a high degree of similarity to the original video in terms of both content and image quality. For detailed experimental settings and discussions, see Section C of the supplementary material.

[Q8]. better to provide the results of NOVA and compare them with the results of the simple baseline.

[A8]. Thank you for your valuable suggestion. We have completed the revisions in the revised version.

[text, flow, bov, x_1, ..., x_28] -> x_29 # training length is reached
[text, flow, bov, x_2, ..., x_29] -> x_30 # infer video extrapolation frames

Dear Reviewer GfSG,

We express our sincere gratitude to the reviewer for dedicating time to review our paper. In our previous response, we took great care in addressing your feedback and have made the following updates and additions:

1. Offered a detailed description of Fig.1.
2. Offered a detailed explanation of frame generation during inference.
3. Offered an unambiguous description of frame numbers.
4. Offered a detailed video extrapolation process.
5. Conducted additional inference time analysis of temporal and spatial layers.
6. Conducted additional experiments to evaluate the limit of the model's video extrapolation capability.
7. Adjustments made to make the visualization results more clearly displayed

We trust that the new experiments and explanations have clarified the contributions and enhanced the quality of our submission. If you require any further clarifications or additional analyses, please feel free to reach out.

Thank you for your time and valuable feedback!

Best regards,

The Authors