VideoMAR: Autoregressive Video Generation with Continuous Tokens

Q1: VideoMAR appears to be a direct extension of masked autoregressive (MAR) frameworks.

We respectfully disagree with this assessment. VideoMAR represents a significant advancement over standard MAR frameworks by specifically tailoring the masking mechanism to the unique properties of video.

First, our core contribution lies in the exploration and integration of video-specific masking strategies. We systematically investigated multiple approaches, including a total mask strategy that randomly masks entire frames, to find the optimal design. This thorough investigation goes far beyond a direct extension.

Second, the paper dedicates significant discussion to justifying our choice of the next-frame diffusion loss. This design is not arbitrary. It was chosen because it uniquely eliminates the training-inference discrepancy common in generative models. Our next-frame diffusion loss training fits the best with inference paradigm of next-frame generation, without training-inference gap.

Third, we clearly distinguish VideoMAR from related works. As for the mentioned related works, NOVA and MAGI, we comprehensively discussed their differences in Line 37-42, and Line 73-77. NOVA proposes to decompose the temporal and spatial generation via generating the coarse features frame-by-frame and refines each frame with a spatial layer, but complicates the framework and weakens the temporal smoothness. MAGI mitigates this issue by appending a complete copy of video sequence during training, but doubles the the sequence length and training cost. MarDini is similar to NOVA in design, and proposes that MAR handles temporal planning, while DM focuses on spatial generation. We don't include MarDini in the paper since it is actually different from MAR. MAR focuses on token-wise modeling with a small diffusion MLP (less than 10M), while MarDini focuses on image-wise modeling with a large transformer architecture (288M).

Q2: Presenting progressive training as a primary contribution.

We respectfully argue that this perspective misses the specific context of our contribution. The innovation is not progressive training in isolation, but its strategic and effective integration within a masked autoregressive system for video generation.

(1) Contextual Novelty: While progressive training is indeed prevalent in diffusion models, its application to autoregressive video models is notably rare and poses unique challenges. To our knowledge, our work is the first to successfully implement and scale progressive training for masked video generation. Unlike prior methods, our system demonstrates its viability and effectiveness at a large scale, which we believe is a novel contribution.

(2) Systematic Impact and Efficiency: The use of progressive training is not an isolated trick but a cornerstone of our system's design. This strategy is directly responsible for the remarkable efficiency gains that allow our model to surpass the strong Cosmos baseline using significantly fewer computational resources. The from-scratch training cost of our method is highly competitive and sets a new benchmark for efficiency in this domain. Furthermore, the short-to-long curriculum aligns perfectly with the sequential nature of autoregressive modeling, making it a principled rather than an ad-hoc choice.

Q3: The discussion in Lines 109–121.

Thanks for your attention in this discussion. We wish to clarify that the primary focus of this paragraph is to analyze different masking strategies within the MAR framework, not to contrast VideoMAR with non-MAR models. The temporal and spatial separation we critique refers specifically to MAR-based models like NOVA. As we demonstrate, such designs can lead to severe temporal flickering and inconsistency. We did not include MarDini in this specific comparison because its image-wise modeling paradigm falls outside the scope of MAR's token-wise approach, and it remains closed-source without a public benchmark for evaluation.

Q4: The categorization in table 3.

Our categorization in Table 3 follows established practices in generative modeling (e.g., Fluid [1] and RandAR [2]), based on the fundamental modeling scope (e.g. token-wise modeling or not).

(1) We classify models as ''diffusion-type'' if they operate holistically at the image or clip level, and ''autoregressive-type'' if they operate sequentially at the token level. This is a standard distinction.

(2) Under this principle, VideoMAR, which models continuous tokens sequentially, is correctly categorized as autoregressive, consistent with prior work like Fluid [1] and RandAR [2].

(3) Conversely, MAGI-1 performs clip-wise modeling and is thus considered a diffusion-type model. This also explains its strong performance, as it can leverage powerful pre-trained diffusion backbones, an advantage not available to token-wise models trained from scratch.

(4) Cosmos is explicitly described as an autoregressive world model in its original paper. Its applications in robotics are downstream fine-tuning tasks, not a reflection of its core generative architecture. We also note that Cosmos was trained with massive resources (e.g., 10,000 H100 GPUs), which underscores the efficiency and scalability of our approach.

[1] Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokens

[2] RandAR: Decoder-only Autoregressive Visual Generation in Random Orders

Q5: Parameter about frame rates and others.

Thanks for your interest in these details. We are happy to clarify. The motion dynamics in our generated videos are primarily determined by the training data distribution. To validate VideoMAR's capability on more complex scenarios, we direct the reviewer to our supplementary material (Section 8). As shown in Figure 14, by fine-tuning on specialized datasets, our model successfully replicates challenging motions like ''blinking'' and ''blossoming''. The generated videos have a frame rate of 12 fps. During training, we subsampled the source videos by a factor of 2 (i.e., sampling one frame for every two).

Q6: The term of video-to-video.

Thank you for the excellent suggestion. We have corrected this term in the revised manuscript.

Q1: The degree of novelty and technical depth.

We respectfully disagree. Our paper's contribution is a novel masking framework specifically engineered for video.

(1) Contrary to the reviewer's claim, our work moves beyond simple adaptations. We conducted extensive experiments on various video-specific masking strategies to validate our final model design. For example, we experiment with the ''total mask'' strategy, which randomly masks each frames of the video (Line 96-116, 253-256, and Figure 6)

(2) Our central innovation, the next-frame diffusion loss, is explicitly designed to solve a fundamental problem: the mismatch between training objectives and inference-time generation. By perfectly aligning the training loss with the next-frame prediction task, our method eliminates the common training-inference gap. This principled solution is the primary contribution of our paper and represents a non-trivial advancement in video modeling.

Q2: Presenting progressive training as a primary contribution.

We respectfully argue that this perspective misses the specific context of our contribution. The innovation is not progressive training in isolation, but its strategic and effective integration within a masked autoregressive system for video generation.

(1) Contextual Novelty: While progressive training is indeed prevalent in diffusion models, its application to autoregressive video models is notably rare and poses unique challenges. To our knowledge, our work is the first to successfully implement and scale progressive training for masked video generation. Unlike prior methods, our system demonstrates its viability and effectiveness at a large scale, which we believe is a novel contribution.

(2) Systematic Impact and Efficiency: The use of progressive training is not an isolated trick but a cornerstone of our system's design. This strategy is directly responsible for the remarkable efficiency gains that allow our model to surpass the strong Cosmos baseline using significantly fewer computational resources. The from-scratch training cost of our method is highly competitive and sets a new benchmark for efficiency in this domain. Furthermore, the short-to-long curriculum aligns perfectly with the sequential nature of autoregressive modeling, making it a principled rather than an ad-hoc choice.

Q3: Qualitative comparisons with competing methods.

Thanks for the comment. Our selection of baselines was deliberate and follows the established standards in autoregressive video generation. We specifically chose Cosmos as our primary comparison because it represents the state-of-the-art and is the most relevant baseline for our image-to-video task. We are confident that this comparison provides a clear and rigorous evaluation of our method's performance. If the reviewer has other specific methods in mind, we would be happy to discuss their relevance.

Q4: The exploration of error accumulation.

Thank you for the feedback. We will break down our response into three parts:

On Our Temperature Strategy: We argue that the perceived simplicity of our temperature strategy is its primary strength, not a weakness. To our knowledge, it is the first training-free method proposed to mitigate error accumulation in this context. This is a significant advantage over the few existing alternatives, which typically rely on complex and costly training-based perturbations. Our approach is therefore highly novel, practical, and demonstrably effective, representing a valuable contribution to the field.

On the Mention of Framepack: We must firmly state that citing Framepack is inappropriate and falls outside standard reviewing principles. Framepack was publicly released on April 17th, less than one month before the NeurIPS submission deadline of May 15th. It is a classic case of concurrent work. More importantly, from a technical standpoint, Framepack's approach is fundamentally incompatible with our model's principles. Its anti-drifting method sacrifices the strict causal prediction chain by incorporating bi-directional context, meaning it is no longer a purely autoregressive model. This design also limits its flexibility, as it demands specialized training for different sampling methods. Our method, in contrast, preserves the causal structure and requires no such modifications.

On ''Similar Training Strategies'': We find the comment on ''similar training strategies'' too general to be actionable. For a constructive discussion, we would have appreciated specific references that the reviewer believes are relevant. Without concrete examples, we are unable to provide a direct comparison. We remain confident, based on our thorough literature review, that our system design is novel and state-of-the-art for this specific problem domain.

Q5: NTP explanation in Table 2.

Thanks for the suggestion. We will clarify in three points:

(1) The ''NTP'' term in table 2 represents replacing the mask-based generation paradigm of VideoMAR with the vanilla next-token-prediction paradigm, while keep the other parts identical (e.g., the same architecture and token-wise diffusion loss). The ''w/o KV Cache'' term in table 2 represents following the mask-based generation paradigm design of VideoMAR, while without the incorporation of frame-wise KV Cache. The comparison between the ''NTP'' and ''w/o KV Cache'' thus reflects the inference speed difference between the next-token-prediction paradigm and mask-based generation paradigm. We will also append the above explanation in the revised manuscript.

(2) This is a common operation in previous MAR-series methods (such as MAR and Fluid), where NTP and mask-based paradigms are extensively compared and discussed.

(3) 672 is a typo. It should be 671, consistent with Table 2. Thanks for pointing out this.

Q6: Different compression ratios.

Thanks for the suggestion. This is a deliberate design choice to maintain computational feasibility for higher resolution training. By increasing the compression ratio, we effectively manage the token sequence length, ensuring that training remains efficient and fits within our available GPU memory budget. Our strong results at these resolutions validate the success of this practical strategy.

Q7: The summarized limitation from reviewer.

The basic concept of autoregressive generation is causality. It is also the core concept in our method that our generation is causal in temporal dimension and bidirectional in spatial dimension. Therefore, the reviewer's claim that our design '' discards MAR’s bidirectional capability along the temporal axis '' is incorrect. Our model, by design, never had temporal bidirectionality, as this is antithetical to causal generation. We believe the reviewer has confused our model's spatial bidirectionality with its strict temporal causality.

Q1: Exploration on diverse datasets or complex video scenarios.

Thank you for this insightful question regarding generalization. We agree that generalization is fundamentally linked to the scale and diversity of the training data. While our model excels on distributions seen during training, pushing the boundaries of generalization further is a broader challenge for the field, and we believe expanding datasets with richer motion is a critical direction for future work.

In this paper, we demonstrate our model's robust generalization capabilities through both state-of-the-art benchmark performance and targeted experiments on high-motion scenarios. (1) Benchmark Performance: VideoMAR achieves state-of-the-art results on the comprehensive VBench benchmark. This performance across diverse scenarios inherently validates our model's strong generalization on established tasks. (2) Adaptation to Complex Motion: We specifically tested the model's capacity to handle high-motion dynamics. As detailed in Supplement Section 8, after fine-tuning on specialized datasets (e.g., eye blinks, flower blossoming), our model successfully captures these complex movements (Figure 14). This proves the architecture's ability to adapt and generalize to new, challenging motion profiles when provided with targeted data.

Q2: Comparison between curriculum and fixed schedule.

Thank you for the question about our training schedule, we are happy to elaborate on the rationale. Our adoption of a short-to-long curriculum schedule is a deliberate design choice, grounded in both a clear rationale and decisive empirical validation from our early-stage experiments. We selected this approach for two primary reasons. First, it significantly improves training efficiency, and its progressive nature aligns perfectly with the sequential modeling principles of autoregressive frameworks. Second, crucially, this rationale was confirmed by direct empirical comparison. We trained two models from scratch for 20 epochs at a 25x256x256 resolution using 16 H20 GPUs at our early-stage experiments. The model with the curriculum schedule began to converge and generate roughly plausible videos. In stark contrast, the model trained with a fixed-length schedule failed to converge and produced no meaningful output under the identical setting. This result confirms that the curriculum is not merely a minor optimization but a fundamental component that enables the successful training of our model.

Q3: Comparison between VideoMAR and NTP.

We appreciate the opportunity to clarify our comparison strategy and the rationale for our model's design. Our work aligns with findings from prior researches (e.g., MAR, Fluid), which have consistently shown that mask-based generation with bidirectional context outperforms causal NTP for visual tasks under fair comparisons.

To provide a robust empirical comparison, we benchmarked our model against Cosmos, a leading NTP-style baseline. We were pleased to find that our method achieves superior results on VBench while being significantly more resource-efficient. Given these results and the considerable resources required to train a new large-scale NTP model from scratch, we believe this comparison provides a clear and fair assessment of our method's advantages.

Q4: Variance of the VBench score.

Thank you for emphasizing the importance of statistical rigor. We provide a three-part analysis that confirms the robustness and significance of our results.

(1) Score Stability and Robustness: VBench is already designed for reliability by averaging over five runs per sample. To provide further validation, we conducted three additional evaluation runs for both our model and the Cosmos baseline. As shown in the table below, the variance is negligible (less than 0.1), confirming that the reported scores are stable and the performance gap is consistent.

(2) Significance of the 0.6 Point Margin: We wish to emphasize that a 0.6 point difference on VBench is significant in the context of state-of-the-art models. On the official VBench leaderboard, top-performing methods are often separated by margins of less than 0.1 points. Therefore, our lead of 0.6 points represents a substantial improvement.

(3) The Overarching Context of Efficiency: Our model achieves this superior score while demanding only a fraction of the resources: 0.5% of the training data and 0.2% of the GPU hours compared to the baseline. This monumental leap in efficiency, combined with a statistically significant performance gain, validates the efficacy and novelty of our approach.

Q5: Quantization of per‑frame error accumulation.

Thanks for your valuable suggestions. We have performed the suggested LPIPS analysis, which decisively confirms the effectiveness of our temperature schedule in combating temporal drift.

(1) We list the LPIPS drift (lower is better) comparison in the following table. Experimental setting: (a) we adopt VideoMAR to generate videos of resolution 49x512x768. (b) The experiments are conducted on the VBench-I2V with 246 samples. (c) We generate the corresponding videos with and without the temperature schedule. (d) We calculate the LPIP distance between the i-th frame and the first frame (0-th frame), and average across all the 246 videos. Experimental conclusion: the quantization results are consistent with our claim, that temperature schedule helps smooth the generated video.

(2) We also quantize the performance gain of the temperature schedule via VBench scores in table 4 of the main manuscript, where temperature schedule elevates VBench score from 80.72 to 82.56.

Q1: Exploration on diverse datasets or complex video scenarios.

Thank you for this insightful question regarding generalization. We agree that generalization is fundamentally linked to the scale and diversity of the training data. While our model excels on distributions seen during training, pushing the boundaries of generalization further is a broader challenge for the field, and we believe expanding datasets with richer motion is a critical direction for future work.

In this paper, we demonstrate our model's robust generalization capabilities through both state-of-the-art benchmark performance and targeted experiments on high-motion scenarios. (1) Benchmark Performance: VideoMAR achieves state-of-the-art results on the comprehensive VBench benchmark. This performance across diverse scenarios inherently validates our model's strong generalization on established tasks. (2) Adaptation to Complex Motion: We specifically tested the model's capacity to handle high-motion dynamics. As detailed in Supplement Section 8, after fine-tuning on specialized datasets (e.g., eye blinks, flower blossoming), our model successfully captures these complex movements (Figure 14). This proves the architecture's ability to adapt and generalize to new, challenging motion profiles when provided with targeted data.

Q2: Scaling efficiency for larger spatial-temporal resolutions.

Thanks for highlighting efficiency. Our model's efficiency is empirically demonstrated by its ability to outperform a powerful baseline like Cosmos with a fraction of the resources. Furthermore, we have validated its scalability by successfully training at high resolutions (49x512×768). The fact that this was achieved under standard hardware constraints is not a limitation but rather a powerful validation of our model's efficient design. We contend that our architecture is fully equipped for further scaling, which is a matter of resource allocation, not a fundamental challenge to our method.

Q3: 3D RoPE to extrapolation ability

We appreciate the opportunity to elaborate on our model's extrapolation capabilities. This ability is thoughtfully designed and stems from two key elements working in concert. First, the autoregressive design of VideoMAR provides a natural foundation for generating longer videos by modeling them sequentially. This sets the stage for extrapolation. Second, our experiments, detailed in Section 1 of the supplement, show why 3D-RoPE is so important. We found that models with absolute position encodings could not extrapolate to new lengths, whereas our use of 3D-RoPE, a relative encoding, successfully enabled this capability. This demonstrates that it is a critical part of our model's design for achieving robust generalization in sequence length.

Q4: Inference speed compared to modern diffusion-based methods.

We append the inference speed comparison with modern diffusion-based methods, including HunyuanVideo and Wan2.1, in the following table. All the values are achieved with single batch size on single H20 GPU. Our method demonstrates competitive inference speed. We will also append this result in the revised version. Note that the inference speed of VideoMAR can be further improved with smaller mask steps and diffusion steps.