RIFLEx: A Free Lunch for Length Extrapolation in Video Diffusion Transformers

We sincerely thank Reviewer DWAu for the recognition of our work. The further questions are addressed as follows.

Q1: If only 20,000 videos are needed, how to select them? Does the selection make any difference?

The 20K videos in this paper were randomly sampled without selection. Fine-tuning aims to adapt the model to modified frequencies, which theoretically requires no data bias. To validate this, we add an experiment where we independently train models on two distinct randomly sampled datasets. We then perform three sampling runs with different random seeds and apply a two-sample $t$-test to compare the performance of the two models. As demonstrated in Rebuttal Table A, the statistical analysis reveals no significant performance difference between the models at the $95$% confidence level (all two-sample $t$-test $p$-values > $0.05$, $α=0.05$).

Rebuttal Table A. Performance comparison between Model A and Model B, trained on two independent randomly sampled datasets based on the CogVideoX-5B architecture. Evaluation is conducted on a 165-sample subset of VBench with three sampling runs, reporting the mean ± standard deviation. The p-values are derived from a two-sample $t$-test.

We thank reviewer Cp7v for the valuable comments. We address the concerns as follows.

Q1: The experimental results shown in the supplementary materials display that some cases may still suffering from temporal inconsistency and may lead to the camera to switch in the playing video. Could the authors give some explanation?

We appreciate the reviewer's attention to this detail and would like to clarify that this may be a potential misunderstanding. In fact, multi-scene generation with camera transitions is a desirable and essential capability for video synthesis. To achieve this, HunyuanVideo is specifically designed to curate a training dataset that includes diverse scene transitions (see Dense Description in Section 3.2 of HunyuanVideo[1]). As demonstrated on HunyuanVideo's project page[2] (e.g., the example in Row 2, Column 1), HunyuanVideo excels at "breaking monotonous cinematography with seamless director-level shot transitions." This capability represents a significant advancement in video synthesis.

Our method preserves the base model's ability to generate both multi-scene videos (Videos 1-3,9 in the supplementary materials) and single-scene videos (Videos 4-8). Importantly, even in multi-scene generation, our method still maintains long-term temporal consistency—for instance, Figure 1 demonstrates consistent identity preservation across two distinct scene transitions. We will clarify this scene transition capability in Section 4.2 of the final version.

[1] Kong, Weijie, et al. "Hunyuanvideo: A systematic framework for large video generative models." arXiv preprint arXiv:2412.03603 (2024).

[2] https://aivideo.hunyuan.tencent.com/.

Q2: Why is the effect not good when the insertion multiple is greater than 3? So how do we choose the appropriate multiples according to different models?

Q2-1: Why is the effect not good when the insertion multiple is greater than 3?

The limitation of a 3× insertion multiple stems from a fundamental trade-off in positional encoding dynamics: an excessive reduction in frequency leads to a diminished ability to discriminate between sequential positions. Specially, a larger extrapolation factor $s$ leads to smaller frequencies $θ_k$ (see Eqn.(8)). When $θ_k$ becomes excessively small, the position difference term $∆ = \cos((p + 1)θ_k) − \cos(p\theta_k)$ diminishes to near-zero values, thereby losing positional discriminability. Empirically, we find the threshold occurs at $θ_k′ \le 2\pi/3L$ when $s=3$. Furthermore, this upper limit is inherent to the positional encoding mechanism and is consistent across existing pretrained video diffusion models based on our experiments. We will add the above detailed explanation in "Maximum extent of extrapolation" part of the final version.

Q2-2: So how do we choose the appropriate multiples according to different models?

We clarify that we do not select different $s$ for different models. Instead, the extrapolation length is determined by the user's requirements, provided it remains below the upper limit. For example, in the paper, we demonstrate results for generating videos with multipliers of $2$, $2.3$, and $3$. Based on the analysis in Q2-1, the extrapolation limit is consistent across current models, eliminating the need for specific designs tailored to different models.

We hope you might find the response satisfactory, and we would be delighted to clarify any further concerns you might have.

We sincerely thank Reviewer EXby for the valuable suggestions. We have thoroughly addressed the detailed comments as follows.

Q1: Explain the proposed method is slightly worse than the best method in automatic metrics.

We kindly clarify that only through a comprehensive consideration of multiple metrics can we objectively assess video generation quality, rather than relying solely on a single metric. For example, while PI and YaRN achieve the highest score in the single NoRepeat Score metric, it performed significantly worse in the Dynamic Degree metric (PI/YaRN vs. our RIFLEx in Table 1), resulting in poor video quality and rendering other metrics meaningless. See Figure 5 for the evidence. To highlight this, we mark severe issues in red for clarity (see lines 330-332). We emphasize that our method is the only one consistently highlighted in the green zone across all 5 settings in Table 1.

Q2: The core difference between this method and others.

Our method differs from prior works in two key aspects:

• We determine which frequency in RoPE should be modified—specifically, the intrinsic frequency whose period aligns with the first observed repetition frame in a video (Eqn. (7)). As discussed in the main text (lines 306–311), modifying only this frequency is sufficient: adjusting higher frequencies disrupts fast motion, while altering lower frequencies has negligible impacts.
• We derive how to adjust this frequency—ensuring it remains within a single period after extrapolation (the non-repetition condition, Eqn. (8)).

Existing approaches modify multiple frequency components in RoPE, but they may target incorrect components. For instance, methods like YaRN and PI mistakenly adjust high frequencies, which results in slow motion (see lines 295–299). Furthermore, even when some methods include the correct components, their modifications can be flawed. For example, the adjustment of intrinsic frequency in NTK does not satisfy our non-repetition condition, resulting in repetition issues (see lines 326–329).

In summary, our work establishes principled guidelines for position embedding design in length extrapolation. We will make it more clear in the "Principled Explanation for Existing Methods" section (Section 3.4). We hope this response clearly articulates our contributions and would be very happy to clarify further concerns (if any).

Q3: Explain the gap between theory and practice and why fine-tuning can alleviate this.

We understand that the gap you are referring to is that fine-tuning the model yields better results than not fine-tuning. This arises from a training-testing mismatch, where the position embeddings used during inference slightly differ from those in training due to modified frequencies. While this discrepancy does not undermine the conclusion about our non-repetition condition, it may affect visual quality since the model lacks explicit training on these specific position embeddings. Fine-tuning helps bridge this gap by adapting the model to these variations, thereby improving visual quality. We will incorporate the above explanation of the training-testing mismatch into Section 3.3, where we discuss whether fine-tuning is a necessary.

Q4: Why does this set of ideas work in spatial extrapolation?

This is because video diffusion transformers typically apply 1D RoPE independently to both spatial and temporal dimensions (see Section 2.2, "RoPE with Multiple Axes"). This shared mechanism results in similar challenges during extrapolation for both dimensions. As illustrated in Figure 2:

• Spatial Repetition ↔ Temporal Repetition: Both phenomena occur when intrinsic frequency components exceed a single period after extrapolation.
• Blurred Details ↔ Slower Motion: These effects arise from interpolating high-frequency components, which are essential for spatial details in spatial domain and fast motion in the temporal domain.

Therefore, our method can be extended to spatial extrapolation, providing a unified framework for extrapolation in diffusion transformers. We will add above explanation in "Extension to other extrapolation types" part of the final version.

Q5: Line 5 in Algorithm 1 should refer to $\theta_k' =\frac{2\pi}{Ls}$.

We will correct Line 5 of Algorithm 1 to use $\theta_k' =\frac{2\pi}{Ls}$ in the final version. Thank you for the careful review.

Q6: Provide the definition of the subscripts multiple times.

As suggested, we will explicitly define the subscript $j$ as indexing frequency components of RoPE multiple times in Section 3.1 to prevent confusion with frame indices in the final version.

Finally, we sincerely appreciate the reviewer for the constructive suggestions, which help to further improve the quality of our work. We hope you may find the response satisfactory. Please let us know if you have any further feedback.

We appreciate Reviewer JAfh for the acknowledgement of our contributions.

Q1: Missing related work on repetition issues in autoregressive video generation models

Unlike diffusion models, autoregressive video generation models typically quantize videos into discrete tokens and generate video content through next-token prediction in an autoregressive manner. Previous works have demonstrated great performance in such models [1–8]. For example, NÜWA [4] employs VQ-GAN for tokenization and generates videos using a 3D transformer encoder-decoder framework. More recently, VideoPoet [5] tokenizes images and videos with a MAGVIT-v2 encoder and autoregressively generates videos using a decoder-only transformer based on a pretrained large language model.

While autoregressive video models can theoretically extend sequences indefinitely through next-token prediction [9-11], recent studies reveal their tendency to degenerate into repetitive content generation[5,11]. In this work, we present a principled approach to video length extrapolation that effectively generates novel temporal content in diffusion-based frameworks. Although our method is developed for video diffusion transformers, the underlying mechanism governing position embedding periodicity may also offer insights for addressing repetition challenges in autoregressive video generation.

Thank you for highlighting this important direction. We will incorporate the above discussion in related work section in the final verion. We would appreciate any additional references the reviewer could suggest that we may have missed.

[1] GODIVA: Generating Open-Domain Videos from Natural Descriptions

[2] VideoGPT: Video Generation using VQ-VAE and Transformers

[3] CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers

[4] NÜWA: Visual Synthesis Pre-training for Neural Visual World Creation

[5] VideoPoet: A Large Language Model for Zero-Shot Video Generation

[6] VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generation

[7] Generative Multimodal Models are In-Context Learners

[8] Emu3: Next-Token Prediction is All You Need

[9] Loong: Generating Minute-level Long Videos with Autoregressive Language Models

[10] NUWA-Infinity: Autoregressive over Autoregressive Generation for Infinite Visual Synthesis

[11] Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer

Q2: Limited exploration of extrapolation factors beyond 3×

Thank you for the suggestions. In this work, we primarily focus on achieving length extrapolation at minimal cost based on a pre-trained video diffusion models. As discussed in the main text, the 3× limitation stems from diminished ability to discriminate sequential positions due to excessive frequency reduction. To further extend beyond the 3× extrapolation, it is promising to investigate the mechanism of positional encoding during training, specifically tailored for extrapolation. We believe our findings colud provide valuable insights for this direction, and we will include a more detailed discussion in Section 5.

Q3: The identification method for intrinsic frequency relies on visual inspection rather than a theoretical analysis.

Thank you for the insightful comment. In our current work, we primarily adopt an empirical approach—visual inspection—for intrinsic frequency identification when adapting the pre-trained video diffusion transformer. While this approach is effective for adaptation, we agree that establishing a theoretical foundation for intrinsic frequency identification is crucial. Achieving this would require fundamental research into how intrinsic frequencies emerge during the pre-training process, potentially analysis from a training-from-scratch perspective. We sincerely thank the reviewer for highlighting this direction, and we will address this in our future work. We will add the above points to the discussion section.