VideoRoPE: What Makes for Good Video Rotary Position Embedding?

Dear Reviewer 3qrY,

Thanks for your valuable feedback. We sincerely thank all reviewers for acknowledging that this paper is novel (t9dP, juMQ), well-motivated (t9dP, 26sL, 3qrY), significant improvements (t9dP, 26sL, 3qrY), well-written (t9dP, juMQ, 26sL, 3qrY), and easy to follow (t9dP, 3qrY). Below we address your questions:

Q1: L266-270, second column. Why such collisions make the model susceptible to distractor influence?...Can the authors provide either logical explanations or more direct empirical evidence on this?

We appreciate your request for clarification. As shown in Fig. 4 and Appendix F, we have linked position embedding collisions in M-RoPE to distractor influence. Due to high-frequency oscillations, distant positions can share nearly identical embeddings (red planes in Fig. 4), making the model vulnerable to distractors. In contrast, VideoRoPE avoids such oscillations, enabling more robust temporal modeling. We have also conducted an ablation study (Table 5), confirming that LTA(Low-frequency Temporal Allocation)—by preventing temporal embedding collisions—improves video understanding performance.

Q2: L413-414, first column: It is quite arguable that V-NIAH-D is more challenging than V-NIAH by merely looking into Fig. 7. Can the authors provide quantitative measure of challengingness of these two datasets?

In the table below, we provide the quantitative performance gaps observed in various models, including our Video-RoPE, tested on both V-NIAH and V-NIAH-D datasets:

These results indicate a significant performance drop on V-NIAH-D for most methods (-10.67 for LLaVA-NeXT-Video-7B and -10.44 for LongVA), supporting our claim that V-NIAH-D is more challenging than V-NIAH. Also, our VideoRoPE effectively mitigates the misleading influence of distractors in V-NIAH-D (-0.22 merely). We will include this quantitative comparison in our revised manuscript.

Q3: I list a few errors, typos, and awkward sentences below.

Thank you for highlighting language issues and awkward expressions. We will carefully revise the manuscript to correct these:

• "uses dimensions" (L214-216) will be corrected to "uses different dimensions".
• "The visualized relationship between the periodicity, monotonicity, and temporal modeling" (L271-273) will be corrected to "The relationship between periodicity, monotonicity, and temporal modeling is visualized in Fig 4."
• "modes" will be corrected to "models" (L357).

Q4: In Fig. 2 right side, what do the axes mean and each cell color mean?

Thank you for pointing that out. Fig. 2 and the last row of Fig. 7 are indeed the same, both showing M-RoPE and VideoRoPE performance on V-NIAH and V-NIAH-D benchmarks. The x-axis indicates context length, the y-axis shows frame depth, and the green-to-red color bar reflects needle retrieval accuracy—from perfect to zero.

Q5: In Fig. 3 of the main paper, and Fig. 9 of the supplementary material, How do we interpret the results? For instance, in L165-176, second column, what does it mean by “the needle is located primarily through vertical positional information”? I do not understand this paragraph. Can the authors elaborate on this paragraph?

In Fig 3, we can derive the fact from different component of the attention score, that M-RoPE locates and retrieves the needle using the component with respect to the feature dimension capturing vertical positional information, namely the product of the last 48 dimension of Q and K in self-attention. This goes against the design intention of M-RoPE.

Q6: In L298-303, first column, what does it mean by “creating a stack in the bottom-left corner”? I do not understand this paragraph as well. Can the authors elaborate on this paragraph? Can the authors give more intuitive explanation on Fig. 6?

The three subfigures in Figure 6 represent the schema of Vanilla RoPE, M-RoPE, and VideoRoPE when encoding text-video-text input. The three dimensions of each subfigure represent the sequential indices (both textual and temporal indices), and the horizontal and vertical positions in each frame. We find that each dimension of Vanilla RoPE increases with the input token indices, regardless of the existence and the spatiotemporal characteristics of video, which is a straight diagonal line in 3D space. After that, M-RoPE uses part of dimensions to represent the spatial information of the video, but the horizontal and vertical position indices do not change with the video frames. Therefore, it appears as a vertical stack in the bottom-left corner in 3D space, deviating from the diagonal direction, different from the pre-trained features in the text. Comparatively, VideoRoPE allows the horizontal and vertical position indices of each frame to change with the frame indices, while maintaining the diagonal features, which makes it easier to transfer LLM to the video mode.

Dear Reviewer 26sL,

Thanks for your valuable feedback. We sincerely thank all reviewers for acknowledging that this paper is novel (t9dP, juMQ), well-motivated (t9dP, 26sL, 3qrY), significant improvements (t9dP, 26sL, 3qrY), well-written (t9dP, juMQ, 26sL, 3qrY), and easy to follow (t9dP, 3qrY). Below we address your questions:

Q1: Temporal allocation...showing some specific failure cases of high-frequency allocation...

Failure cases of high-frequency allocation in M-RoPE have shown in Fig. 3 and Appendix E. Specifically, the M-RoPE responses illustrate its limitations, with attention visualizations revealing that high-frequency encoding hinders long-range temporal dependencies, favoring local patterns instead.

Q2: Spatial symmetry...more empirical qualitative study of how spatial inconsistency becomes problematic

We have quantitatively shown the impact of spatial symmetry in Tab. 5 (second row), with a +1.65 gain on MLVU under 64k context over M-RoPE. To further validate its importance, we evaluated it on 4 benchmarks with consistent improvements supporting our claim.

Q3: While using only low frequencies...potentially risks missing short-term or local dynamics...

See Q4.

Q4: Why the temporal encoding frequencies are not interleaved with the spatial frequencies ... For example, ... [t t x y x y x y … t t x y x y x y] or [t x y t x y x y … t x y t x y x y] to preserve the ability to capture both long- and short-term variations.

We trained a model using the "[t t x y x y x y]" format and conducted additional comparative experiments with varying context lengths on the LongVideoBench benchmark, which features a wide range of video scenarios—including both rapidly changing dynamic content and slowly evolving scenes. Our results show below that, on average, low-frequency temporal allocation consistently outperforms the "[t t x y x y x y]" arrangement. This suggests that our frequency design effectively balances the modeling of both global context and local dynamics across diverse video conditions. As for the "[t x y t x y x y … t x y t x y x y]" pattern, we plan to report on it in future updates.

Q5: ...The performance seems to be sensitive to the δ on LongVideoBench. Does the optimal δ vary significantly across benchmarks, or is it stable? Did you tune it separately for each dataset?

Table 6 presents the ablation study of the scaling factor on LongVideoBench. Additionally, we report the average results on other benchmarks, MLVU and VideoMME, in the table below. Our analysis indicates that the optimal δ remains consistent across various tasks. We would also like to clarify explicitly in the revised manuscript that we did not tune this δ separately for each individual dataset.

Q6-1: The core findings and solutions (i.e., diagonal layout) for spatial symmetry was already proposed in (Su et al, 2024b). Personally, I think the proposed method essentially extends RoPE-Tie-V2(Su et al, 2024b) to the temporal domain.

This is not true. RoPE-Tie-V2 (Su et al, 2024b) introduced the concept of spatial symmetry but did not provide experimental validation, nor did it discuss the diagonal layout. Additionally, RoPE-Tie-V2 cannot handle streaming video inputs (as discussed by reviewer juMQ regarding streaming scenarios)

Q6-2: The newly featured low-frequency temporal allocation and temporal spacing are technically less significant or already proposed in (Li et al., 2024), respectively.

This is not true. Li et al. (2024) primarily focuses on enhancing video models from a data perspective, using only temporal reasoning data in pure text. Their work does not address low-frequency temporal allocation or temporal spacing—both of which are novel contributions introduced by our method. We will clearly highlight these distinctions in our revised manuscript.

Q7: [a] Heo et al., should be cite.

We will cite this work in our related work section. This paper is designed for 2D image tasks. On the contrary, our VideoRoPE is designed for 3D video tasks.

Q8: Captions in Fig.2

Thank you for pointing that out. Fig. 2 and the last row of Fig. 7 are indeed the same, both showing M-RoPE and VideoRoPE performance on V-NIAH and V-NIAH-D benchmarks. The x-axis indicates context length, the y-axis shows frame depth, and the green-to-red color bar reflects needle retrieval accuracy—from perfect to zero.

Q9: Typo in Eq.7

Thank you for identifying this typo. We will correct and clarify this equation in our revised submission.

Dear Reviewer juMQ,

Thanks for your valuable feedback. We sincerely thank all reviewers for acknowledging that this paper is novel (t9dP, juMQ), well-motivated (t9dP, 26sL, 3qrY), significant improvements (t9dP, 26sL, 3qrY), well-written (t9dP, juMQ, 26sL, 3qrY), and easy to follow (t9dP, 3qrY). Below we address your questions:

Q1: The paper covers most of the essential references related to position embeddings for transformer models and video understanding. However, there are a few relevant works that are not cited or discussed that could provide additional context.

Thank you for the suggestion. We will incorporate citations and a discussion of Auroracap, MVBench, and VidComposition into the revised manuscript. These benchmarks, which evaluate video large language models, offer valuable insights into detailed video captioning (Auroracap), general video understanding (MVBench), and compositional structure analysis (VidComposition). We also report the performance of our VideoRoPE on these benchmarks, see Q2 below.

Q2: The paper could potentially be enhanced by discussing additional benchmarks that, while not included in the current comparison, might provide valuable insights, such as those works referenced in the "Essential References Not Discussed" section.

Thanks for your suggestion. We evaluate our method on the Video-Detailed-Caption (VDC) benchmark proposed by Auroracap and MVBench. Since VidComposition has not yet been open-sourced, we will evaluate it once the benchmark is released.

Q3: How would VideoRoPE need to be adapted for streaming video understanding tasks where the full temporal context isn't available upfront? Since there are many new datasets and benchmarks [1, 2] for streaming video understanding, which is more and more important.

Thank you for this insightful question regarding the adaptation of VideoRoPE to streaming video understanding tasks. We acknowledge the growing importance of streaming video understanding, as it closely aligns with how humans naturally perceive the world. Accordingly, we will reference the works you mentioned in our manuscript and discuss the significance of streaming modeling for video understanding tasks.

By design, VideoRoPE effectively supports streaming input scenarios, in contrast to RoPE-TIE-V2, which requires prior knowledge of the video's total length and thus cannot accommodate streaming data. Although VideoRoPE was not explicitly trained on streaming video data, we evaluated its effectiveness for streaming modeling on the StreamingBench Benchmark[2]. Specifically, we focused on the Real-Time Visual Understanding (RTVU) setting relevant to streaming modeling, as presented in the table below. The results demonstrate that VideoRoPE still exhibits superior performance in streaming video scenarios.

Dear Reviewer t9dP,

Thanks for your valuable feedback. We sincerely thank all reviewers for acknowledging that this paper is novel (t9dP, juMQ), well-motivated (t9dP, 26sL, 3qrY), significant improvements (t9dP, 26sL, 3qrY), well-written (t9dP, juMQ, 26sL, 3qrY), and easy to follow (t9dP, 3qrY). Below we address your questions:

Q1: Only one LLM i.e., QWen2-VL, is evaluated. It would be more convincing to show improvement over more base LLMs.

Thanks for your suggestion. We have updated the table below with results on QWen2.5-VL. Performance is reported across three categories: long video understanding (LongVideoBench, MLVU, VideoMME) with a 64k context length, short video hallucination at an 8k context length, and average retrieval accuracy. Across all six benchmarks, VideoRoPE's performance remains consistently superior to other RoPE variants, including Vanilla RoPE, TAD-RoPE, and M-RoPE. We will add these results to our revised version.

Q2: More recent (SOTA) baselines should be compared. For example, LLaVA-Video, LLaVA-OneVision with different sizes.

We appreciate your feedback. We have added the experimental results of Video-RoPE with the SOTA models including LLaVA OneVision, LongVU, Apollo and LLaVA-Video with different sizes (3B and 7B).

Results with 3B size models:

Results with 7B size models:

Notably, our method achieves better results with significantly less fine-tuning data (0.33 million) than prior SOTA approaches (8.8 million for LLaVA OneVision, 3.75 million for LongVU, 3.2 million for Apollo, and 2.7 million for LLaVA-Video). Our results highlight the benefits of using our RoPE design in Video Large Language Models.

Q3: The performance degradation at 64k context length (Tab. 2) is not well explored in the main paper.

In the Tab. 2 of main paper, the performance of 64k (65.56 for MLVU, 61.33 for VideoMME) is slightly worse that 32k (66.02 for MLVU, 61.67 for Video-MME) on some benchmarks. The reason is that simply increasing the context window doesn't guarantee improved performance. It is challenging for the model's ability to effectively process and utilize extremely long contexts. For example, in Figure 5 of the technical report of QWen2-VL based on M-RoPE, the performance of 64k (70.4) is also slightly worse than 48k (71.3). We further tested the performance on our VideoRoPE on 48k and 80k context length. The results at 48k and 80k showed performance fluctuations, which are consistent with Figure 5 of the QWen2-VL technical report.