Visual Context Window Extension: A New Perspective for Long Video Understanding

Thank you so much for the thoughtful questions and suggestions. We hope that our response below will address your concerns.

Q1: …. Could you clarify how extending the visual context helps to narrow this gap?

R1: Thank you for your insightful comment. In fact, extending the visual context window does not directly narrow the modality gap between visual and language embeddings. Instead, the primary goal of this approach is to scale the positional embeddings of visual tokens into an effective visual context window in a training-free manner. This enables the extension of LMM trained on short videos to handle long videos, without requiring additional training to align the two modalities.

Recent works [1] have attempted to align visual and language embeddings, but these approaches typically require extensive retraining and do not specifically address the challenges of long video understanding.

[1] Jiang T, Song M, Zhang Z, et al. E5-v: Universal embeddings with multimodal large language models. arXiv preprint arXiv:2407.12580, 2024.

Q2: How does the clustering visualization of visual and language embeddings change after implementing the proposed window extension strategy?

R2: Following your advice, we further visualized the clustering of visual and language embeddings at different layers of the LLM decoder after applying the visual context window extension. The results are presented in Figure 7 of Appendix A.5. As shown in the visualization, the visual and language embeddings still exhibit two distinct cluster centers after applying the context window extension.

This observation further supports our response to Question 1. The visual context window extension is not designed to address the modality gap between visual and language embeddings. Instead, its primary purpose is to scale the positional embeddings of visual tokens into an effective visual context window in a training-free manner. This allows LMM trained on short videos to be extended to long videos at a low computational cost.

Q3: …. Could you explain why the first frame is considered to preserve rich information?

R3: We apologize for the confusion caused by our imprecise wording. The statement that the first frame "preserves rich information" is made in comparison to other frames within the same group, which are compressed using a larger pooling stride.

We argue that continuously sampling frames from a video often results in adjacent frames containing highly similar background information, leading to redundancy in the visual content and increased memory consumption. To address this, we apply a smaller pooling stride to the first frame of each group, preserving its fine-grained visual details, while compressing the other frames within the group more aggressively to reduce memory usage.

Regarding the uniform grouping strategy, we acknowledge that dynamic content in videos may not always align perfectly with uniform temporal divisions. However, compared to event-based segmentation, a uniform grouping strategy is simpler and avoids the need for additional event segmentation models, which are themselves challenging to design and computationally expensive. Furthermore, event-based segmentation often leads to non-uniform frame sampling across different events, introducing additional complexity in hyperparameter tuning.

In summary, to ensure simplicity and efficiency, we adopt a uniform grouping strategy as an empirical design choice. This approach avoids the overhead of event segmentation while still enabling a reasonable balance between preserving critical information and reducing memory consumption.

Thank you very much for your positive and constructive feedback! We hope that our response below will address your concerns.

Q1: The authors fixed the input frames to 256/512. Could this approach be extended to handle more frames?

R1: In theory, our approach can be extended to handle even longer sequences of frames. However, processing more than 512 frames introduces significant memory overhead. Therefore, we only tested longer sequences (1024 frames) on the VISUAL NEEDLE-IN-A-HAYSTACK benchmark. As shown in Appendix A.1, our model achieved a 100% retrieval accuracy with 1024-frame inputs.

Q1: What are the results of fine-tuning on 256 frames and then extending to a higher number of frames?

R2: Due to the quadratic computational complexity of attention mechanisms and the associated memory consumption, fine-tuning on 256 frames is extremely challenging. The input sequence length exceeds 50K tokens, which is computationally prohibitive, even in large-scale LLM training. As shown in Table 1, LongVILA [1] optimizes attention computation and trains on long videos with input lengths exceeding 256 frames, utilizing 256 A100 GPUs. In contrast, our proposed visual context window extension allows a model trained on only 32 frames to extend to 512 frames or even longer without additional training, while achieving superior performance.

Our method is specifically designed to avoid such exorbitant training costs.

[1] Xue F, Chen Y, Li D, et al. Longvila: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408.10188, 2024.

Q3: Could the authors provide results on MVbench? …

R3: In Table 3 of the LongVideoBench evaluation, we observed that for short videos, sampling 512 frames led to a performance drop. This is because sampling 512 frames for 8-15 second short videos introduces significant visual redundancy. A straightforward solution to mitigate this issue is using fps-based sampling instead of fixed-parameter sampling. For LongVideoBench, we sampled frames at 1 fps, with a maximum of 512 frames. The results in the table below demonstrate that our method achieves significant performance improvements on short videos:

Additionally, following your suggestion, we evaluated our method on MVBench, and the results are summarized below. Results marked with * are reproduced using the same configuration:

For short videos, dense sampling (e.g., 128 frames) results in performance degradation due to increased redundancy. However, when fixed to 64 frames, our method still delivers performance improvements.

Thank you for your response. Some concerns have been addressed. I'd like to keep my rating.

Thank you so much for your suggestion. We hope that our response below will address your concerns.

Q1: The progressive pooling strategy may also result in some loss of information.

R1: We reported the results of the ablation study on the progressive pooling strategy in Table 5. Under the (2,8),4 pooling strategy, our method achieved even better performance compared to the model without the progressive pooling strategy, while also reducing memory consumption by approximately 45%.

This improvement is due to the fact that continuous sampling of video frames can lead to similar backgrounds between adjacent frames, which generate redundant visual tokens. This redundancy results in longer input sequences, which can cause attention distraction and ultimately degrade model performance.

The proposed progressive pooling strategy not only reduces memory consumption but also further improves the model's performance by mitigating the impact of redundant information.

Q2: The paper lacks a detailed overall framework diagram to showcase the entire process.

R2: In this paper, we proposed two key innovations: the Visual Context Window Extension and the Progressive Pooling Strategy. The former does not involve modifications to the model architecture, as it primarily optimizes the position embedding computation method for long-video understanding tasks, making it difficult to directly reflect in the model structure. As for the latter, the detailed processing pipeline of the progressive pooling strategy is visualized in Figure 3.

Q3: The paper may not be entirely fair in comparison with methods that use fewer frames, as a higher number of frames inherently contains more information.

R3: Our comparison includes both short-video LMMs, which support a small number of input frames (e.g., 4, 8, 16, 32, 64), and long-video LMMs, which support a higher number of frames (e.g., 128, 256, and 2048). Considering that the backbone of our method is a short-video LMM (LLaVA-OneVision), we included short-video LMMs in the comparison to establish a comprehensive evaluation benchmark.

Moreover, as shown in Figure 1 (a) and Table 4, increasing the number of input frames for short-video LMMs can actually lead to a performance decline.

Thank you for your responses!

Thank you so much for the thoughtful questions and suggestions. We hope that our response below will address your concerns.

Q1: … In the context of video question-answering, which of the following interpretations is correct?

R1: Thank you for your suggestion. In video question-answering tasks, both the visual and textual components share the same visual context window for interpolation. We conducted experiments on both approaches and observed that when the visual component uses the visual context window for interpolation while the textual component uses the language context window, the model fails to generate complete words. We believe this issue arises because the relative positional relationships between visual tokens and language tokens are disrupted. Considering that textual sequences in video question-answering tasks are typically short, and to ensure our method remains simple and easy to integrate into existing training and deployment frameworks, we opted to use the same visual context window for both visual and textual components. In fact, implementing visual context window extension only requires modifying a few lines of code in existing open-source models. This allows for extending LMMs trained on short videos to handle long videos effectively.

Q2: According to the citation of allava dataset, it seems to be an image dataset. How can it be used to fine-tune the video understanding model in the paper?

R2: Thank you for your suggestion. In our experiments, we used the LLava-OneVision backbone, which supports both image and video understanding tasks by treating videos as sequences of sampled frames. The reason we utilized the ALLaVA image dataset is to align with the methodology of LongVA [1], which achieves long video understanding by training solely on images. Fine-tuning on an image dataset also reduces computational resource requirements. For example, our approach only requires an RTX 3090 24G GPU to fine-tune the model, which can be particularly advantageous in resource-constrained scenarios. Moreover, using an image dataset aligns with the core motivation of this paper: achieving strong performance on long video understanding tasks without fine-tuning the model on long video data.

[1] Peiyuan Zhang, Kaichen Zhang, Bo Li,et.al. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852, 2024.

Q3: In the ablation study of progressive pooling strategy, why does (2,4),4 perform better in some cases compared to (2,2),0?

R3: Thank you for your suggestion. In the ablation study of the progressive pooling strategy, we observed that (2,4),4 outperforms (2,2),0 on short videos. We believe this phenomenon is caused by attention distraction. When densely sampling short videos (e.g., 256 frames), a large number of redundant tokens are introduced. Previous studies [1][2] have shown that excessive context irrelevant to the query can lead to attention distraction, resulting in a significant drop in performance. This issue is even more pronounced in long-video understanding tasks, where video content tends to be highly redundant, and the relevant segments may only appear for a few seconds. During inference, when the input is fixed at 256 frames, the number of visual tokens under the progressive pooling strategy (2,4),4 is 21,952, whereas for the (2,2),0 strategy, the number of visual tokens is 50,176. The significantly larger number of tokens in the latter increases redundancy and exacerbates attention distraction, which explains the performance difference.

[1] Shi F, Chen X, Misra K, et al. Large language models can be easily distracted by irrelevant context. ICML, 2023: 31210-31227.

[2] Li Z, Zhang Y, Pan T, et al. FocusLLM: Scaling LLM's Context by Parallel Decoding. arXiv preprint arXiv:2408.11745, 2024.

Thank you for your response. However, I still have some concerns regarding the limited novelty of the paper, so I decide to keep my original score.