Long Context Transfer from Language to Vision

We appreciate reviewer DYq4’s valuable suggestions. We have addressed them in detail in the following response.

[W1] Less valuable due to llama 3.2.

We appreciate the reviewer’s feedback, but we would like to clarify that the primary contribution of our work is not simply proposing SoTA model. Instead, our focus is on uncovering, analyzing, and rigorously documenting the long context transfer phenomenon and introducing a synthetic benchmark for long video understanding. These contributions are foundational and intended to inspire future advancements in this domain.

Regarding LLaMA 3.2, it is important to note that this model—released after the completion of our work—was not available during the development of LongVA. We actually believe the release of llama 3.2 is a good thing. We advocate for a standard practice to either extend the language model or directly use the long context language model for training large multimodal models.

[W2] Reliance on synthetic benchmark.

We appreciate the reviewer’s concern and would like to clarify that V-NIAH is not the sole benchmark used in our evaluation. In addition to V-NIAH, we have conducted extensive evaluations on real-world datasets such as Video-MME and MLVU. The key advantage of V-NIAH lies in its scalability—it can accommodate arbitrarily long videos without the need to recollect or reannotate benchmark datasets as video lengths increase. This makes V-NIAH an efficient and effective tool for quickly assessing a model’s visual context length capabilities. By combining V-NIAH with evaluations on real-world datasets, we aim to provide a balanced and comprehensive assessment. The two approaches complement each other.

[W3] The paper could delve deeper into potential limitations or failure cases of the long context transfer phenomenon.

We acknowledge the importance of discussing the potential limitations of the long-context transfer phenomenon. We offer some below and will add a limitation section in the updated manuscript:

Long context transfer is not unlimited. LongVA’s performance saturates when we increase the input frames to 384 on VideoMME.

Dear reviewer DYq4,

We sincerely appreciate your feedback. We take every comment seriously and hope our response can address your concerns. If you have any further questions, we’d be more than happy to respond.

We thank reviewer kGWU for the comments!

[W1] Limited contribution

We believe the key contribution of our paper is in demonstrating and analyzing the transfer of long-context understanding capabilities from text to video. This is a novel and important observation.

Reviewer 9tBS and Reviewer 92h6 have raised concerns about how long video understanding can emerge from training solely on long text and image-text data. This highlights that the phenomenon of long-context transfer is not obvious and may even seem counterintuitive. This makes our findings both significant and worth further exploration.

Through leveraging long-context transfer and conducting comprehensive experiments, we show that LongVA achieves strong performance in long-context video understanding. This is achieved with lightweight training on single-image datasets.

[W2] Long context language model, obvious improvements.

We respectfully disagree with the assertion that using a long-context language model to enhance video understanding is "obvious." Concurrent work (Wu et al.) highlights that existing open-source models struggle to process videos longer than 64 frames effectively.

In contrast, LongVA is the first open-source model (outside of proprietary efforts from companies like Google and OpenAI) to show consistent performance improvements with increasing input frames, maintaining robust accuracy even at 384 frames. We run more experiments on VideoMME of other open-source LMMs:

Table: VideoMME Results of Open-Source LMMs

This ability to achieve state-of-the-art results with no video data during training represents a non-obvious and impactful contribution to the field. We hope this clarification helps to better contextualize the novelty and significance of our work.

H. Wu, D. Li, B. Chen, and J. Li, “Longvideobench: A benchmark for long-context interleaved video-language understanding,” arXiv preprint arXiv:2407.15754, 2024.

[W3] Where is the transfer mentioned in the title of the article reflected?

We acknowledge the importance of clearly articulating how the "transfer" in the title is reflected in our work. The core contribution lies in demonstrating modality transfer from long-text and image training to video understanding. We intentionally do not include: 1. Short or long video data. 2. Multi-image data in the training process of LongVA. Thus, the long video understanding capability is zero-shot transferred from the long text capability and image understanding capability.

The success of this transfer is demonstrated through extensive experiments, as presented in Table 4 and Figure 4. These results highlight how LongVA leverages its long-text and image training to achieve competitive performance in long video understanding tasks without requiring extensive video-specific training.

We thank reviewer yaCh for the insightful comments! Below, we address the reviewer's concerns in turn.

[W1] Visual Encoding Ablations

We compare UniRes and AnyRes in Figure 4. UniRes demonstrates significantly better length extrapolation capabilities for long videos than AnyRes. We further run LongVA (AnyRes) on VideoMME and compare it with LongVA (UniRes). Please refer to our general response for the detailed results.

In Table 7, we observe that AnyRes outperforms UniRes on certain benchmarks, such as AI2D and ChartQA. This may be because UniRes does not include a base image feature and applies 2×2 pooling by default, resulting in fewer visual tokens per image. However, on InfoVQA—a high-resolution benchmark requiring detailed image understanding—UniRes outperforms AnyRes by a large margin. This result further confirms the exceptional long-context extrapolation capability of the UniRes encoding scheme.

[W2] Limited context analysis.

We further evaluate LLaVA-Next-Qwen2 (LongVA without UniRes nor extended LM backbone) on our V-NIAH benchmark. The comparison between LLaVA-Next-Qwen2 and LongVA (AnyRes) in Figure 4 demonstrates the effect of extending the language model’s context length.

The V-NIAH heatmap result is here: https://postimg.cc/v4dSXYYd

[W3]

Thanks for catching those typos! We will update those in the camera-ready version.

[Q2] Image grid, high-resolution images, spatial to temporal informatio.

UniRes is designed to unify the representation of images and videos, enabling the transfer of knowledge from spatial to temporal domains. This is a key strength of UniRes, as it allows LongVA to effectively handle video understanding tasks while being trained on images alone. By leveraging this unified representation, LongVA can bridge the gap between image and video modalities without requiring extensive video-specific training.

We believe that the primary factor behind LongVA’s visual length extrapolation capability lies in its training on long-text data, rather than the resolution of the images used. For our training, we utilized the exact same dataset as LLaVA 1.6, which predominantly contains low-resolution images, with the exception of some OCR data. Importantly, no additional high-resolution image data was collected for this work.

To further illustrate this point, the average pixel count of images in the LLaVA-NeXT SFT dataset is 336,759, which corresponds to an image resolution of approximately 580 × 580 pixels. This demonstrates that the UniRes strategy was fully applied to only a small portion of the training data, emphasizing that the success of LongVA is not reliant on high-resolution images but rather on the effectiveness of its unified representation and long-text training.

[Q3 & Q4] Why image-text training helps long video understanding.

Image training is mainly for aligning the vision and language modalities, and to develop visual understanding capabilities for longva. Image-text training alone is not sufficient for understanding long videos. The key is extending the backbone language model's context length using long text data. This is a separate training phase from image-text alignment. For example, models like LLaVA-Next-Video-32K, which only perform image-text alignment, perform poorly on V-NIAH, as shown in Figure 4

Please refer to our response to Reviewer 9tBS’s Weakness 1 for an intuitive explanation of why long-text training helps long-video understanding. In brief, this phenomenon arises from the effective modality alignment and the long-context capabilities transferred from text training.

[Q5] No advantage in long video understanding.

LongVA clearly shows its strong performance in long video understanding. LongVA is able to increase its performance up to 128 frames, after which the model successfully maintains its performance even when extended to 384 frames. We are the first open-source model (outside of closed-source models from companies like Google and OpenAI) to demonstrate that performance can improve as the number of input frames increases and maintain its performance at 384 frames. This is also verified by a concurrent work (Wu, et.al), which shows that existing open-source models fail to understand videos longer than 64 frames. LongVA also achieves the best performance on VideoMME (Table 4) and MLVU ((Table 9) among 7B models.

H. Wu, D. Li, B. Chen, and J. Li, “Longvideobench: A benchmark for long-context interleaved video-language understanding,” arXiv preprint arXiv:2407.15754, 2024.

We run more experiments on VideoMME of other open-source LMMs:

Table: VideoMME Results for LongVA (AnyRes vs UniRes)

We appreciate reviewer 92h6’s valuable suggestions. We have addressed them in detail in the following response.

[W1] LongVA’s performance on ActivityNetQA and VideoChatGPT

We offer the following explanations for these observations:

1. Benchmark Suitability: ActivityNetQA and VideoChatGPT primarily consist of relatively short videos (see Table 3). For evaluating long video understanding capabilities, we recommend focusing on benchmarks like VideoMME (Table 4) and MLVU (Table 9), which are better aligned with the objectives of LongVA.

2. Dataset Bias: LLaVA-NeXT-Video models are trained on video datasets generated by ChatGPT, and the evaluation for ActivityNetQA and VideoChatGPT is also conducted using ChatGPT as a judge. This introduces potential bias, favoring models trained on ChatGPT-generated data. Despite this, we include these datasets in good faith to demonstrate that LongVA performs relatively well on them, even without any specific video training.

3. Performance Improvements with Light Video Training:

After incorporating some video-specific training (which remains significantly lighter than that of LLaVA-NeXT-Video), LongVA-DPO achieves notable performance improvements on these benchmarks. This partially validates our hypothesis stated in Section 2.

[W2] Training details

Section 3.1 of our paper provides substantial information about the training process for the long-context LLM on pure text. For information on image-text alignment, please refer to our response to Reviewer 9tBS's Weakness 2, where we detail the pretraining and fine-tuning setups, including datasets and hyperparameters.

We acknowledge the need for further clarity and will include more comprehensive training details in the appendix of the camera-ready version. Additionally, our code and datasets will be open-sourced, ensuring full reproducibility of our results.

[W3] Video understanding ability and UniRes v.s. AnyRes

We would like to clarify that the primary goal of LongVA is not to achieve state-of-the-art performance but to demonstrate the effectiveness of the long context transfer phenomenon. To this end, we intentionally exclude video training data to highlight that LongVA’s video understanding capabilities are transferred from two sources: (1) its long-text understanding capabilities and (2) its image understanding capabilities. This allows LongVA’s video capabilities to be evaluated in a zero-shot setting.

While excluding video training data provides a clear demonstration of long context transfer, it places LongVA at a disadvantage compared to models specifically trained on video data. This is especially on datasets where response format plays a significant role or when evaluations are judged by ChatGPT. Despite these challenges, LongVA achieves the strongest performance among 7B models on both VideoMME (Table 4) and MLVU (Table 9). We acknowledge that LongVA performs relatively less optimally on certain image datasets and short video datasets, and we believe that incorporating additional image and video data during training could improve its performance. However, this is beyond the primary scope of this paper.

For AnyRes v.s. UniRes, please refer to our response for Q6.

[Q1] Can training only be performed on a single image? How is the long text obtained in image-text training? Because the trained LLM backbone has a context length of 224K. The training details are not provided.

We can perform training on multi-image and video datasets. However, as the response in W3, we explicitly exclude the following during image-text alignment to clearly demonstrate the long-context transfer phenomenon:

1. Multi-image data
2. Video data
3. Long-text data

This design ensures that LongVA's ability to process long video contexts is entirely transferred from its text-training phase, while image-text alignment focuses solely on adapting the model to process visual signals without introducing long-context information.

Dear reviewer 92h6,

Thank you again for finding long context transfer interesting. We are also encouraged by your comments on V-NIAH and our rich experimental results.

We truly value your insights and hope our responses have addressed your concerns. If you have additional feedback or suggestions, we would greatly appreciate hearing them to ensure we have fully resolved the points.

We thank reviewer 9tBS for the insightful comments! Below, we address the reviewer's concerns in turn.

[W1] transfer from text-only data to long video understanding.

As stated in line 195 of our paper, we propose the hypothesis: “if the modalities of vision and language are effectively aligned, the capability to handle long contexts could transfer from text to vision.” LongVA builds on this idea by utilizing CLIP, a well-established image model that generates features inherently aligned with text representations. Through further alignment between the LLM and CLIP modalities, we ensure that these representations integrate into the LLM's input space. LongVA is designed as a preliminary effort to test and validate this hypothesis, providing a foundation for further exploration in this direction.

[W2] Image-text alignment details.

As stated in line 215, we closely follow the training methodology proposed by LLaVA-1.6. Specifically, our setup is as follows:

Pretraining (Feature Alignment):

• Objective: Train only the projection layer connecting CLIP and the LLM while keeping the rest of the model frozen.
• Loss: Language modeling loss conditioned on the image.
• Dataset: LAION-CC-SBU 558K with BLIP-generated captions.
• Hyperparameters:
  • Learning rate: 1e-3
  • Batch size: 64
  • Optimizer: AdamW
  • Epoch: 1

Visual Instruction Fine-Tuning:

• Objective: Train the projection layer, language model, and ViT model jointly.
• Loss: Language modeling loss conditioned on the image.
• Dataset: LLaVA-1.6 (760K samples, image-text pair only).
• Hyperparameters:
  • Projection layer and LLM learning rate: 1e-5
  • ViT learning rate: 2e-6
  • Batch size: 32
  • Epoch: 1

We will incorporate this information into the camera-ready version of the paper and provide a detailed training setup in the appendix. Furthermore, our code will be open-sourced to facilitate reproduction.

[W3] Length generalization of input frames.

Please refer to our response to [W1], which addresses the hypothesis of modality alignment enabling the transfer of long-context capabilities. Specifically, the effective alignment between CLIP and the LLM allows representations to extend beyond the limitations of input frames during training, leveraging the LLM's inherent capability to process longer contexts.

[W4] Those results seem to imply that the input frames and video-specific training are not that important for video understanding.

We fully agree with the reviewer’s assessment on this matter. We believe VILA-1.5’s high performance could be due to: (1) Some questions in Video-MME may require understanding only a few frames to arrive at the correct answer. (2) There may be biases in the questions and answer choices, allowing a strong language model backbone, such as the 30B+ model used in VILA-1.5, to correctly guess answers without relying on video input. These limitations were part of the motivation behind our proposal of V-NIAH. Unlike benchmarks where sparse sampling might suffice, V-NIAH is designed to eliminate this shortcut. Since the needle’s location is not given as prior knowledge, models relying on sparse sampling are likely to miss the needle frame entirely. In V-NIAH, the model must process the entire video (spanning thousands of frames) to locate the relevant information. Additionally, the questions in V-NIAH are intentionally counterintuitive, ensuring the language model cannot rely on textual priors alone and must utilize the video input effectively.

[W4] Ablation with the same LM backbone of different context lengths and video training.

To ablate the impact of the backbone language model’s context length on video understanding, we further run LLaVA-Next-Qwen2 on V-NIAH: https://postimg.cc/v4dSXYYd. Please refer to Table 5 for the relation between LLaVA-Next-Qwen2, LongVA, and LongVA(AnyRes). All three models are trained with exactly the same data and hyperparameters during image-text alignment. Our newly added results together with Figure 4 show the text context length of the language backbone (LLaVA-Next-Qwen2 v.s. LongVA(AnyRes)) significantly impacts the visual context length on V-NIAH

Due to limited computational resources, we were unable to include ablation with heavy video training. That said, we fully acknowledge that incorporating video training would likely improve the benchmark scores. The decision to exclude video data during training was not due to any disregard for its value, but rather to highlight the long context transfer phenomenon. By omitting video-specific training, we aimed to ensure that LongVA’s performance on video understanding tasks could be classified as zero-shot, offering a clearer validation of the proposed hypothesis.