Frame-Voyager: Learning to Query Frames for Video Large Language Models

W1&Q1: data scaling for longer videos

Response:

Frame-Voyager demonstrates strong generalization capabilities to longer videos through two key aspects:

(1) First, Frame-Voyager inherently possesses generalization capabilities, as the essence of video understanding is transferable. It is aligned with most of the existing Video-LLMs’ practices, where models are trained primarily on short videos, yet can also generalize to long videos during testing, such as VILA and LLaVA-One-Vision [1][2].

For example, considering temporal reasoning tasks where the question requires locating the beginning of a video (case study shown in Fig. 6, Lines 445-485), the Frame-Voyager consistently focuses on the video openings regardless of total duration. Similarly, for content identification tasks, e.g., finding ingredients as shown in Fig. 9 (Lines 922-68), it selects relevant frames containing ingredients independent of video length.

In the main paper, we sample 4 frames from 32 frames (or 2 frames from 16 frames) in training, and we directly choose 8 frames from 128 frames in inference. To further demonstrate the generalization capabilities of Frame-Voyager, we augment experiment on LLaVA-One-Vision (please kindly refer to General Response of First Round Rebuttal (2/4) for the details and the experimental results). The results show that Frame-Voyager shows consistently better performance when varying the number of frames from 2 to 64.

(2) Second, our data collection process can be further optimized. We implement two optimization approaches (please kindly refer to General Response of First Round Rebuttal (3/4) for details and experimental results). The first leverages the frame-to-frame similarity and temporal proximity to filter out redundant combinations, and the second one (suggested by the Reviewer qbz8) uses the CLIP to reduce search space. The two optimizations significantly reduce the Frame-Voyager’s data construction time to only 4.4% of the original duration, while maintaining comparable performance across all benchmarks.

In summary, the elaborated two key aspects demonstrate Frame-Voyager's scalability for long video analysis through both its inherent design and optimized data collection pipeline.

[1] VILA: On Pre-training for Visual Language Models. CVPR 2024

[2] ​​LLaVA-OneVision: Easy Visual Task Transfer. ArXiv 2024.

W2: 8 or 16 frames for short video benchmarks

Response:

We appreciate the reviewer's suggestion regarding the short video evaluation. We have expanded our experiments to include 8 and 16-frame evaluations on NextQA and ANQA datasets. The results are shown below.

Our results show Frame-Voyager consistently improves performance with 8-frame processing across both model sizes and datasets. Note that with 16 frames, Frame-Voyager's behavior matches uniform sampling as no frame selection is required.

We will add these expanded results in our revised paper to provide a more comprehensive evaluation of our method's effectiveness across different frame settings.

---

W3: Addition results on more benchmarks

Response:

We are currently conducting the relevant experiments and will provide the corresponding results very soon!

---

W3: Addition results on more benchmarks

Response:

Thanks for the waiting, we provide experimental results on MVbench in the table below. The experiments are conducted with 8 frames as suggested by Weakness # 2 of Reviewer ULTS.

---

Please feel free to let us know if you have any additional questions. We would be happy to address them.

W4: Efficiency comparison between Frame-Voyager and LongVILA

Response:

We appreciate the reviewer's concern regarding efficiency comparisons.

• First, we would like to clarify that Frame-Voyager and LongVILA differ fundamentally in their approaches: LongVILA requires complete model re-training with five stages, while Frame-Voyager functions as a modular enhancement to the existing VILA architectures.
• Second, to facilitate a thorough comparison, we present detailed efficiency comparison below.

• Resource Efficiency: Frame-Voyager requires only ~64 GPU hours of training compared to LongVILA's estimated 5,400+ hours, making it significantly more accessible and cost-effective.
• Minimal Parameter Increase: Frame-Voyager adds only 20M parameters (0.2% increase) to the base VILA model.
• Inference Latency: Frame-Voyager demonstrates superior computational efficiency with a 5x reduction in latency (1.696s vs 8.654s) and a 3x reduction in memory consumption (23,069 MiB vs 74,293 MiB) compared to LongVILA.

These results highlight Frame-Voyager's practical advantages in terms of deployment flexibility, resource utilization, and computational efficiency, while maintaining competitive performance on video understanding tasks.

---

Q4: Latency of the method

Response:

Thanks for the valuable suggestions. Please kindly refer to the General Response of First Round Rebuttal (4/4) for the analysis of overall efficiency.

In summary, the Frame-Voyager only introduces a tiny computational overhead. The minimal increases in parameter count and memory usage suggest that our method scales efficiently while achieving superior video understanding capabilities.

---

Please kindly let us know if you have any additional follow-up questions, and we would be happy to address them.

W1.1&Q1: Cost and detail for data collection

Response:

For the data collection, our choice of smaller frame combinations (M=16/32, T=2/4) aligns with current practice. The leading Video-LLMs, such as VILA and LLaVA-One-Vision, are primarily trained on short video segments [1, 2], but can still generalize to long videos during testing. Specifically, generating frame combinations based on C(32, 4) for the VideoChatGPT (100k) dataset requires 32 A100 GPUs over approximately 40 hours. For the NextQA (35k) dataset, using C(16, 2), it takes 8 A100 GPUs about 1 hour.

[1] VILA: On Pre-training for Visual Language Models. CVPR 2024

[2] ​​LLaVA-OneVision: Easy Visual Task Transfer. ArXiv 2024.

---

W1.2&Q5: Data collection efficiency

Response:

Thanks for the valuable suggestions. First, we would like to highlight that our data collection process is automated and one-off, allowing the resulting data (generated by VILA-8B) to be directly used by other Video LLMs, i.e., results of VILA-40B in Table 1 (Lines 292-293) and results of LLaVA-One-Vision-7B in the General Response of First Round Rebuttal (1/4). We will open-source our algorithm and also the generated dataset after the paper's acceptance.

Furthermore, to address the efficiency concern, we implement two optimization approaches (please kindly refer to General Response of First Round Rebuttal (3/4) for details and experimental results). The first leverages the frame-to-frame similarity and temporal proximity to filter out redundant combinations, and the second one (suggested by the Reviewer qbz8) uses the CLIP to reduce search space. The two optimizations significantly reduce the Frame-Voyager’s data construction time to only 4.4% of the original duration, while maintaining comparable performance across all benchmarks.

---

W2.1 More input frames can enhance QA performance?

Response:

We respectfully disagree with the claim regarding the benefits of additional input frames for QA performance.

We agree that arbitrarily increasing the token length of the model may not always lead to improvement [1] (Lines 038-040). This observation is also aligned with the LongVA-7B's results on the Video-MME benchmark (Table 4 in LongVA paper [2]), where performance plateaus or degrades with more frames.

For instance, on short videos, the LongVA-7B’s accuracy peaks at 61.1% with 32 frames, marginally improves to 61.4% with 64 frames, but drops back to 61.1% at 128 frames. Similarly, the LongVA-7B’s overall performance shows minimal gains from 52.4% (64 frames) to 52.6% (128 frames), and even declines to 51.8% with 384 frames.

[1] Lost in the middle: How language models use long contexts. TACL 2024.

[2] Long Context Transfer from Language to Vision. ArXiv 2024.

---

W2.2&Q2 Generalization ability of Frame-Voyager

Response:

Thanks for raising this important point. Please kindly refer to General Response of First Round Rebuttal (2/4) for the comprehensive analysis and experimental results regarding the generalization of Frame-Voyager on higher numbers of frames. It demonstrates that Frame-Voyager maintains superior performance across frame numbers ranging from 2 to 64.

Additionally, with the optimized data collection pipeline mentioned in General Response of First Round Rebuttal (3/4) and the response to W1.2&Q5 above, we believe that the generalization ability of Frame-Voyager could be further enhanced through increasing the numbers of frames in data collection.

Dear Reviewer qbz8,

We hope this message finds you well. We would like to express my gratitude for the time and effort that you have dedicated to the review process of our submission.

Furthermore, we are writing to kindly remind you that the rebuttal period is coming to a close. Your feedback and evaluation are crucial to the progress of our work, and we value your expertise and insights immensely.

During rebuttal, we have thoroughly addressed all the queries and concerns you raised regarding our submission. Each point has been carefully considered, and we have provided detailed responses.

We are eager to engage in further discussions with you on the responses and any other aspects of our study. Your feedback plays a pivotal role in shaping the direction of our research, and we are keen to hear your thoughts and suggestions.

Thank you for your attention to this matter.

Best regards, Authors of submission #13344

Thank you for your follow-up questions and comments. We truly appreciate the time and effort you’ve taken to engage with our work in such detail. Your feedback has been very valuable in helping us improve the manuscript.

Below, we provide detailed responses to your follow-up queries:

---

Q1: Marginal effect of more frames

In the Reviewer qbz8’s original review, it is said that:

Reviewer qbz8: prior research (e.g., LongVILA [1], LongVA [2]) has shown that additional input frames can enhance QA performance.

In the Reviewer qbz8’s latest review, it is mentioned that:

Reviewer qbz8: The authors argue that more frames do not always lead to improved performance, this contradicts the findings of LongVILA.

---

We still argue that more frames do not always lead to improved performance.

First, we delineate the results from three SOTA backbones, i.e., LongVA, LLaVA-One-Vision and LongVILA, to demonstrate that the improvement benefiting more frames has an upper bound.

Second, we give the explanation from the angle of information entropy along with an intuitive example to further illustrate the diminishing marginal effect.

---

Section #1: Quantitative Results

In Response to the Reviewer qbz8 (First Round-1), we have already cited the results from the LongVA paper [1] which actually supports our argument, i.e., the improvement benefiting more frames has an upper bound.

< For instance, on short videos, the LongVA-7B’s accuracy peaks at 61.1% with 32 frames, marginally improves to 61.4% with 64 frames, but drops back to 61.1% at 128 frames. Similarly, the LongVA-7B’s overall performance shows minimal gains from 52.4% (64 frames) to 52.6% (128 frames), and even declines to 51.8% with 384 frames.

For convenience, we copy the results from the Table 4 of LongVA paper [1] below.

In General Response of First Round Rebuttal (2/4), adopting LLaVA-One-Vision as backbone, we have included the results on Video-MME with different numbers of frames from 2 to 128 on uniform sampling and Frame-Voyager (128 candidate frames). The results also indicate that there are clear upper bounds. Frame-Voyager shows consistently better performance when varying the number of frames from 2 to 64.

In LongVILA [2], the same conclusion could be made based on Figure 3 (a). It is clear that the gains provided by each additional frame are diminishing. Here we copy the averaged results from Figure 3 (a) in the LongVILA paper for reference.

The results from LongVA, LLaVA-One-Vision, and LongVILA demonstrate that increasing the number of input frames yields diminishing returns, with performance improvements reaching a plateau.

References:

[1] Long Context Transfer from Language to Vision, ArXiv 2024.

[2] LongVILA: Scaling Long-Context Visual Language Models for Long Videos, ArXiv 2024.

W1: Adaptive/dynamic frame selection

Response:

Thank the Reviewer 6u8S for the valuable and insightful suggestion. We agree with the observation about varying information density across different videos and query types. Our Frame-Voyager method can indeed conduct dynamic frame selection without requiring additional training or supervision. This can be achieved by normalizing the rewards of candidate frames to a 0-1 range and selecting frames that exceed a specified threshold.

To validate this capability, we conduct experiments on Video-MME (without subtitles) using VILA-8B. We set a normalized reward threshold of 0.7 and maintain our original constraints (maximum 8 frames, minimum 1 frame) for a fair comparison. The experimental results are shown as follows.

From the results, we can find that Frame-Voyager with dynamic selection can improve overall performance while maintaining a similar average number of used frames. Specifically, the fixed-number selection (4-frame combination) achieves a result of 47.9, whereas the dynamic-number selection achieves a result of 49.6, with an average of 4.1 frames. This demonstrates that Frame-Voyager is also capable of conducting dynamic frame selection.

Building on this experiment, we further examine the required information density (defined by the number of dynamically selected frames in our model) from three perspectives: video duration, video content domains, and query types. We present our experimental results below.

The results reveal some patterns in how our dynamic selection adapts to different scenarios:

• Video Duration: Longer videos naturally require more frames (Short: 3.72, Medium: 4.07, Long: 4.61 frames on average).
• Video Content Domains: The correlation between frame requirements and video domains is less obvious. Sports videos require the most average frames (4.43) due to frequent action changes, while other domains maintain relatively consistent frame requirements (3.89-4.31).
• Query Types: Task complexity does influence frame selection. Complex tasks like counting (5.36 frames) and reasoning (4.58 frames) require more frames for comprehensive analysis. Note that counting tasks, despite having lower reasoning demands, usually requires answering questions like "How many different people appear in the video?", which strongly depend on multiple frames. Simpler tasks like perception (3.01 frames) and recognition (3.84 frames) need fewer frames.

The results confirm Frame-Voyager's effectiveness in dynamically selecting frames by considering both the video's information density and the specific query needs. We will further include these bonus results and analysis in the revised version.

W2.1: Generalization process is unclear

Response:

It is “directly apply the model to a higher frame number”. The definition of the generalization here is: when training, we sample 4 frames from 32 frames (or 2 frames from 16 frames), and when inference, we directly choose 8 frames from 128 frames. These numbers are different, so it is generalized from smaller numbers to larger numbers.

Additionally, for more analysis and results regarding this generalization process, please kindly refer to the General Response of First Round Rebuttal (2/4) comment. In this general response, we explain why Frame-Voyager is able to generalize to long videos and larger numbers of frames. We also add experimental support based on the LLaVA-One-Vision.

As for the parameter K, we believe the Reviewer 6u8S has confused between K and T. The K is only used in training, and it means the number of selected combinations for ranking loss optimization. The T is used both in training and inference, and it means the frame number in each combination.

---

W2.2: How Frame-Voyager makes good use of global video context

Response:

Frame-Voyager is built upon the selection of frame combinations instead of retrieving individual frames separately, and it naturally requires a good understanding of the global context of videos (Lines 54-56). In terms of the question of “How Frame-Voyager makes good use of global video context”, we believe there are two points:

(1) For the model architecture of Frame-Voyager, we feed all the candidate frames and utilize the bottom layers of the LLM whose attention mechanism enables the simultaneous processing of frame-to-frame and frame-to-query relationships.

(2) For the training of Frame-Voyager, we design a ranking loss (Equation 3, Lines 250-253) to optimize the selection of frame combinations. It further encourages the model to capture global information for identifying the better combination regarding the given query.

---

W3: Comparison fairness

Response:

Thank the Reviewer 6u8S for raising this point about comparison fairness. We would like to clarify two key aspects of our experimental design and present additional results that demonstrate our method's effectiveness.

(1) First, regarding fairness of comparison: while our Frame-Voyager initially observes 128 frames, it ultimately uses only 8 frames for final processing by the LLM, similar to all the baselines. Both methods have access to the full video - the difference lies in how they select frames. Uniform sampling uses a straightforward distribution strategy, while our method employs selection based on the modeling of the global context between the video and the query. We also compare our method against several frame selection baselines (rather than uniform sampling), as shown in Table 2 (Lines 324-337). These baselines include:

• Rule-based shot boundary detection methods: Histogram, Edges Change Ratio, and Motion, which select frames based on significant frame transitions in texture, structure, and motion.
• Cluster-based method: This approach extracts histograms from all frames and employs K-means clustering to select the most representative frames near cluster centers.
• Frame-text matching methods: These two methods retrieve frames by calculating cosine similarity between query inputs and individual frames using VILA and CLIP embeddings.

The results demonstrate that the Frame-Voyager outperforms all these baseline methods, showing superior performance in both frame subset selection and efficient video understanding.

(2) Second, regarding best uniform sampling configuration: due to the limit of VILA-8B, please kindly refer to General Response of First Round Rebuttal (2/4) for more results conducted on LLaVA-One-Vision. We could observe that Frame-Voyager shows consistently better performance when increasing the number of frames from 2 to 64 (with a best performance of 60.5%). The uniform sampling method plateaus at 32 frames (with a best performance of 58.2%).

Q1: Results on LLaVA-One-Vision

Response:

Please kindly refer to ​​General Response of First Round Rebuttal (1/4) for the results of Frame-Voyager on LLaVA-One-Vision trained with the same data generated by VILA-8B.

From the results, it is shown that Frame-Voyager trained with data generated by one Video-LLM (in our case, VILA-8B) can be zero-transferred to another Video-LLM, e.g., LLaVA-One-Vision-7B. This demonstrates the generalizability and plug-and-play ability of Frame-Voyager.

---

Q2: Comparison with LLM agents

Response:

We appreciate the reviewer highlighting these related papers. We would like to highlight that the two agent-based methods are not directly comparable to Frame-Voyager due to the large language models and complex interaction steps they have, such as multi-round interactions with an LLM-agent and the use of closed-source GPT-4 as the final reasoning tool. We elaborate more on the key conceptual differences as follows.

1. VideoAgent's approach begins with uniformly sampled frames and uses CLIP for text-based frame retrieval. This differs from Frame-Voyager's comprehensive frame combination strategy, which considers the relationships between multiple frames rather than relying solely on text matching of VideoAgent.

2. VideoTree employs a hierarchical approach, first clustering frames and then using language model analysis of cluster captions to guide further exploration by GPT-4. While this method implicitly considers frame relationships through caption analysis, Frame-Voyager takes a more direct approach by explicitly evaluating and ranking complete frame combinations.

Note that both methods rely on running complete Video-LLM models multiple times, requiring significant computational resources. These comparisons highlight Frame-Voyager's unique contribution: a streamlined, efficient approach to frame selection that directly optimizes for informative frame combinations rather than relying on intermediate steps or external reasoning tools. We will also include the appropriate citations and discussions in our revised manuscript.

References:

[1] VideoAgent: Long-form Video Understanding with Large Language Model as Agent

[2] VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos

---

Q3: Experiment on temporal grounding

Response:

Thank the Reviewer 6u8S for this suggestion. While we appreciate the recommendation to evaluate our method on grounded QA (temporal grounding), it's important to note that Grounded QA and our approach address different tasks of video understanding.

• Grounded QA: focuses on identifying specific continuous temporal segments directly related to a question. It’s more suitable for local questions.

• Our approach: focuses on selecting multiple frames with temporal relations (but no need to be continuous) that capture broader temporal relationships across the video. It allows us to handle both local and global questions. To provide a fair comparison, as mentioned in the response to W3, we adapt the grounding method in TempGQA [1] to identify relevant video segments based on the question, and then sample frames from these segments for answer generation with VILA-8B. On the VideoMME (without subtitles) benchmarks, the results are:

  • TempGQA-based approach: 46.4;

  • Baseline uniform sampling with VILA-8B: 47.5;

  • Our Frame-Voyager method: 50.5.

These results demonstrate that while grounding approaches are valuable for their specific use case, our method's broader frame combination strategy achieves better performance on general video understanding tasks. We will add this additional baseline method in the revised manuscript.

[1] Can i trust your answer? visually grounded video question answering. CVPR 2024.

---

Please kindly let us know if you have any further questions or comments. We would be happy to address them.

Q5: Observed issues may stem from limitations in the model side (VILA)

Response:

Thank the Reviewer pMFu for this valuable discussion and feedback. Please kindly refer to General Response of First Round Rebuttal (2/4) for the comprehensive analysis regarding the generalization of Frame-Voyager on higher numbers of frames.

We agree with the Reviewer pMFu that model capability is another factor contributing to the pattern in Figure 3, and our current explanations (Lines 413-433) is not completed. We will revise this experimental analysis part in the manuscript.

---

Please feel free to let us know if you have any additional follow-up questions, and we would be happy to address them further.

W1: Reliability of the pseudo-label strategy

Response:

Thank the Reviewer pMFu for raising these important concerns. We agree that in certain cases LLM can intuitively generate correct answers even with wrong localized frames, primarily due to inherent biases in language models such as language shortcuts and spurious vision-language correlations [1].

We already consider this biased issue of video-LLM, as mentioned in Lines 173-177. To avoid this, our method incorporates the following designs, each compared with the alternative method strongly affected by such bias:

1. We implement necessary filterings against language model biases in our data preparation pipeline (Lines 175-177). We analyze the variance in losses and find that pairs with low variance in the losses across different combinations are not sensitive to the quality of combinations. One of the reasons might be that correct answers may be generated based on the Video-LLM’s inherent language prior without referring to any input content. Thus, we exclude these cases where the model appears to rely primarily on language priors. The effectiveness of this filtering step is validated through ablation studies, as shown in Table 3 (Lines 331-332), comparing VideoChatGPT with and without filtering (49.1 vs 48.3).

2. Our framework adopts a comparative approach that ranks different frame combinations based on their relative effectiveness, rather than aiming to identify a single "perfect" combination, which is usually infeasible.

• Instead of using binary rewards (correct/incorrect answers) or selecting only the lowest-loss combination, we implement a ranking-based loss function that compares pairs of frame combinations. From the perspective of RL, this continuous reward structure provides more stable training signals compared to discrete alternatives [4].
• Our data collection method goes beyond simple frame-question pairs for answer prediction, which cannot differentiate between the inherent biases of LLMs and the usage of provided frame combinations. Instead of directly evaluating answer correctness, we use the internal loss to measure the model's confidence during the answer generation. This loss reflects the model's confidence [2, 3] in producing the answer given the frame combination and question, where lower loss indicates higher confidence.
• Our experimental results in Table 3 (Lines 333-336) demonstrate that this comparative approach (Lines 334-336, setting (5-6) and Frame-Voyager with K=4) consistently outperforms methods trained to select only the single best frame combination (Line 333, setting (4) Top-1 Rank).

In summary, our method produces reliable training signals while minimizing the impact of language model biases. We include additional ranking data examples in this anonymous link to further demonstrate the quality of our generated training data for reference.

[1] Can i trust your answer? visually grounded video question answering. CVPR 2024.

[2] On Calibration of Modern Neural Networks. ICML 2017.

[3] Language Models (Mostly) Know What They Know. ArXiv 2022.

[4] Reward function design in reinforcement learning. Eschmann, J. Reinforcement Learning Algorithms: Analysis and Applications 2021.

---

W2: Complex pre-training data design due to the sensitive signal

Response:

Based on the specialized design mentioned in the response of W1, we believe that the quality of the generated ranking data is guaranteed. Meanwhile, we also observe that the combinations with redundancy or blurring frames tend to have lower ranking due to the lack of information. For example, we can observe that the combination with similar frames has lower rank in the second case shown in the anonymous link.

Dear Reviewer pMFu,

We hope this message finds you well. We would like to express my gratitude for the time and effort that you have dedicated to the review process of our submission.

Furthermore, we are writing to kindly remind you that the rebuttal period is coming to a close. Your feedback and evaluation are crucial to the progress of our work, and we value your expertise and insights immensely.

During rebuttal, we have thoroughly addressed all the queries and concerns you raised regarding our submission. Each point has been carefully considered, and we have provided detailed responses.

We are eager to engage in further discussions with you on the responses and any other aspects of our study. Your feedback plays a pivotal role in shaping the direction of our research, and we are keen to hear your thoughts and suggestions.

Thank you for your attention to this matter.

Best regards, Authors of submission #13344