We sincerely thank you for the valuable comments.

Q1: Could Query Decouple become a performance bottleneck?

A1: Empirically, Query Decouple resulting in minimal impact on overall performance as it's a straightforward task for LVLMs. This conclusion is supported by two key observations:

• Consistent improvements across model capacities: As illustrated in Table 1 of the paper, Video-RAG provides stable performance enhancements when implemented with various 7B models, with less powerful models showing proportionately greater gains. Notably, on the Video-MME-mini benchmark (as shown in Supplementary Section 3), even the lowest-performing baseline model, Video-LLaVA-7B, achieved approximately 90% query decoupling accuracy across 270 queries.

• Deliberate noise robustness testing: We conducted experiments involving random text noise injections to the decoupled outputs. The results revealed that LLaVA-Video-7B-64frames, when paired with Video-RAG, maintained performance within acceptable limits even at significant levels of noise, as shown on the Video-MME benchmark. These findings collectively demonstrate that Query Decouple does not limit Video-RAG's effectiveness.

Q2.a: The article should discuss the impact of different external tools.

A2.a: As suggested, we conducted ablation studies on various components of Video-RAG on Video-MME, and we observed only minimal declines in performance. For each component, we assessed two alternative implementations.

Specifically, we have substituted EasyOCR in the OCR component with Tesseract OCR[1] and the OCRAD API. In the DET component, we have replaced APE with Owl ViT[2] and GRiT[3]. Additionally, in the ASR component, we have replaced Whisper with the SpeechRecognition library and the Vosk API. The sustained performance demonstrates the robustness of Video-RAG to component variability.

Q2.b: The article should discuss the impact of different hyperparameters of the external tools.

A2.b: Regarding hyperparameters, due to time constraints, we limited our evaluation to different confidence thresholds in the APE of the DET component and EasyOCR of the OCR component in Video-RAG. The results demonstrated that Video-RAG can maintain strong performance across a range of component hyperparameter combinations, indicating its robustness to the settings of these hyperparameters.

Q3: For long-video understanding, another class of methods calculates the relevance between frames and the question, then samples frames based on relevance to answer user queries. The article should discuss and compare these in related work.

A3: Thank you for your constructive suggestions. Our research identifies a total of six recent studies that can be categorized in this domain (e.g., ASK[4], Nar-KFC[5], etc.). Below, we outline the key distinctions between Video-RAG and these existing methods:

• Motivation: The primary objective of these studies is to identify and select frames that are pertinent to the query for LVLM. In contrast, Video-RAG aims to enhance the capabilities of the LVLM by providing contextually relevant textual input that directly relates to the query.

• Methodology: The studies above utilize algorithms to calculate image-text similarity to extract frames. Conversely, Video-RAG employs a refinement process that consolidates outputs from external tools through the RAG approach.

• Effectiveness: The performance improvements in these studies are relatively modest. In comparison, Video-RAG showcases a substantial enhancement in performance, achieving a +6.7% boost with Video-RAG in contrast to a +1.2% boost with ASK[4] in the Video-MME using LLaVA-Video-7B with 64 frames.

Furthermore, we have conducted experiments on Video-MME by integrating the frame selection method ASK[4] (CVPR 2025) with Video-RAG. The limited performance gain suggests that Video-RAG does not significantly depend on keyframe selection. This system demonstrates resilience to the absence of keyframes by utilizing complementary modalities to support the reasoning process of the LVLM.

Q4: Would using a larger model for Query Decouple improve pipeline performance?

A4: Utilizing larger models for query decoupling does result in performance enhancements, but the gains are marginal. Specifically, we conducted experiments by substituting Video-RAG's Query Decouple model with more advanced 72B LVLMs, and we observed only incremental improvements:

Q5: For the raw video, what sampling method did the article use—uniform sampling, similarity-based sampling, or another method?

A5: We employed uniform sampling for two primary reasons:

• To ensure alignment of our frame sampling strategy with most LVLMs, thus demonstrating the plug-and-play capability of our method without reliance on key frames.
• To facilitate a fair comparison. Empirical evidence supports that uniform sampling provides the most computationally balanced and generalized approach for understanding long videos.

Q6: Minor issues about the code repository link and lack of citations to clarify which specific methods.

A6: Thanks for your reminder. We have updated the valid code repository link and added the relevant citations to clarify the specific methods in line 39.

[1] R. Smith. An Overview of the Tesseract OCR Engine. Ninth International Conference on Document Analysis and Recognition. 2007.

[2] M Minderer, et al. Simple Open-Vocabulary Object Detection with Vision Transformers. ECCV 2022.

[3] Jialian Wu, et al. GRiT: A Generative Region-to-text Transformer for Object Understanding. ECCV 2024.

[4] Tang X, et al. Adaptive keyframe sampling for long video understanding. CVPR 2025.

[5] Bo Fang, et al. Threading Keyframe with Narratives: MLLMs as Strong Long Video Comprehenders.

We sincerely appreciate the reviewer’s feedback. We have refined our response to further clarify the roles of our Video-RAG components, particularly in the challenging long-video context, and to ground our design in established information theory.

Q1: Why do imperfect Query Decouple and imperfect tools reduce accuracy only slightly?

A1: The minimal impact of the Query Decouple model stems from the strong performance of both the 7B and 72B models in this task, with only negligible differences.

Additionally, Video-RAG's inherent robustness, primarily through the RAG module, ensures resilience to imperfections in upstream tools. Additional experiments confirm this robustness: replacing Whisper ASR tool with the weaker Vosk API reduces overall accuracy by just 0.6%. However, removing RAG while using the weaker ASR causes drops of 3.5% overall and 6.3% on long-video subsets with more noise potential. This highlights RAG's effectiveness in filtering noise and ensuring relevant context for the language model.

This robustness is further enhanced by two factors.

• Information redundancy across modalities; even if ASR fails, OCR or object detection often provide overlapping semantic cues.
• Even 'weak' tools provide a low signal-to-noise output rather than pure noise, the RAG retriever is adept at matching the core semantic 'signal' from a query to these noisy context, effectively ignoring the errors.

All models are based on LLaVA-Video-7B with 64-frame inference input on Video-MME dataset.

Q2: On the Necessity of Query Decouple and External Tools. (Does that mean they are dispensable?)

A2: While our Video-RAG is robust, this should not be mistaken for component redundancy. The necessity of Query Decouple and the external tools becomes undeniable when we consider their total contribution to the performance gain, especially on long videos scenes.

• Necessity of External Tools: The tools (OCR, DET, ASR) are absolutely critical. The – drop all tools ablation shows that removing them causes the long-video performance to decrease from 63.8% -> 51.8% (-12.0%) and overall performance from 70.0%->63.3% (-6.7%). The tools provide essential textual context that is simply unavailable from video frames alone, and our Video-RAG's success is fundamentally dependent on them.
• Necessity of Query Decouple: The Query Decouple module is also a vital contributor. Removing it (– remove Decouple) lowers the long-video score by 1.2%. It makes it a significant and necessary part of the architecture, particularly for breaking down complex queries about event-rich videos.

Q3: The Theoretical Contribution of Video-RAG.

A3: Theoretically, Video-RAG’s architecture is a principled implementation of the Information Bottleneck (IB) principle [1], which seeks a compressed representation $Z$ of a source $X$ that preserves the maximum possible information about a target variable $Y$.

This is formally expressed as minimizing the following Lagrangian:

$\mathcal{L}_{IB} = I(X; Z) - \beta I(Z; Y)$ where $I(\cdot;\cdot)$ denotes mutual information and $\beta$ balances the compression (minimizing $I(X; Z)$) and prediction (maximizing $I(Z; Y)$).

Our Video-RAG maps directly onto this principle:

• Source ($X$): Noisy multimodal video information (raw frames, OCR, DET, ASR outputs).
• Bottleneck ($Z$): Concise textual context selected by the RAG retriever.
• Target ($Y$): The correct answer to the user's query.

Retriever acts as the mechanism that creates the bottleneck, compressing the large source $X$ into a small context $Z$ by selecting passages that are maximally informative for generating the final answer $Y$.

The Query Decouple module enhances this process by breaking down complex queries into focused sub-queries, improving retrieval precision and bottleneck quality. This design's robustness stems directly from its alignment with information-theoretic principles, extending RAG to effectively address the scale and noise of video understanding in Video-RAG.

[1] Tishby et al., The information bottleneck method. 1999.

Thanks for your constructive suggestions! Here are our clarifications.

Q1: Using ASR for extracting video events combined with RAG is a standard technique in video analysis, the novelty lies potentially in using OCR and object detection with video frames.

A1: It is worth noting that our Video-RAG introduces a straightforward yet innovative workflow by leveraging RAG to consolidate outputs from various multimodal tools. This approach equips LVLMs with a comprehensive and information-rich context, significantly enhancing performance in video understanding tasks.

Although all models employ ASR and utilize RAG to fetch relevant textual information to improve video understanding, our Video-RAG has the following highlight:

Differences between Video-RAG and the referenced works:

• While these efforts focus on combining agents for video comprehension, Video-RAG specifically emphasizes how to further enhance the capabilities of LVLMs through information-rich auxiliary context inputs.

• Many of these methods are complex and rely heavily on GPT, utilizing tools to convert video content into text for video QA. In contrast, Video-RAG prioritizes simplicity and flexibility, centering on LVLMs for visual content processing, allowing for plug-and-play integration.

• Video-RAG maximizes existing LVLM performance with minimal resource increase compared to the referenced works that demand higher resource consumption.

Core Contributions of Video-RAG:

• Video-RAG retrieves high-density auxiliary texts, directing the LVLM’s focus to relevant visual cues, thus promoting cross-modality alignment.

• Video-RAG's thoughtful design enables effective connections between LVLMs and external tools through RAG, making it adaptable to various scales of LVLMs while enhancing performance.

• Video-RAG with open-source LVLMs demands minimal additional GPU memory and inference latency, often outperforming proprietary models, especially notable with a 72B model.

In summary, the integration of tool outputs for targeted tasks and the generation of high-quality contextual inputs represent a significant research avenue within the LLM domain. Recent methodologies addressing these themes, including Agentic-like approaches and context management strategies, have seen notable interest, with approximately 10%–12% of accepted NeurIPS 2024 submissions aligning with this critical area of inquiry.

Q2: Are there any synergistic effects resulting from the components of Video-RAG?

A2: Based on our literature review, we define a [synergistic effect] as [Gain from Component A + Gain from Component B] > [Gain from Component A alone] + [Gain from Component B alone].

An analysis of Long-LLaVA-7B-32frames with Video-RAG in Video-MME indicates that:

• Table 4 in the manuscript indicates that OCR and DET demonstrate synergistic effects under RAG conditions in Video-RAG, with a gain of +3.7 exceeding the expected gain of +(0.9+2.3=3.2), resulting in a synergy of Δ=+0.5. This suggested that RAG forces OCR and DET to spatially align (e.g., only reading text on detected objects), eliminating noise from irrelevant regions.

• Our additional experiments further demonstrate synergistic effects between ASR and DET without RAG conditions in Video-RAG, with a gain of +6.9 exceeding the expected gain of +(0.9+5.4=6.3), resulting in a synergy of Δ=+0.6. The results reveal that ASR provides temporal action context (e.g., "opening the box") while DET locates the corresponding object, creating grounded semantics.

We carefully considered your insightful comments and designed rigorous experiments to more effectively demonstrate the synergistic interactions between components, thereby highlighting their interdependence as following:

Q3: The authors show mainly positive cases in its main part and supplementary (Fig. 4). An error analysis would increase the contribution of this paper to the community.

A3: Thank you for your valuable feedback! In response to your suggestions, we performed an error analysis on approximately 100 samples from Video-MME. This analysis has led us to identify two primary challenges that contribute to the errors and bottlenecks experienced in Video-RAG:

• Bottleneck 1: Key Frame Selection Errors in Object Detection. The selection process for key frames sometimes fails, resulting in either the omission of relevant objects or incorrect identifications. This leads to the loss of essential object information or the introduction of erroneous data into subsequent processing stages.

• Bottleneck 2: Challenges in Complex Spatiotemporal Reasoning. The current approach exhibits limitations when addressing questions that necessitate complex spatiotemporal reasoning, particularly those requiring iterative referencing and the synthesis of information from diverse segments of the video.

To address these limitations in future work, we propose the following directions:

• For Bottleneck 1: The existing key frame selection process within the DET component relies on a CLIP-based similarity retrieval approach. While this method provides speed and versatility, it exhibits limited sensitivity to high-level semantic textual information. To enhance this process, we propose substituting CLIP with a smaller LVLM, such as InternVL3-1B, which would evaluate frame relevance through a binary choice format (e.g., "Is this frame relevant to the query? A. Yes / B. No"). The model's confidence score (e.g., the logit associated with "A") could then serve as the criterion for key frame selection. Preliminary experiments suggest that this adjustment leads to performance improvements, which is worth future exploration:

• For Bottleneck 2: The existing single-round question-answering paradigm achieves a particular balance between performance and inference latency. However, our experiments reveal that multi-turn QA significantly outperforms single-round approaches in long-video comprehension. Future research will aim to enhance the complex reasoning capabilities of LVLMs by implementing designs that incorporate Chain-of-Thought strategies, more sophisticated processing pipelines, and expanded utilization of analytical tools.

[1] Trivedi H, et al. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. ACL 2023.

A1.

innovative workflow by leveraging RAG to consolidate outputs from various multimodal tools. [a] and [b] are using video frames and ASR results as RAG sources. Using them together is incremental and this reviewer believes the proposed method is not enough significant. Adding OCR result is interesting, but combining all of them is still not enough innovative.

A2.

[synergistic effect] as [Gain from Component A + Gain from Component B] > [Gain from Component A alone] + [Gain from Component B alone].

Yes, and the results show it has synergetic effect. However, its mechanism still unclear. As general novel techniques cannot be reliable unless its mechanism is supported by careful analysis, just reporting these result is unsatisfactory for NeurIPS standard.

A3.

Thank you for the analysis. These analysis is valuable. This reviewer thinks thoroughly analyzing effective multimodal tools for this multimodal RAG is an interesting direction. Knowing what current task-specific models are effective and what is missing will clarify important missing piece for video understanding.

Dear Reviewer YJ27, 

Thanks again for your great efforts and constructive advice in reviewing this paper! As the discussion period progresses, we expect your feedback and thoughts on our reply. We look forward to hearing from you, and we can further address unclear explanations and remaining concerns if any. 

Best regards, 

Authors

Thank you for your continued engagement.

Q1: Basis for the novelty of Video-RAG.

A1: We believe the current evaluation of novelty may not fully capture the core contribution of our work. The comparison relies on [a], an blog post without quantifiable results, and [b], a speech-focused ICASSP paper that does not address video task, neither of which adequately represent our advancements within the video-LVLM domain for top conference NeurIPS standards.

Specifically, [a] introduces a tool that incorporates RAG and ASR into video understanding in a rather preliminary manner. However, it does not provide benchmark-level analyses or results. As such, it remains unclear whether their approach using RAG in video-related tasks is truly effective. On the other hand, [b] employs RAG in speech tasks, without addressing video understanding. Neither establishes peer-reviewed baselines, video-LVLM task setups, or reproducible results under video benchmarks.

In contrast, our proposed Video-RAG is, to our knowledge, the first work to successfully integrates RAG into the video-LVLM domain. We showcase its efficacy through extensive experimentation on multiple well-established video benchmarks (e.g., Video-MME, LongVideoBench, and MLVU), achieving significant performance improvements.

Consistent with NeurIPS practice, we kindly suggest judging novelty against peer-reviewed work in the same domain and task setting, using comparable evaluation protocols and reproducible quantitative evidence.

[a] MultiModal RAG for Advanced Video Processing with LlamaIndex & LanceDB, 2024-02-17

[b] Do Jung et al., "Speech Retrieval-Augmented Generation without Automatic Speech Recognition," arxiv, 2024.12

Q2: Reporting the results without detailed mechanistic analysis is unsatisfactory for NeurIPS standard.

A2: While we understand the value of mechanistic insights, we respectfully note that NeurIPS has long embraced contributions beyond theoretical analysis, especially in the context of application-driven research and systems-level innovations where robust experimental validation is key.

According to the official NeurIPS guidelines, contributions can take multiple forms, not limited to theoretical analysis but including systems, datasets/benchmarks, and application-driven methods. For instance, several papers accepted by NeurIPS in recent years, such as:

[1] Patrick Lewis, et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS 2020.

[2] Zaijing Li, et al., Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks. NeurIPS 2024.

[3] Junyang Wang, et al., Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration. NeurIPS 2024 Poster.

demonstrate that the core contribution can lie in the novelty of the method's design and its robust experimental validation, rather than a deep mechanistic theory explaining why it works. In this vein, Video-RAG contributes a novel system-level integration of RAG into video-LVLMs, substantiated by robust benchmarks and ablations.

We therefore believe Video-RAG aligns with NeurIPS recognition of novel system designs and practical impact, empirically grounded systems work.

Q3: On Error Analysis.

A3: We appreciate the reviewer’s recognition of our error analysis. This component was essential in identifying limitations of current LVLMs in handling fine-grained video semantics (e.g., temporal causality, OCR fusion). We hope this will motivate further work toward deeper understanding of complex video-language reasoning.

Dear reviewer YJ27:

We appreciate this opportunity to provide a clarification on the contribution of our work, as we believe this is crucial for a fair assessment.

We understand and respect your distinction between novelty and significance. However, we would like to clarify that our primary contribution is not merely the proposal of an idea, but its rigorous systematic implementation and empirical demonstration, which transforms a high-level concept into a robust, benchmarked, and reproducible methodology.

Specifically, we contend that "knowing the direction" is fundamentally different from "making it work." The blog post you reference provides a conceptual sketch but critically lacks the quantitative results and ablation studies on established benchmarks required to validate the approach for the research community.

Just as the research community spent the period from 2017 to 2020 turning the plausible idea of "using Transformers for vision" (as seen in early works like Image Transformer and Non-local networks) into scalable, state-of-the-art models ViT, our Video-RAG's contribution follows a similar trajectory. We contend that transforming a nascent idea into such an empirically-proven system represents a significant contribution, fully aligned with NeurIPS's standards.

To our knowledge, our work moves beyond conceptual pointers to deliver the first rigorously validated implementation of integrating multi-modal tools via RAG into video-LVLMs. By achieving state-of-the-art results on multiple challenging benchmarks (e.g., Video-MME, LongVideoBench), we establish a solid baseline and provide a practical, reproducible framework.

Our visualizations in the main text provide direct qualitative evidence for Video-RAG's quantitative gains. They reveal the underlying mechanism: retrieved text sharpens the LVLM's cross-modal attention on crucial visual details and aligns vision-text semantics, which directly accounts for the performance improvements.

In summary, we respectfully maintain that Video-RAG represents a significant and practical contribution by being the first work to systematically implement and validate this multi-tool RAG approach for video-LVLMs on standard benchmarks. We hope this clarification adequately addresses your remaining concerns and underscores the value of our work to the NeurIPS community.

Thank you again for your time and meticulous review.

Best regards,

Authors

We sincerely thank you for the constructive and valuable comments. The concerns are addressed as follows.

Q1: Video-RAG's overall performance remains fundamentally constrained by the initial uniform sampling strategy. How robust is Video-RAG to frame sampling errors especially in ultra-long videos?

A1: Thanks for your constructive suggestions. We employed uniform sampling for two primary reasons:

• To ensure that our frame sampling strategy aligns with the majority of LVLMs, thereby demonstrating the plug-and-play capability of our method without reliance on key frames selection.
• To facilitate a fair comparison. Empirical evidence indicates that uniform sampling provides the most computationally balanced and generalized solution for video understanding, as corroborated by state-of-the-art open-source models such as Qwen2.5-VL and LLaVA-Video.

In response to your question about how robust Video-RAG is to frame sampling errors, we conducted additional experiments incorporating the ASK[1] (CVPR 2025) keyframe selection method alongside Video-RAG. The results from Video-MME reveal Video-RAG demonstrates robustness against the loss of keyframe information, particularly in the context of long videos. This finding underscores two important implications:

• Negligible orthogonal effects between keyframe selection and Video-RAG.
• Video-RAG highlights the system's ability to compensate for the absence of critical visual cues through auxiliary text support.

Q2: Could the authors elaborate on how spatial relations are computed in the scene graph from raw object coordinates?

A2: The spatial relationships are determined by calculating the positions of bounding boxes generated through object detection. The computational workflow is as follows:

Input: Accepts a list of objects, each characterized by an ID, class label, and bounding box represented as [xmin, ymin, width, height].

Processing:

1. Constructs a directed graph using NetworkX, where nodes represent objects and edges represent spatial relationships.
2. Performs pure geometric computations (without external dependencies) to determine pairwise relationships between objects:

• Overlap (determined by bounding box intersection)
• Relative positioning (including left/right and above/below relationships, where one bounding box is completely situated in the designated direction relative to another without overlap)
• Prevents duplicate edges by enforcing a node1 < node2 ordering

Output: Produces natural language descriptions based on spatial relationships, formatted as follows: "Object 3 (dog) is to the left of Object 7 (cat)."

Q3: Why is “Obj Relation” missing in Figure 2 despite an apparent object interaction?

A3: This is because the figure illustrates a reasoning process where the LVLM determined that addressing this particular question required only positional and number information in the query decouple phase, making object relations unnecessary (hence NULL).

We would like to further clarify:

• Different models may utilize varied reasoning paradigms. For example, Long-LLaVA-7B would consider object relations to answering the same question.

• Object relations are fundamentally derived from the positions of objects for better understanding by LVLM. More advanced models generally depend less on explicit relation annotations, favoring instead a direct analysis that utilizes raw bounding box data.

Q4: Questions about temporal reasoning of Video-RAG.

Q4.a: Although the method enhances static visual-textual alignment, it does not model temporal dynamics or sequence-level reasoning, which are crucial for many long video tasks.

A4.a: In long video understanding, modeling temporal dynamics or sequence-level information is indeed crucial. However, we deliberately avoided explicit modeling of these aspects to ensure improve performance in video understanding across diverse durations while keeping modeling complexity to a minimum. The temporal modeling techniques you referred to are indeed complementary to our work, and we sincerely appreciate your suggestions for pursuing this avenue in future research.

Q4.b: Has the method been evaluated or extended to handle temporally structured queries?

A4.b: An example can be found in Video-MME, which includes temporal reasoning tasks such as "What does Y do after X occurs?" as well as tasks involving the ordering and reasoning of temporal events. In these contexts, Video-RAG exhibits significant enhancements in both temporal perception and reasoning tasks.

Q4.c: How would Video-RAG handle such temporal dependencies given its current frame-wise processing?

A4.c: The enhancement of temporal perception and reasoning task of Video-RAG arises from the use of temporally structured text encoding, which involves the chronological integration of auxiliary texts (such as frame-wise detection transitioning to tracking, incremental ASR facilitating semantic coherence, and sequential OCR reflecting text dynamics). This approach effectively captures video dynamics without requiring explicit temporal modeling.

[1] Tang X, et al. Adaptive keyframe sampling for long video understanding. CVPR 2025.

Dear Reviewer BTw3, 

Thanks again for your great efforts and constructive advice in reviewing this paper! As the discussion period progresses, we expect your feedback and thoughts on our reply. We look forward to hearing from you, and we can further address unclear explanations and remaining concerns if any. 

Best regards, 

Authors

Dear Reviewer,

please respond to the authors, so that the authors know their rebuttal has been read.

Best regards, AC

We sincerely thank you again for your review and are happy to see that the concerns you raised have been resolved💗.

Thank you for the time, thorough comments, and nice suggestions. We are pleased to clarify your questions step-by-step.

Q1: The errors or inaccuracies in transcription, OCR, or object detection that will inevitably propagate and potentially mislead the LVLM.

A1: Follow your suggestion, we replaced the components of Video-RAG with alternative components that demonstrated lower performance, and conducted ablation studies on the Video-MME, revealing only a negligible drop in performance, demonstrating the robustness of Video-RAG to component variability.

Specifically, we have substituted EasyOCR in the OCR component with Tesseract OCR[1] and the OCRAD API. For the DET component, we have replaced APE with Owl ViT[2] and GRiT[3]. Additionally, in the ASR component, we have replaced Whisper with the SpeechRecognition library and the Vosk API.

Q2: Are there cases where auxiliary texts introduce noise, and how does the pipeline mitigate such negative effects?

A2: To investigate this, we injected controlled noise into the text outputs of external tools and evaluated LLaVA-Video-7B-64frames equipped with Video-RAG in Video-MME. Results indicate that Video-RAG demonstrates robustness in handling noise. The underlying mechanism is that the key contextual information retrieved through the RAG process has a high information density, which helps effectively mitigate performance degradation due to noise and redundant content by filtering out such interference.

Q3: The inference latency is higher compared to end-to-end models, which slows down the inference speed for practical applications.

A3: We appreciate your observation regarding the additional latency introduced by the tools associated with Video-RAG. It’s important to note that our analysis suggests that a significant portion of the inference latency for long videos is primarily due to the LVLM itself. The contribution of these tools to the overall inference time decreases notably with the increase in the number of sampled frames. As highlighted in Supplementary Table 3, when processing videos with 384 frames, we draw the following conclusions:

• The base end-to-end LVLM (LongVA-7B [4]) incurs an approximate inference time of 20 seconds per sample on Video-MME.
• With the cost of additional ~11 seconds inference time increasing, our Video-RAG can achieve a substantial performance improvement of 9.9%.

Q4: This paper focuses on single-turn process which limits the performance for iterative reasoning or multi-turn conversations.

A4: Following your suggestion, we have conducted experiments involving Video-MME with IRCoT[5], demonstrating that employing a multi-turn scheme in Video-RAG resulted in a 1.3% enhancement in performance for long videos.

Consequently, we assert that the examination of multi-round question answering warrants further investigation, and we intend to pursue additional research in this domain in the future. Notably, we focused our experiments on long videos (exceeding 30 minutes) because multi-turn question answering is particularly effective for long videos that require complex reasoning.

Q5: This paper lacks experimental/qualitative comparisons with previous VideoAgent-like methods.

A5: Following your suggestion, we have conducted further comparative experiments involving the VideoAgent-like methods VideoAgent[6], VideoAgent[7], and LLoVi[8]. The results indicate that Video-RAG consistently outperforms other leading VideoAgent-like methodologies. Given the considerable API resources required by agent-based approaches, these experiments were executed using our Video-MME-mini benchmark, as detailed in Supplementary Section 3. We would like to clarify that VideoRAG[9] was not initially designed for video understanding tasks, and OmAgent[10] incorporates a highly complex architecture specifically tailored to leverage GPT-4o. We attempted to substitute GPT-4o in OmAgent with LLaVA-Video-72B, but this resulted in inaccurate model outputs. As a result, we were unable to perform comparative experiments on VideoRAG and OmAgent. We intend to incorporate these experimental findings into the main text in future work.

We use LLaVA-Video-72B in 64 frames as the LVLM in Video-RAG, while other Agent-like methods are all driven by GPT-4.

Q6: The novelty of this paper is somewhat limited and this paper focuses on the engineering techniques for video understanding. The proposed method is training-free but this paper lacks further research on how to train video agents with tools.

A6: We appreciate the reviewer's insightful observation. While Video-RAG incorporates some established components, we wish to elucidate its principal contributions as follows:

• Essence of Video-RAG: Video-RAG fundamentally establishes a straightforward yet innovative workflow. By leveraging RAG, it assimilates outputs from a variety of multimodal tools, thereby equipping LVLMs with a more comprehensive and information-rich context. This integration significantly ameliorates performance in video understanding tasks.

• Value proposition of Video-RAG: The primary significance of this work stems from our recognition that the potential of LVLMs remains underutilized. Video-RAG illustrates that supplying a sufficiently dense and relevant context can more effectively orient the LVLM’s attention towards crucial visual elements. This enhancement promotes improved multimodal interaction and fusion. Future research could expand on Video-RAG by integrating additional external tools, facilitating adaptations to diverse application scenarios.

• Broader research relevance: It is important to highlight that the design of such workflows that integrate tool outputs for targeted tasks or the distillation of high-quality contextual inputs represents a significant research avenue within the broader LLM domain. Recent methodologies, including Agentic-like approaches, context compression techniques, and context state management strategies, fall within this thematic area. Our analysis reveals that papers fitting this category comprise approximately 10%–12% of the accepted submissions at NeurIPS 2024, emphasizing the critical importance of this area of inquiry.

• Future directions regarding training: We acknowledge the reviewer’s pertinent observations concerning the significance of training methodologies, such as recent works (e.g., ToolRL[11]) that utilize reinforcement learning strategies to enhance the tool-calling capabilities of LLMs. Your suggestion provides a valuable trajectory for future optimization: investigating how to further fine-tune LVLMs so they can more adeptly incorporate high-quality contextual information sourced from external tools.

[1] R. Smith. An Overview of the Tesseract OCR Engine. Ninth International Conference on Document Analysis and Recognition. 2007.

[2] M Minderer, et al. Simple Open-Vocabulary Object Detection with Vision Transformers. ECCV 2022.

[3] Jialian Wu, et al. GRiT: A Generative Region-to-text Transformer for Object Understanding. ECCV 2024.

[4] Peiyuan Zhang, et al. Long Context Transfer from Language to Vision.

[5] Trivedi H, et al. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. ACL 2023.

[6] Fan, et al. VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding. ECCV 2024.

[7] Wang, et al. VideoAgent: Long-form Video Understanding with Large Language Model as Agent. ECCV 2024.

[8] Ce Zhang, et al. A Simple LLM Framework for Long-Range Video Question-Answering. EMNLP 2024.

[9] Jeong, et al. VideoRAG: Retrieval-Augmented Generation over Video Corpus. ACL 2025.

[10] Lu Zhang, et al. OmAgent: A Multi-modal Agent Framework for Complex Video Understanding with Task Divide-and-Conquer.

[11] Cheng Qian, et al. ToolRL: Reward is All Tool Learning Needs.

Dear Reviewer aLCV, 

Thanks again for your great efforts and constructive advice in reviewing this paper! As the discussion period progresses, we expect your feedback and thoughts on our reply. We look forward to hearing from you, and we can further address unclear explanations and remaining concerns if any. 

Best regards, 

Authors

Dear Reviewer,

please respond to the authors, so that the authors know their rebuttal has been read.

Best regards, AC