W1: Higher computational costs and computational overhead comparison with other agent-based approaches.

A1: Thanks for the comment. Though this work focuses on advancing the boundaries of long-form video understanding capabilities by test-time scaling, we acknowledge the importance of computational efficiency. We compare our DVD agent with representive VideoAgent [1] in terms of efficiency. To fair comparison, we reproduced VideoAgent[1] in our codebase with all same models, denoted as VideoAgent-fashion. Then we conducted experiments on LVBench without transcripts. The results demonstrate that our DVD achieves significantly less search steps while superior accuracy, compared to VideoAgent-fashion.

Under 15 max steps, DVD achieves 74.2% accuracy with only 7.3 average steps, while VideoAgent-fashion requires 11.3 average steps to reach 64.1% accuracy, demonstrating both higher accuracy and better efficiency.

W2: Lack of cases studies to illustrate the effectiveness and versatility of the integrated tools.

A2: Thank you for your suggestion. We have included the case study in Appendix B.2, wherein we present representative examples for five distinct behavioral patterns: (1) Global Browse Only, (2) Simple Action, (3) Iterative Search, (4) Frame Inspect Trap, and (5) Clip Search Trap. The Iterative Search case demonstrates the agent’s use of various tools and query formulations in pursuit of the correct answer. In contrast, the Frame Inspect Trap and Clip Search Trap illustrate scenarios in which the agent encounters difficulties. Specifically, when initial clip searches yield limited relevant information, prompting the agent to repeatedly invoke the same tools and ultimately leading to incorrect answers.

We are committed to incorporate more qualitative visualization to facilitate a more nuanced analysis of the agent's behavior in our revised version.

W3: The video processing itself relies on existing methods and may not push the boundaries of video processing techniques.

A3: We appreciate your feedback and would like to clarify our core contribution. While our framework indeed employs standard video processing techniques (e.g., clip segmentation, caption generation), our primary innovation lies in the agentic search workflow orchestration that fundamentally changes how the search strategy is utilized for video understanding.

Unlike previous video agent that rely on fixed, pre-defined workflows, our framework empowers the agent to dynamically construct query-adaptive workflows. The agent intelligently selects and composes different search tools based on each query's unique requirements, resulting in more efficient exploration of video content and significantly improved understanding accuracy.

To illustrate this conceptual difference:

This adaptive approach represents a paradigm shift from traditional hand-crafted, fixed video processing workflow. Rather than pushing the boundaries through novel low-level video processing techniques, we demonstrate that agentic orchestration of existing tools can achieve substantial performance gains, a 10.1% absolute improvement over VideoAgent-fashion as shown in Table of A1.

Q1: Generalization on different types of video content.

A1_: Thanks for the suggestion. We are commited to add more analysis on video content into the revised paper. Here we provide the performance on different video content categories of LVBench. It shows that our DVD agent achieves strong performance across all categories, including ego-centric self-media, cartoon with massive scene changes, documentary with long shot and carefully edited TV show. This demonstrates the generalization capability of our framework on diverse video content types.

Q2: Detailed use cases of tool use.

A2_: Thanks for the question. Please refer to our response of W2. We provide the specific case study of search workflow orchestrated by our agent in Appendix B.2. For quantative analysis, as shown in Figure 3 of our main body, with OpenAI o3 as the orchestrator model, for 2.8% of queries the agent only uses the Global Browse Tool then draw the final answer. For most of the queries, the agent uses the Global Browse Tool to find relevant event and subjects, then uses the Clip Search Tool to localize specific time ranges, and finally uses the Frame Inspect Tool to inspect the frames to find more fine-grained information to confirm the answer. We are committed to add more qualitative visualization to facilitate a more nuanced analysis of the agent's behavior in our revised version.

Q3: Computational overhead comparison with other agent-based approaches.

A3_: Thanks for the question. Please refer to our response of W1.

Reference:

[1] VideoAgent: Long-form Video Understanding with Large Language Model as Agent, ECCV 2024

Thanks for your comment. Since MR.Video[1]'s code has not been open-sourced, we reproduce MR.Video and evaluate it on LVBench without transcript under the same model. The results are shown in the following table. The peak memory is calculated on Qwen3-32B with peak token number.

MR.Video achieves the highest computational cost in both running time and peak GPU memory usage. This is primarily due to its Segment Intention Analysis component (described in Section 3.3.1 of its paper), which requires querying with extensive scene captions for all video segments. Our DVD demonstrates superior performance compared to MR.Video while maintaining comparable peak GPU memory usage to VideoAgent.

[1] MR.Video: “MapReduce” is the Principle for Long Video Understanding, 2025.04 Arxiv

Please let us know if you have any further concerns. We would be happy to provide any needed clarification.

Thank you for your reply. We want to point out that your understanding is seriously mismatched with original MR.Video paper. MR.Video indeed requires significantly more model calls than two, as stated in its original paper.

According to the original paper, MR.Video consists of four components, each potentially involving multiple model calls. For instance, in the "Map: Segment Intention Analysis" component (Section 3.3.1), for every single segment it triggers once model call to analyze the segment. As stated in the MR.Video paper: "Within each segment, we instruct MR. Video to focus on ... and generate a paragraph of analysis". It means if the video has 30 segments (as exemplified in its own paper), this component alone triggers 30 model calls. This design is the primary source of the higher runtime shown in our comparison table. Our DVD demonstrates superior efficiency compared to MR.Video, achieving both lower runtime, reduced peak GPU memory usage and much higher accuracy.

Q1: Limited Novelty in Video Retrieval & Search

A1: Thanks for the comment. We want to clarify that our key innovation distinguishing our work from previous approaches like VideoAgent [1] lies in our agentic search workflow orchestration through selective combination of different search tools.

Unlike previous hand-crafted, fixed workflows, our framework empowers the reasoning model to dynamically compose workflows tailored to each specific user query, resulting in flexible and query-adaptive search strategies.

To empirically demonstrate the effectiveness of our agentic search workflow orchestration, we conducted an ablation study replacing our adaptive workflow with a hand-crafted fixed workflow.

As shown in the table below, the hand-crafted approach results in a significant 7.5% absolute accuracy drop while requiring more steps. Even when scaling up to 11.1 average steps (52% more than our DVD), the hand-crafted workflow achieves only 70.2% accuracy, still 4.0% below our adaptive approach. This comparison highlights the superior search efficiency of our dynamic workflow design.

Q2: Inefficient Model Inference

A2: We appreciate the efficiency concern regarding our framework. We discuss the efficiency of our DVD from two perspectives: (1) the trade-off between pre-building and online-generating captions, and (2) overall efficiency comparison with VideoAgent.

(1) Trade-off between pre-building and online-generating captions:

The comparison with VideoAgent reflects an efficiency trade-off between two database construction strategies: pre-building captions versus online-generating captions. This trade-off is not specific to any single video understanding agent but rather represents a setting that can be applied to various frameworks, including our DVD. These strategies serve different scenarios with online/offline efficiency profiles:

Offline preprocessing efficiency: VideoAgent precomputes CLIP embeddings for all frames, while DVD generates captions and embeddings using video clips as basic units. During this phase, pre-building captions incurs higher computational cost compared to online-generating while similar time cost benifited from parallel processing across clips.

Online inference-time efficiency: Without pre-generated captions, they must be generated during inference. Here, online-generating becomes substantially more expensive than pre-building, with latency accumulating across iterative reasoning steps.

The choice between these approaches depends on usage patterns:

• For videos queried once, online-generating may be more efficient.
• For videos queried by multiple times, pre-building significantly reduces cumulative inference cost. For example, a video uploaded to YouTube may be queried by thousands of users with different questions, where pre-building can significantly reduce inference cost and improve overall efficiency.

While DVD employs pre-building in this submission, it can also operate in online-generating mode. To evaluate this alternative, we conducted an experiment replacing our original pre-building database with an online-generating approach. The results show that this replacement leads to a slight performance drop. This further clarifies that the two strategies serve different purposes: online-generating is well-suited for scenarios where videos are queried once with relatively simple queries, while pre-building is better suited for scenarios where videos receive multiple queries and achieving optimal performance is prioritized.

(2) Overall efficiency comparison with VideoAgent:

We reproduce VideoAgent using the same LLM and VLM in our codebase (denoted as VideoAgent-fashion). Our DVD framework achieves superior performance with fewer iterations, demonstrating our agent design's efficiency:

Under 15 max steps, DVD achieves 74.2% accuracy with only 7.3 average steps, while VideoAgent-fashion requires 11.3 average steps to reach 64.1% accuracy—demonstrating both higher accuracy and better efficiency.

Q3: Narrow Model Evaluation on models

A3: Thanks for the suggestion. Due to limited access to proprietary models, we evaluated our framework across three open-source models with varying architectures and sizes to demonstrate generalization capability. We tested DeepSeek-V3 (671B MoE without reasoning), DeepSeek-R1 (671B MoE with reasoning), and Qwen3-32B-thinking (32B dense model with reasoning) as orchestrator models on LVBench without transcripts.

All three models significantly outperform the OpenAI o3 Vision-Language Model baseline, validating our framework's effectiveness across different model architectures and scales. Notably, our DVD framework with Qwen3-32B-thinking is already on par with original OpenAI o3, which is the top-tier commercial reasoning vision-language model. When we equip our DVD with OpenAI o3 as the orchestrator model, we achieve a 17.0% absolute accuracy improvement over original OpenAI o3. The consistent improvement demonstrate that DVD's gains stem from the framework design itself rather than simply using stronger models. We will include these results with further analysis in our revised paper.

Q4: Heavy Reliance on Zero-Shot LLM Capabilities and Generalization on Small Model

A4: Thanks for the suggestion. We agree that fine-tuning LLMs for video understanding agent is a promising and effective approach to improve performance. However, it is non-trivial to collect training data of long video understanding or build up reinforcement learning environment to fine-tune a smaller LLM. We plan to explore this direction in future work.

Here we focus on evaluating the generalization ablity of our DVD on smaller LLMs. We conduct experiment with latest Qwen3-32B as the orchester model in our DVD. It shows even with 4.7% parameter compared to the DeepSeek-R1, DVD can still show impressive performance surpassing the Vision-Language Model OpenAI o3. It demonsrates our framework works well even with smaller LLM's zero-shot capability.

Q5: Lack of Qualitative Analysis

A5: Thanks for the suggestion. We will add more qualitative visualization for better agent behavior analysis in our revised version to supplement the textual case study in our Append B.2. As quantitative evidence, we want to highlight Figure 3 of our main body, which shows common error patterns including:

• Repeated tool calls ("Clip-Search Trap" and "Frame-Inspect Trap"): The agent loops on the same tool, wasting budget and lowering accuracy by 8.0 percentage.

• Behavioral collapse: when replacing OpenAI o3 with GPT-4o, the agent adapts the same simple action strategy for most of queries, resulting accuray lower than average. the policy converges to a short “search-query-answer” routine covering 54 % of queries but with sub-average accuracy.

Reference:

[1] VideoAgent: Long-form Video Understanding with Large Language Model as Agent, ECCV 2024

Q1: The trade-off between the boost in accuracy vs. additional cost.

A1: Thanks for the valuable comment. Here we provide our evaluation results under different max step numbers with OpenAI o3 on LVBench without transcripts.

We are committed to add these results with further analysis in our revision.

Q2: Detailed setting of max step number.

A2: In our paper, the max step (iteration) number is set as 15 for the best performance via test-time scaling. As shown in the table of response 1, the smallest iteration with better performance than our baseline Vision-Language Model (VLM) is 4. The accuracy scales effectively, reaching 74.2% at 15 max steps, a 17.1% absolute improvement over the strong baseline.

Q3: Why longer reasoning chain makes the results within the same model even worse?

A3: Thanks for the valuable question. For the same query, the growth of iteration steps generally improves accuracy, as shown in the table of response 1. However, across different queries, the agent may follow different reasoning modes depending on the video and query content, where we observe that queries involving longer reasoning chains often correspond to lower accuracy.

Supported by our case study (Appendix B.2) and quantitative data (Figure 3), we identified two typical failure modes that result in longer iterations with a decrease in accuracy, including:

• Frame Inspect Trap: The agent repeatedly utilizes the Frame Inspect Tool to pose various queries about the same temporal segment of the video. This behavior suggests a cognitive fixation, wherein the agent presumes that crucial information must exist within that specific time range. This fixation prevents it from exploring alternative video content, resulting in a 140.7% increase in iteration steps and an 11.1 percentage point absolute drop in accuracy.

• Clip Search Trap: The agents repeatedly employ the Clip Search Tool to find information with similar queries. This pattern typically occurs when there is little information in the video database corresponding to the user question. This behavior leads to a 152.6% increase in iteration steps and a 16.1 percentage point absolute decline in accuracy.

Thanks for your comment. We compare our DVD agent with previous video agent including MR.Video[1] and VideoAgent[2]. Since MR.Video[1]'s code has not been open-sourced, we reproduce MR.Video and evaluate it on LVBench without transcript under the same model. The results are shown in the following table. The peak GPU memory is calculated on Qwen3-32B with peak token number per query.

MR.Video achieves the highest computational cost in both running time and peak GPU memory usage. This is primarily due to its Segment Intention Analysis component (described in Section 3.3.1 of its paper), which requires querying with extensive scene captions for all video segments. Our DVD demonstrates superior performance compared to MR.Video while maintaining comparable peak GPU memory usage to VideoAgent.

[1] MR.Video: “MapReduce” is the Principle for Long Video Understanding, 2025.04 Arxiv [2] VideoAgent: Long-form Video Understanding with Large Language Model as Agent, ECCV 2024

Please let us know if you have any further concerns. We would be happy to provide any needed clarification.