Tool-Augmented Spatiotemporal Reasoning for Streamlining Video Question Answering Task

We sincerely thank you for your selfless efforts and constructive feedback, which greatly helps us improve the quality of this work. We are especially grateful for the recognition of our categorization of tool types, as well as the performance improvement. Below, we provide point-by-point responses for your concerns.

W1

Discrepancy in reported Video-MME score and lack of prompt transparency

In Table 1, the score of 71.9 corresponds to GPT-4o evaluated with a sampling rate of 1 fps, with up to 384 frames sampled per video. The score of 61.8 corresponds to GPT-4o evaluated with a fixed sampling of 32 frames per video.

The accuracy of video question answering is closely tied to the number of sampled frames—higher sampling rates generally yield better performance. Therefore, fair comparisons should be conducted under comparable sampling conditions. Since our method samples an average of 30.2 frames per video, we compare against GPT-4o’s 32-frame setting (61.8) and mark 61.8 accordingly.

The sources and evaluation method of the results reported in Table 1 are as follows, all under the without subtitles setting:

All time measurements are conducted using either official APIs or open-source models, following the Video-MME recommended protocol (via the lmms-eval GitHub repository). Each measurement is repeated multiple times (at least 3) on 8 NVIDIA GeForce RTX 4090 GPUs.

W2

Categorization inconsistency of Video Summarizer between the main paper and the supplementary materials

There is a classification error in Figure 3: Video Summarizer should be categorized under General Tool rather than the Both category. The information provided in the appendix is correct.

General Tools are only used at the end of the toolchain, when neither temporal nor spatial tools can resolve the question. In contrast, Both tools can function as either temporal or spatial tools within the toolchain. The rationale for classifying Video Summarizer as a General Tool is that it is implemented using a 3B MLLM and is intended for use only as a last resort. In typical cases, we prioritize lightweight tools to improve computational efficiency. We will correct this classification in the later version.

W3

Lack of clarity in runtime reporting and missing efficiency details

1. About the missing Runtime values

There is a strong positive correlation between runtime and the number of sampled frames. The runtime value is missing for that particular data point in the table because it was taken from another paper that did not report the number of sampled frames, making it impossible to accurately estimate the runtime.

This includes two data points in Table 1—Qwen2-VL-7B and Qwen2.5-VL-7B—which are sourced from Table 8 of the InternVL3 technical report, as well as two data points in Table 4—ViperGPT and VideoChat—which are sourced from Table 2 of the DoraemonGPT paper.

We thank the reviewer for pointing out this limitation. In the later version, we plan to replace these externally reported results with those obtained from our own reproduction, thereby eliminating the missing runtime values.

2. About the vague description of "<30s"

These runtime measurements are inherently difficult to obtain accurately, as they are sensitive to various environmental factors: (1) The speed of API is affected by network conditions; (2) Even with the same GPU model, different compute clusters may exhibit variations in video processing speed due to factors such as CPU memory, system age, and overall hardware configuration.

Moreover, runtime is highly dependent on the number of frames, making fine-grained time comparisons meaningless when the number of frames differs slightly.

Therefore, we run each baseline experiment multiple times to obtain a rough performance range, e.g., <30s. Regardless of the interference factors, the timing of these methods generally falls within this range. For our own method, STAR, we conduct more extensive testing (around 10 runs) under different environments to obtain a more accurate average reported in the table, e.g. 15.8s.

Our intention is to determine approximate performance ranges—such as <30s, 30s–60s, or >10min—in scenarios where precise runtime measurement is challenging. Models falling within the same range can be considered to have comparable computational efficiency, as small differences within a range are unlikely to impact practical usage.

Our intention is also not to underrepresent baseline performance, but rather to show: (1) STAR (averaging 15.8 seconds per video with 30.2 sampled frames) operates in the same time scale as models like GPT-4o (0–30 seconds for 32 sampled frames). (2) STAR achieves comparable accuracy to GPT-4o (>10 minutes for 384 sample frames) but completes processing much faster.

In future revisions, we will conduct additional experiments across diverse environments to obtain more precise runtime measurements wherever possible.

Thank you for your thoughtful and encouraging feedback. We are especially grateful for your recognition of STAR’s portability, superior performance across multiple methods, and strong results on diverse benchmarks with clear visualizations. The followings are points-to-points respones to your concerns.

Weaknesses

W1

Model not being able to learn from previous experiments

We thank the reviewer for pointing out this limitation. We acknowledge that tool-integrated post-training is an important direction and will leave it as future work to be explored in follow-up studies.

This paper focuses on a training-free, plug-and-play paradigm, where tools are dynamically selected and invoked without additional fine-tuning. In practice, we have found that current LLM such as GPT-4o already exhibit strong capabilities in instruction-following and zero-shot tool use. As a result, training-free approaches can achieve competitive performance without the need for additional fine-tuning. We believe this paradigm offers a complementary alternative to tool-integrated post-training, with each approach having its own trade-offs.

W2

Why some results on Video-MME is higher than 80%? Is STAR's 70% accuracy a good performance? Details about Video-MME evaluation

This is primarily due to two reasons:

(1) Video-MME provides two evaluation modes: with subtitles and without subtitles. Models generally perform better in the with subtitles setting. For example, on the official Video-MME leaderboard, Gemini 1.5 Pro achieves 81.3% accuracy with subtitles, compared to 75.0% without subtitles. Our work currently focuses exclusively on pure video understanding, without leveraging subtitle information. Therefore, in Table 1, we report results under the without subtitles setting. In contrast, many entries on the PapersWithCode leaderboard either report with subtitles results or do not specify the evaluation setting, making direct comparison less reliable.

(2) The accuracy of video question answering is closely related to the number of sampled frames—higher sampling rates generally lead to better performance. Therefore, fair comparisons require evaluations under the same sampling rate. Some results reported on PapersWithCode may be obtained under higher sampling rates.

To ensure fairness, we also report our performance under increased sampling rates as follows. Due to API budget and time constraints, we randomly sampled 1,000 examples from the Video-MME test set for this evaluation. The following results demonstrate that the STAR framework continues to improve performance as the number of sampled frames increases and reach 77% performance under the without subtitles, 1fps sampling, up to 384 frames condition.

Questions

Q1

Tool number discrenpency between the main paper and appendix

The number 22 is the number of tool types, while 30 is the number of tool implementations. A tool may have multiple implementations to accommodate different computational resources or deployment environments. For example, the Frame Selector has 3 variants:

• Vanilla version: The frame selector is an LLM that takes as input the IDs and textual descriptions of all visible frames (i.e., the Visible Frame Dictionary described in the paper), such as captions generated by an image captioner. It outputs the selected frame IDs. The prompt is directly instructing the LLM to select relevant frames.

• AKeyS-inspired version: This variant shares the same input and output format as the Vanilla version but employs specially designed prompts to encourage the LLM to consider task goals and scene segmentation.

• T*-inspired version: In this case, the frame selector is an MLLM, and its input additionally includes the actual images of visible frames. These images are further arranged into an image grid to facilitate selection. The output remains the same.

Similarly, the Object Detector has both YOLO- and GroundingDINO-based versions tailored for low- and high-resource settings, respectively. The Image Captioner and Image QA Tool have different implementations based on BLIP2 (~1B parameters), LLaVA (~7B), and GPT-4o.

The number 30 mentioned in the paper refers to the total number of tool implementations. We acknowledge that this may cause confusion and will revise in later version to clarify this distinction and eliminate ambiguity.

Q2

Performance with other backbone models

We also evaluated our framework with other (M)LLMs, and the results are as follows. Due to API budget and time constraints, we randomly sampled 1,000 examples from the Video-MME test set for this evaluation.

The results show that both closed-source models (e.g., GPT-4o, Gemini 2.5 Pro) and open-source models (e.g., Qwen2.5-VL-72B, DeepSeek-R1) achieve strong performance. Compared to their tool-free counterparts, these models demonstrate a 7%–8% improvement in accuracy, highlighting the effectiveness of our work across model types.

Q3

Error correction after initial temporal misalignment

It is normal for our STAR framework to initially focus on an incorrect video segment. However, our STAR framework is equipped with backtracking ability that enables correction and prevents cascading errors. When the LLM Planner first invokes the frame selector and the Visible Frame Dictionary is still empty, the selector performs a coarse, uniform sampling over the entire video, dividing it into several segments. It then chooses to explore one segment in depth. If, after comparing the question with the retrieved information, it turns out that the current segment is not relevant, the frame selector is able to shift its focus to other segments. The exploration process terminates only when the LLM Planner has gathered sufficient information to answer the question accurately.

For example, in a case study from the Video-MME test set (question_id=301-2), the question is: “Why can Cusco and Tenochtitlan be easily found while Amazonian settlements cannot?”. The associated video is a YouTube video titled “How the ‘lost cities’ of the Amazon were finally found”. Initially, STAR selected frames from the segment between 1:20 and 1:40, which only mentioned Cusco and Tenochtitlan, but did not address why Amazonian settlements were hard to find. Detecting the lack of relevant information, STAR then shifted to another segment and ultimately identified the correct time span (6:10–6:20), which states: “Because Amazonian settlements were built with wood and earth.”

Thanks for the responses. The responses have successfully addressed my concerns. Although the paper is not particularly interesting, it is solid and well-written, so I will raise my score to 5.

We sincerely thank you for your thoughtful comments and careful consideration. We truly appreciate your decision to raise the score. Your constructive feedback has been very helpful in improving our work.

We sincerely thank you for your constructive feedback. We are especially grateful for your recognition. Below, we provide point-by-point responses for your concerns:

Weaknesses

Typos

1. Thank you for your feedback. We will revise this paragraph to ensure greater logical coherence.
2. The "(3rd place) Toolchain Shortcut" means that the Toolchain Shortcut method performs worst among the three approaches in figure 1. For a more detailed description and corresponding results, please refer to Section 4.2 and Table 5 in the main paper.
3. DoraemonGPT use tools to processes all frames of a video to obtain information, which are then stored in a SQLite database. An LLM subsequently translates natural language queries into SQL statements to retrieve answers from the database. However, since DoraemonGPT operates on all video frames—unlike the STAR framework, which temporally narrows the search scope—it is less efficient. Moreover, due to limitations in text-to-SQL accuracy, the generated SQL queries in DoraemonGPT sometimes fail to execute, compromising reliability. In contrast, STAR avoids such failure-prone intermediate query steps.
4. The original intent of line 106 is to emphasize that the Video Toolkit should be capable of handling both individual frames and video sub-clips—this is considerations along the temporal dimension. Correspondingly, along the spatial dimension, Video Toolkit should support operations on both full images and localized image patches. In the revised version, we will explicitly discuss both the temporal and spatial dimensions to reflect comprehensive analysis.

Core Concerns and Questions

1. About custom-designed tools

Our current Video Toolkit does not yet include custom-designed tools. However, when develop the Video Toolkit, we consider various classic problem classes such as counting, action reasoning, text reasoning problems, and so on. The Video Toolkit support the full solution pipeline needed to address these representative problems. In future work, we plan to develop custom-designed toolkits and solution pipelines, and integrate them with the LLM Planner’s autonomous tool-calling mechanism.

2. When and How to add frames?

The following temporal tools, once invoked and executed, each apply their own selection logic to expand the set of visible frames.

• Frame Selector is an (M)LLM-based tool that, guided by single-turn or multi-turn prompting, analyzes the current Visible Frame Dictionary and identifies additional unseen frames that are likely to be informative for answering the question. These selected frames are then incorporated into the visible set to support further reasoning.

• Temporal Grounding Tool and Action Localization Tool: When invoked by the LLM Planner, the tool takes a textual description as input and grounds it to a corresponding temporal segment in the video. Frames from this segment are then sampled at 1 fps and added to the set of visible frames.

3. Toolchain shortcut definition

More rigorously, toolchain shortcut refers to cases where the toolchain invoked by the LLM Planner is shorter than the ground-truth toolchain. The ground-truth toolchain represents the most computation-efficient sequence of tool calls required to solve the task. Cases where “a general tool alone suffices” should not be categorized as toolchain shortcuts. In the upcoming revision of this paper, we will manually annotate the ground-truth toolchains for the dataset and compare them with those generated by the LLM Planner, thereby enabling a more precise characterization of the toolchain shortcut phenomenon and examining whether there are cases that unnecessarily complicate the process.

4. Spatialtemporal interleaving detail

The older is flexible as we mention in line 172. In other words, the first tool in the chain can be either a temporal or a spatial tool, but the subsequent tools are required to alternate between temporal and spatial tools.

5. Comparison with "query decomposition" works

There are 2 main differences:

• Prior works [1, 2] about query decomposition is better suited for complex multi-step reasoning tasks where each step requires a different tool. For example, Chain-of-Action [1] focuses on retrieving real-time information from heterogeneous sources. In contrast, our approach focuses on decomposing the process of locating key regions into a step-by-step procedure, which is better suited for real-world long-form video QA tasks depending on accurate localization of key frames and regions.
• STAR determines which tool to invoke and what action to take step by step during the exploration of the video environment, rather than predefining the entire reasoning trajectory based solely on the query. This design makes the framework more flexible and dynamic.

In future work, we will further investigate this comparison and aim to validate it experimentally.

References:

[1] Chain-of-action: Faithful and multimodal question answering through large language models.

[2] ToolACE-DEV: Self-Improving Tool Learning via Decomposition and EVolution.

6. Performance with other MLLMs

Due to API budget and time constraints, we randomly sampled 1,000 examples from the Video-MME test set for this evaluation. The results show that, compared to their tool-free counterparts, these models demonstrate a 7%–8% improvement in accuracy, highlighting the effectiveness across model types.

Questions

Experimental Comparisons

1. Why STAR faster than raw GPT-4o?

The main reason is that in our work, the video is not directly fed into GPT-4o. Instead, GPT-4o—serving as the LLM Planner—analyzes the question along with intermediate textual results stored in the Visible Frame Dictionary to decide which tool to invoke next. The LLM Planner does not process video directly. Therefore, In our work avoids processing a large number of image tokens.

2. What cause the reduction in processed frames?

We believe the key lies in selective reasoning.

• Regarding metadata: Our framework does not introduce any additional metadata—the input remains a single video and a question.
• Regarding captions: If the LLM Planner invokes an tool (e.g., an image captioner) on a particular frame, that frame is counted as a sampled/visible/processed frame. Therefore, captions are not considered auxiliary inputs but rather intermediate tool results.

As such, the system does not rely on auxiliary inputs. However, it can be interpreted as using tools to extract different kinds of information from selected frames. This spatiotemporal reasoning approach enables the QA system to selectively fuse and utilize diverse information sources, allowing it to reduce the number of processed frames while maintaining high accuracy.

Tool Quality & Harmony Assurance

How to improve tool quality and harmony? How to measure?

We use the following methods to improve tool quality and ensure robust tool execution:

• Structured tool invocation: We leverage LLM's (e.g., GPT-4o) capability for structured output by requiring the LLM Planner to generate tool parameters in JSON format. These parameters are then parsed to ensure syntactic correctness.
• Time range check: We ensures that temporal manipulations such as frame sampling or trimming are accurately bounded within the valid time range of the video, preventing out-of-bounds errors.
• Resilient workflow design: The workflow is designed to be fault-tolerant. A single failed or ineffective tool invocation does not halt the process. For instance, if a spatial tool like an object detector fails to detect the target object, the planner may subsequently invoke a temporal tool to adjust frame sampling and continue exploration.
• Fallback General Tools: We include general-purpose tools, such as the Video QA Tool and Video Summarizer based on video-language models, as a fallback mechanism. These tools serve as a safety net when specialized tools fail to yield sufficient information.

In terms of metrics, we adopt the following approaches to evaluate tool execution and usage quality:

• Tool Success Rate: We measure the proportion of successful tool executions—defined as invocations where input parameters are correct and the tool returns valid outputs. On the Video-MME test set, the success rate is 97.2%, and on the LongVideoBench test set, it is 96.5%, indicating that tools are generally executed reliably.
• Tool Usage Distribution: To assess the balance and harmony of tool usage, we analyze the distribution of tool calls across different tools and tool categories. This helps us evaluate whether certain tools are disproportionately relied upon, and whether the LLM Planner effectively utilizes the full range of available tools. The results of this metric can be found in Figure 3 of the main paper, as well as in our rebuttal to Reviewer QhNP, Weakness 3.

We sincerely thank you for your constructive feedback, which greatly helps us improve the quality of this work. Below, we provide point-by-point responses for your concerns:

Weakness 1

Some missing implementation details

• For STAR framework，we use gpt-4o-2024-08-06 as the controller LLM. For STAR-mini, we use gpt-3.5-turbo-0125 as the controller LLM to ensure a fair comparison with baselines.

• For videos longer than 16 seconds, we extract 16 initial frames uniformly; for videos with a duration of 16 seconds or less, initial frames are extracted at a rate of 1 fps.

• For proprietary models such as GPT-4o, we use their official APIs. For open-source models such as Qwen, we benchmark models up to 8B on 8 NVIDIA RTX 4090 GPUs with vLLM, and benchmark 72B models on 8 NVIDIA H100 GPUs with vLLM. For both prior work such as T* and our proposed STAR framework, experiments are conducted on 8 NVIDIA RTX 4090 GPUs using the same official APIs, ensuring a relatively fair comparison.

  Why are some runtimes very vague? This is primarily due to three reasons:

  1. The speed of API is affected by network conditions;
  2. Even with the same GPU, different compute clusters may exhibit variations in speed due to overall hardware configuration;
  3. Processing time is highly dependent on the number of frames.

  Therefore, we report a performance range obtained from multiple runs of the baseline methods, rather than a specific runtime. For our own method, we conduct more extensive testing (about 10 runs) under different environments to obtain a more accurate runtime reported in the table. Our intention is to show:

  1. STAR (averaging 15.8 seconds per video with 30.2 sampled frames) operates in the same time scale as models like GPT-4o (0–30 seconds for 32 sampled frames).
  2. STAR achieves comparable accuracy to GPT-4o (>10 minutes for 384 sample frames) but completes processing much faster.

• Specifically, we implement 3 variants of the frame selector for different computational resources.

  1. Vanilla version: The frame selector is an LLM that takes as input the IDs and textual descriptions of all visible frames, such as captions generated by an image captioner. It outputs the selected frame IDs. The prompt is directly instructing the LLM to select relevant frames.
  2. AKeyS-inspired version: This variant shares the same input and output format as the Vanilla version but employs specially designed prompts to encourage the LLM to consider task goals and scene transition.
  3. T*-inspired version: In this case, the frame selector is an MLLM, and its input additionally includes the actual images of visible frames. These images are further arranged into an image grid to facilitate selection. The output remains the same.

  The selected frame IDs produced by the frame selector are subsequently added to the Visible Frame Dictionary. It is important to note that the frame selector and the video trimmer are two distinct and independent temporal tools, both scheduled by the LLM planner. We will revise and highlight this point in the later version.

Weakness 2

The performance with more frames

We conducted experiments to evaluate the impact of increased frame sampling rates. Due to API budget and time constraints, we randomly sampled 1,000 examples from the Video-MME test set for this evaluation. The accuracy results are as follows. They demonstrates that the STAR framework continues to improve performance as the number of sampled frames increases. For example, under a denser sampling setting (1 fps, up to 384 frames), the framework achieves a 5.2% accuracy gain.

Weakness 3

How STAR solves the "unbalanced number and diversity of tools" issue? How to balance tools within category?

The STAR framework addresses the issue of unbalanced tool quantity and diversity primarily by removing the tools that the LLM Planner tends to over-rely on under the no-constraints setting (e.g., the Video QA Tool and Image QA Tool). With these tools no longer “monopolizing” the calls, the distribution of tool usage becomes more balanced both across and within tool categories.

Below are some supporting data:

Below is the distribution of tool usage frequencies on the Video-MME test set for both the No Constraints setting and the STAR framework. The No Constraints setting setting is described in line 253-255 in the paper.

From this raw data, we can compute the variance of tool usage counts within each tool category：

It can be observed that after adopting the STAR framework, the variance of tool usage counts within each category generally decreases, indicating a more balanced utilization of tools within each class.

In future work, we plan to further explore how to achieve balanced tool usage within the same category, potentially by manually annotating the ground-truth toolchains for each problem. This would allow us to analyze the ideal distribution of tool usage within categories and design appropriate constraints or incentives to guide the LLM planner toward balanced usage.

Weakness 4

Missing failure cases

We thank the reviewer for pointing out this limitation. In future versions, we will include representative failure cases along with an analysis of their underlying causes.

The main types of failure cases are as follows:

1. Missing or ambiguous visual information. In some cases, the video alone does not provide sufficient information and must be supplemented with subtitles or audio cues, e.g., Video-MME test set 302-1. We plan to address such issues by incorporating subtitle inputs and developing tools for audio understanding.
2. Incomplete understanding of the video's main theme. Our STAR framework sometimes fails to fully capture the main theme of a video due to sparse frame sampling. We aim to improve performance on such global reasoning tasks by encouraging denser sampling in future versions.
3. Difficulty in inferring underlying motivations behind human actions. STAR struggles with questions that require understanding the deeper intent behind actions. For example, Video-MME test set 312-2: “Why does Michael Bierut mention religious symbols?” This suggests that capturing fine-grained causal reasoning remains a fundamental difficulty in video understanding.