MSR-ViR: Modularized Self-reflected Video Reasoner for Video Question Answering

Weakness2

In our work, the interpretability we emphasize refers to clearly demonstrating how, step by step, the relevant temporal segments are located from the video based on a given question, and how the spatial regions relevant to the question are then inferred from the frames extracted from these temporal segments. The interpretability mentioned by the reviewer refers to the process of step-by-step reasoning from visual information and the question to the final answer. These represent two different definitions of interpretability and can also be seen as two essential steps in completing the VideoQA task: first, locating the temporal and spatial segments, and then reasoning the answer based on the localized content. Our work focuses on the former. In future work, we will incorporating the internal reasoning process of the LLM.

Weakness3

Based on the reviewer’s suggestion, we introduced SeViLa’s test results on NExT-QA, STAR, and NExT-GQA, as well as GCG’s test results on NExT-QA for comparison with the Llava-Next version of our MSR-ViR framework. The results, presented in the table below, demonstrate a significant improvement in VideoQA accuracy, surpassing existing grounding-based methods such as SeViLa and GCG.

We sincerely thank the reviewer for taking time to review our paper and providing thoughtful feedback and insightful suggestions. We address the weaknesses as follows:

Weakness1

The MoST-Grounding module introduces a temporal grounding tool, UniVTG, and a spatial grounding tool, YOLO-World. These two models are relatively small compared to our base LLM, with UniVTG having 41.3M parameters and YOLO-World 48M parameters. Additionally, both models have fast inference speeds, so the MoST-Grounding module does not introduce significant computational costs. In fact, the primary source of computational overhead in our framework is the question parser based on Qwen2-7B, which is quite common in VideoQA tasks based on LLMs. Many reasoning-based VideoQA models(like MoReVQA[1], LLoVi[2] and VideoTree[3]) rely on LLMs, which inevitably introduce additional computational overhead.

When selecting the small modules for the MoST-Grounding module, we consider not only the models’ ability to perform grounding tasks but also their operational efficiency, as highlighted by the reviewer. If the small modules were slow, they would substantially affect the framework's overall efficiency. The chosen UniVTG and YOLO-World models strike a balance between strong grounding capabilities and high efficiency.

Following the reviewer’s suggestion, we replace UniVTG with two temporal grounding models, $R^2$-Tuning[4] and Moment-DETR[5], on the NExT-GQA dataset as a further ablation study. The results are presented in the table below. As shown, both models exhibit lower temporal grounding accuracy compared to UniVTG (especially in the IoP metric), leading to reduced VideoQA accuracy and Acc@GQA for the MSR-ViR framework.

Since spatial grounding must be performed for every temporal-grounded frame, the efficiency requirements for the spatial grounding model are even higher. YOLO-World’s exceptional efficiency makes it particularly well-suited to our needs, offering both high efficiency and high grounding accuracy—features that other open-vocabulary object detector models lack[6].

[1] Min, Juhong, et al. "MoReVQA: Exploring Modular Reasoning Models for Video Question Answering." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Zhang, Ce, et al. "A simple llm framework for long-range video question-answering." arXiv preprint arXiv:2312.17235 (2023).

[3] Wang, Ziyang, et al. "VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos." arXiv preprint arXiv:2405.19209 (2024).

[4] Liu, Ye, et al. "$R^ 2$-Tuning: Efficient Image-to-Video Transfer Learning for Video Temporal Grounding." arXiv preprint arXiv:2404.00801 (2024).

[5] Lei, Jie, Tamara L. Berg, and Mohit Bansal. "Detecting moments and highlights in videos via natural language queries." Advances in Neural Information Processing Systems 34 (2021): 11846-11858.

[6] Cheng, Tianheng, et al. "Yolo-world: Real-time open-vocabulary object detection." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Dear reviewer, thank you again for the review and we hope that our response and the uploaded revised paper have addressed your concerns. We would greatly appreciate your feedback and please feel free to let us know if you have any other questions.

We sincerely thank the reviewer for the insightful feedback and suggestions. The analyses and experiments according to the reviewer's suggestions further improve the quality of our paper, which have been added to the revised version of the paper.

We sincerely thank the reviewer for taking time to review our paper and providing thoughtful feedback and insightful suggestions. We address the weaknesses as follows:

Weakness1

We use the UniVTG model, pretrained on several temporal grounding datasets, as the temporal localization tool in the MoST-Grounding module. However, we have not conducted further training for UniVTG on our dataset, NExT-GQA. As shown in Table 2, while our temporal grounding results are slightly weaker than Temp[CLIP] and VGT in the IoP@0.5 metric, our performance is significantly stronger across all other grounding metrics. We believe this demonstrates that MSR-ViR with the pretrained temporal grounding module UniVTG has superior temporal localization capabilities.

Weakness2

Since the temporal grounding module cannot achieve 100% accuracy in identifying the time segments relevant to the question, sampling video frames solely from the grounding results may still result in missing some information. Therefore, the global representation $g_v$ serves as a necessary "back door". However, $g_v$ is not the most critical source of information—frames sampled from the temporal grounding results and spatial grounding results are more important. To demonstrate this, we have supplemented the ablation study on NExT-QA by removing the temporal grounding frames and spatial grounding frames, leaving only the global representation $g_v$. The results show a significant drop in question-answering accuracy, proving the importance of the grounded frames.

Weakness3

Based on the reviewer’s suggestion, we introduced SeViLa’s testing results on NExT-QA, STAR, and NExT-GQA for comparison with the Llava-Next version of our MSR-ViR framework.

We sincerely thank the reviewer for the response and feedback. The insightful and constructive suggestions further improve the quality of our paper.

Dear reviewer, thank you again for the review and we hope that our response and the uploaded revised paper have addressed your concerns. We would greatly appreciate your feedback and please feel free to let us know if you have any other questions.

We sincerely thank the reviewer for taking time to review our paper and providing thoughtful feedback and insightful suggestions. We address the weaknesses and questions as follows:

Weakness 1

Our MSR-ViR framework employs Qwen2-7B as the question parser and Qwen-VL as the multimodal answerer, both of which are based on LLM architectures. However, many recent works utilize LLMs to tackle VideoQA tasks. For instance, MoReVQA[1] leverages Palm-2, which is also a large language model, for four times in order to reason and get the answer to one question. Reasoning tasks often require step-by-step reasoning to derive multiple intermediate results before arriving at the final answer, which distinguishes them from end-to-end models that directly predict the final answer. This difference also explains why reasoning-based approaches typically cost greater computational overhead.

Weakness 2

To enhance the interpretability of the VideoQA task, the MoST-Grounding module employs UniVTG as the temporal grounding model and YOLO-World as the spatial grounding model, which indeed introduces additional parameters. However, both grounding models have relatively small parameter sizes: UniVTG has 41.3M parameters, and YOLO-World has 48M parameters, which are only a tiny fraction of the 7B parameters in LLMs. Therefore, the MoST-Grounding module does not significantly increase the overall parameter count of the framework.

The MoST-Grounding (Modularized Spatial-Temporal Grounding) module serves to localize the original question temporally and spatially within the video, providing the Multimodal LLM with the most relevant information to the question. We sincerely appreciate the reviewer’s valuable suggestion, as it is true that the current modular network employs only one temporal grounding model and one spatial grounding model. While experiments have already demonstrated satisfactory localization performance (see Table 2), incorporating more modules into the modular network to enhance its functionality could potentially further improve localization and answer accuracy. We will try to introduce more modules into our framework in future work.

Weakness 3

Based on the reviewer’s suggestion, we provide experimental results on inference speed and parameter size as follows:

The inference speed results were tested on two NVIDIA A100 GPUs. As shown, MSR-ViR introduces a larger parameter size compared to end-to-end Multimodal LLM. Additionally, since MSR-ViR requires an LLM to first generate a policy, followed by the MoST-Grounding modular network for grounding, and finally the multimodal answerer to provide the answer, its inference speed is noticeably slower than that of end-to-end Multimodal LLM.

Actually, the additional time cost is normal for reasoning tasks. As tested and reported in [2], the inference speed of VideoTree is about 0.13 qa pairs / s, while the inference speed of LLoVi[3] is about 0.26 qa pairs / s. Similarly, GPT-O1 takes significantly longer time to answer questions than GPT-4o because it performs detailed reasoning process while generating responses.

Question 1

We sincerely thank the reviewer for providing this insightful suggestion! In our current work, we leverage feedback from the Multimodal LLM as a reward to guide the training of the question parser through reinforcement learning. As shown in the ablation study in Table 3, this reinforcement learning approach improves the ability of the question parser to generate reasonable policies. However, reinforcement learning training may not be as stable or effective as direct supervised learning. If a Chain-of-Thought dataset could be constructed to enable supervised training of the question parser, it might further enhance its policy generation capability, thereby improving the overall performance of our framework. We sincerely apologize that, due to time constraints, we are currently unable to construct a high-quality CoT dataset and conduct training. We will explore this direction in our future work.

[1] Min, Juhong, et al. "MoReVQA: Exploring Modular Reasoning Models for Video Question Answering." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Wang, Ziyang, et al. "VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos." arXiv preprint arXiv:2405.19209 (2024).

[3] Zhang, Ce, et al. "A simple llm framework for long-range video question-answering." arXiv preprint arXiv:2312.17235 (2023).

Dear reviewer, thank you again for the review and we hope that our response and the uploaded revised paper have addressed your concerns. We would greatly appreciate your feedback and please feel free to let us know if you have any other questions.

Dear reviewer, thank you again for the review and the insightful feedback. We hope that our response and the revised paper have addressed your concerns. As the discussion stage is about to end, we would greatly appreciate your feedback and please feel free to let us know if you have any other questions.

We thank the reviewer again for the further feedback on our responses. Below, we provide further responses to the reviewer’s concerns:

1. Regarding the concern about inference speed: We would like to emphasize again that reasoning-based inference and end-to-end inference are fundamentally different. It is entirely normal and common for reasoning-based inference to be significantly slower than end-to-end inference, as it not only provides answers but also generates step-by-step reasoning paths to deliver more accurate and interpretable responses.

Still taking MoReVQA[1] as an example, this work shares similarities with ours, solving reasoning-based VideoQA tasks. Its framework includes four LLMs, two additional vision-language models and a video captioning model, with the reasoning process spanning four stages. Clearly, the computational cost of the MoReVQA framework is far greater than that of our MSR-ViR framework. Although the MoReVQA paper does not report inference speed and the code is not open-sourced, it is foreseeable that its inference speed is not significantly faster than our framework. Compared to its direct baseline JCEF, MoReVQA introduces considerable computational complexity and time overhead for its multi-stage reasoning process. However, it achieves a 2.5% accuracy improvement on NExT-QA, which is comparable to our MSR-ViR framework(see the table below). Just like MoReVQA and our MSR-ViR, most reasoning methods share similar characteristics, with increased computational costs compared to their base models. Contributions of reasoning methods extend beyond improving question-answering accuracy. Providing interpretable reasoning paths(and also evidence of answers in videos like MSR-ViR) is also a critical contribution, which inevitably brings greater computational overhead.

It is worth noting that grounding-based methods also often have slower inference speeds. For example, SeViLa[2] has an inference speed of only 0.30 qa pairs / s(as mentioned in SeViLa paper's appendix), which is comparable to the inference speed of our MSR-ViR framework. Note that SeViLa is not designed for long-form videos.

2. Regarding the concern that the accuracy improvement of MSR-ViR on VideoQA datasets is marginal: On the STAR-sub dataset, MSR-ViR(Qwen-VL-based) achieved a 3.4% improvement over its baseline (63.0 → 66.4). Specifically, on the Interaction and Sequence subsets, where temporal and spatial relationships are critical, it improved by 4.4% and 2.5%, respectively. We believe this significant accuracy improvement demonstrates the effectiveness of the MSR-ViR framework and is far from being marginal.

On the NExT-QA dataset, as shown in the table below, we compared the performance improvement of our method and other grounding-based methods and modular methods relative to their respective base models. For example:

• SeViLa improved by 1.2% over its base model BLIP-2$^{\text{concat}}$.
• MoReVQA improved by 2.5% over its direct baseline JCEF.
• Our MSR-ViR(Qwen-VL-based) improved by 1.7% over its base model Qwen-VL.
• Our MSR-ViR(Llava-Next-based) improved by 1.8% over its base model Llava-Next.

The improvements achieved by our method are comparable to those of existing grounding-based methods and modular methods, which we believe are not marginal.

[1] Min, Juhong, et al. "MoReVQA: Exploring Modular Reasoning Models for Video Question Answering." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Yu, Shoubin, et al. "Self-chained image-language model for video localization and question answering." Advances in Neural Information Processing Systems 36 (2024).

We sincerely thank the reviewer for taking time to review our paper and providing thoughtful feedback and insightful suggestions. We address the weaknesses and questions as follows:

Weakness1

Due to the lack of datasets with policy annotations, we are unable to directly perform supervised training for the question parser. As such, using feedback from the Multimodal LLM as rewards to train the question parser via reinforcement learning is a relatively straightforward approach. Based on the results of the ablation study (Table 3) and specific examples (Figures 4, 9, and 10), it can be observed that reinforcement learning enables the question parser to generate more reasonable policies that better support locating spatio-temporal segments relevant to the question in the video, demonstrating the effectiveness of our RL-based Alternate Self-reflection Training Strategy. We will experiment on larger VideoQA datasets with more complex scenarios in future work to evaluate the scalability of our RL-based training strategy.

Weakness2

We are sorry that, due to time constraints, we are unable to provide experimental results on the SOK-Bench and Complex-TV-QA datasets at this stage. Most questions in the Complex-TV-QA datasets focus on predicting future events based on the scenarios in the video, while the SOK-Bench includes many counterfactual reasoning questions. For these types of questions, grounding may not be helpful to improve Multimodal LLM to answer the question. As a result, they may not be suitable for evaluating our grounding-based framework MSR-ViR’s performance. It is worth noting that statistical analysis[1] shows that the average question length in the NExT-QA and STAR datasets is among the longest in commonly used VideoQA datasets, demonstrating the ability of our framework to handle long and complex questions effectively.

Weakness3

We test some more grounding-based model (SeViLa and GCG) and compare the results with our Llava-Next-based MSR-ViR framework, which are shown in the table below. As seen, MSR-ViR framework surpasses SeViLa and GCG on NExT-QA dataset and SeViLa on STAR-sub dataset. To further demonstrate the effectiveness of our framework, we test the base Multimodal LLM Llava-Next on NExT-QA and STAR-sub. As seen in the table below, the VideoQA accuracy has been significantly improved by utilizing MSR-ViR framework(from 73.1 to 74.9 on NExT-QA, from 69.9 to 71.0 on STAR-sub). What's more, for Acc@GQA and grounding metrics, MSR-ViR surpasses SeViLa on NExT-GQA dataset, showing a stronger grounded-qa capability.

Weakness4

Our MSR-ViR framework is implemented based on the Swift[2] framework, and all experimental results are fully reproducible. We plan to open-source our code after the review and provide detailed documentation to facilitate the reproduction and application of our framework.

[1] Fu, Chaoyou, et al. "Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis." arXiv preprint arXiv:2405.21075 (2024).

[2] Zhao, Yuze, et al. "Swift: a scalable lightweight infrastructure for fine-tuning." arXiv preprint arXiv:2408.05517 (2024).

Dear reviewer, thank you again for the review and we hope that our response and the uploaded revised paper have addressed your concerns. We would greatly appreciate your feedback and please feel free to let us know if you have any other questions.

Dear reviewer, thank you again for the review and the insightful feedback. We hope that our response and the revised paper have addressed your concerns. As the discussion stage is about to end, we would greatly appreciate your feedback and please feel free to let us know if you have any other questions.