Divide and Conquer: Exploring Language-centric Tree Reasoning for Video Question-Answering

Thank you for carefully reviewing our paper, acknowledging its strengths, and providing valuable suggestions for improvement If you have any further concerns, please feel free to raise them during the second-round rebuttal phase. As recommended by the official FAQ, we provide all figures and tables via the anonymous link.

Domain Adaptability and Finetuning on Domain Specific Tasks

In general, finetuning MLLMs in a specific domain significantly enhances performance by providing a more solid basis for reasoning. We further detail finetuning for different modules as follows:

• Divide with Top-down Recursive Checking: Finetuning is generally unnecessary for this stage since question decomposition primarily relies on language understanding, with visual content serving as a supplementary element where coarse understanding is sufficient for most scenarios. However, when encountering different question distributions or videos with distinct characteristics (e.g., 360-degree videos or extremely long videos), finetuning may offer significant benefits.
• Perceptual Leaf Question Answering: This step can be readily finetuned using any VideoQA dataset, which would improve the accuracy of leaf question answers and, consequently, enhance subsequent reasoning stages.
• Video-Aided Logical Reasoning and In-Process Answer Verification: Although finetuning these steps can benefit the framework, it requires additional effort to construct corresponding reasoning data for the specific domain.

To further validate the effectiveness of finetuning at different stages, we conducted experiments on AGQA-Decomp by finetuning our entire framework, leveraging the compositional graph and detailed answers provided in AGQA-Decomp. In Table1, we compare the performance of Qwen2-VL and VideoChat2 on AGQA-Decomp with and without finetuning. For the main question, our LTR framework improves the baseline MLLMs by 3%–4%, and the finetuning strategy further boosts LTR by an additional 3%–4%, leading to a total improvement of 7%–8% in accuracy. For sub-questions, the improvement is even larger, 12%–13%, because the Perceptual Leaf Question step is more amenable to enhancement in a specific setting. Moreover, the improvement in terms of $c$-$F_1$ is approximately 15%–16%, largely attributable to the increased sub-question accuracy and the hierarchical information aggregation and logical inference in Video-Aided Logical Reasoning. We thank the reviewer for this suggestion and will incorporate these experimental results and discussions in the following version of the paper.

We thank the reviewer again for the detailed reviewing, and hope our rebuttal have solved the proposed concerns and would strengthen the reviewer's confidence in judging positively for this paper.

Thank you for carefully reviewing our paper, acknowledging its strengths, and providing valuable suggestions for improvement If you have any further concerns, please feel free to raise them in the rebuttal comment. As recommended by the official FAQ, we provide all figures via the anonymous link.

Automated Pipeline

We agree that automation is critical for practical applications. Our two-stage approach, first generating the language-centric logical tree, then performing bottom-up reasoning through tree validation, handles VideoQA automatically. For each question, the system recursively decomposes it into simpler sub-questions until all leaf nodes are perceptive, using retrieved few-shot examples as guidance. The resulting tree is processed bottom-up to analyze both the video and question, yielding interpretable answers. Furthermore, while LTR is designed for full automation, it also allows for human intervention in sensitive scenarios, enabling users to adjust intermediate responses for better accuracy.

Applicable Question Types

We provide two figures to illustrate how LTR framework handles summarization and negation in multiple-choice scenarios. For the summarization question(Figure1), LTR naturally decomposes the video into parts and generates a Language-centric Logical Tree that prompts a summary for each segment, leading to a comprehensive summary of the main question. In the case of negation in multiple-choice questions(Figure2), the framework breaks down the question by checking each statement individually and then integrates the results to determine which statement is false, closely mirroring human reasoning.

However, for questions that are linguistically simple yet require complex cognitive visual reasoning, such as “Is there a thief?”. LTR may struggle to generate a reasonable Language-centric Logical Tree. In these cases, the question is often misclassified as perceptual due to its simplicity, even though answering it requires detailed motion analysis and a comprehensive understanding of the video content. We will include these discussions in the following version.

More Qualitative Results

We further provide three cases to illustrate LTR’s performance in different settings. In Figure3, for “Questions unanswered with LTR,” the counting sub-question misidentify background objects due to perceptual miscounting and a logic trap, resulting in an incorrect final answer despite a transparent reasoning process. In Figure4, for “Questions correctly answered with LTR and incorrectly answered by baseline,” LTR accurately deduced the stationary object’s properties by intersecting responses from multiple sub-questions and leveraging visual-aided reasoning, whereas the baseline relied solely on perceptual cues. In Figure5, for “Questions correctly answered by baseline and incorrectly answered with LTR,” the baseline produce a correct answer without any explanation. However, an erroneous leaf response in LTR led to an incorrect final answer, even though the intermediate video content analysis remained robust.

Errors in Intermediate Nodes

Errors or hallucinations in reasoning are inevitable, but prior study [1] show that providing more context or sampling multiple answers can help suppress them. In LTR, detailed sub-question decomposition offers comprehensive context that reduces hallucinations and minimizes cascading errors, as evidenced by improved sub-question accuracy. So, an error at an intermediate node is less likely to cascade and adversely affect the final answer.

The In-Process Answer Verification stage further mitigates errors by prompting the model to check both reasoning logic and consistency between answers and visual content. Although no system can fully eliminate errors, this verification step increases the reliability of intermediate responses, as confirmed by experiments in Table5 of the submission. We will include these discussions in the following version.

[1] Fei, H. et.al. Video-of-thought: Step-by-step video reasoning from perception to cognition. In ICML, 2024

Performances on Closed-Source Models.

To valid the effectiveness of LTR on closed source models, we conduct experiments using GPT-4o on EgoSchema and MVBench, with 16 uniformly sampled frames per video. As summarized in Table1, LTR yield significant accuracy improvements on these benchmarks, demonstrating its generalizability across various models.

Minor Issue

All typos will be revised in the following version.

Thank you for carefully reviewing our paper, acknowledging its strengths, and providing valuable suggestions for improvement If you have any further concerns, please feel free to raise them during the second-round rebuttal phase. As recommended by the official FAQ, we provide all figures and tabels via the anonymous link.

Comparison with More Baseline Methods.

To further validate the effectiveness of our LTR, we compare the it with three baselines (CoT, Major Voting, and Best of N), across AGQA-Decomp, MVBench, and CausalVidQA using VideoChat2 as the MLLM, as shown in Table1. For fair comparison, the repetition count (N) for all baselines is set to match the number of MLLM executions in LTR for various questions. For CoT, we carefully construct five text-based examples as prompts generated by GPT-4o. For Best of N, VideoChat2 also serves as the reward model. These comparisons will be included in the following version.

In Table1, although CoT, Major Voting, and Best of N improve model accuracy on three datasets, their gains remain relatively moderate compared with our LTR, largely due to the lack of detailed information in these methods. Specifically, the Major Voting and Best of N do not provide detailed sub-question analysis and the CoT offers only simple reasoning examples that constrain the reasoning ability of MLLMs, however, our LTR delivers a structured, comprehensive context and a detailed reasoning process that clearly demonstrates how the answer is deduced, thereby enhancing both accuracy and interpretability.

Furthermore, the compared methods contribute little to the compositional consistency of models, as they lack intra-question information exchange. In contrast, our LTR employs a hierarchical information aggregation and logical inference procedure that yields a consistent and accurate reasoning process, ultimately achieving superior performance in accuracy and compositional consistency on several VideoQA benchmarks (i.e., AGQA-Decomp, MVBench, and CausalVidQA) while also offering enhanced interpretability and controllability.

More Complex Qualitative Results and Module Analysis

To validate the effectiveness of our LTR framework, we provide two complex qualitative examples in Figure 1 and Figure 2 in our response, along with a detailed analysis for the first example.

In Figure1, the video depicts a scenario where a man is struck in the crotch by a dog, causing him to bend over and hold his crotch. In Stage 1, the Divide with Top-down Recursive Checking successfully decomposes the main question into a language-centric logic tree with simpler sub-questions that precisely target perceptual aspects (e.g., detecting the man, the dog, and the actions performed). In Stage 2, the Video-Aided Logical Reasoning module integrates both logical cues and visual evidence to infer answers for each non-leaf node. Crucially, the In Process Answer Verification stage is applied to ensure consistency across intermediate responses; for instance, it corrects the answer for non-leaf question [inter_2] by cross-validating the reasoning with the visual content.

Furthermore, we also update more qualitative results along with comprehensive discussion, covering both successful and failure cases, to better illustrate the impact of each module within our LTR framework. The revised figures and detailed descriptions are available in our response to Reviewer DfhN. We will include these examples and discussions in the following version.

Explanation of System-2 Reasoning

The term system-2 originates from psychology and cognitive science, where the dual process theory delineates human reasoning into two distinct processes: system-1, which is fast, intuitive, situational, and perceptual, and system-2, which is slow, logical, abstract, and cognitive [1]. Notably, CoT, Major Voting, and Best of N are also typical examples of system-2 reasoning models. We will explain more on system-2 reasoning and its relevance to computer science in the follow version.

Minor Issue

The MLLM employed in the qualitative analysis is VideoChat, and all the typos will be corrected in the following version.

[1] Evans, J. S. In two minds: dual-process accounts of reasoning. Trends in Cognitive Sciences, 7(10):454–459, 2003. ISSN 1364-6613.