When Thinking Drifts: Evidential Grounding for Robust Video Reasoning

Thank you for the valuable feedback. We are happy that you appreciated our identification of the “visual thinking drift” phenomenon as an interesting and underexplored challenge, as well as our empirical findings, the theoretical insights, and the breadth of our experimental evaluation across diverse tasks.

To address your concerns, we provide the following response:

Q1. “explicitly investigate why CoT fails for video tasks even when the same Bayesian mechanism applies to both modalities”

Thank you for the insightful comment! The “visual thinking drift” phenomenon is indeed specific to video reasoning and deserves clearer explanation.

In L168, we note that language priors outweigh visual likelihoods in practice, but did not elaborate further. In today’s video-LLMs, the language-projection matrix $W_{lang}$ tends to have a much larger norm than its visual counterpart $W_{vis}$. This is because the model inherits a massive, pre-trained language backbone trained on trillions of text tokens, while the vision pathway is typically a small, newly added adapter, fine-tuned on multimodal data that is orders of magnitude smaller. Moreover, during inference, far fewer visual tokens influence gradient flow compared to language tokens. This asymmetry in initialization, data scale, and gradient dynamics leads to a significantly smaller Frobenius norm in $W_{vis}$, causing language priors to dominate during autoregressive decoding and contributing to the observed drift.

Another important point is the modality mismatch in visual reasoning. In text tasks, the question, answer, and reasoning all happen in the same modality—language—so the model "thinks" in the same space as it reads and answers. But in video tasks, the model must reason about visual content using language as a conduit. This indirect reasoning makes it easier for the model to drift away from the actual video, especially when visual grounding is weak.

We will update this explanation in the text accordingly.

Q2. Generalization test across diverse model architectures (e.g., LongVA, LLaVA-OneVision) or parameter scales

Thank you for the constructive feedback. As noted in [1], LLaVA struggles to generate high-quality, coherent reasoning paths—an observation consistent with our own findings (noted in the footnote under L145). Per the reviewer’s suggestion, we also tested LongVA and observed a similar issue. We conjecture that this limitation stems from the use of a weaker underlying language model (Qwen2), which appears insufficient for reliably supporting structured reasoning prompts.

To our knowledge, Qwen2.5-VL is currently the most stable open-source model for following structured prompts and generating reasoning traces, as supported by recent literature [17, 32] and frequently used open-source projects [2, 3]. This capability is crucial for applying RL-based fine-tuning, which relies on consistent intermediate reasoning.

That said, we appreciate the reviewer’s concern regarding the generalizability of Video-VER. To address this, we conducted additional experiments using the Qwen2.5-VL base model with different parameter scales, 3B. As shown in the table below, our VER method still provides consistent accuracy improvements over the 3B model, demonstrating that our approach generalizes well across model scales.

[1] Yang, L., et al. (2025). GRPO-for-Llava. GitHub repository.

[2] Wang, X., et al. (2025). Open-R1-Video. GitHub repository.

[3] Feng, K., et al. (2025). Video-R1. GitHub repository.

Q3. Potential hallucination introduced by the 72B visual evidence generator

We understand that the reviewer's concern lies in the potential biases or hallucinations introduced by the external Qwen2.5-VL-72B as a visual evidence generator. We address this concern in Supplementary Section E. While large models can indeed hallucinate during visual reasoning, our approach is specifically designed to mitigate this issue. We employ several mechanisms to ensure that the generated visual evidence is grounded and visually anchored:

• Offline Use of Teacher Model (supp L381-384): The external 72B MLLM is used only during training to generate question-specific visual evidence. Our Video-VER model learns from this data but operates independently at inference time, ensuring that any potential errors from the teacher model are not propagated or relied upon during deployment.

• Binary Reward Signal (supp L385-391): Instead of requiring exhaustive annotations, we use a binary reward that simply checks whether the generated reasoning includes any verifiable visual fact. If the evidence is missing or incorrect, the model receives a zero reward. This effectively discourages hallucinations without reinforcing incorrect details.

• Inverted Prompting Strategy (supp L392-410): Unlike standard chain-of-thought prompting, we condition the 72B model on both the question and the correct answer, prompting it to produce concise, verifiable visual facts that support the answer. This tightly constrains generation, encouraging relevance and factual grounding over generic or speculative content.

To validate the robustness of our visual evidence extraction method, we provide an ablation study comparing the use and non-use of our Inverted Prompting Strategy (IPS) in generating visual evidence. The results show that IPS significantly improves the quality of visual evidence, which in turn enhances the overall performance of VER across all benchmarks.

In conclusion, through these design choices, our model is trained to prioritize visual grounding over reliance on linguistic priors, directly addressing the root cause of “visual thinking drift.”

Q4. Performance on videos exceeding 32 frames

We appreciate the reviewer’s interest in evaluating our method on longer video sequences. While our main experiments (Table 3) focus on input lengths of 8, 16, and 32 frames, we have conducted additional experiments with 64-frame inputs to assess scalability. These results, presented in the table below, show that our method effectively leverages extended temporal context and continues to perform well, demonstrating strong scalability and generalization to longer sequences.

Due to computational constraints, we are limited to evaluating up to 64 frames. However, we agree that handling longer videos is crucial for real-world applications. As highlighted in our failure case analysis (Figure 14), missing key visual cues due to limited sampling or encoding can impair reasoning. Thus, the VER mechanism not only enhances visual reasoning but also stands to benefit from continued advances in long-context video modeling.

Q5. Comparison with [46] or [38]

As noted in L38–39, both [46] and [38] focus on curating extensive spatio-temporal annotations to support supervised fine-tuning (SFT), where the model learns from human-labeled examples for specific subtasks. In contrast, our work centers on reinforcement fine-tuning (RFT), which directly optimizes visually grounded reasoning through an RL reward signal. This approach enables the model to learn end-to-end reasoning without relying on handcrafted subtask-specific annotations.

We have included additional evaluations of [46] and [38] across four benchmarks. For both baselines, we report their best published results. Our Video-VER model consistently outperforms these methods, suggesting that our visually grounded GRPO framework offers a competitive—and potentially more generalizable—alternative to these modular, annotation-heavy approaches.

For Q3, we thank the reviewer for noting that our additional Video-QA results already lend indirect support to the pipeline. Below we provide the requested direct evaluation, combining a controlled quantitative audit with qualitative inspection.

Quantitative human audit. We randomly sampled 200 videos from our RL post-training dataset and generated the visual evidence examples with our IPS and without. Three expert annotators independently labelled each sentence as Aligned (fully supported by at least one frame) or Not-Aligned (hallucinated). Ties were adjudicated through majority voting.

IPS therefore cuts hallucinations by 14%, confirming that our approach substantially improves visual-evidence quality. Combined with the binary reward that zeroes out any residual hallucinations during training, this demonstrates the robustness of our evidence-generation process.

Qualitative check. As shown in Fig. 5 and Fig. 11, our Inverted-Prompting Strategy (IPS) generates visually grounded evidence that correctly references both the relevant event and its temporal context.

Taken together, we think these new audits address the reviewer’s original comment: “the generation of visual evidence relies on a strong external MLLM (Qwen‑2.5‑VL‑72B), which could introduce biases or hallucinations itself. While the paper mentions filtering, the robustness of this process is not fully validated.” Concretely, our IPS‑driven filtering offers a 14 % reduction in hallucinations indicating improved robustness. We appreciate the suggestion; this additional analysis will strengthen the paper.

To further substantiate this point, we now provide additional theoretical and empirical evidence supporting the presence of this asymmetry—specifically, how the dominance of the language component underlies the phenomenon we refer to as visual thinking drift.

Theoretical Justification. We present a simple closed-form derivation to explain why fewer gradient updates to the vision encoder naturally lead to a smaller Frobenius norm for the visual projection matrix, compared to that of the language model.

Assume a trainable matrix evolving under stochastic gradient descent (SGD) as:

where the updates $\Delta_t$ have zero mean and second moment $\mathbb{E}\bigl[\lVert\Delta_t\rVert_F^{2}\bigr]=\sigma^{2}$.
Then the expected squared Frobenius norm becomes:

This shows that the expected Frobenius norm grows with the square root of the number of updates $N$.

In modern video-LLMs, the language pathway typically consists of a massive pre-trained backbone trained on trillions of text tokens (e.g., ~18 trillion for Qwen 2.5). In contrast, the visual pathway often involves a newly introduced adapter module trained on significantly fewer multimodal examples—on the order of tens of billions of paired tokens (e.g., ~40B for Flamingo). As a result, the number of gradient updates $N$ for the visual projection is much smaller, leading naturally to the observed asymmetry: $\lVert W_{\text{vis}}\rVert_F \ll \lVert W_{\text{lang}}\rVert_F$.

Empirical Evidence. To validate this asymmetry, we inspected two publicly available video-LLM checkpoints—Qwen2.5-VL-7B and LLaVa-OneVision-7B—and measured the Frobenius norms of the visual and language projection matrices. The results are as follows:

These findings empirically validate the asymmetry discussed above: in both models, the language projection layer has a significantly larger Frobenius norm than its visual counterpart. This suggests that the language signal is considerably stronger during decoding, leading to a dominance of language priors over visual likelihoods [1].

[1] Attention is Not Only a Weight: Analyzing Transformers with Vector Norms, EMNLP 2020.

We hope this additional theoretical and empirical analysis provides a clearer understanding of how language priors can overpower visual inputs in current video-LLMs, contributing to the phenomenon we term visual thinking drift. Please do not hesitate to let us know if you have any further questions or would like additional clarification.

For Q1, we understand that the reviewer’s primary concern relates to the Bayesian analysis presented in Section 3.2. While our theoretical insights were considered “intuitively reasonable”, the level of detail provided may have appeared moderate, as suggested by the reviewer.

The core question raised is why the same Bayesian mechanism applies to both language and visual modalities, yet the performance degradation introduced by CoT is observed primarily in video-related tasks. In essence, the reviewer is seeking a more nuanced explanation of the root cause of this performance degradation, particularly as it pertains to the unique challenges in video-based tasks. We appreciate this thoughtful question and are happy to provide supplementary justification to further support better understanding of our analysis.

Consider a question $q$, condition $\mathbf v$ (which represents either the video features in video LLMs or the "prefilling text" in text-only LLMs), reasoning tokens $c_{1:T}$ and final answer $a$, the joint probability under an autoregressive video-LLM is

We now derive an explicit form for each conditional $p\\bigl(c_t \mid c_{<t}, q, \mathbf v\bigr)$.

In video LLMs, at step $t$ the decoder’s last hidden vector can be written as

where $A_{\text{vis}}\in\mathbb R^{d\times d_{\text{vis}}}$ is the vision adapter (either a linear layer or MLP), and $\mathbf r_t^{(\text{vis})}$ is the visual embedding.

During decoding, every token passes through the same vocabulary projection $W_{\text{out}}\in\mathbb R^{d\times|\mathcal V|}$. For a candidate token $w$ with one-hot encoding $\mathbf e_w$, the logit is computed as $z _w= \mathbf h _t^{\\top} W _{\text{out}} \mathbf e _w$.

Substituting into Equation (2),

where we define:

The conditional distribution is given by a softmax over all tokens $w$:

Removing the common denominator yields the unnormalised expression:

Here $W_{\text{lang}}\equiv W_{\text{out}}$ and $W_{\text{vis}} \equiv W_{\text{out}}A_{\text{vis}}$.

In our previous response, we noted that the dominance of language priors in video-LLMs stems from asymmetries in model initialization and data scale. These asymmetries lead to a substantially smaller visual projection norm, causing language signals to overpower visual inputs during decoding—formally, $\lVert W_{\text{vis}}\rVert_F \ll \lVert W_{\text{lang}}\rVert_F$.

In contrast, for text-only LLMs, there is no visual term $A_{\text{vis}}(\mathbf r _t^{(\text{vis})})$ due to the absence of a vision encoder. As a result, the same "thinking drift" mechanism does not apply in purely textual settings.

We sincerely appreciate the reviewer’s thoughtful comments and are pleased that our responses have satisfactorily addressed the concerns regarding Q2, Q4, and Q5. We would be happy to offer further clarification on Q1 and Q3, as detailed in our separate comments below.

Thank you for appreciating the interesting and well-founded insight into the “visual thinking drift” phenomenon presented in our work, as well as our overall clear presentation.

We understand that the reviewer's main concern lies in the less well-explained cause of the observed accuracy improvements in our experimental results. To address this, we provide the following response.

Q1. Comparison Setup: Is the gain from GRPO or VER? – “most significant issue affecting my decision”

Thanks for the thoughtful feedback! We agree that providing a clearer breakdown of the contributions from each model component is important for evaluating the effectiveness of our approach. In response, we present detailed ablation results below that compare the requested configurations: Baseline + SFT, Baseline + SFT + GRPO, and Baseline + SFT + GRPO + VER (full model). Baseline here is Qwen2.5-VL-7B model.

Our findings show that fine-tuning the baseline model with SFT alone does not consistently improve performance. Adding GRPO yields moderate gains across most benchmarks. Notably, incorporating our Visual Evidence Reward (VER) on top of GRPO leads to substantial additional improvements. All performance deltas are reported relative to the original baseline model.

These results help isolate the contribution of each component and demonstrate that the gains observed in our full model are primarily driven by VER, validating its effectiveness in promoting grounded and accurate reasoning.

Q2. Reliability of Visual Evidence Source

We understand that the reviewer's concern lies in the reliability of Qwen2.5-VL-72B for generating visual evidence, particularly given its potential to overlook or hallucinate visual details—an issue also observed in models like GPT-4o. We address this concern in Supplementary Section E. While large models can indeed hallucinate during visual reasoning, our approach is specifically designed to mitigate this issue. We employ several mechanisms to ensure that the generated visual evidence is grounded and visually anchored:

• Offline Use of Teacher Model (supp L381-384): The external 72B MLLM is used only during training to generate question-specific visual evidence. Our Video-VER model learns from this data but operates independently at inference time, ensuring that any potential errors from the teacher model are not propagated or relied upon during deployment.

• Binary Reward Signal (supp L385-391): Instead of requiring exhaustive annotations, we use a binary reward that simply checks whether the generated reasoning includes any verifiable visual fact. If the evidence is missing or incorrect, the model receives a zero reward. This effectively discourages hallucinations without reinforcing incorrect details.

• Inverted Prompting Strategy (supp L392-410): Unlike standard chain-of-thought prompting, we condition the 72B model on both the question and the correct answer, prompting it to produce concise, verifiable visual facts that support the answer. This tightly constrains generation, encouraging relevance and factual grounding over generic or speculative content.

To demonstrate the effectiveness of our visual evidence extraction method, we provide an ablation study comparing the use and non-use of our Inverted Prompting Strategy (IPS) in generating visual evidence. The results show that IPS significantly improves the quality of visual evidence, which in turn enhances the overall performance of VER across all benchmarks. In conclusion, through these design choices, our model is trained to prioritize visual grounding over reliance on linguistic priors, directly addressing the root cause of “visual thinking drift.”

Q3. Regarding Evidence Generation as Distillation for VideoQA

Thank you for the insightful question! Distillation is an interesting interpretation, and indeed the larger 72B model aims to ensure the most reliable visual evidence signal as possible. Nonetheless, our method goes well beyond standard knowledge distillation paradigms. Specifically, our model is not trained to mimic the final outputs or answer distributions of the 72B model through supervised fine-tuning (SFT). Instead, the 72B model is used once, offline, to generate minimal, question-specific visual evidence that supports correct answers. This evidence is then used to define a binary, fact-level reward signal that guides the training of the 7B Video-VER model via reinforcement learning.

This setup differs fundamentally from conventional knowledge distillation in several key ways. First, there is no direct imitation of the teacher’s answers. The 7B model is not learning to reproduce the 72B model’s outputs or full reasoning traces. It is rewarded only when its own reasoning includes verifiable visual facts aligned with the evidence. Second, the supervision is task-specific and grounded. The visual evidence highlights what matters visually in the context of each question, teaching the model how to reason rather than simply what to predict. Third, the 72B model’s role is limited to only helping the smaller model identify and attend to relevant visual cues. There is nothing more.

As a result, the 7B Video-VER model learns to ground its answers in concrete visual evidence, enhancing both interpretability and robustness. In short, our method uses the 72B model as a generator of visual signals that shape the reward function, enabling the smaller model to learn how to reason visually. This distinction is central to the effectiveness of our VER training approach.

I thank the authors for their efforts in providing detailed responses and additional experiments. The added experiments in the answer to Q1 comparing individual gains indeed show that VER can present unique gains compared to naive GRPO training. The answers to Q2 and Q3 also directly addressed my concerns regarding whether this method ties the learning to the teacher model (72B model) or can learn beyond their visual evidence. While I also have to discuss with other reviewers, I intend to raise my score.

Thank you very much for your valuable comments. We are pleased that you appreciated our “solid demonstration of the initial phenomenon”, “analysis of existing models”, “simple but effective fix to this problem”, and “comprehensive set of baselines evaluated”.

Q1. Results of Video-VER without chain-of-thought.

As shown in the table below, we evaluated our Video-VER model using two types of prompts: a chain-of-thought (CoT) prompt, which encourages the model to reason step by step before answering, and a direct answer (DA) prompt, which requires only a final answer without intermediate reasoning. Both the base model Qwen2.5-VL and our Video-VER model have 7B parameters.

Overall, as the reviewer anticipated, Video-VER achieves more consistent performance improvements when prompted with CoT, which aligns with its training objective. While DA prompting also yields performance gains over the base model on most benchmarks, the improvements are less consistent and generally smaller than those observed with CoT. These results support our hypothesis that VER rewards in GRPO training encourage grounded, step-by-step reasoning.

Q2. Writing nitpick.

Thank you for the insightful suggestion! We will revise the text to remove superfluous adjectives.

Thank you for the valuable feedback. We appreciate the acknowledgment of our analysis of the visual thinking drift phenomenon, the further exploration from a visual grounding perspective, and the proposed reinforcement learning solutions, as well as the extensive experiments and ablation study.

To respond to your questions:

Q1: “Maybe more reasoning models should be added as baselines, such as OpenAI o1/o3.”

Thank you for the suggestion. As shown in Table 1, our baselines span the full spectrum of video-LLM capabilities: proprietary (GPT-4o) and open-source; generalist (LongVA, Video-UTR, LLaVA-OneVision, Kangaroo, Qwen2.5-VL) and specialist video-reasoning models (TinyLLaVA-Video-R1 and Video-R1). Nonetheless, we appreciate the reviewer’s interest in advanced proprietary reasoning models such as OpenAI’s o1/o3. To address this, we evaluated o3 (released April 2025) under both direct-answer (DA) and chain-of-thought (CoT) prompting across all ten video benchmarks. As the table below shows, CoT prompting induces a consistent performance drop—despite o3’s strong reasoning capabilities—mirroring the trends in Figure 2 and reinforcing our conclusion that CoT does not reliably boost video-LLM performance.

(Note: OpenAI’s “Hiding the Chains of Thought” policy prevents o3 from exposing its internal reasoning; in some CoT runs the model interjects messages like “Sorry, I can’t share my private reasoning,” which may contribute to the accuracy loss.)

Q2: Which performs better between using agentic workflow and end-to-end training?

The performance of agentic workflow approaches is often constrained by the capabilities of their component visual specialist models (e.g., key-frame selectors or object detectors). In contrast, end-to-end training leverages a single multimodal LLM trained on large-scale vision-language data, with the vision encoder handling perception and the language model handling reasoning—enabling a more unified approach to video understanding. Our VER model follows this end-to-end paradigm.

As a reference, we include additional evaluations of recent agentic workflow methods [64, 53, 41], which approach video understanding by decomposing the task into sub-problems—such as key-frame selection or temporal grounding—and addressing each with specialized models. Across benchmarks, our 7B Video-VER model demonstrates superior overall performance compared to these baselines.

Nonetheless, agentic workflows offer some flexibility at test time. Their ability to decompose tasks and incorporate external visual models may provide advantages in specific scenarios.

Q3: If the large model can control the behavior in visual space, will there also be hallucinations? Is it consistent with the cause of visual perception illusion?

Thank you for the thoughtful question. If a large model could directly manage visual information—for example, by maintaining spatially accurate representations or consistently referring to visual evidence throughout the reasoning process—it would likely help reduce hallucinations. However, hallucinations can still arise due to inherent challenges in video perception, such as low visual resolution, ambiguous scenes, or limited temporal grounding.

As discussed in Section 3.2, current video language models (video-LLMs) often rely heavily on language-based priors because visual signals tend to fade or weaken during the model’s step-by-step reasoning process (autoregressive decoding). As a result, the model may generate outputs that are only loosely grounded in the actual video content.

This is where our approach becomes especially important. A key focus of our work is to ensure that the model's reasoning actively refers back to visual evidence at intermediate steps. By reinforcing this connection between reasoning and visual input, we aim to reduce hallucinations and improve the model’s alignment with the video. This aligns with our hypothesis that "visual thinking drift" occurs when reasoning becomes unanchored from the visual content, leading the model to fabricate visual details due to insufficient fidelity and control over the visual stream.