More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Models

We would like to express our heartfelt gratitude to the reviewer for recognizing that our proposed conclusions are valuable and actionable, our benchmark and metric are solid and practical, and our presentation is well-organized. We sincerely appreciate the insightful questions raised and have addressed them in detail below.

(W1) Broader Baseline Coverage. Thank you for this suggestion. During our research, we observe most current existing multimodal reasoning models often adopt Qwen2.5-VL as a backbone model and fine-tune on it. Therefore, the set of models we evaluated covers the same model series. To further validate our findings, we conduct a broader evaluation of newer reasoning models across various sizes and their corresponding non-reasoning counterparts. As shown in Table 1, the results align with our findings, further corroborating the universality of the hallucination trends and visual grounding behaviors proposed in this paper.

Table 1: Performance of recently released multimodal reasoning models on perception benchmarks and RH-Bench.

(Q1) Factors Influencing The Reasoning-Perception Balance of R1-OneVision. Thank you for the insightful question. The performance limitations of the R1-OneVision model, built upon the Qwen2.5-VL architecture, are not attributable to inherent architectural constraints of the base model. This is evidenced by the fact that numerous other superior reasoning models also utilize the same baseline. The internal attention mechanism analysis of the R1-OneVision model, as depicted in Fig. 5 of the manuscript, reveals a significant dilution effect on visual information during the reasoning process. This suggests a potential imbalance in the task objectives of R1-OneVision, with the optimization focus skewed towards enhancing reasoning capabilities. Furthermore, the adoption of the SFT+RL training paradigm without integrating additional optimization strategies likely contributes to an imbalance between reasoning and perception abilities.

(Q2) Domain-wise Analysis of RH-AUC Performance. Thank you for highlighting this. To investigate the RH-AUC improvement of math-trained models (e.g., MM-Eureka and MM-R1) on RH-Bench, we categorize the benchmark into math and non-math tasks and separately evaluate the accuracy and RH-AUC scores of each model. As shown in Table 2, the results demonstrate that MM-Eureka and MM-R1 indeed achieve higher RH-AUC scores in pure mathematics tasks, clearly reflecting their superior perception-reasoning balance capability within the mathematical domain. Notably, even on non-math tasks, the RH-AUC of MM-Eureka and MM-R1 surpasses that of some models with extensive training data, such as R1-OneVision. This suggests domain-specific training holds the potential to generalize to other domains, improving perception-reasoning balance, which aligns with our paper's findings.

Table2: Comparison of RH-AUC across math and non-math domains for different models.

(L1) Further Clarification of Interpretive Points. Thank you for your valuable feedback. Regarding the impact of training paradigms and data domains, we present the performance and RH-AUC of various advanced models in Table 1. These results clearly indicate that recent reasoning models no longer rely solely on larger datasets but rather prioritize carefully curated multi-domain data, which aligns with our analysis. Furthermore, most RL-only models exhibit superior performance. For these two training paradigms (RL vs. SFT+RL), our analysis in Tables 3 and 4 below conducts fine-grained statistics on response length and reasoning-related vocabulary. The results reveal a statistically significant tendency of RL models to generate more concise responses while avoiding excessive deliberation, which is consistent with the results in Figs. 8b–c of the manuscript. Moreover, Table 5 presents a more comprehensive comparison of the RL and SFT+RL training paradigms.

Table3: Average response token length.

Table4: Frequency of reasoning related words evaluated on MathVision.

Table5: Performance comparison of two training paradigm models on multimodal benchmarks.

[1] R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimization

[2] MM-Eureka: Exploring visual aha moment with rule-based large-scale reinforcement learning

[3] MM-Eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning

[4] Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models

[5] VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning

[6] SophiaVL-R1: Reinforcing MLLMs Reasoning with Thinking Reward

[7] Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learning

[8] SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models

Thank you for the extensive response. I am satisfied with the answers and will maintain my positive score.

We sincerely thank the reviewer for recognizing our observations as timely, insightful, and important, our comprehensive quantitative evaluation, and our systematic analysis of the reasoning–hallucination trade-off. We sincerely hope that our responses below address your concerns.

(W1) Empirical Analysis and Generalizability Validation. Thank you for pointing this out. We agree that our current analyses are based on off-the-shelf models, and while they offer valuable insights, they should be interpreted as observational rather than causal, this is indeed what we have done in the paper where we show our findings suggest trends and raise hypotheses, such as “RL only outperforms SFT+RL”, we will make this even clearer in the revised version and warrant future controlled studies. To further demonstrate the generalizability of our findings, we conduct a comprehensive performance evaluation of recently released reasoning models. As shown in Table 1 of qZ13 W1, the results not only reveal a consistent trend of amplified hallucinations but also further support the finding that training data volume is not the sole determinant of model performance. As shown in Table 1 below, the performance comparisons across different training paradigms, along with the statistical analyses of reasoning length and reasoning-related terms presented in Tables 2–3, are consistent with the findings reported in our manuscript.

Table1: Performance comparison of two training paradigm models on multimodal benchmarks.

Table2: Average response token length.

Table3: Frequency of reasoning related words evaluated on MathVision.

(W2) Explanation of RH-AUC Computation. Thank you for your insightful suggestion. We strongly agree with the view that models should be able to generate diverse reasoning chains per question, and our Latent State Steering method (Section 4.1) is specifically designed to achieve this capability. This method enables flexible control over diverse reasoning chain generation through a single tunable parameter $\alpha$, dynamically adjusting reasoning length without fixed constraints. Compared to direct generation of diverse reasoning chains, which often yields limited length variation and narrow scope, our steering method achieves significantly greater diversity through $\alpha$ adjustment while preserving the model's original performance (see detailed analysis in Tables 4–5). Appendix Figs. 9–11 show the performance under our steering method. Notably, this method requires only a single hyperparameter adjustment without introducing additional computational overhead.

(W3) Relationship Between Reasoning Length and Problem Complexity. We thank the reviewer for raising this important point. It is true that, in practice, more complex questions may naturally elicit longer reasoning chains. However, RH-Bench is designed to decouple reasoning length from question difficulty in order to evaluate the model’s robustness to reasoning depth under controlled settings. To minimize confounding effects, we fix the question while varying only the reasoning chain length via intervention, allowing us to probe how well the model performs as reasoning length increases even for the same question. This design choice helps us isolate the impact of reasoning length itself, rather than entangling it with question difficulty. We acknowledge this limitation and have clarified this design rationale in Section 5, while also noting that evaluating models under naturally varying question difficulty remains an important direction for complementary future benchmarks.

(W4) Diversity of Base Models. Thank you for your valuable feedback. We respectfully note that most current existing multimodal reasoning models often adopt Qwen2.5-VL as a backbone model and fine-tune on it. Therefore, the set of models we evaluated covers the same model series. To further demonstrate generalizability, we conduct additional experiments on recently released reasoning models and baselines: Qwen2-VL (7B), InternVL2.5 (8B, 38B), and Qwen2.5-VL (32B, 72B). Detailed results are shown in Table 1 of qZ13 W1.

(Q1) Reasoning–Hallucination Trade-off in Larger VLMs. Closed-source models like GPT-4 and O3 struggle to precisely control reasoning chain length, relying mainly on limited prompt engineering. In contrast, our systematic evaluation of advanced open-source reasoning models (32B to 72B) in Table 1 of qZ13 W1 demonstrates that larger models typically achieve higher RH-AUC scores, indicating more robust performance.

(Q2) Length-Control Method Impact on RH-AUC Performance. Thank you very much for your thoughtful suggestions. The RH-AUC evaluation employs the Latent State Steering method, with its implementation details shown in Appendix D. Furthermore, we conduct comparative experiments on the RH-Bench benchmark using three length control methods to assess their impact on model performance. As shown in Tables 4–5 below, by analyzing the accuracy of perception and reasoning tasks and the RH-AUC metrics of different models, the results indicate that the performance differences among the methods on these tasks fluctuate around ±0.5%. Moreover, the RH-AUC metrics demonstrate high stability across all three methods. These results indicate that the reasoning length control methods only affect the output length without compromising the model's performance.

Table4: Accuracy (%) comparison on RH-Bench across three reasoning length control methods: Token Budget Forcing, Test Time Scaling, and Latent State Steering (LSS).

Table5: RH-AUC scores across different models under three reasoning length control methods.

(Q3) Correct Formatting. Thank you for highlighting this. We have carefully reviewed the text and made the necessary corrections in the revised manuscript.

[1] SophiaVL-R1: Reinforcing MLLMs Reasoning with Thinking Reward

[2] Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learning

[3] SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models

[4] MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-Based Reinforcement Learning

[5] VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning

We appreciate the reviewer's thoughtful comments and recognition of our novel evaluation metric and benchmark, systematic analysis of reasoning-hallucination trade-offs, and comprehensive experiments across training paradigms. We sincerely hope our response below can address your concerns.

(W1) Scale of RH-Bench and Effective Evaluation. Thank you for your insightful suggestion. The effectiveness of the evaluation provided by RH-Bench arises from its careful design, rather than solely relying on sample size. As illustrated in Figs. 10-11 of Appendix D, the non-monotonic trend in model performance with varying reasoning lengths observed in RH-Bench closely aligns with trends observed in datasets such as MathVista and MMhalu (Fig. 7 in manuscript), demonstrating the capability of the benchmark to reliably capture model behaviors. Furthermore, RH-Bench effectively distinguishes performance differences among models. This capability results from the design of task diversity, with reasoning tasks involving core domains such as math, visual common sense, and scientific questions, while perception tasks focus on detail recognition and object identification across both multiple-choice and open-ended formats. We plan to further expand the scale and dimensions of RH-Bench to facilitate more comprehensive and in-depth model evaluations.

(W2 & W3) Generalizability Verification Across More Model Families. Thank you for your valuable feedback. In response to W2 and W3 on model size and framework, most existing multimodal reasoning models often adopt Qwen2.5-VL as a backbone model and fine-tune on it. Therefore, the set of models we evaluated covers the same model series. To further demonstrate generalizability, we conduct additional experiments on recently released reasoning models and baselines with different model series: Qwen2-VL (7B), InternVL2.5 (8B, 38B), and Qwen2.5-VL (32B, 72B). These models are tested on four perception benchmarks and the RH-Bench. As shown in Table 1, the results from these broader models are consistent with our findings, providing strong evidence of the general applicability of our study. We will include the new results in the revision of the paper.

Table 1: Performance of recently released multimodal reasoning models on perception benchmarks and RH-Bench.

(W4) Analysis of Hallucination Across Seven Data Types. Thank you for your thoughtful feedback. We agree that performing this analysis could lead to a deeper understanding; therefore, following the suggestion, we conduct an analysis comparing the average hallucination rates of reasoning models to baseline models across 7 categories of hallucination. As shown in Table 2, the results indicate that most reasoning models exhibit higher hallucination rates across different hallucination types, aligning with our previous finding. Specifically, reasoning models demonstrate comparatively higher hallucination rates in tasks like existence, color, counting, OCR, and Chart & Doc, and the gap is larger on OCR and Chart & Doc, which might require more advanced visual capability.

Table2: Hallucination rates (%) of reasoning models across different hallucination types.

[1] R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimization

[2] MM-Eureka: Exploring visual aha moment with rule-based large-scale reinforcement learning

[3] MM-Eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning

[4] Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models

[5] VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning

Regarding generalization, our comprehensive evaluation of existing multimodal reasoning models is highly consistent with the findings in this paper. Specifically, as shown in Table 2, models trained with RL-only methods, including the recently proposed VL-Rethinker and Vision-R1, generally outperform SFT+RL trained models on the key metric RH-AUC. This suggests that RL-only training may be more effective at optimizing the reasoning-perception balance. Furthermore, as shown in Table 3, the recently released MM-Eureka-Qwen surpasses Vision-R1 (trained on 70k samples) despite using a smaller dataset focused on reasoning data. This indicates that simply increasing the training data scale does not guarantee a better reasoning-perception balance. The results in Table 4 consistently show a positive correlation between model size and the reasoning-perception balance. Finally, as indicated in Table 5, all current multimodal reasoning models suffer from an amplified hallucination problem in perception tasks.

We believe these new results provide strong empirical evidence that our framework is efficient and generalizable. Thank you again for your time and valuable insights. We hope that this new body of evidence and our detailed clarifications will be helpful in your final assessment of our work. We would be very grateful if you would consider these new results in your final evaluation.

Table2: Comparison of RL and SFT+RL training paradigms on RH-AUC.

Thank you for your continued engagement and for raising these important questions. We sincerely appreciate the dialogue, as your feedback is helping us to sharpen the positioning and contribution of our work. We would like to address your remaining concerns.

Regarding the concerns about exploring a wider range of model costs, we would like to point out that the cost of evaluating a reasoning model's RH-AUC on our proposed RH-Bench is similar to performing inference of the model on the same benchmark dataset, as our method does not require extra training of the model, and different length of reasoning are independant and therefore can be parallelarized. As shown in Table 1, we compare the computation time of RH-AUC (with different length-control methods) with the total inference time of the original model on RH-Bench. The results clearly show that the average computation time of our steering method (LSS) for RH-AUC differs only marginally from the inference time of the original model on RH-Bench, while demonstrating greater efficiency compared to the other two control methods.

Table1: Comparison of inference time (s) under various reasoning control methods on the RH-Bench for RH-AUC evaluation. Three reasoning length control methods: Token Budget Forcing (Budget), Test Time Scaling (Scaling), Latent State Steering (LSS). Original refers to the model's normal reasoning.

Table5: Comparison of reasoning and non-reasoning models on four perception benchmarks.

Table3: Comparison of RH-AUC under diverse training data scales and types.

Table4: Comparison of RH-AUC across model sizes including 3B, 7B, 8B, 32B, 38B, and 72B.

We thank the reviewer for the valuable feedback, and for recognizing the importance of our proposed problem, the insightful findings provided by our systematic analysis, and the clarity and readability of our paper’s presentation. We sincerely hope our response below can address your concerns.

(W1) Discussion with Relevant Works. Thank you for highlighting these relevant works. We will include the suggested discussion in the revised manuscript. While we indeed follow the attention analysis approach from prior work, the focus of our paper is different. We are not attempting to explain the general source of hallucination in language models; rather, we aim to explain how hallucination in reasoning language models is amplified. Specifically, our attention analysis demonstrates that reasoning models produce longer reasoning chains, which consequently leads to reduced attention toward visual inputs.

(W2) Concrete Technical Contribution. Thank you for bringing up this point. We indeed proposed several techniques to balance the hallucination and reasoning ability of multimodal reasoning language models. Specifically, the Latent State Steering method (mentioned in Sec. 4.1 of the manuscript) is designed to mitigate hallucination while maintaining reasoning ability (see detailed analysis in table3 below) by dynamically adjusting latent representations to control the reasoning chain length. As demonstrated in Fig. 9–11 in the Appendix, our approach effectively reduces hallucinations without sacrificing the original reasoning performance of the model.

(W3 & Q2) Empirical Support for Training Paradigms Claims. Thank you for your valuable suggestion. We agree that some more direct validation would make the argument stronger, and we demonstrated some preliminary analysis in the paper: in Fig. 8b, we demonstrate that SFT+RL may increase reasoning length. Moreover, we illustrate differences in reasoning patterns between these paradigms through a case study presented in Fig. 8c. To further investigate the difference in reasoning behavior between SFT and SFT+RL models, we compare the average token length and the frequency of reasoning-related vocabulary in responses from two model paradigms. As the results in the Table 1 and 2 indicate, SFT+RL models generate significantly more redundant responses than RL models, which aligns with our findings.

Table1: Average response token length.

Table2: Frequency of reasoning related words evaluated on MathVision.

(W4 & Q1) Validation and Analysis of Latent State Steering. Thank you for this suggestion. We would like to clarify that our proposed steering method does not compromise the model's natural reasoning capabilities. Based on the results presented in Table 3–4 below, we evaluate the performance of three length control methods: Token Budget Forcing, Test Time Scaling, and Latent State Steering on RH-Bench. The results show that when controlling for the same length range, the differences in the impact of these methods on model performance in perception and reasoning tasks are minimal. This suggests that the observed performance variations are mostly due to the reasoning length, rather than the specific control method. Furthermore, our steering method uses the hyperparameter $\alpha$ to control output lengths, where setting $\alpha=0$ allows the model to reason without intervention.

Table3: Accuracy (%) comparison on RH-Bench across three reasoning length control methods: Token Budget Forcing, Test Time Scaling, and Latent State Steering (LSS).

Table4: RH-AUC scores across different models under three reasoning length control methods.

(W5 & Q3) Interpretability Analysis and Evaluation Advantages of the RH-AUC. Thank you for your insightful comments. We would like to clarify that RH-AUC is a metric designed to assess the interaction between perception and reasoning across varying reasoning lengths, reflecting an overall balance. This metric is obtained by plotting perception accuracy (y-axis) against reasoning accuracy (x-axis) at various reasoning lengths. As reasoning length increases, perception accuracy typically declines due to diminished visual attention (as shown in Figures 5–6 of the manuscript). Therefore, the area under this curve captures the overall balance between perception and reasoning.

The advantage of RH-AUC over traditional metrics lies in its ability to assess the interaction between perception and reasoning, rather than just measuring their individual accuracy. A simple average of perception and reasoning accuracy inevitably masks their critical interaction under various reasoning lengths. For example, a model with 0.8 perception accuracy but 0.2 reasoning accuracy, and another model with both scores at 0.5, would have the same average (0.5). However, the second model shows better balance. Traditional metrics fail to reflect this, while RH-AUC effectively evaluates the balance between perception and reasoning by testing performance across different reasoning levels.

(Q4) Further Analysis of Error Proportions in Reasoning Models. Thank you for bringing up this point. The distinction between reasoning and perception errors in Appendix Figs. 6a is as follows: perception errors occur at the initial stage of the response, showing wrong identification of visual content (e.g., misidentifying three dogs as two in an image). In contrast, reasoning errors arise during the reasoning after the model has correctly perceived the content (e.g., accurately recognizing the spatial relationship between a red car and a tree but drawing an incorrect answer about whether the car is parked under the tree). The increase in the proportion of reasoning errors further supports our finding that the decline of attention with longer reasoning chains is the key factor exacerbating hallucination, rather than issues in early-stage visual information processing.

[1] MM-Eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning

[2] VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning

Thank you for the authors' detailed and thoughtful response. I have carefully read the rebuttal and appreciate the clarifications provided. The responses to W4 and W5 are helpful and seem to address the concerns raised.

Regarding W1, while the authors’ explanation clarifies the contribution, I still feel that the novelty is somewhat incremental. For W2, I would note that latent space steering has been explored in prior works within the LVLM domain (e.g., [1]), and may not be entirely novel. As for W3, I share reviewer cecd’s concern that the conclusions drawn about the training paradigm may not be entirely robust.

[1] Reducing Hallucinations in Vision-Language Models via Latent Space Steering

We sincerely thank the reviewer for the thoughtful feedback and for carefully engaging with our rebuttal and would like to take this opportunity to clarify the core contributions and positioning of our work. Our work is the first, to our knowledge, to uncover and systematically analyze the phenomenon of hallucination amplification in multimodal reasoning models. This issue, where increasing reasoning length exacerbates hallucinations, has not been previously studied in the literature, and we believe it addresses an urgent and underexplored challenge in the development of reliable vision-language systems. The significance and timeliness of this problem have been positively recognized by reviewers kC9o, cecd, and qZ13.

To study this issue rigorously, we introduce a comprehensive evaluation framework, including novel metrics such as RH-AUC and the RH-Bench diagnostic benchmark. These tools enable precise measurement of how reasoning interacts with visual grounding, providing actionable insights into the reasoning-perception tradeoff. The utility and novelty of this framework have been acknowledged by reviewers qZ13 and cecd. We are also the first to propose multiple techniques for controlling the reasoning length in multimodal reasoning models to investigate the impact of varying reasoning lengths on the balance of reasoning models.

(W2) We sincerely thank the reviewer for mentioning the work in [1]. We'd like to clarify that our method has a different focus from that of [1]. Our steering method focuses on dynamically adjusting reasoning length to investigate its effect on the reasoning-perception balance. Through this method, we analyze how changes in the length of the reasoning chain impact a model's performance stability on both perception and reasoning tasks. In contrast, the method in [1] primarily addresses the single issue of mitigating hallucinations in multimodal models. Furthermore, we have already cited [1] in our paper and will include a more detailed discussion in our revised version.

(W3) We understand the reviewer's concern. We would like to clarify that our paper does not make definitive conclusions about training paradigms. Instead, based on systematic observations of the performance of existing reasoning models, we present analytical findings. In the paper, we used terms such as “may” and “not always” to maintain caution. While these findings are not conclusive, they provide a solid foundation and clear direction for future research.

[1] Reducing Hallucinations in Vision-Language Models via Latent Space Steering