MIRAGE: Assessing Hallucination in Multimodal Reasoning Chains of MLLM

Thanks for your insightful suggestions. We hope this response can address your concerns.

Weakness 1: The dataset contains only single-image questions; no multi-image, video or temporal reasoning is tested.

Thank you for this valuable suggestion regarding the scope of our benchmark. We agree that multi-image and temporal reasoning are important future directions for hallucination research. And we acknowledge that no multi-image, video or temporal reasoning data is the limitation of our paper, as stated in the Appendix G (Limitation). Our primary goal with MIRAGE was to first establish a foundational and high-quality benchmark for reasoning chain hallucinations in the prevalent single-image context. This deliberate focus allows for a deep and controlled analysis of a core challenge in MLLMs. For scenarios that can be adapted, we propose a practical method of concatenating multiple images into a single composite image and annotating object indices, effectively converting a multi-image problem into a format compatible with the current benchmark. Looking ahead, we are committed to extending MIRAGE to include dedicated multi-image and video data in future work to explore hallucinations in more complex temporal and multi-view scenarios.

Weakness 2 and Question 3: LLM-as-a-judge raises concerns about circularity, bias, and stability across evaluator versions.

This is a critical point, and we appreciate the opportunity to elaborate on our comprehensive, multi-pronged strategy to ensure the reliability and validity of our LLM-based evaluation. Our approach is built on three pillars.

First is mitigating circularity. To prevent self-enhancement bias, we intentionally use an older, isolated model version (gpt-4o-2024-08-06) as our primary evaluator, ensuring it has not been trained on the models or data it is evaluating. For open-source evaluators, we designed highly structured, specialized prompts (see Appendix) to constrain the evaluation and minimize potential bias.

Second is human-in-the-loop verification to double-check the validity of LLM-as-a-judge. We do not rely solely on automated metrics. We conducted manual double-checking for our key metrics (factuality and LHS) on representative models. The results demonstrate a strong alignment between our automated scores and human expert judgment. Specifically, for factuality scores in the following table, the automated scores closely match the manually verified scores, confirming their accuracy.

And for LHS, we analyze the absolute difference between automated and manual LHS scores. The results show that the metric is highly stable, with over 75% of samples having a negligible difference of less than 0.1.

Besides, we further validate our framework by calculating the correlation between all metrics in Table 5. The high correlation scores across the board indicate strong internal consistency and robustness.

We believe this rigorous, three-tiered validation process effectively addresses concerns about circularity and stability, ensuring that our findings are reliable. We will add a summary of this validation strategy to the main paper.

Weakness 3 and Question 1: The authors do not provide the rationale behind picking the seven distinct taxonomies for their dataset.

Thank you for this question. Our selection of the seven reasoning taxonomies was a principled decision aimed at ensuring comprehensive coverage of both broad, general-purpose reasoning and deep, domain-specific skills. For general reasoning capabilities, to evaluate skills applicable to everyday scenarios, we included tasks requiring spatial reasoning (object locations, attributes) and statistical reasoning (interpretation of charts and plots). For domain-specific reasoning capabilities, to probe more complex, multi-step logical deductions, we incorporated challenging problems from mathematics (geometry, algebra), science (physics, chemistry, biology), and formal logic (IQ-test style problems).

This structured approach, which balances breadth and depth, is well-aligned with the categorization frameworks used in recent comprehensive surveys on multimodal reasoning [1, 2], confirming the relevance of our chosen taxonomies.

[1] Multimodal Chain-of-Thought Reasoning: A Comprehensive Survey, arXiv 2025

[2] Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models, arXiv 2025

Weakness 3 and Question 2: Section 5 introduces Logos but does not specify which exact model it fine-tunes.

Thank you for pointing out this omission. We apologize for not including these details in the main text. For Logos-7B, the pretrained model is Qwen-2.5-VL-7B-Instruct; and for Logos-3B, the pretrained model is Qwen-2.5-VL-3B-Instruct. The visual encoder is frozen to avoid catastrophic forgetting of visual perception ability. During training, we collect 13K mathematical questions with K12-level difficulty and \sim 1K text-only math questions from LIMO as training data. The batch size is 128. For each training sample, the rollout samples G is 8 by default. The initial learning rate is 1 \times 10^{-6}, both warmup strategy and cosine learning rate scheduler are adopted to stabilize training. We optimize Logos by 10 epochs using AdamW during each stage. The number of CRFT stages is set to 1, which is discussed in the ablation study. Benefiting from the filtration mechanism in CRFT and ORF, the total training time of Logos-7B is less than 24 hours, Logos-3B is faster. We will add these crucial implementation details to the main paper for clarity and reproducibility, while a complete list of hyperparameters has been stated in Appendix B.

Thanks for your insightful suggestions. We hope this response can address your concerns.

Weakness 1: detailed breakdown of the initial data sources

We agree that transparency of data source is crucial. Our strategy is to curate a diverse dataset from high-quality, specialized sources, each chosen to target a specific reasoning skill:

Mathematical Reasoning: We sourced questions from MathVista and MathVision, which are renowned for their high-quality, complex geometric diagrams and mathematical problems.

Logical Reasoning: To assess formal logic, we drew from the official question banks of national civil service examinations, which feature rigorously designed IQ-style test questions.

Scientific Reasoning: We utilized questions from prestigious science exams (e.g., USA Biology Olympiad), which require domain-specific reasoning in science subjects.

Spatial Reasoning: Our data comes from established traffic-related datasets and the synthetic SuperCLEVR dataset, which emphasize the reasoning of complex spatial relationships.

We ensure the inherent difficulty of initial data by combining empirical validation with source selection. For public benchmarks (e.g., MathVision), we confirm their challenge by observing that even powerful MLLMs (e.g., Qwen2.5-VL-7B) achieve relatively low accuracy (25.4%), which provides a strong empirical indicator of their difficulty. For crawled data, we curate questions from sources renowned for their complexity, e.g., Physics / Biology Olympiads and IQ tests, which are by design created to probe the limits of logical deduction. This approach ensures the initial data poses a meaningful challenge to current MLLMs. We will list these sources in the revision.

Weakness 2: limitation in the Logos's effectiveness across all identified hallucination types.

This is a key finding of our work, and we appreciate the opportunity to elaborate on why Logos shows diferent performance on various hallucination types. The behavior stems from two distinct challenges:

Inherent Gaps in Base Models (spatial hallucination): Our analysis in Table 1 and 2 reveals that even top-performing base models have a fundamental weakness in spatial reasoning. Scaling model or data size does not significantly reduce this type of error. Since reinforcement learning in CRFT works by rewarding and reinforcing correct behaviors that the model can already sample, it cannot effectively teach a capability that is fundamentally absent. If the base model rarely generates a correct spatial reasoning path, there are few positive examples for Logos to amplify.

The Nature of RL vs. factual knowledge (factuality hallucination): For strong base models, the remaining factual errors are often subtle and stem from inherent biases in their pre-training data. The goal of Logos's RL framework is to optimize the policy for generating correct reasoning chains, not to inject new factual knowledge. If all sampled reasoning paths from the base model contain the same ingrained factual error (e.g., a slightly incorrect constant), Logos lacks a ground-truth signal to correct it.

Hence, these findings show a crucial insight: tackling spatial and factual hallucinations may require architectural innovations or new data paradigms beyond the scope of post-training methods. These results also suggest that future work might explore better pertaining data, spatial-enhanced network design and advanced framework to mitigate such hallucinations.

Weakness 3: broader range of multimodal reasoning datasets.

To show the generalizability of Logos beyond MIRAGE, we evaluate Logos-7B on three general-purpose benchmarks: MathVista, MathVision, and MathVerse. As shown in the table, Logos-7B not only surpasses its strong base model but also outperforms concurrent state-of-the-art methods, achieving the highest average score. This shows that Logos to mitigating hallucinations also enhances reasoning abilities, leading to comparable performance on academic benchmarks.

Weakness 4: Discuss Logos with DAPO and ADARFT

Thank you for raising these concurrent works. We analyze them and present a comparison of the key differences:

Comparison with DAPO: The primary distinction lies in the policy gradient loss design. DAPO introduces several strategies (e.g., clip-higher, token-level loss) optimized for text-only math problems, which often feature very long reasoning chains. Our experiments found these strong regularizations were less effective and could lead to overfitting in the multimodal context, where reasoning chains are typically more concise (see Appendix D.5). Logos employs a simpler, more robust sample-level policy gradient loss that proved more effective for multimodal training, as shown in Table 8.

Comparison with ADARFT: The key difference is the curriculum learning mechanism. ADARFT uses a static offline difficulty annotation and greedy strategy to select training batches. This approach does not guarantee that all data is utilized effectively. Logos employs a more dynamic hybrid online-offline framework. ORF ensures each batch is filled with high-value samples, while offline difficulty thresholding stages the curriculum effectively. This hybrid approach maximizes data utility and leads to better performance. To provide direct empirical evidence, we compare Logos (w/o CHI) against these methods on MIRAGE and MathVista. Logos demonstrates superior performance on both.

Weakness 5: originality claim by further highlighting how the combination and synergy of these components specifically address the unique challenges

The originality of Logos lies precisely in the synergy between its two core components, CRFT and CHI, which work together to solve a two-part problem in reasoning. The first part is CRFT. Specifically, an MLLM must learn how to reason. CRFT uses a curriculum-based reinforcement learning approach to transform a standard MLLM into a reasoning-capable model, teaching it to generate coherent, step-by-step thought processes. The second part is CHI. Meanwhile, a reasoning model needs guidance on what to reason about for a specific problem. CHI provides this, offering dynamic hints about both the problem-solving structure (h_topic) and the key information to focus on (h_question).

Crucially, these components are mutually dependent and reinforcing. An MLLM without CRFT lacks the basic ability to follow CHI's guidance. Conversely, a CRFT-enhanced model without CHI's targeted hints still struggles with the complex problems. It is a integrated system—a capable reasoning engine combined with intelligent, dynamic guidance—that allows Logos to directly and substantially reduce complex errors like logical (from 64.7% to 49.3%) and fabrication hallucinations (from 25.5% to 15.6%). This holistic, synergistic framework is the core innovation of our work.

Question 2: What measures are in place to ensure that this evaluation accurately ? Any manual double-checking of the step and claim factuality evaluations?

For the first question: Our evaluation is based on the principle that: while the explanatory text of a reasoning chain may vary, the core intermediate factual and numerical results must be consistent across all valid solutions. To accommodate reasoning variations, our evaluation script includes a "REASONABLE" tag (Fig. 9), which treats a step as correct if it is semantically equivalent to the ground truth, even if not perfect reasoning path match. This ensures we credit valid alternative reasoning paths.

For the second question: We conducted manual checking to validate our factuality scores. As the table below shows, the automated scores for two representative models align closely with those from three human experts, confirming the reliability of our evaluation protocol.

Question 3: In-depth analysis of Logose in mitigating spatial hallucinations compared to train-free and SFT methods

We conducted new experiments to compare Logos-7B against other methods specifically on spatial hallucination. The results in the following table are striking. While mitigating spatial deficits remains a significant challenge, Logos is the only method that shows a slight improvement, whereas all other methods, including training-free and SFT-based approaches, actually degrade performance. The reasons are two-fold. Firstly, training-free methods (e.g., Self-Reflection) These methods cannot introduce new knowledge. By forcing the base model, which already lacks spatial reasoning ability, to adhere to rigid prompting rules, they can amplify existing errors. Secondly, SFT-based methods (e.g., R1-OneVision) are often trained on datasets dominated by other domains (e.g., mathematics). This can cause the model to overfit to mathematical reasoning patterns at the expense of its already weak spatial capabilities. In contrast, our Logos avoids these pitfalls. The RL approach enhances reasoning without the overfitting risk of SFT, while our CHI component provides targeted hints (h_topic and h_question) that guide the model on difficult spatial problems. Therefore, Logos is the only method avoiding performance degradation.

Thanks for your insightful suggestions. We hope this response can address your concerns.

Weakness 1: For Table 6, any idea why several of the metrics got worse results for row "+hquestio"?

Based on the description, we assume that the reviewer compares CRFT + h_question only with CRFT + h_topic only in Table 6. This observation points to the nuanced and complementary roles that different hint types play within our Logos framework. The performance difference between +h_question and +h_topic stems from their distinct functions:

Specifically, the primary role of h_question is to guide the model to identify key entities and starting conditions within the question. It excels at grounding the model's initial focus but does not provide a complete, step-by-step problem-solving template or structure. In contrast, h_topic provides a high-level problem-solving schema or template. It offers a clear, structured formulation for the entire reasoning process, ensuring logical completeness from start to finish. The difference leads to performance gap in different metrics. For CRFT + h_question, it has better final accuracy (+0.3) than CRFT + h_topic, but the metrics regarding intermediate steps (i.e., F_step and LHS) are slightly lower (-0.4 and -0.01 respectively). Our Logos enhanced by CRFT is already optimized to generate complete and precise reasoning chains. When this strong base model is augmented with h_topic, it directly benefits from the clear structural guidance, making it easier to construct a perfectly formed reasoning chain. This synergy explains why +h_topic achieves slightly better scores on metrics like F_step and LHS compared to +h_question. This further validates the design of the full CHI setup that combines both h_topic and h_question to achieve balanced improvement across all dimensions.

Crucially, we want to emphasize that CRFT + h_question still significantly outperforms the CRFT only baseline across all metrics. This confirms that guiding the model's initial attention with h_question is a valuable and effective strategy.

Finally, we appreciate you drawing attention to this table. In reviewing it, we identified a typo in the final row ("+full CHI"), which should have all checkmarks as it represents the complete Logos framework. We will correct this in the revision to ensure clarity. We will also add the above analysis of hint functionalities to the paper to further clarify these results.

Weakness 2: Typo of ``Benefiting''.

Thank you for pointing this out. We apologize for this and other potential typographical errors. We will perform a thorough proofread of the entire manuscript to polish the writing in the final version.

Thanks for your constructive reviews. We hope this response can address your concerns.

Weakness 1: The number of data is very limited.

Thank you for your valuable feedback on our submission. We appreciate your insights and the opportunity to clarify several aspects of our work.

Regarding the dataset size, we acknowledge the comparison to existing benchmarks and recognize that many reasoning datasets in the literature typically have around 1,000 examples. However, MIRAGE has been specifically designed to offer a more diverse and comprehensive set of annotations, going beyond just the size of the dataset. As referenced in the latest survey [1], our dataset provides significantly more granular annotations, such as ground-truth answers, reasoning chains, hints, step-by-step claims, and detailed descriptions, which provide a richer context for evaluating reasoning tasks.

In contrast to previous works that primarily focus on answer-level evaluation, MIRAGE supports a multi-level evaluation approach, including assessments at the final answer, intermediate results, and reasoning chain levels. This additional annotation and evaluative capability are integral to improving the robustness of reasoning models by explicitly addressing reasoning hallucinations. The unique emphasis on these detailed annotations is intended to enhance the quality of the reasoning process rather than merely increase the dataset size.

We believe that this holistic approach to reasoning hallucination evaluation makes MIRAGE a valuable contribution to the community, as it provides a more comprehensive and rigorous framework for assessing the true reasoning abilities of models, not just their final answers.

[1] Multimodal Chain-of-Thought Reasoning: A Comprehensive Survey, arXiv 2025

Weakness 2: There are plenty of datasets that already has reasoning chain, like ScienceQA, mathVista, etc.Table 1 is not accurate, or please provide some more explations.

Thank you for your thoughtful feedback on the comparison of MIRAGE with other datasets. We would like to provide further clarification to address the concerns raised.

Firstly, regarding the mentioned datasets like ScienceQA and MathVista include reasoning chains, we would like to clarify that, as indicated in the concurrent survey [1], these datasets do not provide the level of detailed reasoning chains that MIRAGE does. Specifically:

1 MathVista does not contain ground-truth reasoning chains. The focus of MathVista is primarily on final answers, without explicitly providing detailed, step-by-step reasoning processes.

2 ScienceQA includes only high-level lecture hints but lacks comprehensive, fine-grained reasoning chains. The reasoning chains in ScienceQA are not sufficiently detailed or structured to qualify as ground-truth reasoning chains for the purpose of evaluation. The "lecture" annotations provided are simplistic hints, rather than the multi-step, comprehensive chains that MIRAGE offers.

In contrast, MIRAGE stands out by providing rich and multi-level annotations—including ground-truth answers, reasoning chains, step-by-step claims, and descriptions—enabling a thorough assessment of reasoning processes across multiple levels. This distinction makes MIRAGE more suitable for evaluating reasoning hallucinations, as it allows for a detailed analysis of the intermediate steps in the reasoning process, beyond just the final answer.

Moreover, MIRAGE is explicitly designed for hallucination assessment in multimodal reasoning tasks, a specific focus that sets it apart from the aforementioned datasets, which primarily judge the correctness of final answers. This emphasis on hallucination detection at various levels (e.g., final answer, intermediate steps, and reasoning chain) provides a more comprehensive framework for evaluating the performance of reasoning models, making MIRAGE uniquely positioned to address the challenges of reasoning accuracy and hallucinations.

To clarify the difference with previous works, we will add more benchmarks (including Mathvista and ScienceQA) in the revision.

Weakness 3: The contribution is limited other than the dataset.

Thank you for your insightful feedback. We would like to clarify that the contributions of our paper extend well beyond the MIRAGE dataset. While the dataset is a central component of our work, we also introduce several other key contributions that significantly advance the understanding and mitigation of reasoning hallucinations in MLLMs. These contributions are outlined as follows:

1. Insights from Experimental Results on MLLMs. Beyond the dataset, our experimental results reveal several critical insights into how MLLMs handle reasoning tasks: 1)We find that model scale, data scale, and training stages significantly influence hallucinations related to logic, fabrication, and factuality. This highlights the complexities of scaling MLLMs and the challenges they face with reasoning errors; 2)More notably, our experiments show that scaling alone does not improve spatial hallucinations, where models misinterpret spatial relationships. This suggests that current MLLMs exhibit weak visual reasoning capabilities and cannot simply overcome hallucinations by increasing model size or training data. These findings are crucial for guiding future work in hallucination mitigation strategies for MLLMs.

2. The Logos Training Framework. Our third contribution is the proposed Logos training framework, which aims to reduce multimodal logical hallucinations and improve answer accuracy. By leveraging curriculum reinforcement fine-tuning and collaborative hint inference, Logos provides a novel training paradigm that enhances model reasoning capabilities. This framework addresses key issues identified in our experiments, specifically targeting logical and factual hallucinations, and is a promising approach for future MLLM research.

We believe that these contributions—ranging from the dataset to experimental insights and a novel training framework—collectively offer significant advancements to the MLLM research community.

Weakness 4: The author mentioned hallucination types, but not properly defined.

Thank you for your feedback regarding the definition of hallucination types. We apologize for the oversight and appreciate the opportunity to clarify the terminology used in our paper.

To address this, we provide the following clear definitions for each hallucination type, as outlined in the table below:

These definitions aim to provide a clear understanding of the different types of hallucinations we discuss in our work. For further clarification, detailed descriptions of each hallucination type can be found in Appendix A and Table 7 of the paper. Meanwhile, we will also mention the hallucination type definition in the revision of the main paper.

Thank you for the rebuttal and clarifications! It addresses my concerns and questions. I will take that into account on my final rating, after seeing all of the comments and discussions.