Robust Multimodal Large Language Models Against Modality Conflict

We truly appreciate your positive assessment of our paper. We are also grateful for the time and effort you invested in reviewing it.

We appreciate your insightful comments and suggestions. We give a point-by-point response to each of your concerns below. Following the ICML 2025 Peer Review FAQ, we post all additional results to the anonymous link: https://anonymous.4open.science/api/repo/11639-F609/file/Additional_Results.pdf?v=59a00305 for your reference.

1. Other Strengths And Weaknesses

Thanks for your valuable suggestions. We will add discussions about these papers your mentioned in the revised manuscript.

2. The analysis lacks depth in disentangling the root causes. Fundamental questions remain unanswered: Does hallucination primarily stem from the model's tendency to silently correct user query errors? Or does it originate from vision encoders' failure to capture fine-grained visual details?

Thank you for your insightful comments. Our work aims to uncover a fundamental source of hallucination, and while detailing every causal pathway is beyond our current scope, we conduct additional analysis based on our findings.

Firstly, we assert that the hallucination issue does not primarily arise from the model's inclination to silently correct user query errors. In our experiments utilizing a prompt engineering baseline, where the model is instructed to first ascertain the image content before addressing any queries based on that content, hallucinations still occur. In these scenarios, the model is expected not to correct any user errors, yet it frequently misrepresents image content.

Secondly, concerning the vision encoder's capability, MLLMs are generally pre-trained on extensive visual data [1,2], endowing their vision encoders with robust capabilities for recognizing fine-grained visual details. This is supported by their performance on standard benchmarks such as MMBench and MMMU, where the vision encoder has demonstrably captured intricate visual details successfully. Thus, it seems unlikely that hallucinations are due to a failure of the vision encoder in capturing these details.

Finally, we propose that the hallucination may stem from the model's inclination to prioritize certain data modalities when faced with conflicting information. Given the prevalent use of instruction-tuning in training MLLMs [1,2], models might develop a bias towards textual instructions over visual data. This predisposition can lead to hallucination, as the model may overly rely on textual information. We intend to incorporate this discussion in the revised manuscript and plan to conduct more rigorous analyses as part of future work.

[1] Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. In Proceedings of the Advances in Neural Information Processing Systems, 2023.

[2] Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., Fan, Y., Dang, K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., and Lin, J. Qwen2-VL: Enhancing vision-language model’s perception of the world at any resolution. arXiv:2409.12191, 2024.

3. The proposed solutions (prompt engineering, SFT, RL) appear as direct adaptations of existing techniques rather than modality-conflict-specific innovations, which need stronger justification.

The main focus of our work is to reveal the source of hallucination. Most potential modality-conflict-specific innovations would fall into the category of PE, SFT, RL, and the decoding-based method we just supplemented according to other reviewer's suggestion. Furthermore, the prompt template of PE and reward function of RL in this paper are specifically designed to tackle the modality conflict.

In spite of this, we are willing to provide some directions for improvement:

1. Incorporate external tools to detect the information in the image and text, and then use the detected information to guide the model to generate the answer.

2. Construct more fine-grained data, e.g., annotating modality conflict by human, and then use the data to train the model.

We hope to attract more researchers to work on this challenging problem and provide deeper insights into the modality conflict.

4. The evaluation should include conflict-type-specific performance breakdowns, standard benchmark comparisons pre/post training, or maybe chain-of-thought prompting baselines.

We supplement the conflict-type-specific performance breakdowns in Table 2, 3 and 4 of the above linked file. The results show that the model is more prone to hallucinate on Attribute and Relationship conflicts than Object conflicts. Chain-of-Thought prompting baselines are also implemented and compared with our proposed methods, as shown in the linked file.

We sincerely appreciate your thoughtful feedback. If our responses have adequately addressed your concerns, we would be grateful if you could consider raising your score. Thank you once again for your time and effort in reviewing our work.

We are delightful to see your positive remarks on our proposed research topic and experimental designs. We provide discussions about your suggestions as follows. All related results are available at the anonymous link https://anonymous.4open.science/api/repo/11639-F609/file/Additional_Results.pdf?v=59a00305.

1. Some visualization of the proposed dataset is needed so that readers can quickly know the data distribution of the dataset.

We have plotted the distribution of conflict types, and word clouds for text of each type, shown in Figure 1 and 2 of the above linked file. We will supplement these visualizations in the updated manuscripts.

2. Which LLM did the authors use as Judge? Is it aligned to human evaluation? How about the evaluation results of different open-weight LLM and commercial LLM such as GPT-4o and Claude?

We use GPT-4o-mini for Hallu-Rate and GPT-4o for LLM-Judge. These choices are based on the capacity of the models and the budget of the project. Both models are demonstrated to be well-aligned with human evaluation [1] and widely used in the community [2,3]. Since the judgement of Hallu-Rate is a binary classification task, GPT-4o-mini is enough to provide a reliable evaluation while saving computational resources. Judgement of LLM-Judge is a more fine-grained task, and GPT-4o is more suitable for this task.

To address your concerns, we also evaluate the results using an open-weight LLM, Llama-3.3-70B-Instruct. The evaluation results, shown in Table 5 of the above linked file, illustrate consistent conclusions with GPT-4o-mini and GPT-4o. We will include these results in the updated manuscript.

[1] Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems Track on Datasets and Benchmarks. Curran Associates, Inc., 2023.

[2] Guan, T., Liu, F., Wu, X., Xian, R., Li, Z., Liu, X., Wang, X., Chen, L., Huang, F., Yacoob, Y., Manocha, D., and Zhou, T. HallusionBench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2024.

[3] Yu, W., Yang, Z., Li, L., Wang, J., Lin, K., Liu, Z., Wang, X., and Wang, L. MM-Vet: Evaluating large multimodal models for integrated capabilities. In Proceedings of the International Conference on Machine Learning, 2024.

We appreciate your insightful comments and suggestions. We give a point-by-point response to each of your concerns below. Following the ICML 2025 Peer Review FAQ, we post all additional results to the anonymous link: https://anonymous.4open.science/api/repo/11639-F609/file/Additional_Results.pdf?v=59a00305 for your reference.

1. In lines 302 - 303, the paper mentions "Prompt engineering on Qwen2-VL-Instruct series brings significant improvement", but PE doesn't work well on the Qwen2-2B model as the Hallu-Rate increases

We are sorry for the inaccurate description, and we will rewrite the claim as "Prompt engineering brings significant improvement to Qwen2-VL-Instruct-7B, but increases the Hallu-Rate of smaller Qwen2-VL-Instruct-2B model.". And we will carefully check the results and revise the manuscript accordingly.

2. Given the instability of the approaches, particularly for RL, it would be beneficial to report averaged results across multiple seeds for each method to ensure robustness.

We have rerun our SFT and RL approach across all models across three different seeds and computed the mean and standard derivation of these results, shown in Table 1, 2, 3 and 4 of the above linked file. The averaged results demonstrate consistent conclusions with the original results. We will include these results in the updated manuscript and carefully review the results to ensure the robustness of our findings.

3. The MMMC dataset consists of Object, Attribute, and Relationship Conflicts, yet the paper does not provide separate performance analyses for each category. Reporting these results would offer deeper insights into how well each approach handles different types of conflicts.

We analyzed the performance of each approach on Object, Attribute, and Relationship conflicts separately, as shown in Table 2, 3 and 4 of the above linked file. We find that results on each separate conflict types lead to similar conclusions as the overall results. However, MLLMs seem to be more prone to hallucinate on Attribute and Relationship than Object conflicts. This phenomenon may be due to the unbalanced training data distribution, as shown in Figure 1 of the above linked file, or the more abstract nature of Attributes and Relationships. We will include these results and related discussions in the updated manuscript.

4. A more rigorous evaluation, particularly on unseen domains, would strengthen the study. The construction of the current test set split is unclear, but to properly assess robustness, it should include samples with novel images, objects, attributes, and relationships to evaluate generalization beyond the training data.

We split the training and test set based on the image source, ensuring that the test set contains unseen images. Due to the multi-modal input nature of this task, we consider a input image-text pair as a unseen sample if either the image or the text is unseen. And thus the evaluation setting is able to reflect the generalization ability of the model. We will clarify this in the revised manuscript.

5. The paper would benefit from a broader set of baselines for hallucination mitigation, such as decoding-based methods for comparison.

We experimented with a strong decoding-based baseline, SID [1]. To ensure the correctness of our implementation, we run the original code provided by the authors of SID and test on LLaVA-v1.5-7B and InstructBLIP-7B. We also implement methods in our paper on the same models for comparison. The results, as shown in Table 1 of above linked file, indicate that the decoding-based SID gains comparable performance with prompt engineering baselines. We will include these results and discussions in the updated manuscript.

[1] Huo, F., Xu, W., Zhang, Z., Wang, H., Chen, Z., and Zhao, P. Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models. In Proceedings of the International Conference on Learning Representations, 2025.

6. Relation To Broader Scientific Literature

We argue that modality conflict is a cause, rather than a special case of the later scenarios. In other words, we propose the concept of modality conflict to illustrate a source of hallucination, rather than describe the hallucination itself as previous works do. We will clarify this in the revised manuscript.

7. Other Comments Or Suggestions

Thank you again for these valuable suggestions. We will separate second paragraph of Sec 5.1 as a standalone section titled "Hallucinations in MLLMs" and convert Figure 3 to a table in the updated manuscript.

We sincerely appreciate your thoughtful feedback. If our responses have adequately addressed your concerns, we would be grateful if you could consider raising your score. Thank you once again for your time and effort in reviewing our work.