Unsolvable Problem Detection: Evaluating Trustworthiness of Large Multimodal Models

We sincerely appreciate your time and effort in reviewing our work. Below, we provide detailed responses to each of your points. If you have other questions, we welcome further discussion!

Note: Unless otherwise specified, the mentioned line numbers denote that of the original submitted paper version instead of the rebuttal revision version.

W1. Difference in the Definition of Unsolvable Problem

Thank you for your question. The differences from existing settings [1, 2] are described in the Introduction (L64-66). While existing benchmarks primarily address mismatches between images and questions (a small partition of MM-IVQD’ abilities), they have overlooked other critical challenges such as incomplete or missing answer sets (AAD and IASD). To expand the scope of unsolvable problems, we have rigorously defined three types: AAD, IASD, and IVQD. We have also introduced the MM-UPD Bench, which evaluates all these unsolvable problems across various abilities. Our experimental results reveal performance differences among LMMs for each unsolvable problem type (Finding 5, L412-422), underscoring the necessity of comprehensive coverage for all types.

W2. Differences in Design Between Tables 3 and 4 and Rationale for Keeping Them Separate

Thank you for the question. Both Table 3 and Table 4 aim to explore baseline approaches for UPD. The reason we do not merge them is that there is a fundamental difference in the approaches themselves. Table 3 focuses on prompting-based approaches, while Table 4 deals with training-based approaches.

References

[1] Guo+, UNK-VQA: A Dataset and a Probe into the Abstention Ability of Multi-modal Large Models, TPAMI2024

[2] Akter+, VISREAS: Complex Visual Reasoning with Unanswerable Questions, ACL Findings2024

We sincerely thank you for your thorough review of our paper and your constructive feedback on how we can further advance our UPD challenge. We are particularly grateful that you acknowledge our strengths, especially “S2. The findings offer valuable insights into current MLLM’s limitations”. We have carefully considered your suggestions and updated the manuscript accordingly, with changes highlighted in orange. Below, we provide detailed responses to each of your points:

Note: Unless otherwise specified, the mentioned line numbers denote that of the original submitted paper version instead of the rebuttal revision version.

W1. Narrow Definition of Unresolvable Problem Detection

We appreciate your critical question. MM-UPD enables more accurate evaluation than open-ended questions, and we believe that conducting thorough and precise evaluation using multiple-choice questions is a crucial initial step in developing reliable LMMs. While we agree that open-ended questions are important, their evaluations are inherently challenging. Therefore, most established benchmarks use multiple-choice questions, including MMBench, MMMU [1], BLINK [2], and MMStar [3]. However, the problem with evaluating multiple-choice questions is that it is unclear whether the model truly understands the question or is just using a process of elimination, which undermines the guarantee of reliability. We believe that the multiple-choice UPD enables precise evaluation and reveals the reliability of LMMs, which inspires future efforts in this field. We have clarified this description in the revised Future Work section (Section 7).

W2.1 Anticipated Poor Performance of Existing MLLMs

We would like to point out that our findings go beyond mere performance degradation and reveal critical insights that can only be obtained through our rigorous task definition and benchmark construction. For instance, little correlation with MMBench (Table 2), the differences in the effectiveness of prompting strategy (F4, L402-411), performance trends of AAD/IASD/IVQD across MLLMs (F5, L412-422), and the variations in trends across abilities (F6, L422-431) could not have been anticipated without a meticulous problem definition, benchmark design, and establishment of evaluation metrics. We consider that these findings are not trivial and cannot be predicted without this paper.

W2.2 Lack of Innovative Approaches

We would like to emphasize that this study primarily focuses on the rigorous task design of UPD and proposing approaches is not the main contribution of this paper. For a novel task, it is essential to demonstrate the effectiveness and limitations of existing important baselines before proposing innovative methods. As Reviewer gvew also pointed out, we believe our exploration of important baselines will accelerate the community's efforts to propose innovative approaches. We have clarified the description in the Future Work Section 7.

Q1. Manual Dataset Curation Standards

Thank you for pointing out. We have added the details of the curation procedure to the revised Appendix B.5.

The dataset curation was primarily conducted by four annotators among the authors.

• To improve the efficiency of collaborative curation and ensure consistency in quality, we first transcribed the image-question pairs from MMBench into an online editing tool (i.e., Google Docs) and conducted the curation process directly within the platform.
• To enhance the consistency, each question was independently reviewed by two annotators. Finally, the lead author verified the validity of all curation.
• If a problem needed to be refined, the reason was recorded in detail as a comment. For example, in the case of IVQD, which required the most careful curation, one annotator would leave a comment on points such as “The reason the image relates to the question is...” or “If we change this image into ..., the irrelevance is guaranteed". If the other annotator agreed with the comment, the problem was refined. In cases where the other annotator disagreed, all four annotators engaged in discussions to reach a consensus.

We consider that collaborative tools such as Google Docs, double-checking by two annotators, and detailed justifications with collective decisions ensure curation consistency.

Q2. Alternative Potential Solutions for UPD

We consider that one of the promising directions is to mimic the human reasoning process with chain of thought. When humans identify flaws in a problem, they first derive their own answer and then compare it to the problem. By replicating this process, MLLMs may be able to prevent blindly outputting incorrect answers. However, the development and analysis of this new chain of thought techniques clearly goes beyond the scope of this paper. Therefore we position it as a topic for future research.

Q3. Analysis of Open-source MLLMs

Thanks for the suggestion. We have added the same analysis for open-source MLLMs in the revised Appendix E.1. We observe that InternVL2 showed high performance, and the bottleneck lies in image understanding. On the other hand, LLaVA-OV, LLaVA-NeXT, and Qwen2VL showed low performance even when given answers, and the bottleneck was found to be on the language side. This analysis result supports our finding in Table 3 that LLM-driven approaches like chain of thought and self-reflection are particularly effective for LLaVA-OV and LLaVA-NeXT. We consider these will provide valuable insights for the open-source community to develop more reliable models.

References

[1] Yue+, MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI, CVPR2024

[2] Fu+, BLINK: Multimodal Large Language Models Can See but Not Perceive, ECCV2024

[3] Chen+, Are We on the Right Way for Evaluating Large Vision-Language Models?, NeurIPS2024

[4] Yue+, MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark, arXiv2024

W3.1 Limited Benchmark Diversity: Validity of Using only MMBench

Thank you for your question. We consider that using only MMBench does not result in a limited dataset. MMBench is a benchmark specifically designed to objectively and systematically evaluate the diverse capabilities of LMMs, and it has been widely adopted as a reliable evaluation tool in numerous research papers.

In terms of dataset scale, MM-UPD includes 2,095 questions, which is comparable to other recent multi-choice benchmarks: MMMU validation (widely used in academic papers) [1] with 900 questions, BLINK [2] with 3,807 questions, MMStar [3] with 1,500 questions, and MMMU-Pro (vision) [4] with 1,730 questions.

Furthermore, we believe that the sufficiency of a dataset scale is better judged by its ability to produce meaningful insights. In this regard, the evaluation results of MM-UPD have already provided significant findings. Based on these considerations, we consider that MM-UPD offers a dataset of sufficient scale to serve as a reliable benchmark for UPD.

W3.2 Limited Benchmark Diversity: Rationale for Excluding Datasets with Low Standard Accuracy

Thank you for this important question. The reason is that evaluation results from benchmarks with low Standard Accuracy could significantly deviate from the aspect of reliability and potentially cause us to miss important findings. We have added these discussions in the revised Appendix B.6.

To verify this, we conducted experiments with MMMU in the AAD setting.

Evaluation Setup: As preprocessing, we first removed about 24.2% of image-agnostic questions from the MMMU's validation set (900 questions) using GPT-4-based CircularEval. Then, to improve the interpretability of scores, we utilized only multiple-choice questions with four options (which make up the majority of problems in MMMU) and created MMMU-AAD using the same pipeline as in the paper. MMMU-AAD consists of 459 problems. For the evaluation of MMMU-AAD, we applied the same CircularEval strategy as used in MM-UPD.

---

※ The reason GPT-4o's Original Standard performance is lower than its Base Standard is that GPT-4o generates extensive long reasoning for challenging datasets like MMMU, solving problems with a chain-of-thought process. However, this arises from GPT-4o's proprietary tuning strategy and this is unrelated to UPD. Therefore, we omit it from our discussion here.

---

Based on these results, in contrast to MM-UPD, we could not verify the efficacy of either the Option or Instruction approaches. This result reveals that the evaluation using MMMU fails to capture important findings of the effectiveness of these prompting approaches for UPD. Specifically, for expert-level problems, MLLMs do not have accurate answers due to the lack of capability. Therefore, even if it chooses an incorrect option when encountering an unsolvable problem, this only indicates a lack of reasoning ability or knowledge and does not necessarily demonstrate a lack of refusal ability. Additionally, due to the very low overall performance, it becomes difficult to have meaningful discussions based on these minute differences in scores. Therefore, we exclude datasets with low Standard accuracy. However, as the model's capabilities improve much more, it will become increasingly important to evaluate from multiple perspectives using more complex datasets, as pointed out by the reviewer. We consider this will be a challenge for future research.

We sincerely appreciate your thoughtful feedback and valuable insights.

As stated in the Future Work section, we acknowledge the importance of addressing more general tasks, including open-ended questions.

Nevertheless, our multiple-choice UPD serves a valuable purpose—it introduces a novel dimension of reliability assessment within established multiple-choice formats and offers an effective way to measure robust understanding.

Through our carefully designed problem framework, we have demonstrated that models exhibiting poor performance on UPD tasks indicate a lack of prediction reliability, while those performing well demonstrate robust understanding and high reliability. Although many established benchmarks adopt multiple-choice formats [1, 2, 3, 4], they lack methods to assess this depth of understanding.

Furthermore, our benchmark, based on MMBench, accurately evaluates the refusal capability axis, providing precise feedback to enhance reliability (refer to the response to W3.2).

Through UPD’s systematic design and rigorous evaluation framework, we believe our UPD can advance efforts to improve LMM reliability.

W4. Skewed Distribution of MM-IVQD

Thank you for your important question. We consider the current distribution of IVQD questions naturally emphasizes abilities where IVQD situations are more likely to occur, reflecting a natural skew in the IVQD setting. We have added the following discussion in Appendix B.4.

To align with the AAD/IASD distribution, it is possible to increase the number of questions by making the general question (e.g., "Which one is the correct caption of this image?" in #15 Image Topic or "What will happen next?" in #14 Future Prediction) more specific. However, these question types are inherently less likely to encounter IVQD situations, and there is a concern that forcibly modifying the questions might lead to a divergence from real-world IVQD distribution. Moreover, incorporating numerous question types with low IVQD frequency could overshadow the significance of question types that are more likely to occur, thereby compromising the accurate assessment of IVQD performance. Therefore, we chose to exclude these questions rather than modify them.

W5. Lack of the Original Performance Report after Training

Thanks for pointing it out. We have added the Original Standard accuracy after training to the table in the updated manuscript. The score is the following.

LLaVA-NeXT-34B

LLaVA-NeXT-13B

The result shows that the Original Standard performance has decreased after training. Developing a training recipe that maintains the Original Standard performance while increasing refusal ability is a challenge for future work.

Q. About Incompatible Question Detection

Thanks for the interesting question. The primary reason for not incorporating this setup is that the cases where the image and answer set are aligned are limited compared to the three challenges defined in our paper. We consider that ensuring a state where the image and answer set are aligned is challenging and such alignment is primarily limited to specific tasks like image captioning. In most questions, the scenario involves a misalignment between image and answer sets, which can be interpreted as a combination of IASD and IVQD. We judged that analyzing the findings from the combined setting of IASD and IVQD would result in findings similar to those of individual IASD and IVQD. Therefore, considering the practical infrequency of such scenarios and the likelihood of similar findings, we decided not to define this problem setting as a UPD task.

Minor 1. Fix the typo: withcurrent (line 135).

Thanks for pointing out. We have fixed this typo.

Minor 2. Consider reformatting Figure 2 and increasing the text font size for improved readability.

Thank you for pointing out. We will reformat Figure 2 and increase the text font size in the final version.

We sincerely thank you for your insightful and constructive feedback. Your comments are highly valued and critical to help make our paper more robust. We are particularly grateful that you correctly understood the strengths of this paper, specifically noting "S2. comprehensive experiments and analysis", and "S.2. Insightful evaluation of key baselines methods".

We have carefully considered your suggestions and updated the manuscript accordingly, with changes highlighted in magenta. Below, we provide detailed responses to each of your points:

Note: Unless otherwise specified, the mentioned line numbers denote that of the original submitted paper version instead of the rebuttal revision version.

W1. False Exclusions and GPT-derived Bias in Image-agnostic Question Filtering

We appreciate your insightful question on our filtering process. We conducted an investigation and confirmed that our filtering process does not introduce bias in the evaluation of GPT-based models. We have added the following detailed filtering process and the investigation in Appendix B.1.

To eliminate the effect of random guessing, we applied CircularEval, which has proven effective in eliminating random guessing in MMBench, for filtering. As L307-308, CircularEval cyclically shifts the answer options for each question and considers a prediction a success only if GPT-4 correctly identifies the answer for all shifts. Only 11% (124/1164) of the questions were excluded, which is significantly lower than the rate of random guessing (approximately 30%). This indicates that erroneous exclusions due to random guessing were minimized.

Additionally, to investigate GPT-based biases, we thoroughly examined all the 124 questions excluded by GPT-4. As a result, we found that 110 of 124 were questions that could be answered using only the question texts. The remaining 14 questions appeared image-specific but could be answered by GPT-4 using information from its training, such as the frequency of words in the answer options. However, these 14 questions were primarily limited to a small portion of common questions such as "Which Python code can generate the content of the image?" in #13 or "Which term matches the picture?" Therefore, the impact of removing these 14 questions is considered to be negligible. We confirmed that our MM-UPD does not have GPT-4 bias.

W2. Effect of GPT-based Filtering on Manual Effort

Thanks for the question. We consider that GPT-4-based filtering allows for efficient and precise removal of image-agnostic questions. Although we needed to manually review the remaining dataset, initial GPT filtering efficiently reduced oversights compared to checking all questions by hand. Our manual double-checks revealed that GPT-4 with CircularEval performed well. The few questions that GPT-4 could not eliminate were mostly limited to the query on direction (e.g., "What direction is France in the Mediterranean Sea? A: east, B: south, C: west, D: north"). Therefore, manual removals require minimal effort during our final checks.

W3. Dual Accuracy's Limitations in Fair Model Comparison

Thank you for your insightful suggestion. We have added experiments comparing the metrics suggested by the Reviewer and Dual accuracy, and we have confirmed that Dual accuracy is effective for the evaluation. The following discussion has been added in the revised Appendix D.1

We interpret the evaluation metric proposed by the reviewer as the Original Conditional Dual accuracy (OC-Dual) score: OC-Dual = (Success in all Original Standard, Standard, UPD settings) / (Success in Original Standard). If we use Standard instead of Original Standard as the denominator, models like CogVLM with instruction (Standard: 5.4%, UPD: 92.3% from Table H) that reject most questions could achieve high SC-Dual scores. Therefore, we used the Original Standard as the denominator.

The analysis in the revised paper revealed a very strong correlation between the two metrics. This is attributed to the fact that the Original Standard performance of current LMMs shows little variation within the MM-UPD Bench. The OC-Dual score considers the performance under the successes in the original setting. Therefore, even LMMs with very low Original Standard performance might achieve high scores, potentially leading to a gap between the score and practical usability. Given the weakness of OD-Dual, using the Dual accuracy for MM-UPD is the most effective to precisely assess the reliability of state-of-the-art LMMs without compromising real-world applicability.