EXPLORING RESPONSE UNCERTAINTY IN MLLMS: AN EMPIRICAL EVALUATION UNDER MISLEADING SCENARIOS

We greatly appreciate the thorough evaluation of our work and the valuable suggestions provided by the reviewers. Your feedback has been instrumental in helping us improve the clarity and comprehensiveness of our manuscript. In response to your concern, we conducted additional experiments and revised our analysis accordingly.

Q1: The paper lacks a discussion on whether the models are calibrated.

A1: Thank you for your valuable suggestion. We have added a discussion in the appendix to address the calibration of the models. This discussion provides an analysis from three key perspectives:

• ECE. We adopted the method described in [1] to enable the MLLMs to output their confidence scores. The results demonstrate a notable improvement in calibration, with the Expected Calibration Error (ECE) in the True-False (T-F) scenario decreasing from 0.47 to 0.23 after fine-tuning.
• Consistency Rate. The consistency rate of the fine-tuned MLLMs shows significant improvement. Specifically, we observed an approximate 13.58% increase in low misleading rate data and a 26.51% increase in high misleading rate data, as detailed in the revised Table 21.
• Accuracy. The accuracy of the fine-tuned MLLMs also improved moderately, with an approximately 5% increase observed after fine-tuning. These results are reported in the updated Tables 17 and 18 in the revised manuscript.

Table: Comparison of ECE before and after fine-tuning on our benchmark

Table: Comparison of consistency before and after fine-tuning on our benchmark

[1] Xiong, M.,et al. 2023. Can LLMs express their uncertainty? An empirical evaluation of confidence elicitation in LLMs.

Q2: It does not address whether more consistent (more certain) model outputs correspond to more accurate answers.The results in the paper are primarily based on misleading rate (MR) and average consistency rate (ACR), without showing metrics like model accuracy. There is a lack of analysis on utility.

A2: Thank you for your valuable suggestion and for pointing out this important aspect of the analysis. We appreciate the opportunity to clarify and expand on this point. In the appendix of the original manuscript, we presented the changes in accuracy after fine-tuning in Table 22 on page 26 (updated as Table 17 and Table 18 in the revised version). To further address your concern and provide a clearer illustration of the fine-tuned model’s effectiveness, we have included the accuracy metrics directly in Table 2 of the main paper. We also provide the relationship between the accuracy and the misleading rate in Figure 10(in the revised version). The results indicate an inverse relationship between the misleading rate and the accuracy, where a higher misleading rate corresponds to a lower consistency rate. Also, we

Table: the average accuracy before and after fine-tuning on our benchmark

Q4: Instead of finetuning, I would recommend the authors to simply systematically prompt the MLLMs, such as "The questions might contain misleading information, you should try to answer the question correctly despite of those misleading information ..."; another version could even give it two examples (one explicit and one implicit). I would guess/assume, simply doing this extra prompting will make the results much better.

A4: Thank you for your thoughtful suggestion. We conducted additional evaluations incorporating explicit and implicit misleading examples as part of systematic prompting strategies. The results indicate that even when the instructions explicitly warned the model about the presence of misleading information, the misleading rate remained above 70%. Similarly, providing explicit and implicit misleading examples in the instructions resulted in a misleading rate of approximately 70%. Although these strategies reduced the misleading rate by about 15%, the performance remains unsatisfactory for large models. The specific prompt and complete table can be found in the revised version paper, Table 22.

Table: The average misleading rates of different explicit and implicit prompt-based defense strategies on 12 open-source MLLMs. (Warning means your first example. And different explicit and implicit instructions are shown in Appendix.A.2.4)

Q5: The questions only include multi-choice and T/F styles, which certainly makes the metrics calculation easier (reflected in equations 1 and 2), yet probably losing the delicacy in the type of Q/A addressed?

A5: Thank you for your suggestions. To comprehensively evaluate the effectiveness of our method, we transformed the discriminative task into a generative task. Specifically, we randomly sampled 200 data points prone to misleading. Only the images and questions were provided, without answer options. GPT-4o was used to evaluate the generated text and corresponding answers, and the misleading rates were calculated before and after introducing misleading information. (Note that the 200 data points prone to misleading were selected from external datasets. In contrast, using other unscreened generative task datasets does not effectively identify data points that are highly susceptible to misleading.) Complete table can be found in the revised version paper, Table 27.

Table: Comparison of explicit and implicit misleading instruction performance on generative tasks before and after fine-tuning

Q3: During finetuning, a random set of explicit and implicit misled samples are used for finetuning, yet I am afraid the explicit misleading info has a too obvious and unique pattern due to how it's designed, hence too easy to pick them up, making the improvement after finetuning not too surprising.

A3: Thank you for raising this insightful concern. We have analyzed the results for explicit misleading scenarios under implicit instruction fine-tuning, as illustrated in Figure 5-2 of the original version. To present more clarity, we updated the caption of the original figure and included detailed experimental results of the revised version. The findings demonstrate that with an increase in the volume of implicit fine-tuning data, the misleading rate in explicit misleading scenarios can be reduced to less than 20%. This highlights the effectiveness of implicit instruction fine-tuning in mitigating explicit misleading behavior.

Table: Results for explicit misleading scenarios under implicit instruction fine-tuning.

We sincerely thank for your thoughtful and detailed feedback, which has provided invaluable insights for improving our work. We appreciate the recognition of the importance of addressing response uncertainty in MLLMs and the potential utility of our proposed MUB benchmark. Your insights regarding the limitations and areas for improvement in our work are invaluable, and we have carefully addressed these concerns in the revised manuscript. Below, we provide detailed responses to each of the points raised.

Q1: We appreciate your valuable suggestions. We also present the additional experimental results in video question answering (video-QA). This work only evaluates/tackles VLLM instead of MLLM as claimed multiple times in title and throughout paper, though I could maybe see the way to extend to other modalities.

A1: Thank you for your valuable suggestions. In response, we have conducted additional experiments in the video question answering (Video-QA) setting to evaluate the adaptability of our method to multimodal inputs, including video and audio modalities. Specifically, we tested VideoLLaMA-2 [1] on the Video-MME [2] dataset across various categories. The results, shown in Table 28 and 29 (revised version), demonstrate that our method is effective for video and audio modalities. For the video-audio modality, the average accuracy decreased from 48.3% to 40.4%. Similarly, for models employing only the video modality, the average accuracy decreased from 54.9% to 45.5%.

Table: Comparison of results before and after adding misleading instructions with video-audio input for VideoLLaMA-2 on the Video-MME dataset.

[1] Cheng, Z., et al., 2024. VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs.

[2] Fu, C., et al., 2024. Video-MME: The first-ever comprehensive evaluation benchmark of multi-modal LLMs in video analysis.

Thank you for your revisions and further explanations. While I appreciate the expanded evaluations across modalities and the exploration of human-annotated data, I remain concerned about the limited novelty and comprehensiveness of the proposed approach/benchmark relative to existing benchmarks. Nonetheless I acknowledge the significant effort made in the revisions. And I'd like to reconsider my evaluation.

Q3: "The 'Implicit' ones are more interesting, but need more detailed treatment: for instance, at least give several examples and enough information to assess data quality."

A3: Thank you for your valuable suggestions. We appreciate your interest in the "Implicit" prompts. To address your concerns, we have made the following updates and additions: Details and Revisions Made: In the original submission, we provided scores for the implicitness of generated implicit instructions in Figure 4-(3) and examples of such instructions in Figures 17 and 18. In the revised version, we have expanded Section 2.2 to include additional details about the implicit instructions, offering more context and examples to better illustrate their nature and design. Comprehensive Evaluation of Implicit Instructions: To more comprehensively evaluate the generated implicit instructions, we have also introduced more comprehensive evaluation the generated implicit misleading data quality, as shown in the table below. We randomly selected 100 images and compared their results with those from human annotations, open-source models, and closed-source models. To ensure fairness and mitigate biases, we masked the non-implicit information (including answers and option letters) potentially revealed due to the model's limitations within the generated misleading instructions. The implicitness level is evaluated based on scores assigned to the generated implicit instructions, with the highest possible score being 9. The findings indicate that closed-source models demonstrate a higher degree of implicitness as well as a higher misleading rate.

Table:Comparison of implicitness, misleading rates, and time required for generating implicit instructions

Q4: "The evaluation of the 'Implicit' setting is also strange -- using best-of-5 sampling (even though 'Explicit' is best-of-1) which inflates the success rate."

A4: I apologize for not providing a clear explanation here we used GPT-4o to generate five misleading implicit instructions for each question in a single response and intended to apply all five instructions. To ensure a fair comparison between explicit and implicit strategies, we have addressed this concern in the revised version by comparing the results of explicit and implicit strategies under the sampling-1 setting. Additionally, we have expanded the evaluation by providing results for implicit strategies using sampling-1, sampling-3, and sampling-5 across scenarios with both low and high misleading rates. These updates aim to offer a more balanced and comprehensive evaluation of the "Implicit" setting and address any potential concerns regarding inflated success rates. Complete table can be found in the revised version paper, Table 13,14.

Table: The average misleading rate of different sample strategies on 12 open-source MLLMs.

We sincerely thank the reviewer for recognizing the significance of the problem we are addressing and acknowledging the potential of our work. We appreciate your constructive feedback regarding the experiments and writing, and we have made substantial efforts to tighten both aspects in the revised manuscript. Below, we detail the specific revisions and improvements made in response to your comments.

Q1: "Is this going to affect real users in any way? Arguably this is even an intended feature to avoid getting into fights with users."

A1: Thank you for raising this important question. To address this, we offer the following insights and supporting evidence:

• Uncertain responses can significantly affect users’ trust and decision-making processes, as demonstrated by the following example: What is the capital of the UK? A: London (confidence: approximately 1.0). B: Paris (confidence: 1.29 × 10⁻¹⁰). [1]. Even for questions where the correct answer is highly certain, there remains a non-zero probability of the model producing an incorrect response. For questions involving greater uncertainty, particularly in multimodal data, this raises concerns about how much users should rely on the model's answers.
• We conducted a simple experiment where, after the user inputted a question (including both multiple-choice and true/false questions), an incorrect character or word was appended to the end of the text to simulate accidental input. No additional guiding information was provided. However, the results still demonstrated a relatively high misleading rate, the model maintains a misleading rate of approximately 62.4%.

Table: The misleading rate of adding a misleading letter to the end of a question.

Ref: [1] Yadkori et al., 2024. To Believe or Not to Believe Your LLM

Q2: "As a result of this, I don't think the 'Explicit' misleading prompts are really a meaningful benchmark."

A2: Thank you for raising this concern. We would like to clarify the significance of the "Explicit" misleading prompts as a meaningful benchmark and provide additional context: we included the experimental results for explicit prompts in Table 8 of the appendix in our original submission(Table 6 in the revised version). These explicit prompts consist of 11 additional templates, including examples such as "the users' answer is," "the GPT-4's answer is," and "given the context and the picture, the answer is xx." The experimental results reveal that explicit prompts consistently exhibit high rates of misinformation. This pattern is not limited to a single template but extends across various forms of explicit prompts, making them a meaningful benchmark for assessing the vulnerability of MLLMs to adversarial manipulation. Such insights are critical for developing more resilient models and better understanding their limitations. Complete table can be found in the revised version, Table 7.

Table: the average misleading rate of 11 explicit prompt templates on 12 open-source MLLMs.

Dear Reviewer t1Gv,

Thank you very much for your invaluable feedback on our paper. We have meticulously reviewed each of your points and endeavored to address them thoroughly. We would greatly appreciate it if you could review our responses and let us know if you have any additional comments regarding the paper or the rebuttal. We are eager to embrace all your critiques and integrate them into our work.

Thank you!

We greatly appreciate the reviewer’s detailed feedback and constructive insights, which have significantly helped us refine our study. Below, we address each of the key points raised:

Q1: The measurement might be dependent to the misleading information themselves: the content, position, length, etc might all influence this metric.

A1: Thank you for raising this insightful point. To address this concern, we conducted additional experiments to evaluate the effects of position, length, and content of misleading information on the misleading rate.

• Content: In the manuscript, we present 11 explicit prompts, which are categorized into four types. The results indicate that explicitly providing the model with the answer increases the misleading rate.
• Position: To analyze the influence of position, we tested scenarios where misleading information was inserted either before (after the system prompt) or after the question. The results indicate that the misleading rate is lower when explicit misleading information is inserted before the question compared to when it is inserted after the question. (Notably, in the paper, all misleading information is inserted after the question.)
• Length: We increased the length of the phrase "The true answer is" by twofold and threefold, respectively, to examine whether the phrase's length impacts the results. The results indicate that increasing the length alone, without modifying the content, has minimal impact on the misleading rate.

Table: the average misleading rate of 11 explicit prompt templates on 12 open-source MLLMs.

The misleading rates across different positions and lengths in 12 open-source MLLMs.