Unveiling the Tapestry of Consistency in Large Vision-Language Models

We sincerely thank the reviewer for the efforts in reviewing our paper and the positive evaluation. Our responses according to the reviewer's comments are summarized as follows.

---

1. In addition to ConScore[C], how does TDR affect LVLMs' performance on comprehensive multimodal benchmarks?

Our proposed method TDR aims to utilize the discriminative prompt template and VLMs' judgment capability to enhance their open-ended generation ability.

Given the difficulty in evaluating open-ended responses, we randomly selected 100 images from MMMU, MMBench, and MMStar. Then, LLaVA-v1.5-7B was employed to generate initial captions for these images, which were then improved by our TDR method. After conducting five rounds of manual evaluation, we observed that the TDR method resulted in improvements in 37% of cases, no change in 60% of the cases, and a slight decrease in 3% of the cases (GSB = 37:60:3). The results further confirm the effectiveness of the TDR approach.

Furthermore, while not directly for the improvement of discriminative responses, motivated by your question, we also attempted two discriminative benchmarks (rewriting prompts for some low-probability answers). As shown in the table below, we are pleased to see that our method improves LLaVA-v1.5-7B by $0.84$ and $0.92$, and boosts MiniGemini-7B by $0.53$ and $0.79$ on GQA and POPE, respectively. These promising results will inspire our further research.

We sincerely thank the reviewer aYRC for the efforts in reviewing our paper and positive evaluation. Our responses according to the reviewer's comments are summarized as follows.

---

1. Explain the definition of the discriminative and generative domains clearly.

(1) Discriminative questions provide prior information about the image in the text, restricting the model to answer based on this information, making it a closed-ended question.

(2) Generative questions, on the other hand, do not provide the model with specific image priors, allowing the model to generate answers totally based on its own reasoning, making it an open-ended question.

(3) Furthermore, we will carefully review the clarity of other presentations and make corrections in the revised version

---

2. Explain the definition of ConScore[C], metric[C], and metric[D].

(1) ConScore[C]

The ConScore[C] can be clearly defined by the following formula:

    ConScore[C] = (Con[T] + Con[C] + Con[V]) / 3,

where the $Con[X]$ is the Consistency ratio between discriminative answer type $X$ and the caption. The ConScore[C] is a generative domain evaluation metric to indirectly quantifying and assessing the actual user experience.

(2) Metric[C] and Metric[D]

Metric[C] and Metric[D] actually refer to the tables with ConScore[C] and Score[D] as evaluation metrics, respectively. We will provide clearer explanations in the revised version.

---

3. The concerns about using GPT4: (1) Does GPT4 have a certain bias; (2) Does GPT4 also have inconsistent judgment issues; (3) How to eliminate the impact of GPT4's inconsistent judgments?

(1) Initially, we only employ GPT-4 to determine the Consistency of model responses, (e.g., prompts like "Based on the caption, is my response correct? Please answer with yes or no only."). Therefore, this is a binary choice task for GPT-4, and our primary focus is on its accuracy. Regarding this task, we manually checked its performance on five different VLMs by randomly sampling 200 cases in each trial. As shown in the following table, the GPT-4 achieves an accuracy of 95.4% (until August 4, 2024) with standard deviation of 0.0035, which is highly reliable and stable.

(2) Inconsistency refers to the model providing inconsistent answers when faced with different solution spaces of prompts for the same knowledge point. Based on the above, GPT-4 in our evaluation always answers 'yes' or 'no', where the solution space of prompts remains the same, so the concept of Inconsistency does not need to be considered in this situation.

---

4. The comparison between the proposed Trigger-based Diagnostic Refinement and other works (e.g., [1]) in the LLM realm.

(1) Similarity

The similarity between ours and [1] lies in asking follow-up questions based on the historical responses of the model and confirming the final answer, to enhance the quality of responses.

(2) Differences

However, the motivation and technical details of ours and [1] are completely different. [1] utilizes all the accumulated responses as options for the next round of prompts, iteratively searching through historical responses to find the correct answer, which is just effective for mathematical and reasoning abilities. Our method only picks out the uncertain words in model responses, and constructs discriminative templates for questioning, proving effective comprehensive abilities.

[1] Chuanyang Zheng, etc. Progressive-Hint Prompting Improves Reasoning in Large Language Models, 2023.

Dear Reviewer aYRC,

We sincerely thank you for your efforts in reviewing our paper. We have provided corresponding responses and results, which we believe have covered your concerns. We hope to further discuss with you whether your concerns have been addresses or not. Please let us know if you still have any unclear part of our work.

Best,
Authors

We sincerely thank the reviewer AnYL for the efforts in reviewing our paper. Our responses according to the reviewer's comments are summarized as follows.

---

1. It would be beneficial to increase the diversity and scale of ConBench.

We greatly appreciate your attention to the scale and diversity of ConBench. The current version of ConBench contains 1,000 images with 3,000 questions, and covers 19 subdomains, comparable to popular multimodal datasets such as MME [1] and MMBench [2]. In this version, multimodal models have not yet achieved high scores (e.g., the current best model Qwen-VL only achieves 37.00 on ConScore[D]). With further development of VLMs, we will update and scale up the cases of benchmark datasets to tens of thousands, providing a more comprehensive evaluation of Consistency.

[1] MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models.

[2] MMBench: Is your multi-modal model an all-around player?

---

2. Is the Fact 4.4.1. to show that closed-source models put more effort into improving the consistency compared with open-sourced ones?

In the paper, we discovered Fact 4.4.1 to provide an alternative perspective to demonstrate why closed-source models, despite sometimes having lower accuracy on benchmarks, offer a better user experience in practical applications. One possible reason, based on our paper, is that closed-source models have a Consistency bias. Therefore, evaluating the Consistency of VLMs is a feasible approach to indirectly quantifying and assessing the actual user experience.

---

3. Changing generation arguments like temperature, top_p or top_k might have similar effects with the proposed method TDR.

In fact, adjusting these hyperparameters only affects the randomness in sampling, not improving the quality of the generation. As shown in the following table, we conducted comprehensive experiments on LLaVA-NeXT-34B and MiniGemini-34B. The temperature and top_p in the baselines are $0.0$ and $None$, and we set temperature to $0.2/0.8$ and top_p to $0.9/0.7$ in the ablation studies. The results show that tuning the temperature or top_p values can not enhance the Consistency of model responses. In contrast, our approach makes the model aware of where lacks confidence and has errors, leading to corrections that boost generation quality. Therefore, changing generation argument hardly achieves similar effects.

---

4. The paper seems more suitable to be submitted to the Datasets & Benchmarks track instead of the main track.

Our paper differs from the papers in the Datasets & Benchmarks track. It not only introduces a new dataset ConBench but, more importantly, we provide deep analyses of the Inconsistency phenomenon based on that. Our findings establish a relationship between the discriminative and generative realms, highlighting the importance of Consistency between the discriminative answer and caption. Moreover, we propose a solution by forcing VLMs to self-think, where a discriminative prompt is constructed via uncertain words in the caption. It is a comprehensive evaluation pipeline instead of introducing the dataset or benchmark solely.

Dear Reviewer AnYL,

We sincerely thank you for your efforts in reviewing our paper. We have provided corresponding responses and results, which we believe have covered your concerns. We hope to further discuss with you whether your concerns have been addresses or not. Please let us know if you still have any unclear part of our work.

Best,
Authors