Thank you for your valuable advice. Here is our response to your questions:

Q1: Quality of the benchmark

Q1.1: How are these samples collected?

First, we recruit a group of college students as volunteers. We then provide necessary training to these volunteers, equipping them with skills to collect images and create relevant questions and answers for each task. Moreover, we provide them with a range of potential sources to gather images and formulate corresponding questions and choices. Table 7 in the paper showcases these potential sources. Following this, the collected Visual Question Answering (VQA) data undergoes a quality inspection pipeline to ensure the creation of a reliable benchmark.

Q1.2: Quality testing for samples in MMBench

We conduct thorough quality testing on the collected samples through manual review, with a primary focus on two key aspects:

1. Ensuring the accuracy and coherence of the questions, without any logical or factual errors.
2. Verifying the presence of a single correct answer that is pertinent to the question. We filter out any samples that fail to adhere to these aforementioned criteria.

Q1.3: Question can be answered without the corresponding image

Thank you for highlighting this issue. We too identified the same problem after submitting our paper. To determine the number of questions in MMBench that can be answered without an image, we input all the questions into GPT-4 and ensure that it makes reasonable guesses for all questions. We discovered that ~13% of the dev questions and ~10% of the test questions could be answered correctly, which mainly originated from ScienceQA. Subsequently, we removed these questions and used the remaining ones to evaluate all models. As indicated in the table below, the performance of these models remained similar to their performance prior to the image filter, and the rankings of the different models were basically consistent. That suggests that the current MMBench leaderboard remains credible. Furthermore, we plan to release an updated version of MMBench, with the questions that can be answered without images filtered out.

Q1.4 Licensing for collected Internet images

All images within MMBench are available for academic use. During the collection process, we instruct our annotators to document the original URL of each image. Subsequently, we manually review these URLs individually, eliminating any images that are not suitable for academic use. Moreover, should any authors request the removal of their images from MMBench, we will promptly comply.

Q1.5 MMBench is a QA benchmark instead of a VQA one

The conclusion seems overly severe to be considered fair. As demonstrated in Q1.3, a mere 10% of the existing questions can be answered without the aid of images. The remaining majority of questions necessitate the model's accurate comprehension of the image content to provide correct responses.

I found the author responses to my major concern (dataset quality / curation) to be vague.

Response: ...college students as volunteers. We then provide necessary training to these volunteers, equipping them with skills to collect images and create relevant questions and answers for each task...

What does it mean by necessary training and skills to collect relevant questions? I do not find details regarding this important information in the updated version of the paper. Please update the paper with the detailed procedure.

Response: ..we input all the questions into GPT-4 and ensure that it makes reasonable guesses for all questions. We discovered that ~13% of the dev questions and ~10% of the test questions could be answered correctly, which mainly originated from ScienceQA. ... Furthermore, we plan to release an updated version of MMBench, with the questions that can be answered without images filtered out.

How did you use ChatGPT for reasonable guesses? And do you use CircularEval? What is the exact prompt you used for conversing with ChatGPT? I do not find this discussion in the updated paper. Also, given that MMBench is already used in many recent MLLM evaluation, how do you plan to update this benchmark? Do you plan to release a MMBench-2, or simply request all previous works who used the earlier version of MMBench to re-evaluate their models?

The author did not discuss the watermark issue in their response.

I found the author's response regarding to CircularEval to be confusing.

Reading the response, I believe the term used in paper (CircularEval doesn’t necessarily requires N× inference cost.) is just misleading. Inference cost usually refers to the cost of model inference. The authors should update the paper to clarify that this has to do with "ChatGPT calls/querying cost".

Response: sometimes CircularEval even requires less ChatGPT calls compared to VanillaEval. This happens when the heuristic strategy failed to match the VanillaEval prediction with any options, but succeeded in matching one of the CircularEval predictions with a wrong option.

Please clarify this with a concrete example. I find this to be confusing.

Thank you for clarifying the issues of MME. I think this discussion should go into the updated paper to justify why people should use your benchmark instead of MME."

I carefully checked the updated paper. I appreciate that the authors took most of reviewers' feedbacks into consideration for improving the draft. I have a remaining question regarding the use of public data sources.

Quote from paper: "Firstly, our data is gathered from the validation sets of these public datasets, not their training sets. Secondly, data samples procured from these public datasets constitute less than 10% of all MMBench data samples."

I think these statements can be misleading -- I saw that LLaVA is one of the data sources for constructing MMBench; however it does not come with an official validation set (unless you are referring to the 30-images LLaVA-Bench). Also, I do not find the actual distribution of data sources in the current draft.

Our Responses:

1. LLaVA:

   Apologies for any confusion caused. When we talk about "the validation sets", it only applies to datasets that have both "training" and "validation" splits (like COCO or ScienceQA). For LLaVA, we utilize the images in the full set. We will update the paper accordingly to resolve the confusion.

2. Data Source Statistics:

   Since the discussion deadline is approaching, we may not be able to provide the source statistics at this time (some raw data need to be processed to figure out the exact source of each image). We promise that the detailed statistics will be included in the next version of the manuscript.

3. Answer Extraction:

   3.1. ChatGPT-based Answer Extraction:

   Most VLMs and LLMs are designed to chat with human beings: If not finetuned with a sufficient number of multiple-choice problems, most of them can not output the option label for every question perfectly (ie., follow the instruction of solving multiple- choice problems). Our ChatGPT-involved evaluation pipeline is designed to evaluate VLMs solely based on the answer content, and we think that pipeline can be more fair compared to exact matching.

   If we use exact matching during the evaluation, VLMs finetuned on a large number of multiple-choice problems will follow the instructions in MMBench much better, and significantly outperforms VLMs not finetuned on multiple-choice problems. Then it will be difficult to reveal the real multi-modal understanding performance gap with the evaluation results.

   3.2. Poor performance of LLaMA:

   Actually, the overall performance of Vicuna (v1.0, finetuned based on LLaMA) is not SOTA, especially at the current time (2023.11), thus it performs poorly on MMBench answer extraction. In the upcoming version, we will try to utilize more advanced opensource LLMs (XWinLM, UltraLM, Vicuna v1.5, etc.) to see if they can achieve better performance on MMBench answer extraction. If so, we will have a ChatGPT free evaluation pipeline.

4. Training Materials & Annotating Platform:

   We plan to release the training materials we used to guide the annotators. Meanwhile, the release of the annotating platform will be more difficult. We will try our best to deliver an open-source version of the annotating platform. However, we cannot make any promise at this time.

Thank you for your valuable advice. Here is our response to your questions:

Q1: Insight brought by MMBench

Numerous insights can be gleaned from the MMBench leaderboard, and we highlight some of the more apparent phenomena observed during the evaluation:

1. The performance of coarse perception is significantly lower than that of fine-grained perception across all models presented in Table 3 of the main paper. This trend is understandable, given that fine-grained perception necessitates the model's ability to precisely identify and comprehend the various objects present in an image. Furthermore, we discovered that enabling trainability of parameters in the vision encoder positively influences the model's fine-grained perception capabilities. This is because it equips the model with the flexibility to modify its vision feature extraction ability during visual-language pre-training and instruction tuning.

2. The performance of cross-instance finegrained perception is much lower than that of single-instance finegrained perception. The leaderboard of MMBench indicates that a majority of pre-training vision-language datasets primarily include descriptions or questions pertaining to individual objects, thereby neglecting to address cross-object relationships. Furthermore, we've observed that incorporating a referring expression comprehension task can effectively enhance the performance of the cross-instance fine-grained perception task.

3. The performance on perception tasks significantly surpasses that on reasoning tasks. Reasoning is a more complex task compared to perception, as it demands the model to not only accurately identify various objects in an image, but also possess the ability to reason about these objects, including their function and social identity. Therefore, enhancing the complexity of the instruction data, rather than posing simple questions about the color and shape of objects, is anticipated to improve this ability.

Q2: ChatGPT is vulnerable to attack

In the evaluation pipeline of MMBench, ChatGPT is solely utilized to extract answers from predictions and match them to the available choices, and the accuracy is calculated by some predefined functions based on the matching results, with CircularEval strategy. At present, we haven't identified any potential vulnerabilities that could affect ChatGPT within the evaluation pipeline and cannot think any potential attack approaches. If you have any ideas or suggestions on how to perform the attack, please feel free to propose them directly. We are more than willing to engage in a discussion with you.

Q3: The efficiency of this benchmark.

While exact matching has been utilized in numerous previous benchmarks like VQAv2 and GQA, incorporating ChatGPT into the evaluation pipeline may indeed slow down the process. However, when weighing the tradeoff between robustness and efficiency, using ChatGPT for extraction and matching emerges as the optimal choice. This is because it eliminates the issue of generating a high number of false negative samples, and the time required for evaluation remains within acceptable limits. The following table presents the number of ChatGPT API calles required by each VLM under VanillaEval and CircularEval.

Among all VLMs, PandaGPT consumes most ChatGPT API calls (~ 2000). We find that each answer matching prompt is 500 tokens length on average. Based on the current price of ChatGPT (GPT-3.5, which is 0.001 USD / 1k tokens), the price upper bound for each evaluation is only 1 USD (~0.3 USD for all models on average). In practice, we find that for most models, the evaluation can finish in half an hour, with one single OpenAI key and parallel API calling.

Thank you for your valuable advice. Here is our response to your questions:

Q1: How these existing 20 ability dimensions are selected

1. Perception and reasoning are two crucial cognitive abilities in humans. Firstly, we refer to the definitions of perception and reasoning in biology, such as the specific aspects they encompass. The part has been thoroughly discussed in Section 2.1 of the main paper.

2. There is currently no consensus in the academic community regarding the specific taxonomy of sub-abilities in perception and reasoning, and there also lacks a comprehensive theoretical framework. Therefore, we gathered a group of experts from the fields of computer vision, natural language processing, and machine learning to discuss the specific ability taxonomy in perception and reasoning. As a result, we obtained 20 ability dimensions included in MMBench, as well as the ability hierarchy ranging from L1 to L3.

3. The existing 20 ability dimensions are not fixed, and we will continuously update the existing ability dimensions to further improve the ability taxonomy.

Q2: How to classify a question, if multiple ability dimensions are entangled

According to the definition in Page 16, we categorize the "counting" ability as "Object Localization". In MMBench, we do not consider "counting" as a more complex reasoning problem, since for most counting questions, there only exist several objects to be counted.

If multiple ability dimensions are intertwined, we consistently categorize the current question under the more challenging ability dimension. This is because, in most scenarios, the more difficult ability dimension necessitates the model to possess the capability to solve simpler tasks. By doing so, it can effectively utilize the task outputs as intermediate results to tackle more complex tasks. For instance, all questions in "Attribute Comparison" rely on the capability of "Attribute Recognition", since we consider "Attribute Comparison" as a more complex task, we categorize these questions as "Attribute Comparison".

Q3: The results are usually “winner takes all”

Table 3 presents the results of the L-2 ability dimension. Unlike a "winner takes all" scenario, these results on L-3 ability dimension offer a nuanced understanding of the model's capabilities. For further insights, please refer to Tables 9 through 14, which provide additional details on the results. For instance, Shikra demonstrates exceptional performance in Action Recognition, while LLaMA-Adapter excels in Attribute comparison. We hypothesize that factors such as data composition, instruction data quality, and instruction template significantly influence a model's performance across different ability dimensions.

Q4. Bias Analysis

One shortcut might be that the model could provide accurate answers even without relying on image information. To assess the necessity of image data for questions in our dataset, we employed some state-of-the-art LLMs to respond to queries in the MMBench-dev. During the evaluation, we design meta prompts to guide the LLMs try their best to make a reasonable guess, even if a question is not theoretically answerable given only the text part.

As demonstrated in above table, the performance of the text-only GPT-4 model — prompted solely with question text — is notably suboptimal. This limitation is particularly evident in responding to queries that require 'perception' abilities. Such questions, often intrinsically linked to visual information, include examples like "What mood does this image convey?" or "Who is this person?" In contrast, GPT-4 exhibits a degree of proficiency in 'reasoning' tasks, with a notable strength in logical reasoning (LR). The model's performance in LR is comparable to that of other leading open-source models. The effectiveness in LR could be attributed to the model's ability to leverage common sense in answering questions. Notably, some LR data, sourced from ScienceQA, suggest that a portion of these tasks can be effectively addressed using the model's inherent common sense reasoning [1]. In future work, we will meticulously curate our dataset to guarantee that the image data in MMBench is pivotal for addressing every question.

[1] Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering

I thank the authors for the response. I appreciate the GPT-4V results, as well as the results for the bias.

The text-bias result is interesting. In bias results, it's shown that text-only GPT-4 without looking at the images gets 13% accuracy, with a 28% LR (reasoning) performance that is higher than some models with images. It will be interesting to see similar analysis for more models besides GPT-4.

In Table 1, we report the accuracies of GPT-4v and Bard (only answered questions count: Acc = (Number of questions answered correctly) / (Number of questions answered)).

I believe this metric is misleading and leads to misleading high results for models that produces "not answerable" answers for most of the questions. A better metric is to assign random answers for the not answerable questions, then calculate accuracy. For example, Bard refused to answer 31.6% questions, which is not a small percentage.

Additionally, analysis of those rejected questions, i.e. whether they are ambiguous, low-quality, or some other reasons, would be favorable.

Moreover, the response does not address my concerns in a more detailed analysis on analysis of the 20 different ability axis. As most of the results demonstrate "winner takes all". Since the benchmark introduces the 20 dimensions, showing the analysis findings on the 20 dimensions is necessary to show the advantage of reporting 20 numbers over only the overall accuracy.

please refer to Tables 9 through 14, which provide additional details on the results. For instance, Shikra demonstrates exceptional performance in Action Recognition, while LLaMA-Adapter excels in Attribute comparison. We hypothesize that factors such as data composition, instruction data quality, and instruction template significantly influence a model's performance across different ability dimensions.

While the authors provide intensive results in Tab 9-14, the results are less meaningful if no analysis is provided. The analysis provided here is too vague and general to show the necessity for evaluating on different dimensions.

Overall, I thank the authors for their efforts in the rebuttal and the new results. With the remaining concerns, I keep my rating unchanged.

Thank you for your valuable advice. Here is our response to your questions:

Q1: Results for closed source VLMs (GPT-4v, Bard, etc.)

Currently, we have evaluated GPT-4v and Bard on MMBench-Dev, and we will provide the results on MMBench-Test in the final version. Note that due to the strategies set by developers, the closed source VLMs may reject some of the questions: GPT-4v refused to answer 4.2% questions in MMBench-Dev, Bard refused to answer 31.6% questions in MMBench-Dev. In Table 1, we report the accuracies of GPT-4v and Bard (only answered questions count: Acc = (Number of questions answered correctly) / (Number of questions answered)).

Table 1. Model accuracies on MMBench-Dev (all questions). * mean the question numbers used for calculating accuracies may be different from other models.

In Table 2, 3, we provide apple-to-apple accuracy comparisons for GPT-4v and Bard, respectively. In those comparisons, the advantages of GPT-4v and Bard are more significant.

Table 2. Model accuracies on MMBench-Dev (only questions answered by GPT-4v count).

Table 3. Model accuracies on MMBench-Dev (only questions answered by Bard count).

Q2: Performance Analysis & Comparison with closed source VLMs

1. In Table 1, GPT-4v significantly outperforms the leading open-source model (Qwen-VL-Chat) by over 14% overall accuracy on MMBench-Dev. The improvement is primarily evident in reasoning tasks:
   1. In Attribute Reasoning questions, GPT-4v outperforms the second best opensource VLM IDEFICS-80B by over 12% Top-1 accuracy.
   2. In Logic Reasoning questions, GPT-4v outperform1s all opensource VLMs by over 35% Top-1 accuracy
   3. In Relation Reasoning questions, GPT-4v outperforms all opensource VLMs by over 15% Top-1 accuracy Meanwhile, the gap between opensource VLMs and GPT-4v is not that huge on perceptions tasks (3% - 10% Top-1 accuracy).
2. Under apple-to-apple comparisons (only questions answered by the closed source VLM count), the advantage of closed source VLMs is more significant. Generally, the perception performance of opensource VLMs is good and comparable with GPT-4v or Bard (under some preception tasks, opensource VLMs even significantly outperforms Bard). However, the reasoning performance of opensource VLMs still lag far behind GPT-4v or Bard. We attribute the superior reasoning capability of GPT-4v or Bard to their potentially larger and more powerful language models.
3. Most existing opensource VLMs simply map the visual concepts to the language embedding space, and finetune the model with VL instruction data (with a large proportion of questions perception related). To further improve the reasoning capabilities, the developers can: 1. Switch to more powerful language backbones; 2. Design better finetuning algorithms; 3. Built reasoning-related instruction datasets with better quality.