This paper presents a refined and efficient MLLM benchmark called LIME, which enhances the quality of existing benchmarks through semi-automatic refinement. LIME consists of 9,400 evaluation samples across six types of tasks and ten different benchmark datasets. The authors evaluated over 30 models and provided some analyses based on the results.

The paper proposes the LIME, a refined and efficient benchmark for MLLM evaluation. The paper first shows that existing benchmarks contain a large proportion of easy or noise samples that cannot reflect the actual capabilities of MLLM. Then, the paper proposes a three-stage pipeline to filter the existing 10 benchmarks across 6 types. The easy samples, wrong-labeled samples, and answer-leakage samples are removed during this process. The refined benchmark can provide a more rigorous evaluation of the existing MLLMs.

Existing MLLM benchmarks often include overly simple or uninformative samples, making it difficult to effectively distinguish the performance of different MLLMs. This work proposes LIME , a refined and efficient benchmark curated using a semi-automated pipeline. This pipeline filters out uninformative samples and eliminates answer leakage by focusing on tasks that require image-based understanding. The experiments show that LIME reduces the number of samples by 76% and evaluation time by 77%, while it seems promising to be more effective for distinguishing different models’ abilities.

This paper introduces LIME (Less Is More for MLLM Evaluation), a refined benchmark for evaluating Multimodal Large Language Models (MLLMs). The authors propose a semi-automated pipeline to curate a more efficient and discriminative evaluation dataset by filtering out uninformative samples and eliminating answer leakage. The resulting benchmark reduces the number of evaluation samples by 76% and evaluation time by 77% while maintaining or improving the ability to distinguish between different models' capabilities. Key findings include the inadequacy of traditional automatic metrics for captioning tasks and the importance of excluding caption task scores for more accurate overall performance assessment.

Thanks for your interesting work! LIME is very useful. I wonder whether we can also directly use the proposed semi-automated pipeline in LLM domain. Is LIME specifically designed for MLLMs?

We appreciate all the reviewers for taking the valuable time to provide feedback on LIME, which has been very helpful in improving our paper. Since many reviewers raise similar concerns and questions, we will provide a unified response to these issues here, and we encourage all reviewers to read it.

Discussion about solutions for eliminating data leakage:

@Reviewer 1M8b and Reviewer Mt3s

To ‘Eliminate answer leakage’, we classify questions into two categories:

'Text Answerable Questions' are samples where the answer can be directly inferred from the text of the question and options, without requiring the image.

'Seen Questions' refers to samples encountered during the training process. These are easily answerable by MLLMs due to prior exposure.

We use LLMs to Eliminate Text Answerable Questions：LLMs are trained on massive amounts of textual corpus, having excellent textual commonsense reasoning ability. In the text-only setting, we use LLMs to filter out text-answerable questions.

We have eliminated “seen questions” in Semi-Automated Screening Process:During the Semi-Automated Screening Process, we filter the easy samples, which can be answered correctly by most models (with a correct answer rate greater than 6/9). In fact, this subset of deleted data includes “seen questions” samples.

Additionally, to further support our idea. we test the text-only performance of MLLMs on LIME, As shown in below Table, even the SOTA MLLMs(Qwen2-VL-7B ,InternVL-2-8B ) achieve extremely low scores in the text-only setting, which indicates that our pipeline is robust enough to account for all potential data leakage, and there is no more data leakage in LIME.

Overall revision of paper:

Based on Reviewer Pra9's suggestion, we have updated the full results of the data size ablation experiment in the appendix A.4.

Based on Reviewer bgCw's suggestion, we replace "model" with "MLLMs" in line 156 to make the expression clearer and add 19 cases related to MMMU and MMBench in the appendix B.

In response to Reviewer Mt3s's feedback, we have refined the description of Section 2.3 on "ELIMINATING ANSWER LEAKAGE" to prevent any potential misunderstandings by the reviewers.

Motivation of LIME:

LIME is an initial version of a benchmark that embodies two key, enduring motivations:

1. Most benchmarks contain low-quality, noisy data:

As mentioned in Figure 2 of our paper, "Most benchmarks contain low-quality, noisy data, which does not accurately reflect the true capabilities of MLLMs." We need a subset of benchmarks from each benchmark that contains a certain amount of data that comprehensively reflects performance across various aspects, and we provide a stable pipeline for selecting this collection.

2. Existing benchmarks have large gaps with actual user experience:

For MLLM evaluation, it is more important to examine what truly relates to the actual user experience rather than just testing the ability to solve tasks simply. However, existing benchmarks have large gaps with actual user experience, and we focus on the parts of existing benchmarks that are most relevant to the real user's needs.

Contribution of LIME:

1. LIME provides pipeline & guideline for existing benchmarks:

LIME provides an entirely open-source pipeline, which includes three components: "OPEN-SOURCE MODELS AS JUDGES", "SEMI-AUTOMATED SCREENING PROCESS", and "ELIMINATING ANSWER LEAKAGE". By utilizing MLLMs and LLMs, we eliminate data leakage in existing benchmarks, remove potential noise data, and filter out a subset that truly reflects the model's capabilities. Our experimental results show that LIME reduces the cost of benchmark evaluation while maintaining evaluation accuracy, and it better reflects the model's multimodal performance compared to the original benchmark.

LIME is not only a dataset; but also a universal guideline applicable to benchmarks across all domains: Although LIME currently selects only 10 benchmarks as the primary subset, it features a plug-and-play architecture that can be applied to any benchmark in any domain. LIME can also serve as a guideline for creating new benchmarks from scratch, enhancing their quality. Additionally, we commit in our paper to continuously update the sub-tasks included in LIME , even if "core" benchmarks emerge in the future, LIME will be able to detect potential noisy data and improve their data quality.

2. LIME focuses on the parts of existing benchmarks that are most relevant to the user's needs.

As shown in Figure 4 of our paper, we point out that traditional evaluation metrics(CIDEr) for captioning tasks cannot meet the real user needs, as they only focus on the overlap between the model-generated responses and the ground truth. In addition, LIME achieves over 91% correlation with Wildvision-elo, indicating that LIME, as a collection, is very small and static but has extremely high relevance to user experience—possibly the highest among existing benchmarks.

Explanations for some other issues:

"Once someday, the community will realize there is another benchmark needed for the 'core' MLLM evaluation tasks, then the proposed one will become less meaningful"

In the era of rapid MLLM development, the emergence of new benchmarks every day is an inevitable trend. However, most benchmark creation processes face similar challenges(answer leakage,annotation error). LIME offers a robust pipeline designed to eliminate potential errors in the data and uncover the most 'core' aspects of a benchmark. As new benchmarks continue to emerge in the future, we are committed to updating LIME every few months (LIME-v2, LIME-v3, etc.) and providing increasingly valuable test sets.

We would like to sincerely thank all reviewers for their positive recognition of our work. Your feedback is invaluable, not only for improving LIME but also for advancing research in the broader MLLM evaluation field. Below, we provide a comparison of our work with related research and further elaborate on the motivation and contributions of LIME.

@Reviewer Pra9 and Reviewer 1M8b

Systematic Justification

With the rapid development of the MLLMs evaluation field, various benchmarks have emerged. Some works collect data from scratch to create benchmarks, while others filter and process data based on existing benchmarks. Compared to these works, LIME offers a more robust and general pipeline framework, providing higher-quality benchmarks while maintaining lower production and evaluation costs. We use inclusion and exclusion criteria to choose core benchmarks to compare with our LIME:

1. Choose impact benchmarks: filter out papers without 300 citations.
2. Choose general benchmarks: filter out papers to evaluate expert knowledge domains.
3. Choose recent impact publications: add papers on general benchmarks with more than 50 citations released after 2024.

Chartqa [1] collects chart data from four different open-source websites and generates question-answer pairs in the chart domain through a combination of human annotation and language model generation. However, since Chartqa is an early classic dataset, it does not include detailed answer leakage tests or fine-grained difficulty categorization.

Mmmu[2] is a large-scale, multidisciplinary, multimodal understanding and reasoning benchmark, which contains 11,500 samples across six different categories. It focuses on evaluating the logic and reasoning capabilities of MLLMs. However, some studies have pointed out that Mmmu heavily relies on pure text-based knowledge and suffers from potential data leakage issues[5,6], failing to adequately assess the multimodal capabilities of MLLMs. In contrast, LIME employs a rigorous data processing pipeline to eliminate potential data leakage issues, providing more challenging tasks for multimodal models.

Mmbench[3] is designed to evaluate MLLMs across six different capability dimensions, and it leverages LLMs and MLLMs as judges to filter question sources. However, it only removes text-only and incorrect samples, lacking fine-grained difficulty categorization and still posing risks of potential data leakage. As a result, Mmbench is unable toeffectively distinguish the performance differences between different models. As shown in the Table below, compared to Mmbench, LIME can better reflect the capability differences among various models.

MMStar[4] uses LLMs to eliminate data leakage and manually curates a subset of 1,500 data points to mitigate potential data leakage in large multimodal models. However, there are some issues with this approach: 1. It overly relies on manual efforts, and the selection criteria are not fully controllable. 2. The data processing pipeline is not robust enough; it only uses LLMs to address data leakage within LLMs/MLLMs, which may include other potential errors (such as annotation errors, etc.). In comparison, LIME provides a more feasible and generalizable pipeline, which can naturally be extended to other tasks and domains.

[1].Chartqa: A benchmark for question answering about charts with visual and logical reasoning

[2].Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

[3].Mmbench: Is your multi-modal model an all-around player?

[4].MMStar: Are We on the Right Way for Evaluating Large Vision-Language Models?

[5].Mmmu-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark

[6].Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs

Dear Reviewers and Chairs

We sincerely thank all the reviewers and chairs for their efforts during the rebuttal process. Throughout the discussions, we have received positive feedback and valuable suggestions from the reviewers. We are grateful that they acknowledged our method as novel (Pra9), well-motivated (1M8b, bgCw,Mt3s), and effective（Pra9,1M8b,bgCw) supported by comprehensive experiments (Pra9,1M8b, bgCw, Mt3s). We are also pleased that the subsequent discussions successfully addressed the major concerns raised (Pra9, Mt3s).

Compared to similar related work, the core contribution of LIME is:

1. LIME provides pipeline & guideline for existing benchmarks: We have demonstrated that most benchmarks contain low-quality, noisy data, while LIME provides an entirely open-source pipeline. By utilizing MLLMs and LLMs, LIME can eliminate data leakage in existing benchmarks, remove potential noisy data, and filter out a subset that truly reflects the model's capabilities.

2. LIME focuses on the parts of existing benchmarks that are most relevant to the user's needs. Existing benchmarks have large gaps with actual user experience，LIME achieves over 91% correlation with Wildvision-elo, indicating that LIME, as a collection, is very small and static but has extremely high relevance to user experience—possibly the highest among existing benchmarks.

Based on the insightful and thoughtful feedback from the reviewers, we have made the following revisions to the paper:

1. Following Reviewer Pra9's suggestion, we have updated the full results of the data size ablation experiment in Appendix A.4 to demonstrate the impact of dataset size on statistical significance.

2. In response to Reviewer bgCw's suggestion, we have replaced "model" with "MLLMs" in line 156 to clarify the expression.

3. We have added a comparison between LIME and related works such as MMMU and MMBench, and included 19 additional cases related to MMMU and MMBench in Appendix B to address the major concerns raised by Reviewer bgCw.

4. In response to Reviewer Mt3s's feedback, we have refined the description in Section 2.3, titled "ELIMINATING ANSWER LEAKAGE," to avoid any potential misunderstandings.

Once again, we sincerely thank you all for your valuable feedback, dedication, engagement, and suggestion; we truly appreciate it.

Best Regards,

Authors

This paper proposed an approach to reduce the unnecessary evaluation samples from the evaluation benchmarks. Concretely, the author proposed to reduce the samples from the benchmarks by 1. employing multiple MLLMs as judge to remove easy samples. 2. employing text only LLM to remove leaked samples (can just answer the questions from the text only) 3. using GPT-4v+ human approach to check the logic and meaning of those questions that all judge models fail to answer.

Strength:

1. The approach is quite comprehensive for reducing the size of the evaluation benchmark.
2. The empirical results are comprehensive.
3. The paper proposed a totally open-sourced approach for combining multiple datasets as a joint benchmark.

Missing from the submission and my major concern:

After reading the paper, the review, and the rebuttal (general responses 1 and 2), I have one major concern for the usefulness of the proposed approach.

1. Evaluation speed: Surely, we can reduce the evaluation time by reducing the benchmark size. However evaluation speed might not be the bottleneck for the whole MLLM pipeline. The training takes much more time than the evaluation. I wonder whether reducing the size of the benchmark is truly meaningful.
2. Human alignment: I have one major concern of reading table 2. I do understand by removing some samples from datasets, the ranking might change. However I wonder whether this is truly reflect the model's capability in T/F Reasoning, VQA, Infographic understanding, OCR, and Science QA? By reading the table 2, GPT-4o achieves way lower performrance than the open sourced model. This is a big claim. I wonder whether this truly represents the GPT-4o is bad on those capability in real-life user experience? If I read this paper correctly, there is no alignment b/w the human preference and the proposed LIME approach. If the answer for the previous question is no (aka GPT-4o might be better on those capability in real-life user experience,) then the significance for this paper might be diminished. Without human alignment, I cannot justify that the LIME score reflect the model's true capability in those domain.

Given those concerns (specifically point 2), I do not think this paper persuaded me for its significance. I would recommend reject.

Reject