Large Language Models as Automated Aligners for benchmarking Vision-Language Models

Dear Review G18f,

Thank you for your valuable feedback and comments on our paper. We have carefully considered the raised concerns and address them accordingly in the following.

Q1: Comparisons with Visual Instruction Tuning

We appreciate the reviewer's observation on Visual Instruction Tuning. While we aggree that both our methods and Visual Instruction Tuning utilize LLMs to generate data automatically, our methods offer some significant differences as what follows:

(1) Diversity on the ability categories The data generated by Visual Instruction Tuning does not specify the abilities that need to be evaluated. The questions they generate only broadly cover five aspects: object types, counting the objects, object actions, object locations, and relative positions between objects. In contrast, our generation is ability-aware, where we have specific prompts for each ability to generate content that targets each skill individually. For the final generated data, we have covered a total of 17 different abilities (which is more than 4 times of Visual Instruction Tuning). Therefore, our generated data exhibits higher diversity in terms of ability categories.

(2) Diversity on the visual symbolic information Visual Instruction Tuning provides only limited caption and 2D bounding box information to prompt GPT in generating questions. This, to some extent, restricts the diversity of generated questions. For instance, its visual symbolic information lacks OCR information, resulting in the absence of OCR-related questions in its generated data. In contrast, our approach incorporates richer visual symbolic information, such as text, caption, instance relationships, and etc. Therefore, we can prompt GPT to generate data that covers a wider range of information.

Q2: Utilization of GPT

Thanks for highlighting the concurrent works [2] and [3]. While these referenced works also employ GPT for benchmark evaluation, notable distinctions exist between our approach and theirs, as outlined below.

(1) The way of leveraging GPT is different. Concurrent works (e.g., [2] and [3]) solely employ GPT for open-ended evaluation, while our approach integrates GPT not only in automatic evaluation but also in the data curation process. Specifically, we utilized GPT to automatically generate ability-aware questions, spanning a total of 17 different abilities. In contrast, [2] and [3] rely on existing datasets for evaluation, which, to some extent, may lack comprehensiveness. For instance, [2] comprises only 187 images and 205 questions, posing challenges for a comprehensive evaluation. In our exploration, beyond conducting evaluations based on GPT, we also showcased the advantages of using GPT as data curators, highlighting the potential of GPT in this context.

(2) The purpose of leveraging GPT is different. [2] and [3] utilized GPT-4 primarily for automated assessment. In contrast, we aim to present a system employing GPT for both data generation and evaluation, thereby offering a fully automated, scalable, user-friendly, and comprehensive pipeline

Q3: Assessments of different visual capabilities

We acknowledge that assessments of different visual capabilities have been mentioned in previous methods [2, 3, 4, 5]. Therefore, we conducted a quantitative comparison based on the number of assessed capabilities and the quantity of evaluation questions to highlight the diversity of our approach. The table below illustrates that our method significantly surpasses previous methods in the number of evaluation questions. For instance, we exceed the evaluation dataset of MMBench by tenfold (28,500 v.s. 2,974). Additionally, our method is comparable to previous methods in the number of assessed capabilities. We can also extend LLM to assess additional capabilities, thereby scaling our dataset further.

We sincerely thank you for your time! Hope we have addressed your concerns. We look forward to your reply and further discussions, thanks!

Sincerely,

Authors

[1] Haotian Liu, et al. Visual Instruction Tuning. arXiv preprint arXiv:2304.08485, 2023.

[2] Weihao Yu, et al. MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities. arXiv preprint arXiv:2308.02490, 2023.

[3] Shuai Bai, et al. TouchStone: Evaluating Vision-Language Models by Language Models. arXiv preprint arXiv:2308.16890, 2023.

[4] Chaoyou Fu, et al. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models. arXiv preprint arXiv:2306.13394, 2023.

[5] Yuan Liu, et al. MMBench: Is Your Multi-modal Model an All-around Player? arXiv preprint arXiv:2307.06281, 2023.

[6] Yonatan Bitton, et al. VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use. arXiv preprint arXiv:2308.06595. 2023.

Dear Review 69zX,

We appreciate your insightful and constructive feedback. We address your comments below, and discuss the revisions we made accordingly.

Q1: Fairness issue in evaluating VLMs that trained on other GPT-4 generated data.

While both the Auto-Bench and prior datasets (e.g., LLava-i-158K [0]) are curated using GPT-4, distinct variations exist as shown in the following, leading to a domain disparity between our data and that of others, thus ensuring fair evalution.

• Auto-Bench incoporates more visual symbolic information. Auto-Bench integrates a more comprehensive set of visual symbolic cues, including OCR, captions, instance relationships, and others. This diversification marks a departure from existing GPT-4 systhesis data.

• Auto-Bench generates a more diverse set of ability types. Previous benchmark generates questions from limited aspects: object types, object counting, object actions, object locations, and relative positions between objects. In contrast, the dataset in Auto-Bench encompasses a total of 17 distinct abilities, surpassing LLava-i-158K by more than fourfold, for example.

Besides, we observed that not only LLAVA but also most of the models (except BLIP2) compared in the main paper are trained with GPT4-generated data. However, Table 8 indicates there is no strong correlations bwteen the performance and the GPT4-generated data. For example, Blip2 (trained without GPT-4 data) perform better than MiniGPT-4 (trained with limited CC+ChatGPT instruction data) across various sub-skills.

These findings affirm the fairness and balance of our comparisons, addressing concerns about data homogeneity and model bias.

Q2: Fairness issue in evaluation VLMs that trained on coco caption data

Thanks for pointing out this concern. We believe the evaluation is fair due to the following two reasons:

• Compared with COCO caption dataset, Auto-Bench offers unique and previously unseen QAs by prompting GPT4 from diverse visual symbolic information. Though sharing the same COCO images, these unsceen QAs and domain disparities ensure the fair evaluation.

• We found that InstructBLIP is not the sole model trained with COCO captions; all models, with the exception of MiniGPT4, underwent training on COCO captions. Despite this shared training data, we did not observe an obvious positive correlation in performance when comparing models trained with and without COCO captions. This finding also underscores the distinctive and complex nature of the Auto-Bench dataset and indicates the fair evaluation.

We greatly appreciate your suggestions of Q1 and Q2 and have thoroughly discussed these potential issues in Section A.1.5.

Q3: Human verfication in validation data

Actually, we employed a crowdsourcing approach for manual validation, assessing each question against three criteria, including (1) the relevance to the sub-skill topic, (2) appropriateness of the answer, and (3) logical consistency in reasoning. The question is deemed valid only when it meets all criteria. Table 2 in the original main paper presents review statistics for 800 questions across 16 skills. We appreciate your reminder and have updated these details in Section 3.1 of the revised main paper.

Q4: How are images in the dataset categorized by specific subskills?

Thanks for the good questions. For general subskills, we designed questions for all images. For specific skills, we filtered images using COCO labels to ensure scenarios matching the skill. For instance, for physics, we only selected images with people and objects; for biology, we chose images with plants, vegetables, fruits, etc. This case-by-case approach also explains why there are fewer questions generated in the reasoning category. We have updated the details in Section A.1.1 of the Appendix.

Q5: Some illustrations of Open-ended questions

We provide the followig case for a better illustration.

• Image context: A downtown street with people hurrying by has a small store with colorful handicrafts in the window. A young man in sneakers and a sweatshirt stops and gazes at the window.
• Question related to reasoning: Why the man gazes at the window of the store?
• Answer 1: The young man is an art lover who is looking for a unique craft to decorate his home and highlight his taste in art.
• Answer 2: The young man, who is actually a basketball player, saw some unique handcrafted basketballs in a store and he wanted to purchase one as a souvenir.

As we can see, for reasoning tasks, there might be numerous correct answers in an open-ended questions, which challenges the LLM's judgment capability, often leading to additional errors. To mitigate this, we structured the reasoning questions in a multiple-choice format. This approach limits the degrees of freedom, enhancing the stability and reliability of the assessment.

Hi, thanks for answering my questions thoroughly. I am raising my score to 5 as their response alleviates some of my concerns, and will retain it since the idea of the work is not new to me -- GPT-4 has been used for generating instructions and judging responses in prior works separately if not in tandem.

I am adding more comments to improve your current draft.

1. Fairness

• I thank the authors for clarifications on the fairness of the evaluation. According to the authors, most (if not all) of the models have been trained with the GPT-4 generated instructions, COCO captions, and COCO images. I buy the argument that evaluation dataset is (un)fair to all the models equally.

• I do think that this fairness issue needs to be discussed in the main text (even 2-3 lines is fine) instead of completely relegating it to a small Appendix A.1.5

• GPT-4 generated data training: The authors utilize visual symbolic info, diverse set of ability types which they believe is absent in LLaVA-158K dataset. However, all such arguments make sense, I strong empirical evidence is still missing. I suggest the authors to perform similarity analysis between their dataset (and other LMM evaluation dataset) instructions vs LLaVA-158K instructions. It will give you a good sense of the diversity/difference with existing datasets. It will help you understand the nuances of the dataset better.

• COCO Images: Same as above. You can compare the similarity between your dataset (and other LMM eval) dataset images and LLaVA-158K images to better argue the diversity in your dataset.

2. Human verfication in validation data

• If human verification is done on 800 instances, then why do the authors claim 27.5K highly curated samples?

3. How are images in the dataset categorized by specific subskills?

• The authors should provide the heuristics in greater detail in the revised paper.

4. Structure of the reasoning questions

• I do not fully buy the argument provided for keeping reasoning questions as multiple-choice questions. If the LLM is not good as judge on evaluating the responses for the reasoning questions, it is a drawback that should be discussed in the paper. The whole claim of the paper is that LLM can replace humans as judges. But it seems that it is not always the case.

Good luck to the authors for their hard work!

Thanks for the prompt feedback. We are glad to note that we've resolved some of your concerns, including:

• (1) The evaluation on GPT-generated data is fair;
• (2) The diverse visual symbolic information and ability types differs Auto-Bench from other datasets.

Since there are still some concerns that have not been fully clarified, we aim to address your remaining comments in the following detailed responses.

Q1: Discussion of the fairness issue in the main paper

Thanks for the suggestions. We have incorporated the relevant discussions (as shown in the following italicized contents) into the Section 3.4 main of main paper to enhance reader comprehension of fairness issues.

''We aim to evaluate VLMs on the LLM-curated data using LLM's judgments. It is important to acknowledge a theoretical risk of unbalanced evaluation, given that some current VLMs are trained on LLM-generated datasets. However, considering that the Auto-Bench dataset offers substantially distinct datasets compared to previous ones, encompassing more abilities and diverse input visual symbolic information, and given that the selected VLMs are predominantly trained on LLM-generated data, the comparisons can be deemed fair.''

Q2: The strong empirical evidence of the difference between datasets

As suggested, we have provided two empirical evidences to conduct the differences/similarities analysis between Auto-Bench and others, which are shown in the following:

• (1) We have updated Figure 3 by incorporating the statistics of LLAVA-158K, generated by GPT4 as detailed in [1]. The results indicate that while Auto-Bench and LLAVA-158K share similar question lengths, Auto-Bench demonstrates greater diversity, as evidenced by the right part of Figure 3. We attribute this enhanced diversity in Auto-Bench to the incorporation of diverse visual symbolic information and ability types.
• (2) In addition to intra-correlations, we have conducted inter-correlations as outlined in the following table. Specifically, we randomly selected 2,000 questions from two distinct datasets and computed cosine similarities on the corresponding features extracted by the Simcse text encoder. Results (last line in the following table) shows that though both are generated from GPT-4, LLAVA-158K is not the semantically closest (since the score are not the highest) in resemblance to our Auto-Bench. The following table and related analysis have been updated in Section A.1.5 of the appendix.

Dear Review m7V9,

Thanks for your comments and acknowledgment of our novelty. We provide responses to your concerns below.

Q1: The used MS-COCO images may not cover the extensive image domains.

We appreciate and acknowledge the reviewer's observation regarding the limitations of using the COCO dataset, as it may not cover all image domains, such as medical images. The choice of COCO for evaluation was made because of its rich annotations. However, we would like to emphasize that the contribution of this work extends beyond providing a COCO-based dataset for evaluating VLMs.

The primary contribution, in fact, lies in offering an automatic pipeline that is readily extendable to new data domains and scenarios, such as VizWiz. To achieve this, we can leverage off-the-shelf perception models to obtain visual symbolic representations, as mentioned in our paper. Subsequently, we can conduct data curation and model evaluation tailored to the specific domain. We have incorporated this discussion into the revision outlined in Section A.1.1 of the Appendix.

Q2: Meaning of "Rationality" in Section 4.2

'Rationality' in this context refers to the logical coherence of a question in relation to the specified curation topic and the context presented within the image. We have added the descriptions in Section 4.2, for better clarity. Please check the latest revision.

We sincerely thank you for your time. Hope we have addressed your concerns with the above responses. We look forward to your reply and further discussions.

Best,

Authors