AutoBench-V: Can Large Vision-Language Models Benchmark Themselves?

1. "they lack the flexibility xxx" in 053 may involve overclaim. For some charts and flowcharts (ChartQA, ScienceQA), it is difficult to draw these pictures using the T2I model, but this ability is also need to be considered. Many benchmarks consider inputting charts, documents and other pictures (MMT-Bench[1]). Although the author construct many types of tasks, all of these input pictures seem to be natural images. So I don't think this can completely become the reason for the author to claim that he is more flexible than other benchmarks.

2. The questions are all generated through LLM, lacking visual information. Although LLM is very common in constructing QA data, visual information is also very critical in multimodal evaluation. Simply using LLM to generate questions will ignore this aspect (for example, questions like: what is the object in the upper left corner of the picture; what is the red square object in the picture)

3. The author only evaluated seven LVLMs and there is only one open source model. Compared with other benchmarks (such as MME[2]), most of them evaluate 20+ LVLMs, which affects the second point stated by the author, contribution in line 123: "Extensive experiments...", which seems not to be extensive.

4. The author's contribution mentioned that the construction of AutoBench-V greatly reduced the cost of manually constructing benchmarks, which may be an overclaim. Although the author conducted a manual evaluation of the correctness of the automatically constructed benchmark, I prefer to see the manual comparison results of the automatically constructed benchmark and the manually collected benchmarks (like MME[2], MMBench[3], MEGABench[4]) in terms of quality, diversity, etc. This result better illustrates this contribution (in other words, it is not difficult to automatically build a benchmark, but it is difficult to have similar quality to the manually collected benchmarks)

5. The benchmark built by AutoBench-V may be too simple. GPT-4o has an accuracy of 75% on the hard type defined by the author and an accuracy of 90% on the easy type, which is too high compared to other benchmarks, e.g. GPT-4o only has 54% on MEGA-Bench[4].

[1] MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI

[2] MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models

[3] MMBench: Is Your Multi-modal Model an All-around Player?

[4] MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks

However, my biggest concern lies in the experiments. In fact, there is one critical issue that greatly undermines my overall impression regarding the value of this work -

Since all the prompts and the test cases are generated using GPT4o on-the-go, which is the sole Examiner/Judge in the paper's setting, aren't the baseline performances in Table 2 by GPT-4o technically obtained by testing on the training set?

Even if the prevention actions of self-enhancement leakage are done on the image description text encoder factor as stated in Section 4.2, the ad hoc-generated visual examples used in the test cases, as well as the guidelines used along the pipeline, are not fully deprived of the confounding factor that is the prior brought by GPT-4o. Thus, the top-benchmarking performances by GPT-4o, and to a lesser extent GPT4o-mini, in Table 2, are very unconvincing results to me.

The paper should include more discussion on quality control for the dataset, such as avoiding issues with generating incoherent or low-quality images and questions.

Unfortunately the presentation quality is absolutely not ready for publication. The very first sentence of the introduction already contains several errors: Authors cite the “Attention is all you need” from 2017 as a 2023 paper. LLaVA is cited as an LLM even though it is a vision model. And the formulation suggests that the LLM works from 2023 pave the way for NLP works from 2020, which cannot hold.

This continues in the related work section, where this paper is cited as past work: (1) “Deep visual-semantic alignments for generating image descriptions” with then a subsequent study (2): “Microsoft COCO: Common Objects in Context**”** mentioned. However in reality it is the other way around, where paper (1) uses the COCO dataset (2) for evaluation. Then, ImageNet is cited as a “benchmark for assessing LVLMs”, however it is an image classification dataset. Finally ImageNet is cited as an arxiv paper instead of properly citing it as a International Journal of Computer Vision paper. Clearly these citations were not thoroughly examined.

Citations in the references are all lowercased, probably due to incorrect usage of bibtex. English quality is low which makes the paper difficult to read, e.g. the sentences in lines 85-87, and 136-137 could be written more cleary. The paper contains typos like “fine-frained” aspect on line 215.

Method: The benchmark covers “atmospheric understanding”, however the cited paper does not contain the word atmospheric but talks about emotion recognition (one of the 4 aspects under atmospheric understanding). Figure 1 does not properly show which question belongs to which aspect, leaving the reader guessing. Also, the hierarchical structure seems random, e.g. it contains “emotional and psychological” on the left under semantic understanding, and “emotional cues” under “atmospheric understanding”. This hints to problems of automatically generating this hierarchy of aspects using GPT-4. Since it is only a few aspects, those could have been generated manually.

The guidelines chapter is not understandable to me. Sentences like “This guideline acts as a guideline…” do not help. Details are missing: How is the exact procedure? The descriptions of the method are more prosaic than scientific. The reader has to find and read the prompt in figure 23, which is not referenced to in the chapter.

Similarly in the next paragraph, image description with difficulty grading. Even after reading Appendix C.1 it is unclear which prompts are used, and how any of this would not suffer from the common weaknesses of VLMs (e.g. hallucinations, or ignoring some parts of the text when generating images).

In the semantic graph part, the authors describe what the function f could be, but miss describing what it actually is in the end.

A big problem in VQA is that questions can be ambiguous. E.g. in figure 198 the question is formulated in a way that the answer “the stem appears to be pointing upwards” is reasonable but does not match the ground truth.

In figure 19 the ground truth is just wrong, since the child is not wearing a red clothing. Just from these two examples, the generated benchmark seems to be too low quality to provide meaningful results.

Summary: The presentation quality must be improved substantially which would suggest a resubmission. The benchmark quality is at least questionable but in the currently presented form cannot be judged well. Not properly citing related work is a KO criterium.