Dysca: A Dynamic and Scalable Benchmark for Evaluating Perception Ability of LVLMs

Thank you for your valuable feedback! We really appreciate that you found our work is intriguing and worthwhile exploration.

The major concern with this dataset is the use of image generation tool for automatic vision creation. For such images to serve as a gold standard for MLLM evaluation, they must meet a critical assumption: the generated images are perfect and error-free. Unfortunately, even today's state-of-the-art SDXL models, which can produce high-quality images, still blend real and synthetic styles rather than producing wholly realistic images. As humans, we can typically recognize diffusion-synthesized images. SD models excel at creating artistic-style images, but for sure, non-realistic images inevitably introduce bias in evaluations on realistic images. Also, many automatically synthesized images that contain scene texts are often of low quality, i.e., containing inaccurate scene text. Such biases could largely skew model evaluations. Therefore, the reviewer suggests that the authors add ample discussion regarding this issue. For example, the authors could include some in-depth analytical experiments to validate the concerns raised above.

Good point. We understand the reviewers' concerns regarding the utility of image generation tool for automatic vision creation. To ensure the quality of our data, four steps are adopt as follows:

1. First, we manually remove difficult-to-generate foregrounds and attributes, along with backgrounds and styles that could heavily affect image content. We believe this process can serve as a coarse-grained method to eliminate samples that are highly likely to be generated incorrectly.
2. Then, we use CLIP for further image filtering. In the experiment, we randomly choose 1000 samples first and apply filtering with thresholds of 0.65, 0.7, 0.75, and 0.8 individually. Our findings indicate that using 0.75 as the threshold achieve a good balance between image correctness and data scale.
3. After that, we leverage top 6 LVLMs to eliminate any question-answer pairs where the models either answer incorrectly or indicate that the answer is not included among the options. we observe that nearly 100% of the samples filtered out by these models are incorrect.
4. Finally, we analyze the patterns in these incorrect samples, removing the associated vocabulary from our metadata and discarding all related samples.

By meticulously refining the metadata manually and utilizing automated tools to assist in question filtering, Dysca ensures high-quality data synthesis. In the end, we filter out nearly 40% of low quality samples.

Besides, we agree with the reviewer that benchmark creators should aim for a perfect and error-free benchmark. However, in practice, many mainstream benchmarks labelled by human contain significant labeling errors [R1]. For example, the ImageNet [R2] has at least 2,916 errors in its validation set, with an error rate of 6%; the QuickDraw [R3] has at least 5 million errors, with an error rate of approximately 10%. Despite the labeling noise in these benchmarks, it does not affect their validity as widely used evaluation benchmarks. In appendix C, we randomly sample 7.7K images (10% of our data) and manually select images that contain wrong content or potentially lead to incorrect responses. In the end, a total of 167 images (2.2%) are filtered. Compared with these benchmarks [R1], Dysca achieves a lower error rates than these benchmarks.

In addition, Dysca also includes various question-answer pairs generated from prompts with realistic styles, such as “HDR photo” and "iPhone photo". These images are visually indistinguishable from those sampled from the real world. Furthermore, in the section 4.2.2 of the main paper, we find that the distribution between Dysca and real datasets is minimal, indicating that the evaluation results from our VQAs effectively reflect the model’s performance in real-world scenarios. As the reviewer says, "As humans, we can typically recognize diffusion-synthesized images." LVLMs also require the ability to recognize diffusion-synthesized images. We believe that Dysca can serve as a diverse and comprehensive evaluation benchmark and contribute to the community in evaluating LVLMs.

Q3: My doubts about their role in testing seem not to have been addressed by the authors. I would prefer the authors to design some quantitative experiments and analyses in this work to help confirm that “automatically synthesized images are also valuable for testing”.

We appreciate the reviewer's suggestion for design more quantitative experiments and analyses to introduce more insightful observations. Thanks to generative evaluation framework of Dysca, we are able to effectively control variables and conduct a detailed analysis of the model's fine-grained capabilities.

Specifically, we first introduce two metrics to measure the sensitivity of LVLMs on question types and covariate shift. The sensitivity to question type aims to evaluate whether LVLMs exhibit inconsistent performance when facing different types of question types (i.e., multi-choice vs. true-or-false). The sensitivity to covariate shift aims to evaluate whether LVLMs exhibit inconsistent performance when facing the same content and question format, but with variations in image covariates (i.e., image style).

Metrics:

1 - Sensitivity to Question Types:

• True-or-false:

  $$TFTS=\frac{S-50}{100-50}*100\%.$$

• Multi-choice:

  $$MCTS = \frac{S-25}{100-25}*100\%.$$

• Overall:

  $$TS_{Avg} = \frac{TFTS+MCTS}{2},\\ TS_{Var} = \frac{(TFTS-TS_{Avg})^2+(MCTS-TS_{Avg})^2}{2}，$$

where $S$ denotes the score of the models in each question type. $TS_{Var}$ denotes the sensitivity of LVLMs to question types. The lower, the better.

2 - Sensitivity to Covariate Shift:

where $S$ denotes the score of the models in each scenario. Since there are 51 image styles in Dysca, we set $N=51$. $SS_{Var}$ denotes the sensitivity of LVLMs to covariate shift. The lower, the better.

Table below shows the performance of each LVLM on the two metrics, where 'Score' represents the model's perception score across all samples.

Key observations:

1. For the Sensitivity to Question Type : Xcomposer2-VL achieves the best result with a score of 20.9. However, two phenomena are observed: First, the perception ability of LVLM does not show a direct positive correlation with sensitivity to question types. For instance, while the GLM-4V model achieves the highest performance in evaluation tasks, it exhibits higher sensitivity to question types than Xcomposer2-VL. Second, one of the factors influencing the sensitivity to question types may be the inherent biases of the language model. Using the same base language model may result in similar outcomes for this metric. Additionally, it is noted that the Gemini model does not show a significant advantage in this metric, revealing its preference for certain question types.
2. For the Sensitivity to Covariate Shift: GLM-4V achieves the best result with a score of 0.8. However, we also observe that the perception ability of LVLM does not show a direct positive correlation with covariate shift. For example, InstructBLIP-Flan-T5-XXL outperforms InstructBLIP-Flan-T5-XL in terms of performance but shows higher sensitivity to covariate shift.

In addition, since Dysca is able to maintain consistency in both image style and question type, we assess the performance of LVLMs in terms of inter-dimension and intra-dimension differences, where "dimension" denotes the 20 sub-tasks we proposed in Dysca.

In-depth analysis

• Analysis on Inter-dimension:

Here, we perform hierarchical clustering based on the Euclidean distance of the scores across 20 evaluation dimensions for 26 models. Based on the consistency of their performance across different tasks, we categorize the current LVLMs into four levels (first level indicates the best):

We observe that:

1. LVLMs tend to perform better in dimensions that involve well-defined image perception, such as landmark recognition and object recognition. However, dimensions likes style recognition and movie recognition exhibit greater challenges for LVLMs, which may be attributed to the limited training resources in these specific domains.
2. Commercial models exhibit poor performance in tasks related to "race recognition." This is likely due to the additional safety training incorporated into closed-source models, which causes these models to refuse to answer questions related to race-related issues.

Analysis on Intra-dimension:

Thanks to the diverse metadata design, Dysca enables highly granular analysis. Taking animal categories as an example, we analyze the performance of various models within each category. Here, we present the evaluation results of six advanced LVLMs across nine animal categories on multiple-choice questions.

The metadata covers 51 animal categories, and we observed that model performance varies significantly across these categories. For animals like seagull and shrimp, models tend to perform worse compared to other animal categories. This is likely because these animals are harder to collect and incorporate into model training, highlighting the need to direct the model's focus to corresponding domains.

Dear Reviewer 3C5q:

We sincerely appreciate your thoughtful and detailed feedback, which has greatly enhanced the clarity and quality of our paper.

Q1.1: And I still believe that since the process is fully automated, it will inevitably affect the quality of the evaluation testing set.

We acknowledge that using a generated paradigm may introduce some errors. However, we carefully cleaned the metadata and used various tools to ensure the dataset has very few mistakes. Additionally, when we evaluated the models using both manually cleaned and uncleaned VQA data, we found no significant difference in the scores, suggesting that the automatic evaluation results are reliable.

Q1.2: I remain unconvinced that SDXL’s inclination toward generating artistic-style images does not result in significant style and domain biases. Such images can never replace realistic images in evaluations.

We understand the the reviewers' concerns on the potential bias on the model would react between synthesized and real images. In the section 4.2.2 of the main paper, we report the data distribution distance between our dataset and the current existing datasets. We select CCBench, COCO_Val, MMVet, MMBench, MME, MMStar, OCRBench and ScienceQA. The reason why we choose these benchmarks is that they have been widely used in evaluating LVLMs. We use Kernel Maximum Mean Discrepancy (KMMD) to measure the distribution distance. Specifically, we randomly sample 3,000 images from each benchmark (if the scale of the benchmark less than 3000, we use all that data) and utilize CLIP to encode these images. Then, we calculate the KMMD value using an RBF kernel between each pair. The results are shown below:

Each row represents the value of KMMD between two benchmarks. The value of last row denotes the average value of KMMD. As can be seen, the distribution distance between Dysca and real-image benchmarks ranks in the middle compared to all other benchmarks. This demonstrates that the distribution between our benchmark and real benchmarks is similar, indicating that the evaluation results can effectively reflect the model's performance in real-world scenarios.

We sincerely thanks for your positive and constructive feedback. We address your concerns as follows.

How does the proposed benchmark differ from unifying existing benchmarks into one benchmark? Does it evaluate the model behavior that existing benchmarks fail to cover? Combining the existing benchmarks proposed in Table 1 creates a benchmark covering a wide range of perceptual tasks and question types. Therefore, it should clearly show something that existing benchmarks have not been able to do. For example, it would be valuable to show researchers the characteristics of the samples that all VLMs get wrong.

Compared to previous datasets, Dysca provide insightful analyses that others don't:

• The vary subtasks provide the fine-grained weakness detection for LVLMs. For example, in the case of Qwen-VL-Chat , it achieves a score of 96.96% accuracy in landmark recognition task, but obtains a score of 53.48% accuracy in age recognition task. The result suggests that Qwen-VL-Chat may require more fine-tuning in age perception data. The performance variability of models across different sub-tasks enable developers to make targeted improvements.
• Dysca includes diverse questioning types (e.g., Multi-choices, True-or-false, and Image Caption), highlighting the biases models exhibit in response to different types of questions. For example, Otter achieves an accuracy of 47.31% in the multi-choices question type but obtains an accuracy of 82.51% in the true-or-false question type. The problem exposes the bias in the training dataset of LVLMs towards particular question types.

We acknowledge that our benchmark can be seen as the union of perceptual tasks from all existing benchmarks. However, we believe this highlights our advantage: Dysca, as a single benchmark, can complete the evaluation tasks of many different benchmarks do. Although there already exist many benchmarks for LVLMs (as we discussed in Table 1), we provide an end-to-end process from image to Vision-QA generation. Our approach significantly reduces annotation costs compared to manually labeling images (e.g., MME). It also avoids the risk of hallucinate annotations that may occur when using ChatGPT for labeling based on image prompts (e.g., JourneyDB). This novel pipeline enables us to create a benchmark that is easily scalable and adaptable for incorporating new subtasks and scenarios.

We appreciate the valuable suggestion to show more samples that all VLMs fail to answer correctly. In Figure 4 of the main paper, we present the responses of four advanced models to a sample with a print attacking. As shown, when the word "False" is added to the image, all models, including GPT-4o, Gemini-1.5-pro, GLM4-V, and Qwen-VL-Chat, produce incorrect answers. This result underscores the vulnerability of these models in print attacking scenarios. We commit to including additional samples that multiple VLMs fail to answer correctly in the final version of our paper.

It is worth noting that VLMs tend to perform poorly when applied to marine life. This could be caused by the challenges in collecting ocean-related data or possibly to the fact that human understanding of land environments is better than that of the ocean. This observation may indicate a need for more specialized VLMs tailored to oceanic domains.

Regarding reviewer 3C5q's feedback on the Q3 answer, reviewer 3C5q mentioned: "I would prefer the authors to design some quantitative experiments and analyses in this work to help confirm that ‘automatically synthesized images are also valuable for testing.’” I'm in agreement with this response. As such, I have commented on proposing new metrics or presenting results that reveal insights not covered by existing benchmarks. Including these aspects in the main paper would strengthen the work.

Lastly, it is somewhat challenging to identify the changes made in the revised version. It might be helpful to highlight the revisions in a different color.

Key observations:

1. For the Sensitivity to Question Type : Xcomposer2-VL achieves the best result with a score of 20.9. However, two phenomena are observed: First, the perception ability of LVLM does not show a direct positive correlation with sensitivity to question types. For instance, while the GLM-4V model achieves the highest performance in evaluation tasks, it exhibits higher sensitivity to question types than Xcomposer2-VL. Second, one of the factors influencing the sensitivity to question types may be the inherent biases of the language model. Using the same base language model may result in similar outcomes for this metric. Additionally, it is noted that the Gemini model does not show a significant advantage in this metric, revealing its preference for certain question types.
2. For the Sensitivity to Covariate Shift: GLM-4V achieves the best result with a score of 0.8. However, we also observe that the perception ability of LVLM does not show a direct positive correlation with covariate shift. For example, InstructBLIP-Flan-T5-XXL outperforms InstructBLIP-Flan-T5-XL in terms of performance but shows higher sensitivity to covariate shift.

In addition, since Dysca is able to maintain consistency in both image style and question type, we assess the performance of LVLMs in terms of inter-dimension and intra-dimension differences, where "dimension" denotes the 20 sub-tasks we proposed in Dysca.

In-depth analysis

• Analysis on Inter-dimension:

Here, we perform hierarchical clustering based on the Euclidean distance of the scores across 20 evaluation dimensions for 26 models. Based on the consistency of their performance across different tasks, we categorize the current LVLMs into four levels (first level indicates the best):

We observe that:

1. LVLMs tend to perform better in dimensions that involve well-defined image perception, such as landmark recognition and object recognition. However, dimensions likes style recognition and movie recognition exhibit greater challenges for LVLMs, which may be attributed to the limited training resources in these specific domains.
2. Commercial models exhibit poor performance in tasks related to "race recognition." This is likely due to the additional safety training incorporated into closed-source models, which causes these models to refuse to answer questions related to race-related issues.

Analysis on Intra-dimension:

Thanks to the diverse metadata design, Dysca enables highly granular analysis. Taking animal categories as an example, we analyze the performance of various models within each category. Here, we present the evaluation results of six advanced LVLMs across nine animal categories on multiple-choice questions.

The metadata covers 51 animal categories, and we observed that model performance varies significantly across these categories. For animals like seagull and shrimp, models tend to perform worse compared to other animal categories. This is likely because these animals are harder to collect and incorporate into model training, highlighting the need to direct the model's focus to corresponding domains.

Thank you for your valuable feedback! We really appreciate that you found our work is good-quality and comprehensive.

No data leakage is one of the main claimed contribution for the dataset, but there is no comprehensive statistical comparison of Dysca with existing benchmarks. Only the blind experiment in Appendix D, which should be moved to the main content considering its importance, trying to explain this point which is not enough from my perspective.

Good suggestion! We appreciate the valuable suggestion to move the blind experiment in Appendix D and add more statistical comparison of Dysca with existing benchmarks. In MMStar [R1], the authors report widely-used benchmarks enable LVLMs to answer text-only questions with an accuracy higher than random selection, including MMMU [R2], MMB [R3], ScienceQA [R4], AI2D [R5], Seed [R6] and MathVista [R7]. Here, we present the evaluation results of LLAVA-1.5-7B [R8] on Dysca and the six benchmarks, showing the accuracy of responses when only textual questions are provided. Here, we present the evaluation results of LLAVA-1.5-7B on Dysca and the six datasets, along with a comparison to random selection.

As can be seen, LLAVA significantly outperforms random selection when only text questions are provided in the other 6 benchmarks. In contrast, Dysca achieves results closest to random selection, indicating that our work has a limited data leakage issue.

The setup of adversarial attack, i.e., only PGD, is too simple. Compare to the corruption and printing attack setup, only PGD is considered for adversarial attack is not convincible, especially considering there are bunch of emerging studies on the adversarial attack of LVLMs.

We acknowledge that PGD is a simple attack method targeting LVLMs. However, simplicity does not equate to "ineffectiveness." As demonstrated, when using the PGD algorithm for attacks, models experience performance drops exceeding 50% in some cases. For instance, Xcomposer-VL achieves a score of 71.40 on benign samples but drops to a score of 30.28 when evaluated on adversarial samples, a decrease of approximately 58%. We believe that the severe performance degradation caused by even the simplest adversarial attacks is a critical issue that model developers need to address. As a gradually evolving dataset, Dysca will incorporate more diverse attack methods in the future to enrich our evaluation scenarios.

There is no description of the rule-based approach adopted when getting the prompt and QA from the metadata.

The process of generating the prompt and QA is shown in Figure 3. We first construct four lists containing style, attribute, foreground, and background, respectively. The prompt is combined by randomly selecting a word from each of the four lists. Each word is donated as the answer and then classified into one of 20 perceptual tasks. We design specific rules for generating questions for each tasks. For more details, please refer to the QA generation code released on the anonymous GitHub repository at https://github.com/Benchmark-Dysca/Dysca/blob/main/pqa_generate/pqa_generate.py .

No explanation of the metadata itself. Why the metadata is combined with the four aspects? Is there any dataset- or task-specific relationship when considering this?

The combination of the four aspects (including their order) is designed based on the experiment. We find that the combination effectively generates prompts that produce accurate images for text-to-image diffusion models. Additionally, these four components cover enough dimensions for the evaluation tasks and serve as a flexible framework to add new evaluation dimensions. In the future, we will also consider adding more aspects if new task dimensions are required.