A-Bench: Are LMMs Masters at Evaluating AI-generated Images?

First, we would like to thank the reviewer for the constructive and valuable feedback. We have addressed your concerns point-by-point below.

1. How LMMs Reason Before Giving the Choice

Thanks for pointing out the importance of verifying whether LMMs truly understand their choice. Following your suggestions, we randomly sample a small subset which consists of 200 question-answer pairs for validation. Specifically, we modify the question query into:

#User: [Question] [Image Token]
[Choice A] [Choice B] [Choice C] [Choice D] 
Answer with the option’s letter from the given choices and give the reasons for why you choose this option.

The prompt includes the instruction give the reasons for why you choose this option to encourage the LMMs to explain their choices, allowing us to assess whether they truly understand the question or are merely guessing. Considering the complexity of LMM responses, introducing additional LMMs as judges could introduce bias; therefore, we conducted a user study to evaluate the alignment of the LMMs' reasoning processes with their choices. This evaluation is only applied when the LMMs provide the correct option. Specifically, we present observers with the image, question-answer pair, and the corresponding LMM response, asking them to judge whether the LMM’s choice and reasoning are aligned. If aligned, the response is scored 1, if not, it is scored 0. Each evaluation involves two observers. If their judgments match, the result is recorded, if they disagree, a third observer arbitrates, and the majority decision is recorded. We select some LMMs (GPT-4o, Gemini 1.5 Pro, CogVLM2-19B, LLaVA-NeXT-8B) with good performance on A-Bench for this experiment. The results are illustrated below:

From the results, we observe that in terms of semantic understanding, LMMs generally grasp the content of the questions well, with their reasoning process aligning closely with the chosen options. However, in areas related to visual quality understanding—where their performance is already weaker—some correct answers do not align with the reasoning, indicating a few cases of correct guesses. Nevertheless, the overall alignment accuracy remains acceptable, and the LMMs show relatively stable performance across questions within the same category. This suggests that A-Bench testing is still reasonable, accurate, and meaningful.

2. LMMs' Sensitivity to Content

Thanks for your comment. We further conduct a content sensitivity experiment based on your suggestions. We split the content into Human, Animal, and Objects, then we calculate the performance of each content type for illustration.

Interestingly, our findings indicate that LMMs perform best on images with Objects content, while they struggle more with images featuring Animal content. This may be because objects tend to be simpler and more straightforward, making them easier for the models to understand. In contrast, animals often exhibit unusual or complex generated structures and patterns, which can negatively impact the accuracy of LMMs' understanding.

First and foremost, we would like to thank the reviewer for the time and valuable feedback. We are sincerely grateful for the recognition and appreciation expressed. Our point-by-point responses are as follows:

1. Discussion About SelfEval

Thanks for your suggestions. We carefully review SelfEval [1] and find it highlights key weaknesses of using external models, such as: ``Evaluation metrics can vary widely depending on the chosen model, impacting reliability. If the same model is used in both training and evaluation, results may be biased, not reflecting true performance. External models often struggle with certain tasks, such as counting or recognizing specific attributes, making their evaluation scores unreliable'' This provides strong evidence that external models may not be reliable for evaluating T2I models.We will include a discussion of SelfEval in the introduction. Thank you for the reminder.

2. Responses Variance of LLMs/LMMs

Thank you for your question. Your concern is crucial, as the accuracy and stability of the benchmark directly affect the quality of evaluation. Here, we will address this: First, we use a consistent prompt instruction format to minimize any misunderstanding by LMMs and standardize the output. Additionally, we set the model's temperature parameter to 0, meaning the LMM's output will no longer be affected by randomness. As a result, the model will give the same response to the same question each time, eliminating variance.

It’s also worth noting that increasing the model's temperature to encourage more diverse and exploratory answers is indeed an interesting consideration. To further address your concern about the statistical significance of the experiment, which we believe is crucial and important, we repeat the A-Bench experiment for 5 rounds with different temperature settings across several popular 7B-8B LMMs. The performance is listed in the table below, with the results presented as the mean accuracy ± standard error.

Based on the results, we can observe that when the temperature is set to zero, the accuracy results for all LMMs remain consistent across all 5 rounds. As the temperature increases, the average performance declines and the results become more unstable, with higher standard errors. Therefore, to ensure reproducibility and performance stability, we prefer the zero-temperature setting, as it more accurately and reliably reflects the performance of LMMs, making it more suitable for practical applications.

3. Inappropriate Words and Writing Improvement

Thanks for your constructive suggestions. We have changed the word controversial on L272 into counter factual. We have improved our statement in the introduction that `A substantial performance gap of 16% remains between the best-performing LMMs and human evaluators on AIGI assessments, indicating significant room for improvement.'

Thanks again for your valuable suggestions on improving our work.

[1] Sai Saketh Rambhatla, Ishan Misra, SelfEval: Leveraging the discriminative nature of generative models for evaluation

First of all, we would like to thank the reviewers for the time and constructive feedback. We will address your concerns point by point.

1. Scientific Foundations for the Semantic Reasoning

We appreciate your inquiries about the scientific foundations of our selected aspects. We agree that ensuring comprehensive coverage is essential. When evaluating the semantic capabilities of LMMs, it is generally recommended to test from simpler to more complex tasks [1]. Thus, our aspect selection follows a progression from basic to more complex dimensions. Since our primary focus is on exploring LMM-based AIGI evaluations, we draw on previous work that uses LMMs for evaluation metric design [2, 3, 4], where LMMs are typically employed for image-related question-answer tasks. To evaluate the alignment between an AIGI and its prompt, we first assess overall subject alignment, which led us to select the first dimension, Basic Recognition. We then evaluate the attributes, actions, and interactions of objects in the AIGI, leading to the second dimension, Bag-of-Words Pitfalls Discrimination. Finally, given the creative nature of AIGI generation, which often requires external knowledge, we chose the third dimension, Outside Knowledge Realization.

We further refined our dimensions by considering specific aspects of AIGI generation that are particularly relevant:

1. Major Object Recognition and Minor Object Recognition focus on identifying generated objects, which is a fundamental capability of AIGIs [5, 6].
2. Attributes Awareness evaluates the model's sensitivity to object attributes, which is crucial for basic evaluations [1, 7].
3. Nouns as Adjectives Awareness addresses potential issues where T2I models may misinterpret nouns as adjectives, generating objects instead of intended attributes [8, 9].
4. Composition Identification pertains to understanding compositional relationships [10, 11].
5. Number of Objects Counting assesses the model's ability to accurately count objects, which is critical for checking if the AIGI matches numerical specifications in the prompt [12, 13].
6. Specific Terms Recognition involves identifying domain-specific scenes and objects, such as geography, sports, or food, important for external knowledge [14].
7. Contradiction Overcome tests the model's ability to correctly interpret AIGIs even when their content contradicts established world knowledge [15].

Thus, our dimension selection follows a suggested benchmark approach from simple to complex, with sub-dimensions designed to address critical points in AIGI generation and LMM evaluation. While we cannot guarantee absolute comprehensiveness in the semantic domain, we have aimed to cover the most important aspects for evaluating AIGI capabilities with LMMs. In line with your suggestions, we have provided corresponding citations and restructured our framework to align with the current evaluation setup. Thank you for your thoughtful feedback.

2. About the Dataset Size

Thank you for your comment. To begin with, we would like to briefly outline the dataset sizes of some of the poular LMM evaluation benchmarks. For example, the MMBench benchmark, which is widely recognized for its comprehensive evaluation of semantic understanding abilities, contains 2,948 multiple-choice questions (MCQs) [1]. In the domain of perceptual quality, the Q-Bench benchmark, which is quite popular, includes around 2,990 MCQs [16]. Based on these examples, we aimed to keep our dataset size around 3,000 MCQs. Initially, we designed a total of 3,000 MCQs, but during the data cleaning and validation process, we identified and removed some problematic or unsuitable items. As a result, we ended up with 2,864 MCQs, which was not an arbitrary decision. We hope this clarifies our rationale.

Additionally, since the A-Bench dataset is fully manually annotated and requires validation by at least three other humans, the annotation process is both costly and time-consuming. As such, it is quite challenging to scale up. We will acknowledge this limitation in our discussion section. However, we also plan to update and expand the dataset in future work, and we appreciate your understanding on this matter.

Dear Reviewer,

Thank you for recognizing the value of our paper and for providing valuable and constructive feedback. We have carefully addressed the concerns and incorporated additional details based on your thoughtful suggestions in the questions section. We kindly hope that these revisions may merit your consideration for a raised rating.

Should there be any remaining issues or concerns, we would greatly appreciate it if you could kindly point them out, allowing us the opportunity to further discuss and address them.

Thank you once again for your time and thoughtful review.

Best regards, The A-Bench Authors

We would like to thank the reviewer for the time and meaningful comments. First, we would like to kindly clarify a critical point that weakness 5 "Two-thirds of the low-level semantic question-answering data come from other datasets" is a misunderstanding. All the Question-Answering data in A-Bench is entirely original and created by us and none of it comes from any other existing datasets. We will then address your concerns point-by-point.

1. Adding Latest SOTA models

Thank you for your suggestions. We have further tested five additional models. Please note that part of the correct answers in the A-Bench dataset are kept private (to avoid data leak). The updated performance is shown in the table. Additionally, the version of GPT-4o used in the paper is GPT-4o-2024-05-13. For completeness, we have also included GPT-4O-2024-08-06 for comparison.

2. Timeliness of A-Bench

Creating a benchmark involves generating images, collecting data, training evaluators, and verifying data quality, making the process both time-consuming and costly. As a result, it is inevitable that AIGI benchmarks may not always keep pace with the latest technologies or models. However, the insights provided by the benchmark in evaluating AIGI remain valuable and offer useful guidance. We will address this limitation in the discussion section. Of course, we are committed to ongoing updates and expansions to ensure the benchmark remains current. We appreciate your understanding.

3. Question Annotation

Thank you for your comment. While human annotations require significant time and may cause AIGI benchmarks to lag behind the rapid evolution of technology, they remain essential. Human annotations are critical for AIGI benchmarks because they provide accurate, consistent, and context-sensitive evaluations that AI models alone cannot achieve, particularly when assessing subjective qualities such as creativity, coherence, and alignment with complex prompts. Although human involvement makes it more challenging to scale datasets and can delay the integration of the latest model advancements, it ensures reliable ground truth, addresses edge cases, and upholds ethical and cultural sensitivity. Moreover, human reviewers are capable of evaluating complex, nuanced aspects of generated images that current AI models cannot effectively assess, helping to ensure the benchmark remains both meaningful and trustworthy, even as AIGI technologies rapidly evolve.

Thank you for your suggestion regarding using prompts for annotation, we find it very valuable. However, there are exceptions, such as cases involving prompts that require outside knowledge, where the AIGI may not generate outputs as expected, necessitating human verification. Additionally, for visual quality annotations, the prompt itself has limited relevance. We also recognize that prompts could constrain annotators' creativity, potentially overlooking interesting and diverse questions. Nonetheless, we will consider your suggestion in future annotation efforts. Thank you again for your insightful feedback.

Dear Reviewer,

Thank you for your valuable and constructive feedback. We have carefully addressed your comments and revised the paper accordingly. We kindly ask if these revisions have resolved your concerns. If so, we would greatly appreciate your consideration of a raised rating.

If there are any remaining issues or concerns, we would be grateful if you could kindly point them out, allowing us the opportunity to discuss and address them further.

Thank you for your time and thoughtful review.

Best regards, A-Bench authors