EvalAlign: Supervised Fine-Tuning Multimodal LLMs with Human-Aligned Data for Evaluating Text-to-Image Models

I have some concerns on the paper.

First, I am a bit concerned on how the score is derived. From my current understanding, the final scores are derived from an average of the score outputs of several questions. Is ther Human column also obtained by so? If so, this might not be a good enough ground truth.

Second, well in Sec. 5.1 the author states that the test set images do no overlap with train set ones, they do come from the same 16 generation models. As the final evaluation only shows model-wise ranking consistency, this result might not enough exclude overfitting (e.g. memorizing on model specific styles, etc). I would encourage a further testing on several hold-out T2I generators.

Third, a minor question. Using SFT for LMM to score has been discussed by Q-Align (ICML2024), which finds out using logits are better than using model.generate() for scoring. It also has the ability for image faithfulness evaluation, please try to compare with it or discuss with it. Furthermore, for faithfulness evaluation (which is actually image quality, am I right?), the compared baselines are similarity-based metrics (which are, from their design, alignment-related metrics). I would suggest the authors to compare with some baselines related to T2I quality evaluation (inc. Q-Align) in this part.

1. Lack of Justification for Main Contributions

The paper's three key contributions are insufficiently supported by experimental validation and theoretical grounding:

1. Although the dataset is described as having detailed human feedback covering "11 skills and 2 aspects," the experiments primarily focus on the 2 broad aspects. There is little exploration of the 11 specific skills, which would have been valuable given that the 2 aspects have been widely studied in prior works, such as [a].
2. The method is claimed to enable "accurate, comprehensive, fine-grained, and interpretable" evaluations. However, the results mostly reflect the 2-aspect performance, with no evidence of superior fine-grained or interpretability-focused evaluation compared to previous methods.
3. While the paper emphasizes cost-efficiency in terms of annotation and computation, this claim is questionable. The annotation process requires extensive human annotations, which is labor-intensive. Additionally, the method achieves optimal performance with a 34B MLLM model, which is computationally expensive.

2. Unclear Advantages Over Existing Datasets

• According to Table 1, the primary benefit of the proposed dataset seems to be its focus on the two-aspect evaluation. However, several prior datasets such as ImageReward, PickScore, and HPS(v2) implicitly address these aspects as well. While explicit question-based feedback is used in this work, it is not clear how this approach leads to better evaluation outcomes, especially since a vast number of questions would likely be required to cover all image aspects comprehensively.
• The paper does not adequately compare its approach with previous work like [a], which also includes detailed, multi-aspect human feedback via scoring rather than question-answering. The advantages of question-based feedback over scoring are not clearly demonstrated in terms of faithfulness or alignment evaluation.
• While the paper emphasizes cost-effectiveness, the dataset requires 130k annotations to achieve optimal results (Table 1). This annotation volume does not appear more economical than previous datasets.

3. Weak Experimental Results

• It is unclear whether models from other methods were trained on the proposed dataset to ensure a fair comparison, especially in Tables 2 and 3.
• The results in these tables do not consistently support the claims made. For instance, the 500 configuration does not show a clear optimal performance in Table 6, and there is no clear positive correlation between model size and performance improvements in Table 7.

4. Writing and Structure Issues

• Some sentences lack clarity and coherence. For instance, “the utilized synthesized images are treated as real images as they don’t explicitly recognize the problem of synthesized images with low image faithfulness” is confusing, especially since HPS(v2) aims to evaluate generated images.
• Writing structure should be improved. The key novel contributions of the method is unclear.
• There are some repeated sentences with similar meanings. It is better to re-write them to make the paper more concise.

Conclusion

While the paper presents a promising approach with potential contributions in the form of a detailed dataset and a new evaluation method, it currently lacks sufficient support for its claims. The advantages over existing work remain unclear, and the experimental validation needs improvement. Therefore, the paper is not yet ready for acceptance in its current form.

[a] Rich Human Feedback for Text-to-Image Generation, CVPR 2024, best paper.

1. The experimental part of this article has a big problem. Table 3 seems to have done a lot of experiments, but it is actually evaluated at the model level, not the instance level. This is not a challenging task, because everyone knows that PixArt draws well and SD 1.4 draws relatively poorly. Ranking the strengths of 24 models is far less meaningful than scoring a single image, that is, an end-to-end AIGC quality evaluation tool. In other words, which of the two images from the same model has higher quality is more important.

2. This paper only reviews coarse-grained datasets in related work, but does not consider fine-grained datasets. In addition, some AIGC-related dataset such as [1,2,3] was not considered. These datasets have fewer images but more annotations, and each image contains dozens of fine-grained annotations. Since fine-grained annotations are one of the major innovations of this paper, it is not comprehensive to only review coarse-grained datasets (i.e. only two or three annotations, or even less than one per images).

[1] PKU-AIGIQA-4K: A Perceptual Quality Assessment Database for Both Text-to-Image and Image-to-Image AI-Generated Images

[2] PKU-AIGI-500K: A Neural Compression Benchmark And Model for AI-Generated Images

[3] PKU-I2IQA: An image-to-image quality assessment database for ai generated images

1. The evaluation is done at the model level, not the instance level. This is a major flaw. In the actual evaluation process, the community is not only concerned with ranking the strength of the T2I model but whether each AIGC image is good enough. As far as I know, AIGC quality asessment [1,2] uses instance-level, because model-level evaluation is not very challenging (see Vbench [3] Figure 4, the correlation with subjective labels can easily reach 0.9). I hope the author can improve this point.
2. The author considered 24 models. Although the number is large, they are highly homogenized. For example, DeepFloyd IF uses three different conditions, which are considered three models, but they are the same. Including SD 1.4, 1.5, 2.0, and 2.1, the difference in visual effects is quite limited. However, for open-source models such as DALLE 3 and Midjourney, the dataset does not include them. From my subjective perspective, they are still slightly stronger than PixArt, and ignoring them will result in an incomplete dataset.

[1] Depicting Beyond Scores: Advancing Image Quality Assessment through Multi-modal Language Models [2] Descriptive Image Quality Assessment in the Wild [3] VBench: Comprehensive Benchmark Suite for Video Generative Models