What makes a good metric? Evaluating automatic metrics for text-to-image consistency

Thank you for your comment that this work provides an interesting analysis and that it would be a good contribution to COLM! We address your questions below.

• abstractness of some visual metrics: We understand that the abstractness of some of our metrics (e.g. imageability) may seem a little unintuitive. They are however quite standard in the psych/compiling side of things, and since they are underexplored in benchmarks for vision-language modeling, we felt it was important to include them here.
• comparison to existing preprint: Thank you for pointing us to this work! It was a nice read and we’ll absolutely include a discussion of it in our next draft. For example, both their paper and ours address evaluating automatic metrics and consider CLIPScore, but in the end, the two works use different methods but draw complementary yet different conclusions. We focus on how well metrics correlate with linguistic and visual properties and their failure points via robustness to known shortcuts. They focus on how well metrics can separate faithful and unfaithful image generations using a set of existing image/text prompts. We address inter-metric correlation, but they don’t cover that. Both works do agree that more expensive, VQA-based metrics are not necessarily more robust than simpler ones like CLIPScore. We’ll definitely watch these authors’ contributions carefully as well going forward, as it seems we have similar interests!

Thank you for the note about the two titles, somehow that slipped through editing. In the end, we’re going with “What makes a good metric? Meta-evaluating automatic metrics for text-to-image consistency.” We will update this in OpenReview.

Thank you for the thoughtful review! We agree that the desiderata are useful for defining metrics and will be useful for future metrics as well.

• Comment on these results not being surprising: A primary motivation for our analysis is the fact that these metrics are being widely used right now; if there are issues with them, this affects work on creating SOTA models for T2I generation. These metrics are interestingly even used in much more specific analyses beyond just overall “does this image match this text?” For instance, they’re used in benchmarking the compositional understanding of T2I models [1] and for fairness measurements in generated images [2]. This makes having a comprehensive understanding of these metrics – as well as weaknesses and places for improvement – very important.
• Comment on delivering a new metric: We agree that introducing new metrics are important! Our paper intentionally focuses on the first stage of this process, by defining desiderata and doing a deep-dive into existing metrics to lay the foundation for a follow-up work. Our experiments in Section 4 are designed to robustly identify failure points of existing metrics (for instance, a lack of sensitivity to word order in the text prompt and generated questions) that should be ablated in new metrics. Creating a new metric is an exhaustive process that would warrant a separate paper. For what it’s worth, we are exploring an approach to improve the metrics which relies on generating hard negative questions. We’ve discovered that this approach is promising, but we are still working through some of the specifics. We decided to share issues with current metrics before we have fully solidified our positive metric proposal, because these metrics are really catching on in the field—we want to raise the alarm before the field advances much further on somewhat shaky foundations.

[1] A Contrastive Compositional Benchmark for Text-to-Image Synthesis: A Study with Unified Text-to-Image Fidelity Metrics

[2] DIG In: Evaluating disparities in image generations with indicators for geographic diversity

We appreciate your comments that our work is clear, convincing and comprehensive!

• overview figure and illustrating examples: This is a good point about having additional illustrating examples, particularly around the generation. We’ll use the camera ready’s additional page and the appendix. We’ll focus in particular on examples where (i) all metrics perform well (ii) CLIPScore performs well & VQA-based metrics do not (iii) VQA-based metrics do well & CLIPScore does not and (iv) all metrics struggle. We hope this will qualitatively help to interpret each metric’s strengths/weaknesses.
• correlation between #/generated questions and metric scores: Ah, we see how this is a bit confusingly worded. Indeed, you are right, looking only at the #/questions would be a bad T2I metric (according to our desiderata too), and we definitely don't want to propose that. The fact that the #/generated questions appears to affect the measurement, when it shouldn't, suggests the metric is flawed. This is likely because the chances of answering any question in the affirmative is high (as we argue in Section 3.4).
• correlation between CLIPScore and VQA-based metrics: Good question! Here’s our intuition on why VQA-based metrics correlate w/ one another but not with CLIPScore. There are different types of examples where each metric will “thrive”. Let’s take the prompt, “two dogs to the left of a cat”. The cardinality & object position might be difficult for CLIPScore, as CLIP was not trained to understand fine-grained semantics. VQA-based metrics, on the other hand, do leverage fine-grained questions (“how many dogs are there?” “is the cat on the left?”, etc.). Now, let’s consider another prompt, “there’s a white dog and a black dog?” While fine-grained, the generated questions may be underspecified (“Is the dog white?” “Is the dog black?”). The grounding of which dog is being referred to in each question is nontrivial (e.g. the answer to “is the dog white” depends on which dog is being referred to, which is not specified). This is an instance where CLIPScore may perform better. Given these different classes/types of questions where one type of metric struggles where the other may thrive, this could explain the lack of correlation.
• In terms of your comment on the small, focused nature of our work, we have also addressed a similar concern to reviewer KjKK in “Comment on these results not being surprising”; it might be of interest to you to read our response to them below.