Who Evaluates the Evaluations? Objectively Scoring Text-to-Image Prompt Coherence Metrics with T2IScoreScore (TS2)

Thank you for your thoughtful review. We are heartened to hear that you view our meta-metrics as a novel strength, and that you appreciate our comprehensive experiments that really attempted to make the strongest possible case for the QG/A metrics by analyzing multiple backends. We hope we can satisfactorily answer your question.

• On the weakness of evaluating T2I faithfulness metrics as score regressors

You are right to point out that one of the motivations of the QG/A metrics, which we do not inspect in detail here, is that they can provide fine-grained, (ideally) explainable assessments of T2I outputs over defined axes, as opposed to the other metrics providing simple numerical outputs. It is important to note this as a strength.

However, we are not the only ones who give the QG/A metrics a score regressor treatment—in their own works introducing the QG/A metrics, the authors justify their contributions with the same score regressor methodology, against the very same correlation-based metrics we analyze. In their works, using their ad-hoc evaluation sets and simple metrics, they claim significant gains in this score regression performance over the correlation metrics.

Simply by building a more carefully considered test set and metrics that are tailored to evaluate objective, relative errors alone without contamination by other forms of human preference such as aesthetics, we find surprising contrary results—that the regressor improvements claimed in the works introducing TIFA and DSG are illusory when considering structural correctness of the images alone.

That being said, you are right to point out that contextualizing the exact implications of our findings is crucial for inexperienced readers, and we will do so. In our camera ready, we will state the following points regarding the purpose of our evaluation:

1. TS2 is intended to provide a meta-assessment of metric quality that isolates structural correctness in relative comparison of related images
2. TS2's evaluation is orthogonal to human aesthetic preferences, and that a comprehensive meta-evaluation should also take those preferences into account
3. That our findings, while important, surprising, and soundly demonstrated, do not tell the full story of T2I metric quality; some metric strengths such as explainability and contextualization cannot be captured numerically

We appreciate you bringing this point to our attention and look forward to using it to strengthen our camera-ready.

Thank you for your detailed review! We appreciate your praise of our work’s rigorous evaluation, valuable dataset contribution, sound methodology and analysis, and clear writing. We are overjoyed to hear that you really like [our paper] and hope to see it accepted. These are all very heartening! We hope you will find our responses to your questions satisfactory.

Weaknesses

• Omitted metrics

ImageReward did compare itself against CLIPScore and BLIPScore in its introductory paper, so it is reasonable to consider evaluating it in our study. We decided not to and to instead focus on the correlation metric vs. VQA metric angle, and because it is simultaneously attempting to measure the orthogonal characteristics of image aesthetic quality and image-prompt faithfulness as mediated and combined by human preference annotators. You raise a good point that it still can be evaluated using TS2, and that the findings from this experiment may have interesting implications for the trade-off between capturing aesthetic preferences and prioritizing structural correctness. We will discuss this possibility (although we do not have results for it) in the future work section, and investigate including ImageReward in our final leaderboard that will be linked in the camera ready.

VQAScore is very interesting, and we did see the paper late into the writing process, and before its source code was fully ready for our use. While we are unable to evaluate it in the paper, we do look forward to adding it to our leaderboard. Though we do not have results for it, we will include a reference to it in the camera ready.

• Explanation of Ad-hoc evaluations

Yes, we should clarify, the reason we define “ad-hoc” evaluation benchmarks as test sets that were released alongside proposed methods, and call them out in the discussion of related work, is because of the potential concern that these benchmarks—showing the superiority of the metrics they are introduced alongside—may contain some bias (probably unintentional) in favor of the proposed metric they are motivation, and that these benchmarks are not the primary contributions of their introductory works, and thus have less documented design considerations and production methods. By centering the design considerations of our evaluation in this paper, and not introducing a new metric, we are able to focus on the rigor of our evaluation, identify weak points in prior evaluations, and as neutral arbiters demonstrate our surprising finding.

Questions

• Regarding size of TS2

We do believe the size of the dataset is sufficient to defend our findings. In particular, the high total number of comparisons that TS2 enables is its primary strength. By organizing the images along semantic error graphs containing multiple walks, the images are able to be reused much more efficiently than in other datasets, to analyze how well models can order them along different sets of accumulated semantic errors. For example, in figure one the image “1-2.jpg” is simultaneously used to check if a metric attends over a boy missing from the picture, as well as if fruit is missing, by comparing it to both “2-0.jpg” and “2-1.jpg”. This effect, applied over all the SEGs, greatly amplifies the utility of each image.

• Notions of human performance

Though our dataset was produced through human annotation with a high annotator agreement, it is important to note that the human annotators were aware of the implicit ranking task, while the metrics under test are not. We do not think this is a significant weakness of the work, as human performance on the inherently synthetic task of image quality scoring is not as important as performance on ranking along objective errors.

Thus, if human performance were judged on the task of simple Likert scoring of image-prompt accuracy without instructions, humans may not significantly outperform the metrics. However, if the human annotators were instructed to count the number of errors, we suspect they would perform quite well, even without the other images for comparison over which the ranking task is performed.

Thank you for this stimulating idea, we look forward to including this discussion in our camera ready.

Thank you for clarifying. These are great points:

1. TS2 + and orthogonal quality-only eval might actually tease out the degree to which human annotators attend over each consideration by directly comparing their preference correlations to each metric under different annotation schemes. A really cool idea might be to treat those two considerations as principal components a metric could be interpolated between, or enabling a search for a jointly optimized single "best metric." We will definitely investigate adding ImageReward to the final leaderboard given this.
2. Yes, this is another stimulating direction for future work. Using this human baseline also has implications for direct "VLM-as-a-judge" metrics that ask them to provide Likert scores, as only "superhuman Likert-assigning" VLMs would be sufficient to beat humble CLIPScore.
3. Agreed, we will make sure to clarify that QG/A metrics have this interpretability advantage, and in particular that this consideration, alongside TS2 evaluation, helps users choose a metric based on scenario and needs. Eg., in an interactive app (relatively low analysis throughput rate) cost is less of a consideration, and a human would benefit from interpretable analysis. Whereas for an online reward/feedback model or a supervisory post-filter for image generation, cost is an important consideration and interpretability isn't. Grounding metric selection in all of these considerations is best; TS2's contribution is in capturing one important consideration well.

Thanks for the further stimulating discussion!

Thank you for your detailed review! We appreciate your recognition of our important contribution which you find interesting, that enables detailed insights and has a thoughtful discussion. We hope you will find our response to your questions about the counting system satisfactory.

Weaknesses

• Ambiguity introduced in the error counting approach

For count errors, we considered “a X” to remain correct if there is more than one “X”---a picture of multiple flamingoes does contain “a flamingo.” However, when specific numbers were provided in the prompt, like “one flamingo,” containing more than one counts as an error.

You are right to point out that there is not necessarily a single objective answer for how to handle multiple attributes that could be incorrect about an object simultaneously, such as your example where “a boy in a green shirt” has a grey hoodie. In this case, under our annotation scheme, we still counted it as one error—”no green shirt,” but the case could be made for it to be two.

While it is important to clearly document these annotation nuances for reproducibility, these issues are inconsequential to our results, because our meta-metrics only evaluate rankings along descending walks in the error graph. An image of the boy in a grey shirt and the image of a boy in a grey hoodie are not child or parent nodes of each other, instead being related only to images that may also have no boy at all. So regardless of whether a grey shirt or grey hoodie count as the same number of errors, a grey shirt with no fruit has more errors than a green shirt with no fruit, and has fewer errors than no boy and no fruit at all. The relative difference of errors between nodes that aren’t connected on the error graph do not actually matter, as our metrics are only assessed over walks.

We will ensure this deeper documentation of the error counting process, as well as this explanation for the role the error counts play in evaluation (relative to directed connected nodes, absolute values unimportant) are both provided in the camera ready.

• What are our recommendations to future system builders

We have a few thoughts on this front. We think the right way to use TS2 in metric evaluation is to treat it as an orthogonal evaluation axis to human preference. While human preference correlations are great for capturing total output image quality, metrics that are well-correlated with TS2 will be ideal for measuring structural correctness. Authors might want to consider releasing dual metric evaluations, one which performs well on TS2, and another that performs well on image quality and aesthetic human preference correlations, to give more fine-grained feedback to models.

The pareto optimality evaluation plays a particularly important role when introducing metrics that may be employed for automated feedback, such as in a reinforcement learning setup or as a post-filter to select the best candidate image from a set of outputs.

As for metrics that may perform better on our evaluation, we think authors should consider better ways to gain faithful outputs from VQA modules, perhaps using techniques such as chain of thought or self-consistency prompting.

Thank you for the stimulating question, we will include this discussion in our camera ready.

• Difficult to read italics in results table

Thank you for pointing this out. Rather than italicizing we will underline the runner-up scores.

Thank you for your thoughtful review. We are heartened by your recognition of our rigorous methodology and well-designed experiments approaching the important task of T2I metric assessment. We will briefly address your weaknesses and questions:

Weaknesses

• Scope of image-generating models

Indeed there are many other generative image systems out there beyond the set we used to produce SEG-populating images. While it would be interesting to include images from OpenMUSE or aMUSEd in this approach, we believe showing a metric’s poor performance to rank on any set of T2I models is sufficient to demonstrate a need for improvement. Though we do not use a comprehensive set of T2I models to produce the test SEGs, we never claim our test is comprehensive, only that it is sufficient in showing the surprising lack of differentiation between tested methods.

However, in the future images from additional SOTA models should be added to TS2 as advanced metrics are introduced. Our findings stand well with the current images but in future work we will definitely add more.

• Bias in the rank-order metrics

As we note in sec 6.1, Spearman’s rho (rank order corr) is indeed biased in favor of discrete scoring methods over continuous methods such as CLIPScore, because of the way they handle ties. Luckily, the continuous methods are the weak baseline that the QG/A and Caption-based methods (which are discrete) are intended to beat. The ranking score does indeed penalize these metrics---yet they still compete with or outperform the rewarded metrics. This makes the finding even more striking; despite being at a disadvantage, CLIPScore and ALIGNScore still win, to our surprise.

In the future, as we maintain the TS2 leaderboard and other metric papers use it, it will become important to add metrics that do not penalize continuous metrics, as perhaps more advanced ones will be introduced. For the results in this paper though, the bias actually strengthens the conclusion.

We will clarify this point in the camera ready.

Questions

• Pairwise vs multi-image ranking

You ask about multi-image ranking instead of pairwise ranking. To clarify, we do not do two image pairwise ranking, all of our evaluation metrics are multi-image-based. Indeed, this is our work’s primary advantage over all prior T2I metric evaluations.

Our metrics are either full-walk scoring ($rank_m$) or node-pair scores ($sep_m$ and $delta_m$). For the full-walk scores, ranking over each descending sequence of nodes (each of which contains multiple images)

We only refer to pairwise comparisons in Table 1 as a way to distinguish our benchmark against prior work. Our metrics are exclusively multi-image (and our per-equiv pref score in Table 1 represents that on average, each node contains 3.4 images of equivalent correctness)

Future updates to TS2

We are excited to incorporate these points into the camera ready, and your suggestions for additional ranking metrics and more images are great ideas for future contributions as we expand TS2 as a living resource. However, we believe the resource as it currently stands represents a significant and timely contribution that advances our understanding of current text-to-image faithfulness metrics.