GRADE: Quantifying Sample Diversity in Text-to-Image Models

We sincerely thank you for recognizing our paper as well-written, well-motivated, and well-organized. We are pleased that our clear definition of the diversity metric and the systematic approach in Section 3 resonated with you, as well as our analysis linking low diversity to underspecified training captions.

We thank you for your helpful comments. Please see our response below.

W1: “Since GPT-4o is used to generate prompts, attributes, and attribute values …”

We agree that because this version of GRADE relies on GPT-4o, it is not straightforward to compare the exact results of this study to other models. However, this is true to any method that uses GPT-4o, and is not directly related to GRADE. It is seamless to change the underlying model to an opensource version that does not change at all.

We opted for GPT-4o as it was the most reliable and fastest model we could use to analyse images in scale (we have analysed over 720K images).

The nature of our paper does not call for future replicability of exact scores. In the future, researchers who use GRADE to report diversity scores for their own models, may opt to use an open LLM and VQA for better replicability.

W2: “The definition of diversity in this work is focused on specific attribute variations …”

Thank you for pointing this out, we agree that our definition is not all-encompassing and discuss this in the manuscript under the Ethics Statement (lines 540-548).

It is noteworthy that GRADE could be used in addition to other metrics that capture other forms of diversity.

W3: “There's no limitation discussion for example how GRADE might handle ..”

Our framework targets only concepts with visually verifiable properties, as we define attributes in Section 3.1.

We do not find that asking questions that can be visually verified about a concept is inherently limited, but the models that are used to test for this concept (like the VQA), could be. We believe that this will change as LLMs and VQA models improve, and remind that GRADE can be modified to work with new underlying models seamlessly. We have clarified this in a dedicated Limitations section.

Thank you very much for the fantastic suggestions! We agree that compositional diversity and relationships between multiple concepts in the same image are great avenues for research. Indeed, we suggest exactly that in the conclusion (lines 533-537). This is a future work direction which we plan to pursue, but believe it is out of scope for this one.

Q1: “For the concept-specific attributes …”

GRADE proposes the attributes automatically, so there are many other attributes than just shape/color/texture. We discuss this in section 3.2, in lines 216-226.

Regarding consistency across concepts -- each set of attributes is tailored to the concept, so they are unique.

If you're interested, you can view the attributes of each concept in the supplementary material we submitted.

Regarding LLMs and biases: We agree with this point, and have added it to a Limitations section. We note that all methods incorporate some form of bias, and while we agree it is crucial to acknowledge their existence, no evaluation method is free of bias.

Thank you for your attention and dedication in evaluating our manuscript. We hope you have had the chance to consider our response from November 16. In it, we clarify that GRADE is limited by its underlying models, and that reproducing results with GRADE is not a hurdle. These clarifications can also be viewed in our revised paper, in a Limitations section we have added. Furthermore, we have clarified that GRADE measures concept-specific attributes, and not all forms of diversity, as originally stated in our Ethics Statement in the paper, as well as addressed other concerns you have stated.

Please let us know if there are any additional points or questions you would like us to clarify. We are more than willing to offer further explanations.

Thank you,

We thank you for agreeing that our work tackles an extremely relevant open research question, and noting GRADE’s reference-free property to be important as well as its focus on measuring diversity of concepts with respect to relevant factors.

Thank you for the many detailed comments and questions. we respond to them in order. due to response character limits, we respond in two messages

W1: “The proposed metric does not seem to be effective at discriminating between models. … “

While we indeed show all models are not diverse, they do differ from each other in diversity. For example, the difference between FLUX-dev and SD-1.4 is very apparent, both in the scores and anecdotally (see Figure 3a and Figure 4). Thank you for proposing a statistical significance test, we added them and hope the results alleviate your concerns (see below, more detail in the uploaded revision):

Statistical significance:

We run a two-tailed permutation test with 100K permutations between every unique pair of models (66 in total). Our null hypothesis is that the diversity scores of two models are the same and the p-value is compared to a significance level of 0.05.

Results:

The vast majority of pairs are statistically significant. The few which are not are anecdotally also of similar quality. For example, all pair combinations of SD-1.1, SD-1.4, and SD-2.1 are not significant, which is not surprising, seeing as these models for the most part share the same underlying models and training data.

W1.1: “There is no rigorous validation of the proposed metric … “ We respond in W1 and W2 to each claim.

In addition, we note that a unique advantage of GRADE is that the diversity score it returns can be traced to specific attributes that lack diversity, which can then be addressed.

W2: “The authors claim in the conclusion …”

You are right, our experiments do not measure human perception or judgement of diversity. What we do show is:

• The questions can be answered by examining images of the concepts
• The attribute values are acceptable variations to each concept
• The VQA is reliable in identifying the attribute values of concepts approximately > 90% of the time

In other words, we show humans agree that GRADE is accurate in implementing our definition.

Following your observation, we amended the text to reflect this change, as well as state in a Limitations section that our definition of diversity does not directly correspond to human perception of diversity.

While we completely agree with the correction, we would like to point out that (to the best of our knowledge) FID, Recall, and even Vendi Score were not shown to clearly capture diversity as perceived by humans either. Human perception of diversity is an open question in other fields, such as psychology, our definition of diversity over attribute values is less open to interpretation.

W3: “The metric relies on how well the VQA model is able to capture nuances …” Thank you for clarifying that our text is not clear in this regard.

In general, we unify attribute values that have the same semantic meaning. The example we provide in lines 224-225 is of “circular” and “round”, but your example is equally valid.

W3.1: “It is unclear what is the interplay between GRADE and other aspects such as alignment between prompt and generated image, and realism …”

Yes, for the vast majority of cases, the VQA would be able to answer the question correctly.

The VQA is prompted to reply with “None of the above” if the attribute value cannot be extracted from the image, as mentioned in Section 3.2 (lines 228-235). This can happen if the concept is not depicted in the image or the attribute value cannot be inferred. We have not found special issues with unrealistic images (results from prompts “an egg in a zero-gravity environment”). We confirm it responds accurately in such cases and does not hallucinate attribute values, as we show in section 4-b and section 4-c, where humans agree with the VQA on such cases too.

W4: “It is not clear how the approach validity as evaluated …”

Our study does include people, and GRADE did consider ethnicity, skin tone, and gender properties (which can be viewed in the supplementary material).

Example:

people: we have a "person" concept with a question about gender: "Is the person male or female?"

While GRADE did not ask about ethnicity directly, it did ask about skin tone: For our concept "princess", we have the question "What is the skin color of the princess?".

W5: The manuscript lacks clarity in many parts. Please refer to the "Questions" section of this review for more details.

We respond to the questions in a separate comment.

W6: "... missing references ..." Thank you for these references! We have added all references pertaining to diversity metrics.

We note that while these citations improve the related work section, the takeaway of our paper is not changed.

Thank you for your attention and dedication in evaluating our manuscript. We hope you have had the chance to consider our response from November 16. In it, we detail a statistical significance test that shows GRADE captures the difference between models, clarify our statement that GRADE reflects human judgement of diversity, and the motivation to measure diversity independent of the training data distribution, as well as address the rest of the concerns and questions. We have also provided a revised manuscript with highlights that mark the changes.

Please let us know if there are any additional points or questions you would like us to clarify. We are more than willing to offer further explanations.

Thank you,

Thank you for your detailed review. We are delighted that you agree that overcoming the reference-based nature of traditional metrics is important, and that our work offers deeper insights into text-to-image model behavior.

W1: “Generally, there is a common perception that ‘cookies’ are round, …”

Thank you for bringing this up!

We find that for the most part, it is personal preference: Some people would like to get a diversity of results, and some are happy with a common default. Some would like it for specific properties (eg that generic cookies are round by default) but not for others (eg that generic nurses are female by default). Similarly, users in a creative setting may want to use the generation process as a brainstorming tool, and have it generate non-typical variations of concepts.

Regardless of personal preference regarding this model behavior, it is useful to be able to quantify it, which is the focus of our work. An appealing property of our method is that it is also verbose and transparent about the kinds of diversity (or lack of) it found, so users can decide on a case-by-case basis.

W2: “The method uses an LLM to generate …”

By default, GRADE suggests all properties and values automatically but it is straightforward to intervene in the category and values generation step and use other ones.

Because of this modularity, and because the underlying models can be changed with ease, we do not see it as a significant problem.

Thanks to your note, we will clarify in a Limitations section that the diversity score ultimately hinges on the concepts, and data generated by the LLM. However, we remind that we manually validated every set of categories and values that were used in the study, so this limitation is not manifested in our results.

Q1: “The distinction between common and uncommon prompts …”

Other tokens in the prompt affect the depiction of the concept we measure. For example, the cookie mentioned in the prompt "a cookie on a table during Christmas festivities" is more likely to take the shape of a tree, or a gingerbread man, because “Christmas” is in the prompt, but it does not represent the typical cookie that one would imagine otherwise.

Common prompts include scenarios which may affect the variation of the concept, as it places the concept in a typical environment. The uncommon prompts purposefully place the concept in uncommon settings to mitigate such common associations

Following your question, we revised section 3.2 paragraph (a) to include the example above.

Q2: How do the authors recognize and address hallucinations when using LLMs and VQA?

We measure the reliability of the LLM and VQA models we use (GPT-4o for both) in section 4 and find they are very reliable. However, we acknowledge that they are limited and as any generative model, hallucinate.

We updated the manuscript to clarify that GRADE is limited by the performance of its underlying models, such as the LLM and VQA, as stated in our response to W2.

Thank you for your attention and dedication in evaluating our manuscript. We hope you have had the chance to consider our response from November 16. In it, we clarify the motivation to measure diversity independent of the training data distribution, and that the underlying models, like the LLM and VQA limit the performance of GRADE. We also address your questions, such as clarifying the need to distinguish between common and uncommon prompts. We have already updated a revision too, where you can view our changes–highlighted in yellow.

Please let us know if there are any additional points or questions you would like us to clarify. We are more than willing to offer further explanations.

Thank you,

Thank you again for providing us the opportunity to clarify and improve our paper.

Regarding W2, my question was whether the choice of which attributes to measure depends on the LLM. ...

This is exactly one of the strengths of GRADE. As we discuss in lines 47-53, the set of relevant attributes is concept-dependent, and requires world knowledge. We use LLMs to automatically identify that set.

While LLMs might not be exhaustive in identifying all relevant aspects, we expect all strong LLMs to be able to identify a good set of semantically-adequate attributes.

someone might want to measure the diversity of a cookie's ingredients.

We do not claim to be exhaustive. If someone wants to specifically measure the diversity of a particular attribute they can inject it to the pipeline and run it as-is, however, it is optional.

For the revised Fig 1, which attributes should be measured? Why are they considered absolute attributes?

The attributes for “cup of soda” by GPT-4o are shown below. The figure reflects the relevance of these questions nicely, as the images show bias toward specific answers, shown in italic font below.

1. Is there a lid on the cup? No
2. Does the cup have a straw in it? No strong bias either way
3. If visible, what flavor is the soda (based on its color)? Cola
4. Is the cup translucent or opaque? Translucent
5. Can you see ice in the soda? Yes

Can you clarify what you mean by absolute attributes?

Assuming you believe that in the paper we consider the set of attributes for each concept to be encompassing to the diversity of the entire concept, then this is not the case. Please refer to us to the lines that led you to believe that, so we can revise it.

We consider each distribution on its own. I.e., we do not imply the results reflect diversity of unseen attributes. Thanks to your review, our revision states so explicitly in the Limitations section (lines 517-518).

Table 4 shows the average diversity score of a model over all distributions of all concepts, as to show the average diversity score over the attributes we did measure, but we do not generalize our findings of a concept’s diversity beyond the attributes we measured.

If human intervention is required each time, doesn't this diminish the novelty of the proposed automated metric?

You're correct, but human intervention is not required at all, as we state in the abstract (line 16), in Section 4 (Lines 264-265), and in the Conclusion (line 526).

In point 1, ... Is that correct?

No. The normalized entropy is an absolute value within the range [0,1] that indicates how diverse a concept is over a particular attribute (see Section 3.1 for the formulation).

There is no need to compare it to other models at all to interpret the scores, as a score is strictly bounded, with 1 indicating uniform behavior and 0 indicating consistent selection (see lines 195-196).

If so, is it fair to compare the diversity of a prompt that measures "shape" with a prompt that does not?

No, that would not be fair, but we do not do that. If that was understood from our paper, please direct us to the lines so we can revise it.

Repeating for clarity: We never compare models on the basis of a concept alone, but on a combination of concept and attribute (which must be the same).

Ambiguity also arises in distinguishing between common and uncommon prompts. Why must their numbers always be equal?

We kept them equal for our study, to not have an overwhelming representation of a prompt that is inherently biased, like the “Christmas” example, which results in an untypical number of tree cookie shapes, compared to the typical cookie.

Who defines what is "common" or "uncommon"?

We determined commonness by checking the frequency of co-occurrences of content words in captions in LAION-5B (Lines 267-287). We found that the average co-occurrence frequency for a common prompt is 30,655 while for an uncommon prompt just 956.

... It seems that all common and uncommon situations should have an equivalent impact on the object being measured for diversity to be fairly evaluated.

We agree! The choice of any prompt directly affects the diversity score, this is why in addition to common prompts (which may be biased), we have uncommon prompts, which are less associated with the concept (this is stated in lines 211-223). I.e., uncommon prompts prevent manipulation of diversity.

This balance is needed if we want to measure the diversity of the typical cookie. If we are to measure the attribute diversity of a concept in a specific setting (e.g., Christmas), we would not need uncommon prompts (which GRADE can accommodate). But as stated in the abstract and introduction, we focus on underspecified concepts (e.g., cookie and not cookies specific to Christmas).

Thank you again for responding with thoughtful follow up questions, we appreciate it. Please let us know if you still have concerns!