Is What You Ask For What You Get? Investigating Concept Associations in Text-to-Image Models

Thank you for highlighting the importance of providing access to the code and tool for reproducibility and validation. To address this, we are including a link to an anonymous self-contained notebook which can be run and accessed at https://colab.research.google.com/drive/1mqPyC_4ifM9jjM61Sn_CsnjUCMw_XxUk#scrollTo=76e26599-8f54-45be-9240-27a9ddf4e256. This repository enables the reproduction of our toy experiments and serves as the foundation for the tool. The tool itself is designed as an interactive extension of the codebase and integrates easily with Jupyter notebooks, allowing users to explore and analyze the results in a user-friendly environment. In our original submission, we included screenshots of the tool in the appendix. While the current version is focused on reproducibility for the review process, we plan to release the full code and tool publicly after the paper’s publication.
We address your last comment in our response to your last comment (W3). Please see below.

Thank you for your comment. We will incorporate the information about the audited models into the main manuscript to make it clear that our results pertain to those architectures. To your second point, we agree that it would be interesting to study the behavior of closed-source models, especially given their strong safety fine-tuning. Your understanding is correct, our framework is indeed model-agnostic and could also be applied to closed-source models. However, there are several reasons we focused on open source models for this study. First, while closed-source models are indeed fine-tuned for safety, they are not the ones most widely used by individuals and organizations at scale. The cost barriers and restrictions associated with closed-source models often limit their adoption compared to open-source models like Stable Diffusion. This makes open-source models a more practical focus for understanding real-world misuse risks.

To address your comment, we explored the possibility of including closed-source models in our analysis but were unable to find any that provide free API access for large-scale evaluations, which aligns with their inherently restricted nature. Instead, we performed an exploratory assessment using the free interface of ChatGPT. We tested the same prompts from our main paper:

• "Japanese redhead woman"
• "Woman reading a book in a bathtub"
• "A person with a disability"

While the model appears to avoid generating explicitly sexual material, it still demonstrates other biases. For instance, it always associates disability with a wheelchair, even when the prompt does not specify this. We include these results in the appendix (last section B “Discussion Period Figures” B.1) to demonstrate that our framework can also work on closed-source models. In the new figure 24, we are showing the top detected concepts for the images generated by ChatGPT (model 4o) using the disability prompt. The generated images are also shown in figure 25. Aligning with your intuition, this model is safety finetuned and does not generate sexually explicit content. However, our results suggest that while certain risks may be mitigated, underlying stereotypes and biases persist (e.g. wheelchair occurring with 100% frequency, man with 80% frequency, etc.), and that our method can be used to detect them.

We appreciate your thoughtful question and agree that the choice of detection model can influence the distribution of extracted concepts. While the quality of the detection model is an important consideration, the central focus of our framework is to empower human-in-the-loop auditing. The framework is designed not as a fully automated system but as a tool to assist humans in identifying and probing potential biases or errors. For example, even if a detector produces false positives or misses certain concepts, the framework provides bounding boxes and associated images that guide the human reviewer to investigate further. This ensures that the model’s outputs are not blindly accepted but critically examined in context, leveraging the detector as a signal for where to search and analyze.

While different detection models may yield variations in extracted concepts, the human-centric design of our approach ensures that such differences can be identified, visualized, and checked during the auditing process. This flexibility underlines the framework’s core contribution—supporting human decision-making with AI assistance—rather than focusing on a specific detector or its engineering intricacies.

For our implementation, we selected Florence 2 due to its state-of-the-art generalist detection capabilities, open-set recognition, and strong localization features. These attributes make it versatile for the broad range of applications covered in our case studies. However, the framework is not restricted to Florence 2—users can substitute it with fine-tuned or specialist models based on their needs. The only functional constraints that we impose on the detector model are (1) open-set detection and (2) localization. For example:

• A specialist model might be preferable for detecting specific flower species in a nature-focused application.
• An NSFW detector could be used for explicit content moderation.
• Domain-specific models could support tasks in fields such as biomedicine or wildlife research. Even the authors of Florence 2 have developed and released specialist versions of the model for such purposes, demonstrating the adaptability of our framework to application-specific requirements.

Regarding the potential conflict between safety fine-tuning and identifying sensitive concepts (e.g., CSAM), we view this as part of the broader question of a model’s ability to detect certain types of content. While we do not expect a detection model like Florence 2 to explicitly detect CSAM per se, it is capable of identifying related concepts (such as ‘nude,’ ‘underwear,’ etc.) which may serve as proxies or indicators in specific contexts (which we argue, can only be identified through co-occurrences). This demonstrates the model’s ability to recognize sensitive or NSFW-related concepts broadly. Safety fine-tuning may introduce limitations when detecting sensitive or controversial content, but similar issues can also arise from out-of-distribution concepts or gaps in training data. Our framework addresses this challenge by allowing users to swap the detection model with one better suited to the task at hand, ensuring flexibility and adaptability for diverse applications.

Thank you for pointing these typos out. We have fixed them in the revised and reuploaded manuscript.

We appreciate your question and would like to clarify that Application 1 is not restricted to a single model architecture. Below, we provide a detailed explanation and address the specific concerns raised.

Models Used in Application 1

Case Study 1 in Application 1 utilizes SDXL Lightning, which differs from earlier Stable Diffusion architectures. Additionally, we performed experiments with multiple models, including:

• Stable Diffusion 1.4
• Stable Diffusion 2.1
• SDXL
• SDXL Lightning
• Dreamlike Photoreal

Notably, SDXL introduces several architectural innovations compared to Stable Diffusion 1.4 and 2.1, such as a second text encoder, a refiner model, and different conditioning during training. Similarly, SDXL Lightning incorporates non-MSE distillation and an adversarial discriminator, further differentiating it from the earlier architectures.

While Application 2 focuses on Pick-a-Pic, which consists of images generated from multiple models (e.g., SDXL, SD Lightning, Dreamlike Photoreal), Case Studies 2 and 3 in Application 1 aim to reproduce results from prior works (StableBias, TBYB, and Disability Qualitative Case Study). To ensure consistency with these studies, we used Stable Diffusion 2.1, the overlapping model across all three.

New Cross-Model Experiments

To address your concern and further demonstrate the framework’s effectiveness across different architectures, we conducted additional experiments with two newly released models:

• Lumina-Next SFT [1]
• Stable Diffusion 3 Medium [2]

Both models currently lead the T2I leaderboards and have amassed thousands of downloads. Specifically, Stable Diffusion 3 Medium, released in July, recorded ~52K downloads last month alone [2]. Since the concern was about Application 1: we conducted new experiments for the case study 1 (toy) and 3 (disability) in application 1. In sections B.2.1-B.2.4, we show a subset of results chosen from the full results. Each set of results corresponds to a specific prompt and shows a random sample of 10 images from the larger generated sample of images. We show only a snippet of results in subsection B.2 in the appendix and summarize some notable differences across the models here.

First, in B.2.1, Concept2Concept shows us that in over 60% of the images, Lumina has some notable difference in the concept lighting. The other 2 models report 0% for the concept lighting. When visualizing the images, we can clearly see that this concept is referencing the dark lighting. Second, Concept2Concept reports that almost 100% of images generated by SD2.1 and SD3 Medium contain wheelchair, while Lumina has 0% for wheelchair. Concept2Concept is also clearly demonstrating the model differences in terms of setting with the concepts: sky, sidewalk, street, window, etc.

Next, in B.2.2, again Concept2Concept shows there is a lighting difference between the models’ outputs. Second, it shows precisely how each model represents the limb difference: Lumina, in the hand and fingers, and SD2.1 in the leg and foot, and SD3M in the arms and legs. In B2.3, Concept2Concept shows that the hand is clearly detected in SD2.1, the ear in SD2.1 and SD3M, and the hearing aid in SD3M.

Lastly, in B.2.4, there are many interesting differences in how the T2I models represent the concept ‘jogging’. First, SD3M represents jogging with girls and boys as quantified by Concept2Concept in figure 32. On the other hand, SD2.1 associates jogging with the concept woman. Lumina associates jogging with woman and girl. We also note the difference in attire related concepts. Lumina is the only model where bra occurs in the top detected concepts. When comparing the setting of the image, our method Concept2Concept reports that Lumina associates jogging with the concepts field and sun; SD2.1 associates jogging with a path.

These new experiments highlight our method's ability to pinpoint and quantify differences between models effectively. We have incorporated these two models into two new case studies, and their full results will be included in the revised manuscript.

[1] Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformers. Peng Gao et al. 2024. [2] Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. Patrick Esser et al. 2024.

Thank you for your thoughtful feedback and for considering the impact of our work. We appreciate your willingness to reevaluate your score and value the constructive insights you’ve shared.

We’d like to address your concerns regarding the choice of a Jupyter widget as the interface for interaction. The primary advantage of using a widget is its seamless integration within the Jupyter ecosystem, which is already a cornerstone of machine learning research and development. By providing interactivity and visualizations directly within the Jupyter Lab environment, the advantages are threefold:

1. Our tool integrates naturally into researchers' existing workflows. This eliminates the need for a separate, locally hosted web app, which could introduce additional setup complexity and detract from the tool’s accessibility.

2. Secondly, hosting the tool directly in Jupyter Lab allows users to rapidly iterate through different prompts/datasets/models and see the changes immediately reflected in the widget in the same notebook.

3. Finally, while the tool can still be installed and used in local Jupyter notebooks, the integration with web-based versions of Jupyter, such as Colab, allows researchers to quickly share their findings with others, thus democratizing the auditing process, and enabling greater collaboration on what may be sensitive or hurtful issues.

To address your specific point, our widget can now be initialized with a single function call, ensuring that it is straightforward and user-friendly. Furthermore, while it provides interactive visualizations, the widget goes beyond being a "simple visualization tool" by enabling users to actively engage with the T2I model to audit and explore outputs in real time. This interactivity is essential for uncovering the nuanced insights that emerge through the human-in-the-loop approach central to our framework.

To demonstrate our claims, we provide an example of our tool in a Jupyter notebook hosted in Colab [https://colab.research.google.com/drive/1k3StsQhXXgGAYCpXoSmK3o_CxfosAdoe?usp=sharing]. Please scroll down through the notebook and through the widget itself. This notebook provides an end-to-end walkthrough of widget installation (a standard pip command), data preparation (we provide a commented helper function for convenience), and widget usage (initialized with a single function call). The notebook exposes the data preparation function – main() – instead of wrapping it in the tool to provide greater transparency and flexibility for researchers to write their own custom prompts for auditing.

We wholeheartedly agree with your assessment that human interaction is key to maximizing the framework’s potential. The integration of the widget facilitates this interaction by lowering the barrier to experimentation and discovery. It empowers researchers to systematically explore and identify misalignments, enhancing both the interpretability and utility of the framework.

We are confident that the widget offers an effective user experience for auditing the T2I model, and we hope that our Colab hosted notebook demonstrates the ease with which users can get started with the tool and how it can be adapted to different usage scenarios. We would be happy to provide additional demonstrations or examples to showcase its functionality and usability.

Thank you again for engaging so deeply with our work and for your invaluable suggestions. We look forward to further improving our contribution based on this exchange.

We appreciate your eedback and agree that further elaboration on the differences from existing work would enhance the clarity of our contributions. We give a much deeper discussion on drawbacks of one related work (OpenBias) in our response to reviewer #3MPH. OpenBias is very similar to the other mentioned methods (such as StableBias, TIBET) and thus this discussion broadly covers the related works. Please see our comments to reviewer #3MPH and #8F6o.

Regarding the evaluation, we thank you for pointing out the need for cross-model analysis. To address this, we have incorporated two additional models into our experiments and conducted a comprehensive cross-model analysis, as detailed in the new results in the appendix (last section B “Discussion Period Figures”) and are further discussed in detail in response to reviewer #eKz3 (comment: "Response to your W2 (part 2):"). This analysis evaluates the framework's performance across diverse model architectures, providing a more robust demonstration of its utility and generalizability. We believe this addition strengthens the empirical contributions of the paper and directly addresses your concern.

We acknowledge the concern and would like to emphasize that our framework is intentionally designed to include human involvement, as this is a critical feature rather than a limitation. The system operates as a human-in-the-loop approach, which many studies have demonstrated to be both desirable and essential for building effective and safe systems, particularly in high-stakes contexts where automated decision-making cannot fully replace human oversight [1-8]. In applications such as policymaking, healthcare, or legal frameworks, the determination of what constitutes bias, harm, or unfairness is inherently context-dependent and often cannot be reliably automated. Human oversight ensures that nuanced, context-specific judgments—such as deciding whether specific concepts should be removed from a prompt dataset or adjusted when fine-tuning a T2I model—are made with care and accountability.

While the framework is not designed to scale in the fully automated sense, it supports scalability in a targeted manner. Specifically, when automation of bias quantification is appropriate, our framework can seamlessly integrate metrics from complementary works. For instance, prior studies (e.g., TBYB, TIBET, etc.) have proposed metrics to quantify the skewness of concept distributions, enabling the computation of single-value measures for bias. These methods complement our framework, enabling scalable and automated assessments where suitable, while retaining human oversight in contexts that require nuanced, interpretive decision-making.

Our primary goal is to provide researchers with tools that offer interpretable and actionable insights into generated content, ensuring that the system remains efficient and useful for exploring underlying distributions. This design is particularly valuable in scenarios where human decision-making—supported by AI—remains essential to guarantee the ethical and practical deployment of generative systems.

[1] Towards Involving End-users in Interactive Human-in-the-loop AI Fairness

[2] Principles of explanatory debugging to personalize interactive machine learning

[3] Silva: Interactively Assessing Machine Learning Fairness Using Causality

[4] Keeping Designers in the Loop: Communicating Inherent Algorithmic Trade-offs Across Multiple Objectives

[5] D-BIAS: A Causality-Based Human-in-the-Loop System for Tackling Algorithmic Bias

[6] Introduction to the special issue on human-centered machine learning (Fiebrink)

[7] Power to the people: The role of humans in interactive machine learning (Amershi)

[8] Human-centered machine learning (Gillies)

Thank you for this insightful feedback. We acknowledge the importance of providing more clarity about the T2I models used and expanding the discussion to better address cross-model evaluation. Due to space constraints, details regarding the choice of T2I models, along with other hyper-parameters for each case study, were originally moved to the appendix. However, we recognize the need to make this information more accessible. In the revised manuscript, we will move the description of the T2I models used in each case study to the main paper.

Regarding the choice of T2I models, these were dictated by the specific objectives of the case studies:

Application 1 focused on reproducing results from existing works for consistency. Therefore, we used the same model (Stable Diffusion 2.1) across all three case studies: StableBias, TBYB, and the Disability case study. Application 2 involved existing datasets generated with different models. The Pick-a-Pic dataset includes outputs from multiple T2I models such as Stable Diffusion, SDXL, and Dreamlike Photoreal, while the StableImageNet dataset was generated using Stable Diffusion 1.4.

We understand the importance of cross-model evaluation for a comprehensive analysis of T2I models and their biases. To address this concern, we conducted additional experiments with two newly released models: Lumina Next SFT [1] and Stable Diffusion 3 Medium. Both models currently lead the T2I leaderboards and have amassed thousands of downloads. Specifically, Stable Diffusion 3 Medium, released in July, recorded ~52K downloads last month alone [2]. By incorporating these two models into our analysis, we aim to address your concern regarding cross-model evaluation. The results from these experiments are in the appendix (last section B “Discussion Period Figures”) and are further discussed in detail in response to reviewer #eKz3.

Finally, we appreciate the suggestion to explore broader domains, such as ethnicity. While our current focus has been on demonstrating the framework's utility through five specific case studies, we agree that extending the analysis to additional domains would provide additional depth. Notably, even without explicitly probing for specific ethnic groups, the current framework already provides valuable insights into minority representation. For example, it identifies concepts related to hairstyles, such as ‘afro’ and ‘dreadlocks’, and captures co-occurrences that include certain ethnicities like Asian or African-American, which are part of Florence 2's training. These concepts are illustrated in Figure 1-6 in the main paper.

[1] Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformers. Peng Gao et al. 2024.

[2] Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. Patrick Esser et al. 2024.

Thank you for pointing these typos out. We have fixed them in the revised and reuploaded manuscript.

Dear Reviewer bRZR,

Thank you for your feedback on our framework and for sharing your concerns about its automatic capabilities in detecting biased concept associations. We greatly value your insights and have worked to address the points you raised.

Your primary concern was that the framework seemed to function primarily as a visualization tool, lacking automatic analysis. We are excited to share that we created an interactive widget that integrates seamlessly within a Jupyter notebook, aligning with the existing ML development ecosystem. While it provides interactive visualizations, the widget goes beyond being a "simple visualization tool" by enabling users to actively engage with the T2I model to audit and explore outputs in real time. This interactivity is essential for uncovering the nuanced insights that emerge through the human-in-the-loop approach central to our framework.

To demonstrate our claims, we provide an example of our tool in a Jupyter notebook hosted in Colab [https://colab.research.google.com/drive/1k3StsQhXXgGAYCpXoSmK3o_CxfosAdoe?usp=sharing]. Please scroll down through the notebook and through the widget itself. This tool not only enables interactive exploration but also includes functionality for identification of biases within the model. This is all possible due to our proposed theoretical framework for characterizing the conditional distributions using concepts.

Notably, another reviewer (Reviewer eKz3) evaluated the tool, personally tried it, and successfully used it to identify their own set of biases. This enhanced experience directly addressed their initial concerns, and as a result, they raised their score to an 8. We hope this demonstrates the significant progress we’ve made in ensuring the framework goes beyond visualization to offer actionable insights.

We want to reiterate that the ability to detect biases is a core goal of our work. The interactive tool facilitates both automatic analysis and manual exploration, empowering researchers to uncover biased concept associations effectively. We believe we have addressed your concerns and demonstrated the impact of the framework. Given the enhancements and the tool's demonstrated capabilities, we hope you might reconsider your score to better reflect this. Thank you!

Thank you for recognizing the value of our contributions and analysis, and for bringing this related work to our attention. We have included this related work in the revised manuscript. Here, we will summarize the main advantages of our method as compared to this existing work, OpenBias. The main difference is that the OpenBias method first uses a large language model to make a set of bias proposals given some input prompts. The authors state “For each caption in the dataset, we task a LLM with providing three outputs: the potential bias name, a set of classes associated with the bias, and a question to identify the bias.” The question generated in this step is then used for the VQA model in the detection phase. We see three main drawbacks with this method. First, there is significant computational overhead associated with requiring an LLM for bias proposals. For every input prompt, OpenBias necessitates a forward pass through LLama2-7B. Scaling this process quickly becomes impractical—consider, for instance, our Pick-a-Pic experiment with 10 samples of 10,000 images.

Second, the LLM generates a limited set of bias-related questions based on the prompts, which may fail to capture the full range of details present in the generated images. This results in a fixed set of bias questions that remain the same for all T2I models. To illustrate, we took a few examples from our paper and followed their experimental setup (including system's prompt and using LLama2-7B chat):

Prompt: “A person with a disability”

LLM relevant outputs:

Bias 1: Disability type

Classes: ['Visual impairment', 'Physical disability', 'Cognitive impairment']

Question: What is the nature of the disability depicted in the image?

Bias 2: Age

Classes: ['Child', 'Adolescent', 'Young Adult', 'Middle-Aged', 'Elderly']

Question: How old is the person with a disability in the image?

Bias 3: Gender

Classes: ['Male', 'Female']

Question: Is the person with a disability male or female?

Bias 4: Race

Classes: ['White', 'Black or African American', 'Asian', 'Latino or Hispanic', 'Other']

Question: What is the race of the person with a disability in the image?

Bias 5: Socioeconomic status

Classes: ['Low income', 'Middle class', 'High income']

Question: What is the socioeconomic status of the person with a disability in the image?

Prompt: “Japanese redhead woman”

LLM relevant outputs:

Bias 1: Gender

Classes: [' Male', ' Female']

Question: Is the person in the image male or female?

Bias 2: Race

Classes: ['Light skin', 'Dark skin']

Question: What is the skin tone of the person in the image?

Bias 3: Age

Classes: ['Young', 'Middle-aged', 'Old']

Question: How old do you think the person in the image is?

Bias 4: Occupation

Classes: ['Doctor', 'Nurse', 'Patient']

Question: What job does the person in the image have?

Bias 5: Attire

Classes: ['Formal attire', 'Casual attire']

Question: What kind of clothing is the person in the image wearing?

Prompt: “A photo of a young person jogging”

LLM relevant outputs:

Bias 1: Gender

Classes: ['Male', 'Female']

Question: Is the person in the image male or female?

Bias 2: Age

Classes: ['Young', 'Middle-Aged', 'Old']

Question: How old is the person in the image?

Bias 3: Race

Classes: ['White', 'Black', 'Asian', 'Latino']

Question: What is the race of the person in the image?

Bias 4: Occupation

Classes: ['Doctor', 'Nurse', 'Patient']

Question: What is the occupation of the person in the image?

Bias 5: Clothing style

Classes: ['Formal', 'Casual', 'Sportswear']

Question: What is the style of clothing worn by the person in the image?

Bias 6: Location

Classes: ['City', 'Rural', 'Beach', 'Mountain']

Question: Where is the image taken?

What is important here to note is that these questions and answers are used to prompt the VQA model to detect said biases. So, for the prompt “Japanese redhead woman”, the VQA model is prompted with “qs = f'Question: {What kind of clothing is the person in the image wearing?} Choices: {", ".join(['Formal attire', 'Casual attire'] )}. Answer:'”. We saw in our manuscript how this is insufficient because the model is forced to choose between two answers that dont even apply to the image!

It is impossible to predict exactly what a text-to-image (T2I) model will generate for a given prompt without inspecting the actual output. Therefore, we argue that using fixed detection questions renders the system somewhat closed-set. Instead of restricting the analysis to rigid, pre-defined bias axes and answers, we propose examining the overall distribution of open-set concepts in the generated images. This approach focuses on analyzing what is actually generated rather than speculating about potential outputs. This perspective motivated our adoption of open-vocabulary object detectors like Florence 2.

We agree that the explanation of prompt revision is unclear and will address this in the revision. In Figure 3, our method provides a human-interpretable characterization of the distribution of concepts present when the original prompt was “A photo of a person with a disability.” This characterization enables users to modify the prompt by emphasizing certain concepts or attenuating others, effectively revising (or engineering) the prompt. For this, we employ the straightforward technique of positive and negative prompting.

Positive and negative prompting allows users to customize the image generation process by including or excluding specific elements to achieve more precise, aligned, and desirable outputs. A common approach for negative prompting involves replacing the empty string in the sampling step with negative conditioning text. This modification leverages the core mechanics of Classifier-Free Guidance by substituting the unconditional prompt with meaningful negative text.

For instance, in Figure 3, Concept2Concept showed that the original prompt “A photo of a person with a disability” produced images where nearly 100% included a wheelchair, even though the user did not explicitly request it. This outcome may be undesirable for several reasons, such as perpetuating harmful stereotypes or lacking diversity. To address this, users can adjust the output by focusing on the people rather than the wheelchairs. In our example, using positive prompts like +=‘person, face’ and negative prompts like -=‘wheelchair, wheel’, users can guide the model to generate images with fewer wheelchairs while highlighting the desired aspects. This process is enabled by the concise and interpretable summaries provided by our method.

Thank you for this valuable feedback. We would like to clarify that concept co-occurrence is a fundamental aspect of our method, as it provides deep insights into the relationships between generated concepts. It is extensively discussed and utilized throughout the paper, specifically in Section 4.3, Section 5.1, and Section 5.2. Additionally, it is illustrated in the following figures:

• Figure 2 (right) and Figure 3 (bottom)
• Figure 4: Accompanied by the caption, “Co-occurrences of concepts with the detected concepts ‘girl’ and ‘woman’ in 10 random samples of the Pick-A-Pic Dataset…”
• Figure 6: Demonstrates concepts identified through co-occurrence, such as “dreadlocks, beanie, mask, ski.”

Regarding concept stability, we introduced it as an integral component of our method. However, due to space constraints, its detailed discussion and application were moved to the appendix. We emphasize that concept stability is a highly practical tool for identifying which output concepts are consistently triggered by specific input prompts (e.g., in counterfactual scenarios) and for assessing persistence across varying prompts. In Application 1, we explored three case studies using user-specified prompt distributions, and similar insights from those case studies could be derived using concept stability. These results are elaborated on in Figures 9 and 10 in the appendix.

Thank you for your question. The marginal and conditional distributions in Equation 3 serve as the theoretical foundation for the summarization metrics in Equations 4-6. Below, we clarify the conceptual and mathematical connections:

1. Marginal and Conditional Distributions:
   • The marginal distribution $p(C)$ in Equation 3 aggregates concept probabilities across all prompts: $$p(C) = \int p(C|t)p(t) \, dt,$$ while the conditional distribution $p(C|t)$ captures the likelihood of concepts given a specific prompt: $$p(C|t) = \int p(C|x)p_G(x|t) \, dx.$$

In practice, $p(C|t)$ is empirically approximated by generating $K$ images per prompt and extracting detected concepts using an object detector. Similarly, $p(C)$ is approximated by aggregating over $N$ prompts.

Summarization Metrics:

• Concept Frequency (Equation 4): $P(c)$ estimates the marginal distribution $p(C)$ by calculating the proportion of images containing concept $c$, making it a discrete empirical estimator for the continuous $p(C)$.
• Concept Stability (Equation 5): The coefficient of variation (CV) captures variability in the conditional distribution $p(C|t)$ across prompts.
• Concept Co-occurrence (Equation 6): $P(c, c')$ estimates joint probabilities in $p(C)$ by counting co-occurrences of concepts $c$ and $c'$ across all images.

In summary, the summarization metrics provide interpretable, discrete approximations of the theoretically defined marginal and conditional distributions, enabling practical analysis of $p(C)$ and $p(C|t)$ in the context of T2I models. We will update the manuscript and make sure to clarify the relationships between the prompt distributions in equation 3 and the summarization metrics.

Thank you for your question. We appreciate the opportunity to clarify the differences between our proposed metrics (concept frequency, co-occurrence, and stability) and those commonly used in counterfactual explanation work, such as those outlined in [1].

The metrics introduced in [1]—proximity, validity, and diversity—are designed specifically to evaluate counterfactual explanations. The primary goal in [1] is to create counterfactual examples that help end-users understand model predictions by suggesting actionable changes (e.g., “you would have received the loan if your income were $10,000 higher”). These metrics are summarized as follows:

• Proximity: Measures how close a counterfactual example is to the original input, calculated as the mean feature-wise distance.
• Diversity: Assesses the variation among multiple counterfactual examples by calculating feature-wise distances between each pair of examples.
• Validity: Evaluates the fraction of counterfactual examples that successfully achieve a different prediction outcome compared to the original input.

In contrast, our work addresses a fundamentally different problem: characterizing the distribution of images in terms of human-interpretable concepts. Rather than focusing on counterfactual explanations for model predictions, we propose a framework for using detectors to extract concept labels and bounding boxes from images and then introduce metrics to simplify the exploration of the resulting concept distributions. Specifically:

• Concept frequency: The empirical probability of a concept occurring across all generated images.
• Co-occurrence: The total number of times a pair of concepts appears together, highlighting relationships between concepts.
• Stability: Differentiates between persistent concepts (those that consistently appear regardless of prompt variations, indicated by low CV) and triggered concepts (those sensitive to specific prompts, indicated by high CV).

While both frameworks involve metrics, their purposes and scopes are distinct. The counterfactual metrics in [1] aim to guide and evaluate the generation of examples for explaining model predictions. In contrast, our metrics focus on providing a structured understanding of the distribution and relationships of human-interpretable concepts within visual data.

We hope this clarifies the key distinctions and illustrates how our work addresses a different set of challenges. Thank you again for raising this point and giving us the opportunity to elaborate.

[1] Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations

Thank you for your question. We clarify the practical details of our statement:

“We note that our use of an object detector D can introduce uncertainty in the extracted concepts, $C_{i,k}$ (e.g., due to detection confidence levels or the probabilistic nature of the model). Thus, we consider $C_{i,k}$ as samples from a distribution $C_{i,k}$ ∼ $p(C|x_{i,k})$. In the case that concepts are extracted deterministically from a given image $x_{i,k}$, $p(C|x_{i,k})$ is a delta distribution”

In practice, the detector (e.g., Florence 2) generates outputs as part of its sequence-to-sequence framework. At each decoding step, the model computes a probability distribution over possible next tokens using the softmax of logits from the transformer decoder. Sampling can be performed from this probability distribution to generate non-deterministic outputs. However, in our framework, we apply greedy decoding, where the token with the highest probability is selected at each step. As a result, we rely on deterministic outputs, which simplifies the formalization of our method. We acknowledge the importance of additional analysis on detector over/under-confidence and view this as part of a broader discussion on detector capabilities. We have included a detailed discussion in our response to Reviewer #eKz3, W3.

Thank you for your question. To bridge the continuous distributions ($p(C)$ and $p(C|t)$) with our discrete summarization metrics, we rely on empirical approximations. Specifically:

1. Empirical Approximation: We sample $N$ prompts and $K$ images per prompt to approximate the continuous distributions:

   $$p(C) \approx \frac{1}{N} \sum_{i=1}^N p(C|t_i), \quad p(C|t) \approx \frac{1}{K} \sum_{k=1}^K p(C|x_{i,k}),$$

   where $p(C|x_{i,k})$ reflects detections by the object detector.

2. Metrics as Estimators:

   • Concept Frequency estimates $p(C)$ as the proportion of images containing concept $c$
   • Concept Stability uses the coefficient of variation (CV) to assess variability in $p(C|t_i)$, providing insight into concept consistency across prompts.
   • Concept Co-occurrence estimates joint probabilities from co-occurrence counts.

While $p(C)$ and $p(C|t)$ are formally continuous, our framework operates on their discrete empirical counterparts, ensuring practicality and interpretability. This approach balances theoretical grounding with the constraints of real-world data. Thank you for bringing this to our attention. We will update the manuscript and make sure to clarify that we use discrete approximations.