Lost in Translation: Conceptual Blind Spots in Text-to-Image Diffusion Models

Thank you very much for the thoughtful and detailed review. We reply point-by-point here, to begin the discussion. We provide detailed explanations here and will offer a clearer clarification in the updated version of our paper.

1. "Concept Pairs" denotes two main objects A and B in a text segment asked by users. C, while not explicitly indicated, is often bound together with B (or A) in the training dataset. For example, in the case of a teacup (A) and iced coke (B), the frequent association of iced coke (B) with a transparent glass (C) in datasets leads to the replacement of the teacup (A) by the transparent glass (C). The inclusion of this hidden object C is the main focus of our study.

2. Categories and patterns in Table 1 are classified by human experts through meticulous observation.

   2.1 Specifically, Category I represents the misalignment issues involving the hidden object C, as mentioned above. Category II, on the other hand, represents traditional issues involving only A and B, such as requesting an image of a frog and a turtle, but the final image contains only two frogs.

   2.2 Each pattern represents more specific scenarios within each category , such as "Beverage and Erroneous Container", "Local Cuisine and Non-native Location", ... The existence of patterns can better provide GPT with few-shot guidance.

3. "What GPT helps, and human researchers guide?" In this process, GPT helps to come up with more misalignment concept pairs, and human researchers provide instruction prompt, few shot and evaluation on images. Specifically, GPT generates new examples that it deems likely to exhibit misalignment issues, based on given misalignment cases. Human researchers are responsible for designing the instruction prompts and providing examples for GPT's few-shot learning. We have included the complete instruction prompt in the appendix of our paper. Furthermore, the examples provided by GPT are sent to Midjourney and SDXL-1.0 for image generation, and human researchers then assess whether misalignment issues occur.

4. "Elaborate screening process" Of course, GPT first expands each concept pair with five information-rich sentences, and each sentence undergoes four rounds of image generation. This results in a total of 20 images, from which human experts select based on the number of images correctly representing both A and B. A Level 5 rating means all 20 images exhibit misalignment issues, indicating the highest quality samples. All complete instruction prompts are included in the appendix of our paper.

Thank you very much for the thoughtful and detailed review. We reply point-by-point here, to begin the discussion. We provide detailed explanations here and will offer a clearer clarification in the updated version of our paper.

1. Method and Model

   1.1 We believe that such misalignment issues could be mitigated by more precise image-text pairing during the pre-training process , similar to the approach taken by DALLE3[2]. We have also supplemented our data with the performance of DALLE3 and the AAE[1] method, which is based on traditional misalignment problems. The performance of AAE was subpar, while DALLE3 indeed demonstrated superior performance, on par with the SDXL-1.0 model that has been enhanced with MoCE.

   1.2 Our proposed MoCE does indeed encounter some failures. For example, it may overlook the relationship between objects A and B (e.g., when asked for a tea cup of iced coke, it generates a glass of iced coke and an additional teacup), or it merges A and B into a single object (such as generating a panda-themed hairdryer when requested to depict a panda and a hairdryer). We will showcase such examples in the updated version of our paper, possibly in the appendix.

2. Bad Readability

Thank you very much for pointing out the bad readability of our paper. In the updated version of our paper, we will pay closer attention to this issue, aiming to present our study as clearly as possible. This includes providing clear explanations of key terms and detailing the objectives and processes involved in our dataset collection and so on.

3. Human Evaluation Thank you very much for pointing out your concern. In human evaluation, we particularly focus on the Level 5 samples within Category I, total of 173. Specifically:

   3.1 Category I: category I represents the misalignment issues involving the hidden object C, as mentioned above. For example, in the case of a teacup (A) and iced coke (B), the frequent association of iced coke (B) with a transparent glass (C) in datasets leads to the replacement of the teacup (A) by the transparent glass (C). While Category II, on the other hand, represents traditional issues involving only A and B, such as requesting an image of a frog and a turtle, the final image contains only two frogs.

   3.2 Level 5: GPT first expands each concept pair with five information-rich sentences, and each sentence undergoes four rounds of image generation. This results in a total of 20 images, from which human experts select based on the number of images correctly representing both A and B. A Level 5 rating means all 20 images exhibit misalignment issues, indicating the highest quality samples. All complete instruction prompts are included in the appendix of our paper.

[1] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. ACM Transactions on Graphics (TOG), 42(4):1–10, 2023.

[2] OpenAI. Dall·e 3 system card. 2023.

Thank you very much for the thoughtful and detailed review. We reply point-by-point here, to begin the discussion. We provide detailed explanations here and will offer a clearer clarification in the updated version of our paper.

1. "Missing the clear definition of important terms." Thank you very much for pointing out that some important terms are not clearly defined.

   1.1 "Concept Pairs" denotes two main objects A and B in a text segment asked by users. In text-to-image generation, when a text segment contains multiple objects, some may not be accurately represented in the resulting image. In our study, we focus on scenarios where a text segment contains only two objects. We refer to these two objects as A and B, collectively termed a "concept pair."

   1.2 "Conceptual Blind Spots" (CBS) denotes object A (or B) is encroached upon and consequently disappears in the image. In text-to-image generation, "Conceptual Blind Spots" represents the occurrence of object encroachment. Specifically, this denotes the disappearance of object A (or B), which represents the user's requirement, in the final image. Previous studies primarily focused on scenarios where A is encroached upon by B. However, we have observed that in many instances, A can also be encroached upon by a hidden object C, which often appears bound with B in datasets. For example, in the case of a teacup (A) and iced coke (B), the frequent association of iced coke (B) with a transparent glass (C) in datasets leads to the replacement of the teacup (A) by the transparent glass (C). The inclusion of this hidden object C is the main focus of our study.

   1.3 "Socratic Reasoning"[1] denotes the process of problem identification, understanding and reasoning using Large Language Models (LLMs), particularly GPT. We utilize GPT to help us come up with concept pairs that are likely to encounter issues of "Conceptual Blind Spots". The specific procedures for this approach are detailed in Appendix of our paper.

2. Missing detailed explanation of the process of data collection Thank you very much for pointing out that we miss clear explanation for the data collection.

   2.1 "The overall goal of the dataset" is that we aim to collect CBS samples with inclusion of a hidden object C mentioned above, a focus not emphasized in previous work. Typically, earlier studies concentrated on scenarios involving only A and B mutually encroaching upon each other.

   2.2 "The reason behind each round" is that we use a divide-and-conquer strategy, leveraging the extensive knowledge possessed by GPT to expand our dataset. Specifically, human mind has its limits, hence the need for GPT's assistance. However, GPT often cannot complete the task in one step, leading us to divide the data collection process into several rounds. Initially, we provide CBS examples conceived by human experts and use these as few shot for GPT to generate more examples. Subsequently, human experts categorize these examples into patterns, creating stronger few shot for further rounds. Here, human experts are involved only in the initial two stages. Afterwards, GPT operates like a snowball effect, continually expanding the scope of data collection.

3. Missing detailed explanation of our method Thank you very much for pointing out that we miss clear explanation for the our method. We provide a detailed explanation here, and will give an algorithm table in the updated version of our paper.

   3.1 A smaller value of D indicates a higher likelihood of objects A and B appearing together in the image. Specifically, during N-step diffusion sampling, GPT initially suggests the drawing sequence for A and B. Taking the case where A is drawn first, we provide the model with text prompts only for A in the initial x steps and then supply prompts for both A and B in the remaining N-x steps. After the drawing process, we adjust x based on the magnitude and sign of D to find a more optimal x.

   3.2 We employ a binary search algorithm because we believe that the absolute value of D has a linear relationship with x. For concept pairs A and B prone to CBS issues, if x is too small, CBS issues will still occur, whereas if x is too large, it limits the time steps available for drawing B. Therefore, we use a binary search algorithm to quickly find the appropriate value of x.

The 4th, 5th, 6th responses can be found in the next text box.

[1] Qingxiu Dong, Li Dong, Ke Xu, Guangyan Zhou, Yaru Hao, Zhifang Sui, and Furu Wei. Large language model for science: A study on p vs. np. arXiv preprint arXiv:2309.05689, 2023.