Progressive Compositionality in Text-to-Image Generative Models

We thank Reviewer Amfr for their time and feedback! We are encouraged that R4 finds that “our methodology is novel” and “our framework improves compositional performance”. We appreciate the insightful questions, which have helped clarify certain aspects of our work. Below, we will address R4’s feedback point-by-point.

W1: More justification is needed to support the claim of achieving multiple improvements solely from this contribution.

Thanks for the feedback! According to the ablation study in Table 4, fine-tuning COM-DIFF without our CONTRAFUSION framework (which combines the designed contrastive loss with a multi-stage fine-tuning strategy) yields only limited improvements. Introducing the contrastive loss allows the model to effectively utilize contrastive image pairs, while the multi-stage fine-tuning strategy further enhances performance, especially in the complex category.

As it is critical to keep our arguments precise, we will clarify the wording in our revision.

W2: there seems to be potential for generating a higher number of combinations than the 15,000 currently provided..

At 15,000 contrastive image pairs, our Com-Diff dataset is indeed smaller than some other benchmark datasets. However, our aim is to keep high-quality contrastive image-pairs with minimal changes, which limits dataset size due to resource and costs.

A high-level objective of our work is to introduce an automatic pipeline to generate contrastive image pairs and demonstrates that fine-tuning with such image pairs can serve as a realistic and useful setting to improve compositionality of T2I models. We are definitely open to expanding the dataset and improving our dataset quality, and perform scaling law on how data size will impact our model performance in future work.

W3: Limitations from LLM and VLMs

We recognize existing work leveraging earlier LLMs and VQA models may suffer from this problem. However, given a text prompt consisting of a fixed number of objects and attributes and is fixed in pattern in most cases, the GPT-4 generated questions-answer pairs in our method empirically cover basically all/most the elements by decomposing the prompt, indicating current SOTA LLMs is able to handle such tasks more effectively.

To ensure the quality of our dataset, we conduct fine-grained analysis and rigorous experiments below (more details regarding annotation process and interface can be found in Appendix A.6).

• User study of element coverage in the LLM-generated QA pairs.

We sampled 1500 prompts (about 10% of our dataset, proportionally across 3 stages) and performed additional user study by asking annotators if the GPT-4 generated question set covers all the elements in the prompt. Each sample is annotated by 5 human annotators. The interface is presented in Figure 13.

Empirically, we find about 96% of the questions generated by GPT-4 cover all the objects, 94% cover all the attributes/relationships.

• User study of accuracy of question-answering from VQA models.

As noted in the response to W1@Reviewer PM7r, we performed an additional user study wherein for each question-image pair, the human subject is asked to answer the same question. The accuracy of the VQA model is then predicted using the human labels as ground truths.

Results are shown below (more details about data quality evaluation are in Appendix A.6), we observe that the VQA model is able to effectively measure alignment even when the text prompt becomes complex.

Overall, we also see significant improvements in compositionally ability evaluated by T2I-CompBench (Table 3) obtained in our current framework by using LLMs and VQA models.

We appreciate R3's suggestion on evaluating the quality of our dataset, we've added more details and interfaces about human evaluations in Appendix A.6.

W4: The idea of modifying text prompts to match generated images, line 250, is unclear.

Thanks for the feedback! Though it might intuitively look difficult to accurately describe complex prompts by revising captions with VQA models, we empirically show how this procedure works with examples. More details about the procedure can be found in Appendix A.3.

We use the following instruction to revise captions:

Instruction: Given the original text prompt describing the image, identify any parts that inaccurately reflect the image. Then,generate a revised text prompt with correct descriptions, making minimal semantic changes. Focusing on the counting, color, shape, texture, scene, spatial relationship, non-spatial relationship. (also give examples of revised prompts).

Given an image (Figure 10) and a text prompt, “Three puppies are playing on the sandy field on a sunny day, with two black ones walking toward a brown one.”

The prompt is modified to “Four puppies are standing on a sandy field on a sunny day, with three black puppies and one brown puppy facing forward. ”

The revised prompt made minimal semantic changes to the original prompt, but correctly described the images. Even such subtle characteristics like not “ playing on” but “standing” can be detected by the VQA model, attesting the effectiveness of our proposed method. We also found that without specific instruction to ask VQA to focus on the categories we are interested in, although VQA is powerful, the model may occasionally fail to generate precise captions that correctly describe the image.

To rigorously validate our claim and evaluate how accurately the revised caption describes the image, we randomly sampled 500 prompt-image pairs, that are selected by revising the caption, and performed additional user study by asking annotators how accurately the caption describes the image (interface shown in Figure 14). Note we have 5 annotators, each is assigned 100 caption-image pairs.

Empirically, we found that 4% of the samples show that the revised caption similarly describes the image as the original caption. 94.6% of the samples show the revised caption better describes the image. Overall,with the following settings, the average rating of the alignment between revised caption and image is 4.66, attesting that revised caption accurately describes the image.

• 5: The caption perfectly describes the image.
• 4: Has a few minor discrepancies.
• 3: Has several minor discrepancies.
• 2: Has significant discrepancies.
• 1: Does not match at all.

We appreciate the valuable feedback and have added the above discussion in our revision (Appendix A.3 and A.6).

W5: The description of the three stages appears for the first time on page 5, but…

Thank you for your valuable suggestion. Though we briefly mention curriculum learning in the last paragraph of introduction, we agree that we should introduce multi-stage training in more detail earlier. We’ll adjust the order in our revision!

W6: Image Similarity Quality Control

Details about the image similarity quality control are answered to W1@Reviewer PM7r. We manually examined 50% of the dataset, and filtered out approximately 8% image pairs that is either not well-aligned with text prompts or not similar. Empirically, the dataset contains over 95% high-quality contrastive pairs. We are definitely open to improving the quality of our dataset by comprehensively examining it in the future!

• Automatic Metrics:

We also evaluate the image similarity with cosine similarity between the image embeddings (CLIP Image Similarity). The average similarity score across 15,000 image pairs is 85.7%, indicating that the image pairs are highly similar, with only small differences.

Q1: How do you ensure that when generating negative images, only the specified attribute was changed?

Given the positive prompt, we edit it to the negative prompt with only specified attribute and relationship changes/swap, which ensure minimal changes semantically. We also employed image editing to change specific attributes in the image to keep minimal visual changes.

To validate the similarity between positive and negative image pairs, we have human evaluators to filter those image pairs that are not similar. We also show that the CLIP image similarity between image pairs is 86.7%, rigorously attesting the quality of the rest of the dataset.

Q2: In Figure 3, isn't the dog's head in the second image of the first row incorrectly generated?

We appreciate your careful evaluation. We will discuss more about this in the limitation section.

Such artifacts are a common limitation of all current generative models, including SOTA models. However, addressing this issue is not one of the primary objectives of our work. To maintain the quality of our dataset, we implemented rigorous filtering procedures, including both automated checks and manual reviews. However, given the dataset’s scale, a small number of edge cases (<5%, based on a recent audit) may have slipped through. These cases represent only a minor fraction, and the vast majority of examples in the dataset exhibit high fidelity in their compositional representation.

Q3: What is the training cost of the proposed approach? if the performance improvement is significant enough to justify the cost

Our fine-tuning procedure is computationally efficient with 15,000 samples. However, with our pipeline and constructed dataset, the improvement on most evaluation benchmarks are significant, which is worthwhile compared to the costs. We believe our dataset could also benefit future research and the proposed pipeline is also valuable to improve T2I compositionality.

Q4: Are there any adverse effects, such as a decrease in the model's ability to generate diverse images, after fine-tuning

Our paper primarily focuses on addressing compositionality in text-to-image generative models, and as such, diversity is not the main focus of our work. We believe that enhancing compositionality does not necessarily diminish the model’s ability to generate diverse images, as our method targets alignment without constraining the variety of generated outputs.

Empirically, we calculate FID on 10000 images randomly sampled from the COCO 2014 validation set, which might give some insights about the diversity of generated images. The results are shown below.

We observe that the image quality and diversity remain stable, as measured by the FID score of the generated images.

Q5: Given that CLIPScore has several limitations as … isn't it contradictory to use CLIPScore for measuring alignment in a paper that aims to overcome these limitations?

In our evaluation (Table 3), T2I-Compbench incorporates multiple metrics beyond CLIPScore, including BLIP-VQA, Uni-Det and 3-in-1, as noted in the response to Q2@Reviewer cqxf, to complements the limitations of CLIPScore.

Although the limitations of CLIPScore, it’s widely used in existing works and serves as a useful metric for demonstrating improvements in alignment with our method (Figure 4). We also present results with human annotations in Appendix A.6, to provide a more comprehensive evaluation.

We sincerely appreciate R4 for your valuable reviews and suggestions, we have added those elaborations in our revision.

Dear Reviewer Amfr,

Thank you again for your valuable feedback and acknowledgement of our work. We'd like to check in with you and see if our rebuttal has clarified your concerns. If there are still any unclear parts in our work, please let us know and we would be happy to engage further before the discussion period closes. Thanks for your time and effort in reviewing this paper!

W3: More comprehensive details about experiment setup

For fair comparison, we generate 10 images for each text prompt for T2I-CompBench evaluation. We present the standard deviation results in Table 3 and will run other experiments with different seeds to get standard deviation.

• Human Evaluation Details

As answered in the response to W1@PM7r, we tasked 3 annotators in total, each annotator assigned 2550 images out of the total 7650 images (about 50% samples, proportionally across three stages), to check if the positive and negative images align with the text prompt well and are similar with minimal changes. We filtered only 647 images from the selected 7650 images, which is 8.45%, attesting the quality of the dataset.

To further guarantee the quality of our dataset, we perform additional user-studies in terms of coverage of LLM-generated QA pairs, accuracy of question-answering of VQA models and alignment of revised caption with images with 1500 randomly selected samples with 5 annotators. More details and interfaces can be found in Appendix A.6.

W4: Relevant background research

We strongly agree that adding relevant background research is important to the authors to understand this field and our contributions. Due to the limitation of words, we didn’t add related work in the submission.

Both of the references provide insightful ideas regarding mitigating compositionality issues. We thank R3 for providing these two references, and we have incorporated both references into the related work section in our revision (Appendix E).

Q1: Do you have evaluation results for training on the Com-Diff dataset using multi-stage fine-tuning without incorporating contrastive loss?

To further disentangle the impact of multi-stage and contrastive loss, we present the results using solely multi-stage without contrastive loss below.

Solely applying multi-stage doesn’t significantly improve the performance, we hypothesize this is due to the model not being able to fully explore the Com-Diff dataset without contrastive loss. We will add the results in Table 4.

Q2: Recycling simpler compositional prompts

We appreciate R3’s valuable suggestion and find it quite inspiring to prevent the model forgetting the generative ability on simple prompts by recycling simpler samples. We plan to conduct experiments by introducing varying proportions of simple prompts in later stages and will include these results in an ablation study once we have determined the optimal settings.

We thank Reviewer cqxf for their time and feedback! We are encouraged that R3 finds that our work “mitigates a critical issue in text-to-image generative models” and that “our dataset is a valuable resource for further research”. We will respond point-by-point to R3’s feedback below:

W1: Overlap between Com-Diff and T2I-CompBench; Evaluating the model on a completely separate test set

We agree that our work could definitely benefit from a detailed analysis of the overlap with T2I-CompBench and evaluations on other benchmarks. Below, we include those two result:

• Overlap with T2I-CompBench

When constructing our prompts, we consciously avoided using the same ones as those in T2I-Compbench, especially considering some prompts from T2I-CompBench are empirically difficult to generate aligned image (e.g., “a pentagonal warning sign and a pyramidal bookend” as shown in Figure 9), which are not well-suited for our dataset. We have filtered out similar prompts from our dataset using LLMs to identify uncommon combinations of objects and attributes.

Specifically, during the prompt construction process, approximately 1% of our prompts overlapped with those from T2I-Compbench, which have also been filtered out.

• GenAI benchmark

We also evaluate our model on the Gen-AI benchmark. To ensure a fair comparison with DALL-E 3, we fine-tune our model on SD v3. As shown below, ContraFusion outperforms all other models on advanced prompts. While it demonstrates slightly weaker performance in some basic categories compared to DALL-E 3, it still surpasses other models, highlighting the effectiveness of our framework.

We have added those two analysis in our revision (See more details in Appendix A.1 & Appendix C.2).

W2: Explicitly stated the metrics used in Table 3 and other relevant sections …

Thanks for the valuable suggestion! We agree explicitly stating the metrics and important considering the difficulty of evaluating compositionality in generative models. Following [1], we use DisentangledBLIP-VQA[1] for color, shape, texture, UniDet[2] for spatial, CLIP for non-spatial and 3-in-1 [1] for complex categories. In Figure 4 and Figure 5, we use average CLIP score between image and prompt similarity.

We have clarified this point in our revision (Appendix C.1).

[1] Huang, et al., T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation [2] Zhou, et al., Simple multi-dataset detection

Dear Reviewer cqxf,

Thank you again for your valuable feedback and acknowledgement of our work. We'd like to check in with you and see if our rebuttal has clarified your concerns. If there are still any unclear parts in our work, please let us know and we would be happy to engage further before the discussion period closes. Thanks for your time and effort in reviewing this paper!

We thank Reviewer drFx for their time and feedback! We are encouraged that R2 finds that “our contrastive loss is interesting” and “our experiment setups are clear”. We will respond point-by-point to their comments below:

W1: A bit more detail on how the contrastive loss is incorporated would be nice.

Thank you for this valuable feedback! Yes, we use latent encoder representation of positive and negative images at any arbitrary time step t, and apply a small projection head to map the image features to the space where the contrastive loss is applied. During training, the model predicts the noise to be subtracted. After experimenting with different proportions, we found that a weight of 0.1 for the InfoNCE loss yields the best performance.

We believe it’s important to keep our framework clear and precise and will add more details in our revision.

W2: Comparison with other datasets with hard-negative compositional images.

How our model would fare with other hard-negative dataset is indeed an interesting question.We include the results of fine-tuning our model with different datasets, evaluated by T2I-CompBench below..

Although COLA attempts to construct semantically hard-negative queries, the majority of the retrieved image pairs are quite dissimilar in practice, often introducing a lot of noisy objects/background elements from the real images, due to the nature of retrieval from existing dataset. We hypothesize this makes it difficult for the model to focus on specific attributes/relationships in the context of compositionality. We show some examples in Figure 12. Furthermore, they don’t include complex prompts with multiple objects/attributes, and do not account for non-spatial relationships.

Winoground contains high-quality contrastive images, however it only contains 400 effective sample pairs, covering 5 categories, which is too small for the model to learn and we believe the comparison is not fair.

In contrast, our dataset ensures the generated image pairs are contrastive with minimal visual changes, enforcing the model to learn subtle differences in the pair, focusing on a certain category.

We really appreciate your suggestion and agree comparing with other similar dataset is important and have added the above results in our revision (Appendix A.5)!

Dear Reviewer drFx,

Thank you again for your valuable feedback and acknowledgement of our work! We'd like to check in with you and see if our rebuttal has clarified your concerns. If there are still any unclear parts in our work, please let us know and we would be happy to engage further before the discussion period closes. Thanks for your time and effort in reviewing this paper!

Thank you for the review. The experiments on comparing to tuning with other kinds of hard negative data makes sense and highlights the befit of their data. I will keep my rating.

We thank Reviewer PM7r for their time and feedback! We are encouraged that R1 finds that “our approach is well motivated” and “our experiments are comprehensive”. We will respond point-by-point to their comments below:

W1:The data pipeline relies on VQA method's ability to measure alignment ...VLMs also struggle with this,...

We agree that the performance of our approach partially depends on the reliability of question-answering from the VQA model. Nevertheless, as shown through our quantitative results, significant improvements from prompt-image similarity (measured by CLIP score in Figure 4.) and compositionally ability evaluated by T2I-CompBench (Table 3) are obtained in our current framework by using VQA models.

To analyze the accuracy of the VQA model's answering results, we sampled 1500 images (about 10% of the dataset, across 3 stages) and performed an additional user study wherein for each question-image pair, the human subject is asked to answer the same question. The accuracy of the VQA model is then predicted using the human labels as ground truths. Results are shown below (more details about data quality evaluation are in Appendix A.6).

We observe that the VQA model is able to effectively measure alignment even when the text prompt becomes complex. After filtering images with low alignment score, we use CLIPScore as a second check to select the best aligned image, further guaranteeing the quality of our dataset.

• L259 on quality control is also ambiguous

We tasked 3 annotators in total, each annotator assigned 2550 images out of the total 7650 images (about 50% samples, proportionally across three stages), to check if the positive and negative images align with the text prompt well and are similar with minimal changes. We filtered only 647 images from the selected 7650 images, which is 8.45%, attesting the quality of the dataset. The high-level objective of our paper is to propose an automatic pipeline to generate faithful contrastive image pairs, which we find crucial for guiding models to focus on compositional discrepancies. Especially with the advancement of VQA models and LLM models, our pipeline and dataset can benefit from higher alignment scores.

As it is critical to keep our words precise, we have added clarification in our revision (Appendix A.6).

W2: Comparison between synthesized vs. real hard negative dataset

How our model would fare with a real hard-negative dataset is indeed an interesting question.We include the results of fine-tuning our model with COLA, BISON evaluated by T2I-CompBench below (randomly sampled consistent number of samples across datasets).

Although COLA and BISON attempt to construct semantically hard-negative queries, the majority of the retrieved image pairs are quite dissimilar in practice, often introducing a lot of noisy objects/background elements from the real images, due to the nature of retrieval from existing dataset. We hypothesize this makes it difficult for the model to focus on specific attributes/relationships in the context of compositionality. We show some examples in Figure 12. Furthermore, they don’t include complex prompts with multiple objects/attributes, and COLA does not account for non-spatial relationships. In contrast, our dataset ensures the generated image pairs are contrastive with minimal visual changes, enforcing the model to learn subtle differences in the pair, focusing on a certain category. To the best of our knowledge, no real contrastive image dataset only differs on minimal visual characteristics.

We agree the comparison is very valuable and have incorporated the results in our revision (Appendix A.5)!

W3: The user study in figure 8 is not convincing that…, but at the cost of the generation quality.

Since our model is fine-tuned on Stable Diffusion models, which tends to generate lower aesthetic scores compared to DALLE-2 and PixArt-Alpha [1], it is important to note that our method does not compromise aesthetic quality in comparison to SD v3 and SDXL–as shown in Figure 8, after fine-tuning, our method outperforms both in terms of aesthetic quality and alignment.

[1] Chen, et.al., PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis

Dear Reviewer PM7r,

Thank you again for your valuable feedback and acknowledgement of our work. We'd like to check in with you and see if our rebuttal has clarified your concerns. If there are still any unclear parts in our work, please let us know and we would be happy to engage further before the discussion period closes. Thanks for your time and effort in reviewing this paper!

Dear Reviewer PM7r,

As the discussion period will close next week, please take some time to read the authors' rebuttal and provide feedback as soon as possible. Did the author address your concerns, and do you have further questions?

Thanks,

Area Chair

Thanks, I don't have any other questions at this time. I increased my score to a 8, thanks for the new experiments and discussion.