Enhancing Compositional Text-to-Image Generation with Reliable Random Seeds

We appreciate the reviewer's detailed feedback and their positive assessment of our method's novelty, efficiency and experimental validation. Below we address each point raised in the Weaknesses and Questions sections:

The reliance on selected seeds may limit the diversity of generated outputs

Thank you for raising this important point, as it aligns closely with a key contribution in our paper. We specifically address concerns about limiting model diversity by adopting a novel fine-tuning scheme that modifies only the "Query" and "Key" projection layers in the attention modules. This approach is designed to preserve the model’s inherent diversity while improving accuracy (see Table 9 for more details). As shown in Table 4, our method achieves a Recall score of 71.3, which is remarkably close to the original model’s score of 76.7, indicating that diversity is largely maintained. In contrast, other methods, such as LMD (Recall = 49.4), MultiDiffusion (Recall = 36.2), and Ranni (Recall = 46.6), show significantly reduced diversity. Additionally, if diversity were not a major concern, our approach could achieve even higher accuracy, as demonstrated in Table 9.

Lack of an automatic, inference-time method to select reliable seeds for generating accurate compositions. Will the authors consider developing an algorithm to dynamically choose seeds based on prompt characteristics?

Thank you for raising this important point. We would like to clarify that our seed mining method is indeed automatic via utilizing CogVLM. Perhaps the underlying concern here is the efficiency and practicality of applying this approach on-the-fly for a specific testing prompt. We acknowledge that, in its current implementation, the process can be somewhat slow for such use cases, as it involves generating hundreds of images to evaluate the reliability of a random seed for a given prompt. This being said, this approach remains feasible for more efficient approaches, such as one-step diffusion models, where the computational overhead is significantly reduced. Additionally, we find the reviewer’s suggestion of a direct mapping from text embeddings to reliable noise patterns highly intriguing. This could indeed streamline the seed selection process and improve efficiency. However, we believe that collecting the paired training data and training such a model would require substantial effort, making it an excellent avenue for future work. We are excited by the potential of this direction and would be happy to pursue it in follow-up research, building on the foundations established in this study. Thank you for highlighting this compelling opportunity.

Potential decline in image quality is unexplored. Are there measures to prevent potential degradation in image quality or unintended biases? How do the authors ensure that the self-generated dataset maintains high quality compared to real-world or externally validated datasets?

Thank you for raising this concern. A drop in image quality is indeed a challenge for all methods aiming to improve model fidelity, including ours. However, as shown in Table 4, our method achieves a much smaller loss in aesthetic scores—a proxy for image quality—compared to state-of-the-art methods, while still improving accuracy, as can be seen from Table 4:

Aesthetic Scores: SD: 5.19, LMD: 4.65, MultiDiffusion: 4.40, Ranni: 4.43, Ours: 5.13.

This improvement is largely due to our fine-tuning scheme, which modifies only a small part of the model, preserving the overall image quality. In contrast, as shown in Table 9, fine-tuning all model parameters significantly degrades image quality (Aesthetic Score: 4.12).

The approach assumes that data generated with reliable seeds is of higher quality for model fine-tuning, but lacks empirical comparisons with real-world datasets.

We would like to clarify that our approach does not claim that synthetic data is inherently better than real-world data. Instead, our focus is on the economic and practical benefits of using self-generated data, which provides a cost-effective way to fine-tune models, especially when high-quality real-world datasets are scarce or difficult to obtain. While empirical comparisons with real-world datasets can be interesting, a direct apples-to-apples comparison is non-trivial since a sufficiently diverse set of high-quality real-world images for specific prompts, such as "four coins," is not always readily available or easy to assemble. If the reviewer has a specific real-world dataset in mind that could be suitable for comparison, we would be happy to explore it and provide further insights.

Thank you for the suggestion! We have added another qualitative comparison between SD and Ranni for multiple-category numerical composition in Figure 11 in the new revision. If further comparisons are needed, we are happy to provide them.

We appreciate the reviewer's thoughtful feedback and their positive assessment of our method's simplicity, broad applicability, and experimental validation. Below we address each point raised in the Weaknesses and Questions sections:

Comparison with Ranni [1].

We thank the reviewer for bringing Ranni to our attention. Similar to LMD, Ranni employs a two-stage generation process: text-to-panel and panel-to-image. We have conducted comparisons with Ranni (implemented on SD2.1 768x768) on our dataset. As shown in the updated Table 4, while Ranni achieves comparable numerical and spatial accuracy (50.7% vs. our 51.3%, and 35.5% vs. our 36.6%), it does so at a significantly higher cost to aesthetic quality and recall (Recall: 46.6 vs. our 71.3; Aesthetic Score: 4.43 vs. our 5.13). A qualitative comparison has also been added in Appendix A.7. We also would like to note that Ranni uses additional network structures for conditional input, requires extensive fine-tuning (40k steps vs. our 5k), and relies on a much larger dataset (50M samples vs. our 2.5k). These distinctions highlight the efficiency of our method in achieving competitive performance with fewer resources.

What are the "these images" in abstract

Thank you for pointing this out. "These images" in the abstract means the generated images used for training. To make it clearer, we have changes "these images" to "these generated images".

Scalability to out-of-scope prompts/tasks

Thank you for this interesting question. In principle, our method obtains the largest improvement when seed mining is conducted on the same task as that of the text prompts for testing. Nevertheless, our fine-tuned models demonstrate a good ability to generalize and improve the compositional accuracy for out-of-scope but related tasks (e.g., composing more than 6 objects in our case).

To evaluate adaptability to unseen tasks, we tested our method on two new datasets: (1) numerical prompts with required quantities varying from 7 to 8, and (2) multi-category numerical compositions (such as "two cups and four spoons"). To test our method on multi-category numerical composition, we created a test set of 600 prompts, featuring 10 category pairs with diverse backgrounds and numerical instructions (see Appendix A.6.2).

As shown in Appendix A.6.2, A.6.3 and the two tables below, our fine-tuned models extend their improvements to instructions that were never seen during seed mining or fine-tuning.

Table 1: Accuracy comparison when generating 7 and 8 objects

Table 2: Accuracy comparison for multi-category positional composition.

[1] Liu, Haotian, et al. "Visual instruction tuning." Advances in neural information processing systems 36 (2024).

[2] Hertz, Amir, et al. "Prompt-to-prompt image editing with cross attention control." arXiv preprint arXiv:2208.01626 (2022).

[3] Chen, Minghao, Iro Laina, and Andrea Vedaldi. "Training-free layout control with cross-attention guidance." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2024.

We appreciate the reviewer's thoughtful feedback and their positive assessment of our method's simplicity and usefulness. Below we address each point raised in the Weaknesses and Questions sections:

How do we decide if an image is correctly generated in Figure 4? Does this process bring biases?

We utilized CogVLM2 to assess whether an image is correctly composed (details provided in Section 4.2 and Appendix A.3.1). This is an automatic approach, and we agree with the reviewer that a discussion about potential biases is necessary, which we have added in A.3.1. In our particular case, we have spent significant effort to manually verify the correctness of CogVLM2 and found no specific biases in the model. Additionally, we manually reviewed the 600 images generated to create Figure 4 (top) and found very few assessment errors from CogVLM2. The model performed well across various categories, backgrounds, and object arrangement patterns, including scenarios with overlapping objects.

Does CogVLM2 result in biases, for example, preferency towards non-overlapping layout? Are the fine-tuned models overfitted to generating images with non-overlapping layouts?

We would like to clarify that many generated images with overlapping object layouts are correctly annotated by CogVLM2. To illustrate this, we provide several examples in Figure 8 of Appendix A.3.1 (in the revised paper). Figure 4 seems to highlight layouts with non-overlapping objects because the pre-trained Stable Diffusion and PixArt-alpha models tend to generate such images more frequently, not because of a bias in the evaluation model. Notably, our fine-tuned model does not overfit to a specific mode, as demonstrated by its significantly higher recall scores compared to all other baselines (Table 4). To illustrate that the fine-tuned models are not restricted to generating non-overlapping layouts, we showcase examples of images with object occlusions generated by our fine-tuned Stable Diffusion model in Figure 9 of Appendix A.3.1. We would like to point out that this is another strength of our method compared to layout-to-image methods such as LMD and MultiDiffusion, which rely on non-overlapping bounding boxes as input constraints.

Which layer is used for the heatmap in Figure 4?

We follow the established convention in the field (e.g., [2], [3]) by computing the average of the attention maps across all timesteps and all cross-attention modules within the UNet model.

The four coins can be parallel or any position arrangement. Why does the heatmap for "four coins" in Figure 4 is coincidently splited the 4 grids?

We would like to clarify that this heatmap represents the behavior of the original pre-trained diffusion model. The pattern in Figure 4 occurs because the prompt "four coins" is most often composed when arranged in a 2x2 grid. This is likely influenced by the pre-training data, which contains many examples of "four" arranged in this way. However, this is not the only pattern the model can generate; other arrangements, such as parallel ones, as the reviewer suggests, are possible but less representative. The plot in Figure 4 does not emphasize these less frequent cases.

Overlapped result is hard to generate?

We would like to note that our seed selection method does not favor any particular types of images, whether overlapping or non-overlapping. We do not impose any specific mechanism to check for this, and CogVLM does not bias toward any particular case, as far as we have observed. While controlling when the model generates overlapping versus non-overlapping objects could be useful, we believe that this goes beyond the scope of our current work.

In general, how often and accurately a model generates overlapping objects depends largely on its training data. If the training data predominantly contains non-overlapping layouts, the model is less likely to generate accurate overlapping compositions. This is a common challenge with pre-trained generative models and is not specific to our method. Nevertheless, as shown in Figure 9, there are still many instances where overlapping objects are generated successfully.

Do we control the noise by fixing the seed? Why is there a preferency over certain seeds in Section 3.3?

Yes, fixing the random seed effectively fixes the input noise for diffusion models. In principle, our paper highlights that certain noise patterns can be exploited to enhance model performance. In practice, we use the initial seeds as a straightforward and reproducible way to control the noise. Each seed uniquely determines a specific noise pattern, and with a fixed seed, the model consistently starts from the same deterministic noise rather than resampling. This leads to consistent positional arrangements and explains the observed preference for certain seeds in Section 3.3.

The author has addressed my concerns, especially on occluded samples.

The main idea is easy to follow.

The technical contribution is weak, but it will interest most of our community if published.

I raised my rating.