Rare-to-Frequent: Unlocking Compositional Generation Power of Diffusion Models on Rare Concepts with LLM Guidance

We sincerely appreciate the reviewers' constructive comments and positive feedback on our manuscript.

W1. My primary concern is that what is deemed rare for the diffusion model may not be considered rare for the LLM. Since the LLM lacks access to the training distribution of concepts used by the diffusion model, it may substitute rare concepts with other rare ones. Providing the LLM with the concept distribution from LION could enhance the results. This distribution has been published by [1].

This is an excellent question. Because recent diffusion models have been trained on billion-scale text-to-image datasets, we could naturally expect that their distributions are closely aligned with LLMs. This may be the reason why our zero-shot LLM guidance performs well in our experiments. Meanwhile, following your suggestion, we provided captions from LAION-400M to assist LLMs in generating a rare-to-frequent concept mapping.

Setup. Given a rare attribute word ("bearded") in a prompt ("a bearded apple"), we measured the frequency of all the subsequent words in the LAION dataset. For example, if there are 100 captions containing "bearded man" in LAION, the frequency of "man" for the attribute "bearded" is calculated as 100. We then integrated these next-word frequencies into the R2F process in two ways: (1) R2F with the most frequent subsequent word in LAION. For each rare concept attribute word, we extract the most frequent subsequent noun from LAION and directly use it for the noun of the frequent concept. For example, if "man" appears most frequently after "bearded", we use "bearded man" as the frequent concept for the original rare concept such as "bearded giraffe". (2) R2F using Top20 subsequent word frequency in LLM prompt. In this case, we extract the top 20 frequent subsequent words from LAION, and add them to the LLM instruction for identifying rare-to-frequent concept mapping as follows:

...When finding frequent concepts for extracted rare concepts, please consider the words that appeared most frequently after the attribute word of the rare concept in the LAION image caption dataset. The list of the top 20 words is as follows and is in the format of ('next word', 'count'). \n EXAMPLES...

Result. The results for these R2F variants with LAION information are in the table below.

While we expose the LAION information, the performance of these variants is not higher than our original R2F. This may be due to the low quality of LAION captions (e.g., captions are mostly alt texts that are crawled from the web), and because recent models such as SD3.0 are trained on more high-quality image-caption datasets [1], potentially diverging from the distribution of LAION captions. Indeed, the top 5 subsequent words following "bearded" in LAION were ('man', 8772), ('dragon', 5996), ('collie', 3573), ('iris', 2153), and ('dragons', 1087), showing a discrepancy with common sense knowledge.

However, we believe that with access to high-quality diffusion training sets, the performance of LLM guidance can be further enhanced, and we will include this finding in the revised manuscript. Again, we greatly appreciate your insightful feedback.

• We added this analysis in Section M of the revised manuscript.

[1] Improving Image Generation with Better Captions, ArXiv, 2023

Q3. Is the design of R2F+ necessary？ In fact, layout-based methods have outstanding spatial awareness, however, the trade-off is increased computational cost and a decline in image quality (in terms of detail, aesthetics, etc.). First, you need to conduct a comparative evaluation of R2F+ in terms of image quality. Additionally, as noted in Table 3’s T2I-CompBench, R2F achieves higher spatial metrics than both the layout-based method LMD and the LLM-based method RPG. Thus, expanding R2F to a layout-based approach may be unnecessary, as it would only improve spatial performance while significantly compromising image quality.

Image Quality. Per your suggestion, we additionally measured three popular image quality scores (e.g., LAION-aesthetic [5], PickScore [6], and ImageReward [7]) of R2F+ for multi-object cases in RareBench and compared it to R2F. As shown in the table below, there was no significant difference in image quality scores.

• We added this quality analysis in Section N of the revised manuscript.

Necessity of R2F+. R2F+ is useful in many applications where the layout-aware composition is very important. (1) Layout-aware image/poster design. When a user creator wants to place an object in a specific position within an image for poster design, R2F is insufficient because it cannot adjust absolute positions. In such cases, R2F+ becomes essential. (2) Data synthesis for enhancing spatial understanding of foundation models. Recent multi-modal LLMs (e.g., LLaVA [8]) and pre-trained VLMs (e.g., CLIP [9]) are known to exhibit weaknesses in spatial understanding [10], so several studies have attempted to enhance their performance with spatiality-aware image synthesis (e.g., generating images that accurately captures spatial information in text prompts). R2F+ has the potential to enhance the performance of these foundation models by serving as a data synthesis method, as spatial composition is more critical than image quality in this case.

• We added this discussion in Section P of the revised manuscript.

[5] Laion Aesthetic Predictor. https://github.com/LAION-AI/aesthetic-predictor, 2022

[6] Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation, NeurIPS, 2023

[7] Imagereward: Learning and Evaluating Human Preferences for Text-to-image Generation, NeurIPS, 2024

[8] Visual Instruction Tuning, NeurIPS, 2024

[9] Learning Transferable Visual Models From Natural Language Supervision, ICML, 2021

[10] SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities, CVPR, 2024

Q4. You can consider using the IterComp[1], which is a backbone specifically designed for compositional generation and may lead to a more significant performance improvement.

Thanks for introducing an important relevant work. We conducted additional experiments using IterComp as the backbone for R2F, and the results are in the below table.

Overall, R2F_itercomp consistently improves the compositional generation performance of IterComp on Rarebench dataset (i.e., better T2I alignment scores by GPT-4o). These results further demonstrate the flexibility of R2F across the diffusion backbones. R2F_itercomp was generally better than R2F_sdxl but worse than R2F_sd3.0. This is likely because IterComp enhances the SDXL backbone by compositional-aware preference optimization [11], so it might not yet match the generative performance of the more recent SD3.0 backbone. We will include these results in Table 4 of the main paper.

• We included this result in Table 4 of the revised manuscript.

[11] IterComp: Iterative Composition-Aware Feedback Learning from Model Gallery for Text-to-Image Generation, ArXiv, 2024

W1. For applications, rare concept composition generation is still a relatively niche area, although I acknowledge that it is indeed a novel task within compositional generation. Have you considered exploring a broader range of application scenarios?

This is an excellent question. Rare concept composition is essential in various applications that require the creation of creative content, such as designing characters and posters for comics, movies, and games. Creators in these domains should often produce content that has never existed, such as characters with elemental faces (e.g., fire or water), pirate ship-shaped spaceships, or trumpet-like guns. Therefore, rare concept composition could be considered a mainstream area for these creators.

Furthermore, our idea of frequent concept guidance can potentially be extended to other modalities, such as text-to-speech (TTS) [1,2,3] and text-to-music [4]. For instance, TTS models have concept categories including speaker, intonation, and emotion. When generating speech such as "an angry Trump speaking Catalan", we might expose frequent concepts such as "an angry Spanish speaking in Catalan" to improve the composition performance of diffusion-based TTS models.

Thus, we believe that rare concept composition has a broader range of application scenarios.

• We included this discussion in Section P of the revised manuscript.

[1] AudioLDM: Text-to-Audio Generation with Latent Diffusion Models, ICML, 2023

[2] Natural language guidance of high-fidelity text-to-speech with synthetic annotations, ArXiv, 2024

[3] DiTTo-TTS: Efficient and Scalable Zero-Shot Text-to-Speech with Diffusion Transformer, ArXiv, 2024

[4] Simple and Controllable Music Generation, NeurIPS, 2023

W2. For the computational cost, this paper adopts an approach similar to LMD to enhance R2F, resulting in R2F+, which involves substantial latent and gradient computations. A detailed comparison of computational and memory overhead with other methods is essential to assess the feasibility of the proposed approach.

We thank the reviewer for helping us improve our paper. The table below compares the A100 GPU time and memory required to generate a prompt "a horned lion and a wigged elephant", which consists of two rare concepts "horned lion" and "wigged elephant".

Peak memory. While R2F+ involves several latent and gradient computations, there is no significant difference in peak memory compared to R2F since it follows a sequential process. R2F requires approximately 31GB of peak memory, and R2F+ requires approximately 35GB of peak memory where an additional 4GB mostly comes from the gradient computations in cross-attention control.

GPU Time. The time taken for R2F+ is approximately 72 sec, which can be decomposed as 1) masked latent generation via object-wise R2F takes around 42 sec, and 2) region-controlled concept guidance takes around 30 sec. Specifically, for the process of 1), the generation of each object takes around 20 sec (same as R2F) with an additional 1 sec for masking, resulting in a total of 42 sec for two objects. For the process of 2), the majority of the increased computation time compared to R2F is attributed to attention control, which adds around 10 sec. Consequently, for a prompt with N objects, the time complexity of R2F+ is expected as N*(T+1)+(T+10), where T is the inference time of R2F or SD3.

Thus, in our experiments, we generated each image within tens of seconds to a few minutes on a single 40GB A100 GPU, which is feasible.

• We added this efficiency study in Section O of the revised manuscript.

Thank you for your detailed response. After reading all the replies, I plan to keep my score for acceptance. I appreciate this thoughtful discussion.

We sincerely appreciate the reviewers' constructive comments and positive feedback on our manuscript.

W1. The current design for the scheduling of the selection of frequent and rare compositions of concepts is a bit ad-hoc. You always use frequent composition at the beginning and then start randomly selecting of composition after a fixed point. Based on your theoretical analysis, any additional guidance can be included or used to determine the selection of composition of concepts?

We thank the reviewer for your constructive comments. We acknowledge your concerns but respectfully argue that our scheduling approach is not ad-hoc but carefully designed for two reasons. (1) Leveraging abundant knowledge in LLMs. To ensure careful scheduling, we leverage the strong zero-shot ability of LLMs to extract "visual detail level" required to draw each concept and use it for concept guidance, based on prior observations that diffusion models determine rough visual features during the early sampling steps and detailed visual features in the later steps (as explained in lines 237-240). This LLM-guided scheduling approach provides an appropriate concept guidance schedule across prompts with various semantics. (2) Consistent performance improvement across diverse concept categories. With our careful scheduling, R2F significantly improves the composition performance across diverse concept categories, including property, shape, texture, etc, outperforming fixed scheduling as shown in Figure 9.

W2. From your example, each rare composition has only two concepts. How do you generalize your approach to more complicated and rare composition (3 or more concepts, such as adj. + adj. + noun, e.g., an agent rabbit with a gun in a casual suit )

Thanks for your careful comments. RareBench already includes the complicated rare composition cases (as the 'complex' case), consisting of three or more concepts, and R2F still exhibits superior performance on such complex cases as shown in Table 6. Specifically, looking at Figure 6, there is an example "A horned bearded spotted raccoon smiling" from the complex case, and R2F successfully generates the image that accurately follows the prompt. Technically, given examples such as "adj1 + adj2 + noun", R2F finds a noun that more frequently appears in the context of "adj1 + adj2", and uses it for frequent concept guidance.

W3. Have you tried to use rare components at the beginning and then use frequent instead? The intuitive explanation for using frequent one first is needed. It will be good if you have relevant experimental results.

As per your suggestion, we generated images using rare concepts at the beginning and then used frequent concepts at the last steps instead. The generated images can be found in Figure 17 of Appendix L. Overall, the generated image tends to align more closely with the frequent prompt rather than the original rare prompt. This is because diffusion is a step-wise denoising process where the generated image depends more on the lately used prompt. Therefore, it is advisable to guide the process by using frequent concepts first and rare concepts last.

• We included these results in Section L of the revised manuscript.

W4. In the real world, both rare and frequent composition of concepts are considered in generation. Then, the method that improves the quality of rare compositions should not hurt the quality of frequent composition. Without manual determination, how can your approach still maintain a high generation of frequent composition. In other words, it will be good if you can provide discussion on how your method could be adapted to automatically handle both rare and frequent compositions without manual intervention.

Thanks for your constructive comments. Our R2F framework leverages the LLM to determine whether each decomposed sub-prompt has a rare concept and needs frequent concept guidance without requiring manual intervention. For example, as shown in Figure 4, for the sub-prompt "an awful snake" (denoted as $c^2$), the LLM determines it has no rare concepts, and thus, R2F guidance is not applied for this sub-prompt. As a result, on the T2I-compbench, containing a high proportion of frequent concepts, R2F still shows superior composition performance indicating that it can effectively control the quality of frequent concept composition. Furthermore, with the region-guided R2F+, we can ensure that addressing rare compositions does not compromise the quality of frequent compositions.

W1. The method requires alternating among a set of prompts during the denoising process, which makes multiple-step inference inevitable. Therefore, this design might not work well with current state-of-the-art acceleration methods, which reduce the number of denoising steps to 4 steps or even less.

We thank the reviewer for helping us improve our paper. To address your concern, we conducted additional experiments using FLUX-schnell, one of the state-of-the-art diffusion models that can generate high quality images in 4 steps, as the backbone for R2F with 4 steps of denoising process.

The core idea of R2F is that by exposing frequent concepts to the diffusion sampling process its rare concept composition performance can be significantly enhanced. Prompt alternating is one design choice, which is highly effective (in terms of T2I alignment) and efficient (as only one prompt, either rare or frequent, is used for each step) for long guidance length. However, as you pointed out, the efficacy of the prompt alternating may diminish for short guidance length. In this case, we recommend using the Composable approach (detailed in Section 4.4) which blends rare and frequent concepts within the text embedding as an alternative approach.

Configuration. For the short guidance length of 4 steps, we adjusted the interpolation configuration of the Composable method. For concepts with a visual detail level from 1 to 3, we applied the composable method only for the first step, while for those with a visual detail level from 4 to 5, we applied it to both the first and second steps. In the final third and fourth steps, only the original rare prompt was exposed. We set the blending factor of $\alpha$ to 0.3.

Result. The image generation results are presented in the table below, and the generated images are added in Figure 16 of Appendix K in the revised manuscript. Similar to the original results with a longer guidance length, the T2I alignment performance of FLUX-schnell improved in most cases when R2F was applied. Therefore, the frequent concept exposure idea of R2F can be generalized to the latest acceleration methods with 4 steps, which can have a broader impact on many real-world applications where fast inference time is crucial.

• We included these results in Section K of the revised manuscript.

W2. There is a small gap between the theoretical analysis and the empirical method. For the theoretical analysis, the author studies the scenarios of linear interpolation of scores produced by different prompts. For empirical results, the author performs alternating prompts across different denoising steps.

Thanks for your careful comments. Our theory assumes a score estimator for image sampling, which does not include multi-step denoising. For diffusion models with multi-step denoising, both the linear interpolation (of latents or text embeddings) and alternating prompts can be regarded as a way of interpolation. Table 6 empirically compares the effectiveness of these design choices, and the alternating approach is the most effective in multi-step denoising. Additionally, the alternating approach is the most efficient because it only requires to use either the rare or frequent prompt at each step. Nevertheless, both the linear interpolation and the alternating approach are more effective than the vanilla SD3.0.

• We clarified this explanation in lines 213-215 in Section 3.2 of the revised manuscript.