RealCompo: Balancing Realism and Compositionality Improves Text-to-Image Diffusion Models

We sincerely thank you for your time and efforts in reviewing our paper, and your valuable feekback. We are glad to see that the proposed method can be generalized to different models, achieve SOTA generation results, and expand to various application scenarios in a training-free manner. Please see below for our responses to your comments.

Q1: Why propose another method for balancing realism and composability? T2I models differ in realism due to training variations, and control-based models enhance controllability while retaining original realism.

A1: Thank you for your comment. As you mentioned, control-based models enhance controllability while retaining the realism of original T2I models. However, our extensive experiments show that increasing the strength of condition control significantly reduces detail richness and overall aesthetics. Fig1 in our paper and Figs3 and 4 in supplementary PDF illustrate this decline of realism with increased control strength ($\beta$ increase). Additionally, Fig1 and 2 in supplementary PDF demonstrate that even with larger, more powerful backbones, increasing control strength degrades detail and aesthetic quality. Thus, control-based models must sacrifice realism to improve compositionality. The imbalance between realism and compositionality in control-based models is a critical and unresolved issue.

It's correct that different pre-trained T2I models produce varying levels of realism due to different training strategies and data. Hence, by using different stylized T2I models, our method easily achieves stylized compositional generation. As shown in Fig7 in our paper, our method maintains high-quality style preservation with different stylized T2I models.

Q2：How can you ensure that combining T2I and L2I generates real diagrams when T2I images are mostly fake?

A2: One important criterion for determining whether an image appears real or fake is the reasonableness of its composition, specifically whether object placement aligns with real-world physical scenarios. Fig5 of supplementary PDF shows examples where T2I-generated images fail in this regard. For instance, in Fig5(a), the teapot generated by T2I model is visibly suspended in the air, defying physical laws. Our method uses layout constraints to ensure objects are generated within reasonable bounds, maintaining both aesthetic quality and compositional reasonableness. Similarly, the image generated by T2I model in Fig5(b) shows a red chair unnaturally placed on a table, and in Fig5(c) depicts two people too close to each other. These examples indicate that while T2I model excels in aesthetics, it lacks compositional reasonableness. Our method uses LLM to generate conditions that comply with physical laws, guiding the model to generate images with high compositional and aesthetic quality. Thus, our method generates more realistic images compared to T2I models.

Q3: In Fig1 and paragraphs 2-4, the authors present their motivation solely through GLIGEN, which is inappropriate.

A3: We are grateful for your feedback and apologize for our oversight. In supplementary PDF, we provided additional experiments to validate our motivation. Fig3 shows a more apparent trend of changes in image realism using GLIGEN. Fig4 uses InstanceDiffusion to further validate this, with a parameter $\beta$ to control the strength of layout guidance. As $\beta$ increases, the realism of generated images decreases, evidenced by loss of details, decline in aesthetic quality, and unrealistic content. For example, the dog's facial and body details degrade, the cat's eyes differ in size, and the bird has abnormally thin legs.

Additionally, we set the parameter $t_0$ to control the number of steps for layout control during generation. Specifically, when denoising steps are fewer than $t_0$, layout is used for control, and when the steps exceed $t_0$, only text is used. Similar to GLIGEN, when $t_0$ exceeds 20, the generated images of InstanceDiffusion show minimal differences, indicating that layout control's influence is mainly in the early denoising stages. Even if layout guidance is discarded in the later stages of denoising and only text is used, it is still challenging to recover the realism of the images.

Fig1 and 2 in supplementary PDF show our analysis of the aesthetic quality of images generated by GLIGEN and ControlNet at different sizes. The figures demonstrate that as conditional control strength increases, image aesthetic quality declines at any size. Thank you again for your feedback. We will expand on this part of the experiments in the manuscript.

Q4: Existing methods may optimize $z_t$, while this paper optimizes two coefficients. Thus, the contribution is insufficient.

A4: Here we would like to emphasize two points that distinguish us from previous work and add to the significance of our findings in this paper, we provide detailed clarifications in the General Response.

First, we are the first to discover the imbalance between realism and compositionality in generated images, providing experimental analysis and conclusions.

Second, we offer a new perspective on optimization-based generation methods, achieving satisfactory results and generalization using coefficient updates for the first time.

We will clarify these two points further in our manuscript.

Q5: Fig1 lacks detailed explanation.

A5: Thank you for your feedback, and we apologize for any confusion caused. Details about Fig1 can be found in A3. We will update our manuscript to provide more comprehensive and accessible explanations regarding Fig1.

Q6: Sub-sec 3.3 contains limited valuable information since difference between Eq9 and Eq6 is how to obtain a mask/spatial condition.

A6: Thank you for your suggestion. Sub-sec 3.3 is an extended part of our paper, aimed at explaining the generalizability of spatial-aware conditions. We will condense and refine this section in future revisions.

Hello, Reviewer ZHfz, Regarding your concerns, we have provided some responses that you might find useful, including:

• We have highlighted and summarized the innovations and contributions of our method from two perspectives: innovation in task establishment and model design.

• Additional experimental results, incuding using different models to validate our motivation and compare our method with T2I models in terms of the visual authenticity of the images.

• We provide a detailed explanation of Figure 1 in the paper and the motivation of our method.

We sincerely hope our responses address some of your concerns. If you have any further questions, please feel free to ask. Thank you.

We sincerely thank you for your time and efforts in reviewing our paper, and your valuable feekback. We are glad to see that the proposed method is straightforward and flexible, the whole paper is well written, the idea is interesting and promising, and the experiments are extensive. Please see below for our responses to your comments.

Q1: Since the proposed method runs multiple models concurrently and requires gradient-based updates, a discussion on computational efficiency is crucial for its practical application.

A1: Thank you for your suggestion. We compare our method with other T2I models and spatial-aware diffusion models regarding inference time, VRAM usage and complex performance in the table below:

We observed that our method achieves better compositional generation results compared to other training-free approaches, with only marginal increases in inference time and VRAM usage. Our method can be flexibly applied to any T2I and spatial-aware diffusion models without the need for training.

Q2：Fig1 doesn't illustrate the trade-off between realism and compositionality well. I don't see an apparent drop of realism/aesthetics as $\beta$ increases.

A2: We are grateful for your feedback and apologize for any potential confusion caused. We have provided two more intuitive examples in supplementary PDF. As shown in Fig3, we conducted experiments using GLIGEN. We observed that as the layout control increased (with a higher $\beta$) or the number of layout control steps increased (with a higher $t_0$), the realism of the generated images declined. There is a noticeable degradation in both detail richness and aesthetic quality. For instance, the legs of the teddy bear appear unrealistic, as if it is facing backward with strange distortions, and the overall details of the rabbit become blurred and unappealing.

Similarly, as shown in Fig4, we performed experiments using InstanceDiffusion, where we also define a parameter $\beta$ to control the strength of the layout control. It is evident that there is significant quality degradation in the dog's facial and body details. Additionally, the cat's eyes are different sizes, and the bird's legs are abnormally thin, indicating reduced realism in the generated images under the influence of layout control. This suggests that achieving a balance between realism and compositionality in generated images is generally unattainable.

Thank you again for your feedback. We will update the manuscript to make Fig1 clearer and more intuitive.

Q3: Why does RealCompo improve image realism (Table 2, Fig6 left) and style preservation (Fig6 right) beyond the T2I model's capabilities? This is unclear, as the loss function (Eq6) aims only to enforce spatial constraints.

A3: First, we provide a detailed explanation of the concept of realism. Realism refers to the fidelity of details, aesthetic quality, and positional reasonableness of an image. Here, positional reasonableness is different from compositionality. Compositionality refers to the number of objects, spatial relationships, and attribute bindings in the generated images, while positional reasonableness is an important metric for assessing the image realism, specifically whether the placement of each object in the image is reasonable. When the details and aesthetic quality of an image are similar, positional reasonableness becomes a key factor in user selection. In Fig5 of the supplementary PDF, we provide examples from the user study, which demonstrates the advantages of RealCompo over the T2I model in realism. As shown in Fig5 (a), T2I model generates a teapot that is visibly suspended in the air, which doesn't conform to the physical laws of real-world scenes. In contrast, RealCompo generates objects within reasonable bounds through layout constraints, ensuring both the aesthetic quality and positional reasonableness. In Fig5 (b), the red chair generated by the T2I model is unnaturally placed on top of the table, and in Fig5 (c), two people generated by the T2I model are too close to each other. These examples illustrate that although T2I model outperforms in detail and aesthetics, its positional reasonableness needs improvement. Our method utilizes LLM to generate conditions that comply with physical laws, guiding the model to generate images with both high positional reasonableness and aesthetic quality. Therefore, under similar detail and aesthetic quality, RealCompo's more reasonable and realistic composition gives it an advantage over the T2I model in terms of realism.

Second, the design of our loss function (Eq6) is crucial. As discussed in Sec 4.3 and illustrated in Fig8 of our paper, simply weighting the predicted noise from T2I and L2I models disrupts the object positions controlled by the L2I model, despite enhancing the detail and aesthetic quality of the generated images. This disruption occurs because T2I model lacks conditional controlled, leading to arbitrary positioning that interferes with layout control and results in uncontrollable compositionality. Therefore, our loss function is essential to control the T2I model, ensuring it enhances image realism without compromising layout control.

Q4: What is the purpose of initializing the coefficients as random values (Eq. 1)? This seems unnecessary and introduces random bias towards pixels.

A4: Thank you for your comment. Our fundamental goal is to ensure the initialization coefficients of the two models are the same, so it is indeed unnecessary to initialize them with random values. This is not the focus of our method, we will update and optimize this part in our paper.

We sincerely thank you for your time and efforts in reviewing our paper, and your valuable feekback. We are glad to see that the proposed method is flexible, the experimental results are promising, and the explanation of the important concepts is reasonable. Please see below for our responses to your comments.

Q1: The paper uses a balancer to balance two diffusion models, which is very similar to the concept of "checkpoint merge" or "model ensemble". It is necessary to clarify the differences between these two approaches.

A1: Thank you for the comment. We here clarify the differences between our method and checkpoints merge and model ensemble.

Checkpoint merge aims to combine parameters from multiple models; however, this method presents a problem in compositional generation. Checkpoint merge is typically used to blend the realism or stylistic features of two models, requires the model to possess a high level of spatial awareness and precise localization capabilities. Simply merging the parameters of two models, such as SD and GLIGEN, may compromise the strength of layout control, leading to a decline in compositionality. Our method merges the predicted noise from both models, as each contains unique strengths in either rich semantic information or spatial features. This fusion method more effectively leverages the strengths of both models.

Model ensemble, on the other hand, involves a straightforward weighted combination of the output noises from two models. We have discussed the limitations of this approach in our paper. As seen in Section 4.3 and Figure 8, although simply weighting the predicted noise from T2I and L2I models can result in a generated image with higher realism, it disturbs the object positions of the L2I model. The lack of conditioning in the T2I model leads to arbitrary object placements, causing conflicts with the layout's object positions and resulting in uncontrollable composition. Thus, model ensemble has limitations in compositional generation, and dynamic balancer is a crucial bridge for balancing the predicted noise from the models.

Therefore, in compositional generation tasks, both checkpoint merge and model ensemble methods have their own shortcomings when it comes to achieving a trade-off between realism and compositionality.

Q2: Please explain the necessity of the proposed method. If condition-guided models like ControlNet use a stronger backbone, higher quality, larger quantity, and more diverse data and annotations during training, is it still necessary to balance with non-condition-guided text-to-image models?

A2: Even though condition-guided models use more powerful backbones with larger sizes and higher performance, the challenge of achieving a balance between realism and compositionality in generated images still exists. Therefore, our approach is necessary and widely needed in the field of controllable generation. In the supplementary PDF, we provide additional experimental results to validate the necessity of our method. As shown in Figure 1, we conducted experiments using layout-based models (GLIGEN) of different sizes, varying the parameter $\beta$ to control the strength of condition guidance. It is evident that the aesthetic quality of the generated results significantly decreases as the condition guidance strength increases, even with a larger and stronger backbone. This indicates that improving compositionality comes at the expense of realism. Similarly, in Figure 2, we conducted experiments using keypoint-based models (ControlNet) of different sizes, varying the parameter control_scale to control the strength of condition guidance. Again, it is apparent that larger and stronger backbones, trained with more and better data, exhibit superior realism under the same conditions. However, as the strength of the condition control increases, a notable decline in realism is observed. Therefore, the trade-off between realism and compositionality is a common issue.

1. It's recommended to plot the current dynamic coefficient curve.
2. Is there exisits a general coefficient curve that is suitable for most of the prompt? If it is possible, there's no need for an optimization-based method that requires gradient backward, which will be mush easier for real application.
3. It's recommended to show the results of the fixed coefficient (the best selected) instead of dynamic coefficient.