Control and Realism: Best of Both Worlds in Layout-to-Image without Training

We would like to sincerely thank Reviewer Reviewer YGgi for the thorough and constructive feedback on our manuscript. We are more than happy that the reviewer finds our problem definition is reasonabl, theoretical analysis illustrative, and superior performance. We would like to address the concerns as below.

W1. Non-local attention prior.

A1. We acknowledge that for certain object categories—such as coconut trees (As discussed in lines 80-84) and cars —the center of mass may deviate from the geometric center of the bounding box due to their asymmetric or elongated structures, making strict centralization suboptimal. To address this, we adopt a decaying schedule for weight $\rho$ of non-local attention prior (Lines 225–229), which applies stronger centralizing guidance during early denoising and gradually relaxes it as generation progresses. This design allows the model to first stabilize attention and then adapt to the object’s natural shape and pose. As shown in Fig.4, our method successfully generates well-aligned objects like cars and astronauts without rigidly adhering to box center. Moreover, our ablation in Fig.6 shows that maintaining a high prior weight throughout can degrade results, confirming the effectiveness of our decay strategy. In summary, the non-local attention prior functions as a soft, adaptive constraint, enabling both precise spatial control and natural object generation across diverse categories.

W2. Ablation of Nash-MTL.

A2. Adaptive update is a key component of WinWinLay, designed to balance layout controllability and visual quality with minimal computational overhead. As shown in Tab.2, our ablation study demonstrates that incorporating this strategy consistently improves both spatial fidelity and semantic alignment. Moreover, Fig.7 shows that it reduces the number of update steps by dynamically adjusting optimization directions, leading to faster convergence. We further report actual inference times in Tab.3, confirming that adaptive update significantly improves generation efficiency.

W3. Example in Fig.2.

A3. Adequate object coverage within the bounding box is crucial for precise and reliable user control. In our evaluation, metrics like AP depend on the IoU between generated objects and target boxes—if an object covers only part of the box, IoU drops, leading to lower AP. This not only affects quantitative results but also indicates a mismatch with user-defined spatial intent, particularly in scenes with dense object layouts requiring fine-grained control.

W4. Compared with R&B.

A4. To clarify the distinctions and advantages of WinWinLay, we provide a focused analysis of the two core components of R&B: Region-aware Loss (RAL) and Boundary-aware Loss (BAL).

(1) RAL leverages the IoU between the predicted box $\hat{\mathcal{B}}$ and the ground-truth box $\mathcal{B}$ to guide object placement. However, this mechanism introduces two critical limitations:

• The non-differentiability of bounding box operations makes gradient-based updates less reliable and harder to tune.
• Similar to traditional attention energy functions, RAL does not mitigate local bias, leading to incomplete or inaccurate coverage within the target region.

(2) BAL employs a Sobel filter to enhance attention contrast at object boundaries within the box, with the goal of improving object realism. However, it too adopts an energy function-based formulation, and as such, suffers from similar drawbacks:

• Due to the non-differentiability of IoU, BAL reduces to a heuristic energy-based form, which continues to suffer from local bias and incomplete box filling.
• BAL does not account for the underlying latent distribution $z_{t}$, and therefore cannot effectively navigate the trade-off between layout adherence and image fidelity.

Together, these differences underscore the principled, efficient, and robust nature of WinWinLay in addressing the limitations that R&B is inherently constrained by. We have now included this discussion in revised manuscript.

W5. More baselines and metrics.

A5. Please see A1 for Reviewer hxyx.

W6. Details of user study.

A6. To ensure clarity and reproducibility, we conducted the user study on Wenjuanxing, a platform similar to Amazon Mechanical Turk. 150 participants evaluated 50 image pairs, yielding 7500 responses per study. For each pair, users answered two questions:

• Which image better matches the bounding box layout?
• Which image has higher visual quality?

Images were shown side-by-side with layout prompts, and both question order and image positions were randomized to avoid bias. These details have been included in revised manuscript for completeness.

Thanks again for the insightful review. We are happy to discuss any aspects of the manuscript that may require further clarification.

We would like to sincerely thank Reviewer QbsF for the thorough and constructive feedback on our manuscript. We are more than happy that the reviewer finds our paper is clearly structured, comprehensive and insightful analysis, and superior performance. We would like to address the concerns as below.

W1. Other base model.

A1. In line with prior works, we adopt SD1.5 as the default base model in our experiments to ensure a fair and direct comparison with existing training-free Layout-to-Image methods. This choice is consistent with the evaluation protocols followed by representative baselines such as BoxDiff, Layout-Control, and CSG. To further demonstrate the generality and scalability of our approach, we also extend WinWinLay to more recent and powerful SDXL model. As shown in the teaser figure and in additional visualizations provided in the Appendix, our method consistently delivers superior performance under the SDXL setting as well—achieving strong controllability and realism.

W2. Small objects.

A2. In fact, both the COCO2014 and Flickr30K datasets used in our experiments naturally contain a significant number of small object instances, such as mouse, coffee cups, and similar items. Therefore, the quantitative results already reported in Tab.1 and Tab.2 implicitly reflect the model’s effectiveness in handling small bounding boxes. To more directly address the reviewer’s concern, we have conducted an additional focused evaluation specifically targeting small-object scenarios. In this experiment, we filtered the datasets to retain only those bounding boxes that occupy no more than 1/8 of the total image area, following a standard definition of small object regions. As expected, all methods exhibit some performance degradation under these more challenging conditions, due to the intrinsic difficulty of generating fine-grained details in small spatial regions. However, our method consistently outperforms all baselines across multiple metrics in this setting as well, demonstrating its robustness, precise spatial control, and superior fine-detail generation capabilities. Specifically, we observe that when the bounding boxes of multiple objects exhibit unnatural relationships—for instance, when a cat and an avocado are assigned bounding boxes of the same size—most existing algorithms suffer from significant performance degradation. In contrast, our method consistently maintains strong performance under such conditions. Visual results are visible on this link.

Thanks again for the insightful review. We are happy to discuss any aspects of the manuscript that may require further clarification.

We would like to sincerely thank Reviewer hxyx for the thorough and constructive feedback on our manuscript. We are more than happy that the reviewer finds our paper is well-structured, theoretical analysis illustrative, and exhaustive ablation studies. We would like to address the concerns as below.

W1. More Baselines.

A1. As our proposed WinWinLay framework operates in a training-free paradigm for Layout-to-Image generation, all methods selected for comparison in our initial experiments,e.g., BoxDiff and CSG, were likewise training-free approaches. In contrast, methods such as [1] and [2] are training-based, requiring supervised learning on paired layout-image data. These methods operate under fundamentally different assumptions and objectives, making a direct comparison less meaningful within the scope of our setting.

Regarding [3], we acknowledge its relevance. Due to strict page limitations, we initially prioritized comparisons with widely recognized baselines and the most recent SOTA method, CSG (ECCV 2024). Here, to address the concern and to offer the more comprehensive evaluation, we have now included additional experiments comparing our method with [3] and [4], both of which follow training-free paradigms more aligned with our setting.

The extended results demonstrate that WinWinLay consistently outperforms these methods across multiple evaluation metrics, highlighting its superior controllability and visual realism. Visual results are visible on this link. These findings further substantiate the robustness and effectiveness of our proposed method.

[1] Instancediffusion: Instance-level control for image generation. CVPR2024.

[2] Migc: Multi-instance generation controller for text-to-image synthesis. CVPR2024.

[3] Grounded text-to-image synthesis with attention refocusing. CVPR2024.

[4] R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation. ICLR2024.

[5] Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation. NeurIPS2023

[6] ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation. NeurIPS2023

W2. Multiple instances.

A2. We would like to clarify that our method is capable of handling textual prompts and layouts involving multiple instances of the same object. This can be achieved by jointly mapping quantifiers and object names (e.g., "two" + "apples") to multiple bounding boxes, and then enforcing center-based generation constraints within each individual bounding box. Visual results are visible on this link and have been added in the revised version.

W3. Typos.

A3. We have carefully checked the grammar of the entire manuscript and made corrections.

Thanks again for the insightful review. We are happy to discuss any aspects of the manuscript that may require further clarification.

We would like to sincerely thank Reviewer z1Kv for the thorough and constructive feedback on our manuscript. We are more than happy that the reviewer finds our writing well-organized, theoretical analysis illustrative, and quantitative improvement promising. We would like to address the concerns as below.

W1. Object in the center region.

A1. We would like to clarify that generating objects near the center of the bounding box represents a "positive bias", which is a deliberate design choice intended to enhance generation stability and user controllability. Specifically, placing objects at the center of the bounding box offers finer-grained and more predictable spatial control. In contrast, generating objects near the periphery of the box can introduce several undesirable effects:

• Objects placed near the boundaries are more susceptible to spatial overflow, often leading to leakage beyond the given region;

• Non-central placements inherently introduce positional ambiguity, making it difficult to predict or control which subregion of the box will be emphasized during generation.

Such ambiguity increases uncertainty during optimization and often results in instability or inconsistent outcomes. By encouraging central placement within the bounding box, our method achieves a more robust trade-off between layout adherence and generation reliability. This design also enables users to maintain precise control, especially when dealing with dense or complex scenes. To avoid confusion, we have provided additional clarification in both the Method and Limitation sections of the revised manuscript. We emphasize that this center-focused behavior arises from the prior formulated in Equ.13, where the centrality bias is smoothly modulated by the hyperparameter $\rho$ and decayed over generation. This mechanism is not rigid; it merely guides the model toward stable, interpretable spatial alignment without strictly constraining the center of the object to remain fixed at the center (As discussed in lines 80-84).

W2. Adaptive update rule.

A2. While we acknowledge the use of adaptive updates in prior works such as "Taming Transformers for High-Resolution Image Synthesis", we would like to emphasize that adaptive update rule proposed in our method is conceptually and technically distinct, as elaborated below:

• Theoretical Foundation: The adaptive strategies employed in prior works are largely heuristic, lacking a principled formulation. In contrast, our approach is grounded in multi-task optimization theory. Specifically, we formulate the adaptive update as a Nash Bargaining problem between two competing objectives—layout controllability and visual fidelity—leading to a theoretically justified and analytically solvable solution. This formulation offers a deeper understanding of the trade-off and provides a rigorous basis for gradient balancing.

• Gradient Computation: To mitigate computational burden, prior works typically approximate gradients by restricting updates to the final layer or a limited subset of parameters. Our method, however, naturally yields full gradients for each task without introducing additional overhead, as both objectives are defined over the same latent variables in the denoising process. This makes the adaptive update mechanism not only theoretically sound but also computationally efficient and seamlessly compatible with our training-free setting.

Taken together, while the high-level motivation is shared, the underlying methodology, implementation, and theoretical framing of our adaptive update rule differ substantially. We have further clarified this distinction in the revised manuscript.

W3. Other Limitaion.

A3. Compared to Text-to-Image methods, our Layout-to-Image approach typically requires more time—for instance, SD1.5 takes 3.17s, while WinWinLay takes 15.01s. Nevertheless, our method remains more efficient than the current SOTA method, CSG, which takes 19.88s. Hence, improving the generation efficiency of our model will be an important focus in future work.

W4. Use of Vspace.

A4. We have reorganized the content layout in the revised version to ensure improved visual clarity.

Thanks again for the insightful review. We are happy to discuss any aspects of the manuscript that may require further clarification.