Rethinking The Training And Evaluation of Rich-Context Layout-to-Image Generation

We thank the reviewer for his/her review. We use W to denote bullets in weaknesses

Answer to W1: In Rebuttal Section A, we compare the throughput of L2I methods using SD1.5 and SDXL baselines. It is noteworthy that the overall throughput of our method is not significantly hampered. In a typical scenario with 5 objects, the throughput of our method is more than 60% of the throughput of the original backbone model.

Answer to W2 and W3: We argue that conditioning the model on rich-context descriptions requires both a rich-context dataset and a designated conditioning module for complex descriptions. Without a rich-context dataset, the generalizability of the conditioning module from word/phrase-level context to rich-context descriptions can be hindered. Conversely, without a proper conditioning module, the model may perform poorly when handling complex descriptions.

To validate the effectiveness of both the rich-context dataset and the regional cross-attention module, we conducted two additional ablation studies: 1) We retained the regional cross-attention module but replaced the rich-context descriptions with word/phrase-level descriptions obtained using the Recognize Anything Model and GroundingDINO (Ln 213). 2) We replaced the regional cross-attention module with the self-attention modules used in GLIGEN and InstDiff, training them with the rich-context dataset.

The results, presented in Rebuttal Sec D, show that performance drops either when using word/phrase-level descriptions or when removing the regional cross-attention module. This validates the importance of both the rich-context dataset and the regional cross-attention module.

Answer to W4: Our proposed two metrics, although they resemble extensions of existing metrics, are appropriately repurposed to offer significant value for the L2I problem as 1) ​​These metrics are specifically designed to quantitatively measure L2I performance in rich-context scenarios, where existing metrics have failed to do so (Ln 176-183). 2) In addition to proposing these metrics, our reliability analysis is crucial. It demonstrates the conditions under which these metrics are effective and consistent with human perception. We argue that this analysis should be considered a contribution to the field of evaluation metrics.

Answer to W5: In Rebuttal Sec E, we provide a visual comparison to demonstrate the effectiveness of using region reorganization compared to straightforward feature averaging (the baseline described in Ln 279) when dealing with overlapping objects. The qualitative results validate the importance of our proposed methods, as they produce more accurate generated objects that better align with the designated layouts.

For the ablation study on box indicator and image resolution, identifying a failure generation without one of these components can be deliberated, and there may not be a strong sample-wise effect when applying them. Therefore, we recommend referring to the quantitative performance metrics for a more accurate comparison.

We thank the reviewer for his/her review. We use W and Q to denote bullets in weaknesses and questions

Answer to W1: Our proposed two metrics may not seem exceptionally special, however, we are the first to repurpose these metrics to evaluate L2I performance. They offer significant value for L2I problem as 1) ​​these metrics are specifically designed to quantitatively measure L2I performance in rich-context scenarios, where existing metrics have failed to do so (Ln 176-183); 2) in addition to proposing these metrics, our reliability analysis is crucial. It demonstrates the conditions under which these metrics are effective and consistent with human perception. We argue that this analysis should also be considered as a contribution to the field of evaluation metrics.

Answer to W2: In Rebuttal Sec E, we provide a visual comparison to demonstrate the effectiveness of using region reorganization compared to straightforward feature averaging (the baseline described in Ln 279) when dealing with overlapping objects. The qualitative results validate the importance of our proposed methods, as they produce more accurate generated objects that better align with the designated layouts.

Our objective with the four properties outlined in Sec 3.2 is to address the rich-context L2I challenges detailed in Sec 3.1 (Ln 98-110). Specifically, when “Flexibility” is satisfied, the rich-context description can be accurately understood by the model (Ln 99), the “Locality” ensures the objects are positioned correctly within the designated layout box (Ln 103) and “Completeness” guarantees the global consistency in the generated images (Ln 103-104). Finally, “Collectiveness” allows the model to consider and properly represent the interaction of overlapping objects (Ln 109-110).

In practical terms, positioning objects accurately (Locality) and maintaining plausible image quality (Completeness) are fundamental prerequisites for L2I problems. By satisfying the Flexibility, our model can better understand descriptions and generate more satisfactory objects (Figure 1, 5). Additionally, a model that meets Collectiveness can more effectively handle interactions between overlapping objects (Rebuttal Sec E).

We will clarify and elaborate on this information in the revised paper to ensure it is more comprehensible.

Answer to W3: As mentioned in Appendix Section B, our model is trained with an accumulated batch size of 256 for 100K iterations, which is half of the current state-of-the-art (SoTA) L2I method InstDiff, trained with a batch size of 512 for 100K iterations. Additionally, the number of training samples is 2M (after filtering failed downloads in CC3M), which is also half of the number of samples used in InstDiff, which has 5M samples. Overall, the training cost of our work is less than half of the existing L2I methods. Therefore, we believe the performance improvement is not due to larger computational resources or data, but rather from our design. We will include this comparison in the revised appendix.

Answer to W4 and Q: We provide pseudo-code for the proposed two evaluation metrics in Rebuttal Sec C. The CLIP model used is the clip-vit-base-patch32, trained by OpenAI, and the SAM model is the ViT-H checkpoint from the official SAM implementation.

Answer to W5: For reproducibility, our implementation will be released upon the completion of the submission. In practical terms, we implement region reorganization in the dataloader to minimize computation overhead . This ensures that the reorganized region-text correspondence is shared across all regional cross-attention layers. Consequently, the regional cross-attention layer in the denoising model is reduced to a classical cross-attention operation with a special attention mask that indicates the permissible attention region between visual and textual tokens. Essentially, the attention operation depicted in Figure 2 is conducted by passing a pre-computed attention mask when cross-attending visual and textual tokens. We will enhance the clarity of these details in the revised paper.

We thank the reviewer for his/her review. We use W and Q to denote bullets in weaknesses and questions

Answer to W1, Q2: Our solution, like GLIGEN and InstDiff, requires inserting additional parameters into the pre-trained model and enhancing L2I ability through training. Indeed, our module is not plug-and-play as it is not training-free. However, it is compatible with existing pre-trained large models. Specifically, our cross-attention module is inserted into the original diffusion model right after each self-attention layer (Ln 204), similar to GLIGEN. This allows our model to be applied to all modern pre-trained diffusion models with similar structures, such as the Stable Diffusion family, DALLE family, DeepFloyd, GLIDE, etc.

For reproducibility, our implementation will be released upon the completion of the submission. In practical terms, we implement region reorganization in the dataloader to minimize computation overhead . This ensures that the reorganized region-text correspondence is shared across all regional cross-attention layers. Consequently, the regional cross-attention layer in the denoising model is reduced to a classical cross-attention operation with a special attention mask that indicates the permissible attention region between visual and textual tokens. Essentially, the attention operation depicted in Figure 2 is conducted by passing a pre-computed attention mask when cross-attending visual and textual tokens. We will enhance the clarity of these details in the revised paper.

Answer to W2.W5,Q3: Per the reviewer’s suggestion, we evaluated the generation diversity using LPIPS and Inception Scores as proposed in [A], with results provided in Rebuttal Sec B. Using the same model backbone (SD1.5) as previous works, our method can generate images with greater diversity. We also observed that the diversity of the SDXL-based model is less than that of the SD1.5-based model, which we attribute to the inherent differences between the backbones.

Furthermore, while [B] addresses mask-based layouts, our work focuses on box-based layouts. Unlike mask-based layouts, box-based layouts lack dense annotation for every pixel and involve overlapping regions, making it infeasible to directly apply cross-attention-based techniques as in [B]. In the revised paper, we will elaborate on related works and clearly differentiate our approach from previous methods.

Answer to W3: Our method is designed for rich-context L2I, but it does not fall short on simpler description datasets like COCO. Our analysis in Figure 6a indicates that when the description is simple, our performance is on par with existing methods. However, as the description complexity increases, our advantage becomes more pronounced.

Answer to W4, Q1: We provide pseudo-code for the proposed two evaluation metrics in Rebuttal Sec C. The CLIP model used is the clip-vit-base-patch32, trained by OpenAI, and the SAM model is the ViT-H checkpoint from the official SAM implementation.

Above information will be updated in the revised version paper.

[A] LayoutDiffusion

[B] Freestyle Layout-to-Image Synthesis

1. The proposed model can be easily applied to all modern pre-trained diffusion models with similar structures. And the implementation will be released upon the completion of the submission. It is strongly recommended to combine with popular architecture such as diffusers.
2. The rebuttal addressed my main concern about the possible decrease on diversity.

I increased my score from 5:borderline accpet to 6:weak accept.

We thank the reviewer for his/her review. We use W and Q to denote bullets in weaknesses and questions

Answer to W1: The backbone of the current SoTA L2I method, InstDiff, is based on SD1.5. Our proposed method has been validated using both SDXL and SD1.5 backbones (Ln 201). Therefore, our comparisons against baselines using the SD1.5 backbone are fair and justified.

While the performance results in Table 1 appear close, it is crucial to consider the following points: 1) Table 1 illustrates the model’s performance across various description complexities, including easy, medium, and hard. However, our model excels when handling complex object descriptions (see Figure 6a), which shows that as the complexity and length of the descriptions increase, our model’s performance advantage over the baselines becomes more pronounced. 2) As noted in lines 290-295, the evaluation configuration in Table 1 does not fully leverage our model’s generation capabilities. Our model’s performance can be further enhanced with higher resolution. We provide the combined results on RC CC3M of Table 1 and Figure 6b in the following table. By examining our model’s performance at higher resolution (Figure 6b), it is clear that our improvements over the baselines are noteworthy.

Answer to W2: Our method is not intended to be a, as the reviewer suggested, “complete” L2I solution, and there is potential for future extensions to tackle broader L2I problems. Our objective with the four properties outlined in Sec 3.2 is to address the rich-context L2I challenges detailed in Sec 3.1 (Ln 98-110).

Specifically, when “Flexibility” is satisfied, the rich-context description can be accurately understood by the model (Ln 99), the “Locality” ensures the objects are positioned correctly within the designated layout box (Ln 103) and “Completeness” guarantees the global consistency in the generated images (Ln 103-104). Finally, “Collectiveness” allows the model to consider and properly represent the interaction of overlapping objects (Ln 109-110).

In practical terms, positioning objects accurately (Locality) and maintaining plausible image quality (Completeness) are fundamental prerequisites for L2I problems. By satisfying the Flexibility, our model can better understand descriptions and generate more satisfactory objects (Figure 1, 5). Additionally, a model that meets Collectiveness can more effectively handle interactions between overlapping objects (Rebuttal Sec E).

We will clarify and elaborate on this information in the revised paper to ensure it is more comprehensible.

Answer to W3: As noted in lines 203-204, our method is a training-based approach and is not training-free. We insert the proposed regional cross-attention layers after each self-attention layer in the original diffusion model. Consequently, even if applying the cross-attention strategy mentioned in Ln 125-127, the model still requires training because the newly inserted parameters are randomly initialized.

Answer to Q1: “Rich-context” can be considered as an extension of the “open-set” concept. While both “open-set” and “rich-context” scenarios deal with an unlimited number of object classes, the descriptions in the rich-context setting are notably more diverse, complex, and lengthy (Ln 95-97).

We mention “impossible to list all classes” as the limitation of using the closed-set detectors for evaluation(see Ln 176-180). In contrast, the limitation of open-set object detectors is that they are designed to handle inputs at the word or phrase level, but not sentence-level that are required in the rich-context setting (refer Ln 182-183).