LayerFusion: Harmonized Multi-Layer Text-to-Image Generation with Generative Priors

We thank the reviewers for their constructive criticism and acknowledging the novelty in the attention-blending scheme introduced. Please find our responses to the mentioned weaknesses and questions below:

W1: Correction on Notation Used

We appreciate the reviewer pointing out the typographical error in the notation for the attention probability map. The correct notation should indeed be $m \in \mathbb{R}^{M \times M}$, representing a matrix with dimensions M×MM \times MM×M. We correct this error in the uploaded rebuttal.

W2: Evaluation

In our quantitative evaluation (Section 4.2), we focus on preserving the distributions of the foreground and background generation models. To complement this, we conduct a perceptual study to evaluate the harmonization quality of the blended images. Regarding the suggestion to use image segmentation models for evaluation, we acknowledge the potential of segmentation-based methods in assessing foreground-background integration. However, current segmentation models face challenges in accurately extracting RGBA images with their corresponding alpha channels, particularly for generated content. We discuss these limitations in Figure 6, highlighting that such methods may experience difficulties in extracting RGBA content.

W3: Evaluation Benchmark

Since our task involves generating RGBA foregrounds, RGB backgrounds, and their harmonious blending, our evaluation is conducted on generated image triplets. Section 4.2 (Foreground & Background Quality - claims highlighted in yellow in the uploaded rebuttal) specifies that these triplets are derived from our foreground and background generation models.

W4: Problem Addressed in the Paper

While our method addresses challenges in image blending, it also provides valuable insights into the internal representations learned by diffusion models. Specifically, we introduce:

• A novel way of interpreting self-attention maps through a sparsity score for structure extraction.
• The first demonstration of attention-level blending for visual tasks such as image blending.

These contributions extend beyond traditional blending tasks by offering new perspectives on the interpretability and utilization of diffusion models. We believe this aligns with ICLR’s focus on advancing machine learning methods and their applications. Furthermore, the ICLR community has a history of recognizing work on vision tasks that explore creative generation and interpretability, as evidenced by prior publications ([3], [4], [5], [6]).

[3] IMPUS: Image Morphing with Perceptually Uniform Sampling Using DIffusion Models, https://openreview.net/pdf?id=gG38EBe2S8, ICLR 2024

[4] DiffEdit: Diffusion-Based Semantic Image Editing with Mask Guidance, https://openreview.net/pdf?id=3lge0p5o-M-, ICLR 2023

[5] Diffusion in Diffusion: Cyclic One-Way Diffusion For Text-Vision-Conditioned Generation, https://openreview.net/pdf?id=ePOjNlOjLC, ICLR 2024

[6] Generating Images with 3D Annotations Using Diffusion Models, https://openreview.net/pdf?id=XlkN11Xj6J, ICLR 2024

Q1: Definition of "Harmonization" and Its Importance

Harmonization, in the context of our work, refers to the seamless integration of the foreground and background layers such that the final blended image appears cohesive and visually natural. This is particularly important for applications in creative workflows, where spatial editing, object movement, and compositional adjustments rely on well-integrated layers. To clarify the effectiveness of harmonized blending, we provide supplementary results in Fig. 21, that highlights various aspects of harmonization such as shadowing, lighting and grounding.

Dear Reviewer 878i,

Since the deadline for author discussions are approaching, we kindly ask if you may provide any feedback or additional questions you have. We will do our best to address these issues before the rebuttal period closes.

Thank you,

Paper 1337 authors

Dear Reviewer 878i,

Thank you for your contributions to the reviewing process until today. As a final reminder, please consider revising your final score given our responses and the revised manuscript regarding the concerns stated by your review.

We hope to receive your response, as we are now approaching to the end of the review period.

Thanks,

Paper 1337 authors

We thank the reviewers for acknowledging the novelty introduced and constructive criticism. Please find our responses corresponding to the concerns raised:

W1: Dependency on Foreground Structure

We appreciate the reviewer highlighting the limitations in harmonization quality in certain examples (e.g., Figure 4). As noted, our method relies on a baseline model for RGBA image generation. This baseline tends to favor generating large, centered objects, which can limit its ability to produce fully harmonized outputs in all scenarios.

Despite this, our method demonstrates improvements in generating backgrounds that physically align with the foreground in terms of spatial positioning and contextual coherence. For example, the snow effect in Figure 4 showcases meaningful interactions between the foreground and background. We acknowledge that using a baseline model introduces some limitations, which we discuss in section A.2 included in the supplementary material. For ease of reference, we highlight the mentioned discussion in yellow in the rebuttal uploaded.

In addition, to better demonstrate the harmonization capabilities of our approach, we provide additional results involving shadows, lighting and grounding in Fig. 21 in the revised manuscript (highlighted in yellow). Here, we additionally provide comparisons with alpha blending, which highlights the advantages of our attention-level blending approach.

Q1: Trained Parameters

Our proposed method is training-free with respect to both the RGBA generation model (LayerDiffuse) and the RGB generation model (SDXL). However, the RGBA baseline model does require fine-tuning of the attention layer weights using a LoRA-based approach to adapt it for foreground generation. Note that, we use the pre-trained weights of [2], and do not perform any additional training.

Q2: Comparison to Methods Like PCT-Net

Thank you for suggesting a potential extension of our work. Our method focuses on a training-free generation pipeline, adapting it to methods like PCT-Net, which involve training for consistent tone generation across layers is beyond the scope of this work.

[2] Transparent Image Layer Diffusion using Latent Transparency, https://arxiv.org/abs/2402.17113, SIGGRAPH 2024

Dear Reviewer MWZY,

Since the deadline for author discussions are approaching, we kindly ask if you may provide any feedback or additional questions you have. We will do our best to address these issues before the rebuttal period closes.

Thank you,

Paper 1337 authors

Dear Reviewer MWZY,

Thank you for your contributions to the reviewing process until today. We would like to make a final reminder on the possibility of revising your score, given the concerns and questions addressed in the comments we provide and in the revised manuscript.

We hope to receive your revision on your review, as we are now approaching to the end of the review period.

Thanks,

Paper 1337 Authors

We thank the reviewers for acknowledging the novelty introduced in our method. Please find our responses on the raised concerns below:

W1: Comparison with Generation Using Composite Prompts

Thank you for raising this point. While it is indeed possible to generate composite images using a single prompt, our method offers significant advantages by also generating an alpha channel alongside the foreground image. This provides users with flexibility in spatial editing tasks such as moving or resizing objects, which would otherwise be difficult with a directly generated composite image.

Furthermore, the separation of foreground and background layers makes our approach a potential data generation method for tasks such as image inpainting and object removal, where having background and foreground layers are highly beneficial.

W2: Contributions

We understand the concern about the perceived novelty of using attention maps for blending, as this has been explored in prior work. However, our method introduces two key innovations:

• Structure Prior from Self-Attention Maps: We propose a novel interpretation of self-attention maps as structure priors, which is detailed in Section 3.2 (highlighted in yellow) and Algorithm 1. This provides additional guidance during blending, improving spatial coherence.
• Attention-Level Blending Mechanism: To the best of our knowledge, our method is the first to perform blending at the attention level. Unlike methods such as Blended Latent Diffusion, which rely on latent variables for blending, our approach utilizes attention outputs, enabling finer control over the blending process.

These contributions expand the potential of attention mechanisms in generative tasks, which we address in our key contributions listed at the introduction section. For the ease of visibility, we highlight these contributions in yellow, in the introduction section.

W3: Evaluation

As the task of image triplet generation with RGBA foreground and RGB background is relatively new, our method serves as a first approach to enable inference-time blending for such images. While this limits direct comparisons, we have included:

• Results from other RGBA generation-based approaches (Table 1).
• Comparisons with harmonization methods to demonstrate the effectiveness of our blending strategy (Figure 7 - highlighted in yellow). In addition, we conducted a user study to address the limitations of quantitative metrics in evaluating perceptual quality. The results of this perceptual study indicate that our method outperforms the baseline approach requiring additional training for harmonization tasks.

Thanks to the author's reply, although it is said that on this task, this method is the first to perform blending at the attention level, in other AIGC fields, similar operations such as attention level replacement, panning, deflation, weighted fusion, etc., I encountered many times at the review of manuscripts from 2023 to now.

To be honest, the quality of this work is good, but still not good enough for a field like AIGC. I suggest authors try simpler conferences or journals. Or try to explore the value of applying this work in other fields, such as applying it to traditional high-level, low-level tasks or dataset construction, etc., to show its value in serving other tasks, and then submit it to the top conferences.

Dear Reviewer jFaU,

Since the deadline for author discussions are approaching, we kindly ask if you may provide any feedback or additional questions you have. We will do our best to address these issues before the rebuttal period closes.

Thank you,

Paper 1337 authors

Thanks for your suggestions regarding possible improvements on our work. While acknowledging the use of attention features on other applications in AIGC field, we refer to the operation of performing harmonization on attention level as our novelty, instead of utilization of attention features overall.

In addition, our method serves as the first approach to utilize the attention outputs as a tentative alpha channel estimate. Note that the baseline model [1] does not provide any alpha channel representation during the generation process, but outputs the alpha channel from the final foreground image. As our harmonization approach aims to output a foreground and background that are compatible with each other, such an alpha channel prediction is insufficient where intermediate blending steps are necessary for interactions between layers. To resolve this, we propose using generative priors from the attention layers to get a coarse alpha channel prediction in the form of a mask (please refer to Sec. 3.3 and Alg. 3), to be able to perform the blending. This use case of the attention maps also serves as a novelty of our method, compared to other applications in the field of AIGC.

Regarding the novelty introduced with the attention scheme we use for blending, we will clarify the introduced novelty with attention blending in the final version of our paper.

[1] Transparent Image Layer Diffusion using Latent Transparency, https://arxiv.org/abs/2402.17113, SIGGRAPH 2024

We thank the reviewers for acknowledging the contributions of our work and your constructive feedback. Please find our responses to the mentioned concerns below:

W1: Using a Pretrained Model

Thank you for pointing this out. In our work, we focus on leveraging the capabilities of existing pretrained models to extract meaningful information for the blending task. This approach eliminates the need for collecting RGBA foreground data paired with RGB backgrounds, which is an expensive and resource-intensive process. By relying on pretrained models, we can bypass these challenges while demonstrating the flexibility and potential of our framework. Please see section A.2 in the supplementary material for discussions regarding the limitations caused by the pretrained model. We highlight this discussion with the color yellow, for the ease of visibility in the rebuttal uploaded.

W2: Lack of Quantitative Metrics and Evaluation

We acknowledge the importance of quantitative metrics in evaluating the blending quality. Since the task of image generation involving RGBA foregrounds and RGB backgrounds is relatively new, there is limited prior work for direct comparison. However, we have included comparisons with methods capable of performing this task, as well as harmonization models, which address overlapping but distinct challenges. In addition, to address the gap regarding the lack of quantitative metrics available for the task, we also conduct a user study as a perceptual evaluation, which shows the effectiveness of our blending approach.

Dear Reviewer jYPb,

Since the deadline for author discussions are approaching, we kindly ask if you may provide any feedback or additional questions you have. We will do our best to address these issues before the rebuttal period closes.

Thank you,

Paper 1337 authors

Thank you for your insightful feedback, please find the answers related to the adresses concerns below:

W1: Unstable Interaction

We appreciate the reviewer highlighting the effect of the mask threshold. To clarify, the boundary coefficient is used primarily to ensure object completeness rather than to control stylistic or lighting changes. The main effect of the mask boundary coefficient (denoted as d in the calculations) is to ensure complete coverage of the foreground during blending, as demonstrated in the ablations provided in Figure 8(d) (explanations highlighted with yellow in the uploaded rebuttal).

Rather than applying a constant threshold, we propose an adaptive masking strategy based on the distribution of changing mask values across iterations, allowing for a soft decision boundary. This method ensures flexibility and robustness during masking. Regarding the boundary coefficient, we determine the initial boundary coefficient value by integrating the masking strategy proposed by [1]. The algorithm for mask construction is detailed in Section 3.3 and elaborated further in Algorithm 3 of the supplementary material.

W2: Multi-Characters

Thank you for pointing out the importance of multi-character generation. As the baseline model used for foreground generation is biased towards generating a single, centered object, our work has focused on single-foreground scenarios to evaluate the effectiveness of our approach. However, we note that the generated foreground is in RGBA format, which allows for manual construction of scenes involving multiple objects.

We address such limitations in the supplementary material, precisely in section A.2.

W3: Further Interaction

We acknowledge that scenes involving physical interactions and occlusions are a limitation of our current approach. Our method relies on attention outputs to extract a tentative alpha channel estimate, which primarily provides positional information about the generated object. Overcoming this limitation would require baseline models capable of explicitly modeling physical interactions, as our LayerDiffuse framework focuses on generating objects in isolation. We discuss such limitations that occur due to the baseline model used in section A.2 of the supplementary material.

For ease of reference, we highlight the discussions provided in section A.2 in yellow.

[1] Diffusion Self-Guidance for Controllable Image Generation, https://arxiv.org/abs/2306.00986, NeurIPS 2023

Dear Reviewer FJ9h,

Since the deadline for author discussions are approaching, we kindly ask if you may provide any feedback or additional questions you have. We will do our best to address these issues before the rebuttal period closes.

Thank you,

Paper 1337 authors