Generating compositional scenes via Text-to-image RGBA Instance Generation

We thank the reviewer for their comments that increase the quality of our work, and address main concerns below.

Novelty: while we agree with the reviewer that layered-based representation are receiving increasing attention, we argue that our proposed methodology has several unique components which paves the way for multi-layer innovation. Firstly, transparent image generation is an emerging topic, with very few related works dedicated to the task. To tackle this task, we propose a novel modelling strategy with a disentangled latent space and mutual conditioning. In addition, we provide several training insights to achieve better results with this type of generation task. Secondly, we disagree with the reviewer that the level of fine-grained control we are providing can be achieved with contemporary method, as evidenced by our experiments. While common layered methods allow control over location and appearance of objects, they often struggle with patterns and fine-grained instance details, as all components are generated at the same time. We have investigated competing methods with publicly available implementations, and consistently observed difficulties in achieving the right level of details, and/or accurate instance positioning for the type of complex prompts we are showing in our manuscript. We highlight that we are working with substantially more complex image descriptions than e.g. T2I Compbench (reference [16]). In contrast, by generating instances separately, we can control fine-grained attributes and focus on scene coherence at blending time. Lastly, our multi-layer scene composition is also different from pre-existing methodology, due to our pre-generated instances. Here, instead of focusing on generating the right instance prior to a global merger, we focus on instance integration within the scene, through iterative instance addition layer by layer, at each timestep. This allows more natural integration of individual instances in the scene.

Related works: While we focused our evaluation on published work, we agree with the reviewer that LayerDiffusion [1] is highly relevant recent related work, and update our manuscript to add comparisons to this work, both in related works and experiments. We highlight that this work was developed in parallel to ours, and our methodology and handling of transparency is entirely different. Furthermore, we highlight that the multi-layer scene generation process of LayerDiffusion is substantially more restricted, as instance location cannot be specified and is solely controlled by background appearance. We provide quantitative and qualitative comparisons in this rebuttal. Experimental details, and discussion are available in the global rebuttal. LayerDiff [2] is contemporaneous work to ours, and is closer to a multi-layer extension to Text2Layer than our work. We note an important limitation of this work, which is that occluded components are not generated. This makes scene manipulation tasks more challenging, and substantially reduces flexibility. Unfortunately, due to the complex dataset building required and lack of publicly available code, we are not able to provide comparisons to this work, but will integrate it in our related works section.

Experiments: We agree with the reviewer that quantitative experiments would be a great addition. Unfortunately, as discussed in sec 4.2 (Baselines), quantitative evaluation is highly challenging due to the interactive nature of our methodology, as well as the highly complex nature of image prompts we are working with. This would require the development of a novel benchmark, which we leave to future work as quantitative evaluation remains an open problem for controllable generation tasks. We have made efforts to provide many examples highlighting different properties and benefits of our work, and to thoroughly analyse our method’s behaviour in our supplementary experiments. We note that we do provide a non-layered baseline with the PixArt-alpha model (reference [6]) and are happy to additionally provide a Stable Diffusion baseline as well if needed.

Instance interactions: We agree with the reviewer that independent instance generation is a limitation of our work, and an important focus of our future work. Our method in its current state can mitigate these issues with precise instance prompts, and stronger blending freedom at composition time. Highly intricate interactions can be achieved without altering our main methodology, by fine-tuning our RGBA generator with higher quality, fine-grained training data, or making it capable of generating grouped instances.

[1] Zhang, Lvmin, and Maneesh Agrawala. "Transparent image layer diffusion using latent transparency." arXiv preprint arXiv:2402.17113 (2024).

[2] Huang, Runhui, et al. "LayerDiff: Exploring Text-guided Multi-layered Composable Image Synthesis via Layer-Collaborative Diffusion Model." arXiv preprint arXiv:2403.11929 (2024).

We thank the reviewer for their comments that improve the quality of our work. We are grateful for the positive remarks (interesting work, fantastic generation ability, superior controllability), and address the reviewer’s main concerns: comparison to LayerDiffusion and the impact of multi-layer noise blending on realistic generation.

LayerDiffusion: Firstly, we highlight that our work was developed in parallel to LayerDiffusion [1], which is nearly contemporaneous to ours, and our methodological decisions were not inspired from this work. This can be evidenced by our entirely different methodological decision : we explicitly model the transparent channel, and disentangle the latent space; while LayerDiffusion implicitly incorporates transparent information. However, due to the very similar objective of transparent generation, we agree with the reviewer that comparisons are important, and extend our evaluation to include a comparison to LayerDiffusion. Experimental details, results and discussion are available in the global rebuttal.

Besides transparent generation, LayerDiffusion also offers multi-layer generation properties by generating instances conditioned on a background image. We point out here that our blending strategy affords substantially better controllability, as LayerDiffusion does not allow to specify the location of these instances (see Figure 2 of this rebuttal). We will update our manuscript to incorporate discussions on LayerDiffusion in related work, and add experiments provided in this rebuttal.

Multi-layer noise blending can lead to unnatural generation, as evidenced in our supplementary experiments (see Supp. F.2 and Figure 12), when noise injection constraints are too strong. Our method is designed with this challenge in mind, with two important components to ensure natural looking images: 1) iterative integration of instances in the global scene, and 2) imposing constraints in early timesteps only. The latter ensures naturalness of generated scenes by leveraging the intrinsic biases of diffusion models. Imposing constraints only at early timesteps ensures instance locations and key attributes are integrated, while giving enough freedom to the model to generate realistic images. The impact of increasing or decreasing constraints is discussed in more details in our supplementary section F.2. The former component, unique to our multi-layer approach, iteratively integrates new instances in separate layers. Generating separate images allows to smoothly integrate each instance in the scene individually, simplifying the task in comparison to methods that aim to blend all instances in the scene at once.

[1] Zhang, Lvmin, and Maneesh Agrawala. "Transparent image layer diffusion using latent transparency." arXiv preprint arXiv:2402.17113 (2024).

We thank the reviewer for their comments that increase the quality of our work and are encouraged by the positive feedback. As discussed in our limitation section in supp. D, we agree that our more expensive generation is a limitation of our work. However, we expect this cost to be heavily amortised once a critical mass of RGBA assets has been generated. The same principle is also the current practice for 3D digital assets. We further highlight that it is a common limitation of layered representations, where layers require separate generation processes, and consider that the added flexibility and blending quality can outweigh additional generation cost. Nonetheless, we intend to explore more efficient solutions in future works, notably by exploring ways to combine instance generation and blending while maintaining fine-grained control.

We thank the reviewer for highlighting clarity issues in section 3.3. We will update our manuscript to include the following algorithmic description of the section in supplementary materials to facilitate understanding.

Multi-layer scene composition algorithm:

Inputs: $K$ pre-generated RGBA instances $\{I_k,M_k\}$,
A bounding box based scene layout $\mathcal{L}=\{[cx_k, cy_k, w_k, h_k ], \text{for } k \in 1,\cdots,K\}$,
A latent diffusion model (VAE $\mathcal{E}$, diffusion model $\mathcal{D}$)
T generation timesteps
Random noise $\eta \in \mathcal{N}(0,I)$
Number of blending timesteps $n$,
background blending timesteps $b$,
cross-layer consistency timesteps $n_s$
Output: $K+1$ multi-layer images (background, $K$ images with increasing number of instances)

Generation set-up >>Rescale $\{I_k,M_k\}$ according to $[cx_k, cy_k, w_k, h_k ]$
$x^0_k \leftarrow \mathcal{E}(I_k)$
$m_k \leftarrow$ Downsample $M_k$ to latent space dimension
Compute $x^k_t$ at all diffusion timesteps $t \in [1,T]$ using DDIM inversion
Initialise all images with same Gaussian noise: $y^k_T \leftarrow \eta$

Composite scene generation >>For $t=T$ to 0: Loop over timesteps
>>> Predict noise update with diffusion model: $\epsilon_{\theta}(y^k_t,t) \leftarrow \mathcal{D}(y^k_t,t)$
$y^k_{t-1} \leftarrow$ Update according to noise scheduler and $\epsilon_{\theta}(y^k_t,t)$
If $t \geq b$: Background blending optional step >>>> $y^0_{t-1} \leftarrow$ $y^0_{t-1} \cdot m^*+\frac{1}{2} (y^0_{t-1}+ y^K_{t-1})\cdot (1-m^*)$

If $t \geq n$: Scene composition

For $k=1$ to K:

$y^k_{t-1} \leftarrow$ $y_{t-1}^{k-1} \cdot(1-m_k)+x^k_{t-1}\cdot m_k$

Elif $t \geq n+n_s$: cross-layer consistency optional step

For $k=0$ to K:

$y^K_{t-1} \leftarrow y^K_{t-1} \cdot(1-m_k)+y_{t-1}^{k-1} \cdot m_k$

We thank the reviewer for their comments that increase the quality of our work. We are encouraged by the positive comments that our work is useful, our methodology is sound, and our visual results are convincing. We now address the main limitations raised: 1) limited contribution and novelty, and 2) experimental comparisons.

Novelty: we have aimed to design a novel controllable generation method with fine-grained control and interactivity in mind. To this end, we propose to first generate RGBA assets, and then assemble them in a composite scene. Generating images with transparency information is far from trivial (as mentioned by RwSp1) as the concept of transparency is foreign to diffusion models. We introduce novel modelling and training methodology for this purpose. In contrast with LayerDiffusion [1], which injects transparency information in the null space of the VAE to facilitate fine-tuning, we explicitly model transparency through disentanglement of RGB and alpha channels. Explicitly modelling transparency allows for more precise and nuanced generation of RGBA images, and guarantees that we generate images with a transparent background, which is not the case with LayerDiffusion. Text2Layer (reference [55]) is a substantially different approach as well, requiring triplets of background, foreground and saliency maps for training; and predicting all components together. Crucially, this approach doesn’t generate individual instances, but a foreground comprising all instances. Our scene composition leverages these RGBA assets in a novel multi-layer approach. In contrast with existing multi-layer methods which generate layers in parallel before merging them, we build a multi-layer composition process where each layer is dependent on the lower image layer, by integrating new instances in an increasingly complex image one layer at a time. This process allows to smoothly integrate each component individually (rather than trying to blend all instances together at once) and is uniquely afforded by our RGBA asset pre-generation. This further allows scene manipulations very easily, as pre-generated RGBA instances can easily be moved, replaced, and have their appearance frozen.

Evaluation: We have made efforts to compare to highly relevant, widely used, state of the art works, for both RGBA generation and scene composition, at the time of submission. For the former, we reimplemented Text2Layer as the closest published work capable of RGBA generation. We highlight that extension of this work to multiple layers is far from straightforward, as disentanglement of foreground instances through their saliency segmentation approach is non trivial, and is beyond the scope of this work. We agree with the reviewer that nearly contemporaneous work LayerDiffusion is an important comparison, and provide quantitative and qualitative comparisons in this rebuttal. Experimental details, results and discussion are available in the global rebuttal.

Regarding scene composition evaluation, we have made efforts to compare to open-source, widely used state of the art works designed for compositional generation. As discussed in section 4.2 (Baselines), we have compared to layout-based generation method GLIGEN, which in our experiments, more consistently outperformed boxdiff and ReCO (references [47] and [50]) and therefore constituted our best baseline bounding box based method. Similarly, for multi-layer scene generation, we have chosen multidiffusion, due to its strong performance and available implementation. Unfortunately, we were not able to reimplement Layered rendering diffusion model for zero-shot guided image synthesis (LRDiff, reference [36] in our manuscript) for comparison, due to missing implementation details and no public code. We highlight that LayerDiff[2] is contemporaneous work without available opensource code. Due to the expensive data construction and training costs, we are unable to provide comparisons to this work. While our focus is on generating all image components, including occluded areas, LayerDiff only generates visible pixels. This in turns, strongly limit scene manipulation abilities, which is one of the key benefits of our representation. Finally, while inpainting and editing techniques are relevant related work, we highlight that the focus of our work is not editing, but interactive and controllable scene generation. Our scene manipulation experiments highlight intrinsic benefits of using RGBA assets to easily manipulate scene content with strong content preservation. However, we do not aim to directly compete with editing methods, as we have not introduced any explicit editing mechanisms.

[1] Zhang, Lvmin, and Maneesh Agrawala. "Transparent image layer diffusion using latent transparency." arXiv preprint arXiv:2402.17113 (2024).

[2] Huang, Runhui, et al. "LayerDiff: Exploring Text-guided Multi-layered Composable Image Synthesis via Layer-Collaborative Diffusion Model." arXiv preprint arXiv:2403.11929 (2024).