Compose and Conquer: Diffusion-Based 3D Depth Aware Composable Image Synthesis

We thank you for acknowledging the contributions of our paper, mainly:

• Our proposed DDT is the first method to leverage image triplets to distill the relative depth positions of objects in a diffusion model.
• Our proposed soft guidance is a novel method to localize global semantics.
• Our experiments demonstrate the model's capabilities of relative placement of objects and its effectiveness in preventing concept bleeding.

[W1, Q1, Q2] Correct me if I have misunderstood. I'm confused about the details of the soft guidance. I understand that the work wants to leverage the mask map to selectively impose the foreground embedding and background embedding for the cross-attention mechanism. However, I'm uncertain as to whether this process should take place during the computation of similarity or subsequent to it. I would appreciate it if the author could elucidate the specific dimensions and computational details related to both the cross-attention mechanism and the soft guidance technique, to enhance reader comprehension. Please refer to the questions for additional context on my confusion.

1. In my understanding, the size of $S$ is $i\times j$, where $i$ is the number of queries, and $j$ is the number of keys; the size of $W_K \cdot y_{full}$ is $j \times C$ and the size of $W_Q \cdot z_t$ is $i \times C$. However, I am puzzled as to why $j$ needs to be greater than $2N$.
2. Additionally, I am uncertain about whether the mask should be applied along the dimension of $C$. It perplexes me that the mask is utilized on the calculated similarity rather than during the actual computation of similarity.

[W1] Firstly, we would like to address your questions on soft guidance and why the mask is utilized on the calculated similarity rather than during the actual computation of similarity. Assuming “during the computation of similarity” means before $S$ is rescaled with softmax and multiplied with $V$ (eg. $S'=(QK^T/\sqrt{d})\otimes M'$), and “calculated similarity” means after S is rescaled with softmax and multiplied with $V$ (eg. $S'=\operatorname{softmax}(QK^T/\sqrt{d})\otimes M'$), we would like to clarify that soft guidance is indeed applied during the computation of similarity, or before the softmax operation. It seems that this confusion stems from us omitting the softmax operation in the paper, where we state that “…the attention operation $S' \cdot V$ is completed.”, and whereas we intended $S$ to be understood as the raw similarity scores, readers would assume that $S$ means the scaled similarity score (or attention score, depending on sources). We thank you for your detailed insight on this matter. We’ve updated the equations to be clear that soft guidance is indeed applied during the similarity computation, and the reshaped, flattened, repeated boolean mask $\varphi(M) \in \{0,1\}^{i \times N}$ to $\varphi(M) \in \mathbb{B}^{i \times N}$ to avoid confusion.

[W1] Secondly, we would like to illuminate the specific dimensions used in the soft guidance technique. We bring up an exact example of how soft guidance is applied in the first cross-attention layer latents pass through, when generating an image of size $512 \times 512$. In our implementation, we use 8 attention heads, and set $N=4$, or the number of tokens $\mathbf{y}\_\text{fg}$, $\mathbf{y}\_\text{bg}$ are projected to. Since we use the CLIP text encoder, the given text prompt is encoded into $77$ tokens with a dimension of $768$. By $\mathbf{y}\_\text{full} = \operatorname{concat}(\mathbf{y}\_\text{text}, \lambda\_\text{fg}\mathbf{y}\_\text{fg},\lambda\_\text{bg}\mathbf{y}\_\text{bg})$, $\mathbf{y}\_\text{full}$ becomes a tensor of shape [1, 85, 768]. The latent $\mathbf{z}\_t$ is a tensor of shape [1, 4096, 320] because it has been processed by the subsequent self attention layer. $\mathbf{z}\_t$ and $\mathbf{y}\_\text{full}$ are then projected into $Q$, $K$, $V$, with respective dimensions of Q=[8, 4096, 40] , K=[8, 85, 40], and V=[8, 85, 40]. The raw similarity scores are calculated by $S=QK^T/\sqrt{d}$, resulting in S=[8, 4096, 85]. We then reshape, flatten, repeat the mask $M$ used for soft guidance (with an initial dimension of [1, 512, 512], to a shape of fg_mask=[1, 4096, 4], which serves as the mask for the foreground token part of $\mathbf{y}\_\text{full}$. We then take the inverse of the mask, bg_mask=[1, 4096, 4], as the mask for the background token part of $\mathbf{y}\_\text{full}$. We then create a base mask of only ones base_mask=[1, 4096, 77], and concat the three masks to create $M'$ of shape [1, 4096, 85]. Then the operation of $S\otimes M'$ is conducted across all attention heads (eg. masking), to get $S'$. Then, the rest of the attention operation, $\operatorname{softmax}(S')\cdot V$ is performed.

We thank you for acknowledging the main contributions of our work, stating that:

• Our method poses as a feasible solution in the challenging problem of compositional image generation.
• Our novel training scheme may be of interest to the community in a broader context.
• Our experiments demonstrate superior qualitative and quantitative results.

[W1] Calling the model "depth-aware" is misleading. I would rather say it learns to compose two spatial layouts and generate a coherent image. Using the teaser figure as an example, the cat can appear either in front of or behind the cake given the same depth maps, and similarly, the castle can either occlude or be occluded by the mountain. In other words, the exact ordering of objects is not induced by the depth maps. This phenomenon is likely because the depth produced by MiDaS is scale and shift invariant (i.e., it is not metric depth, and background can appear closer than foreground).

[W1] We agree on your insight stating that the exact ordering of objects are not induced by the depth maps. The ability to order objects into their successive foreground/background stems from the proposed depth disentanglement training (DDT), where information about the otherwise occluded background is retrieved via our image triplets and distilled into our local fuser. The depth maps produced by MiDaS are indeed scale/shift invariant, but they do hold attractive properties that lead us to choosing them as our local condition, instead of other representations (eg. segmentation maps). Because depth maps are single channel representation of an object's shape and holds the relative depth representation of an object in the scope of the image, they hold an inductive bias that we sought attractive to leverage. To summarize, the "depth-aware" property of our model doesn't stem from the depth maps themselves (although we do leverage its inductive biases), but its ability to process each depth map in their respective foreground/background stream in our local fuser and place them accordingly.

[W2] Since all that matters is generating a coherent image, I would imagine that other types of spatial conditions (e.g., segmentation masks for both foreground and background) can work equally well if used for training.

[W2] We agree that different spatial conditions can work equally well during training. We have added an additional experiment that elucidates this case in our revised paper, section [A.4] Ablation Study, of the appendix. We explore the use of canny edges instead of depth maps, due to the fact that canny edges hold very different inductive biases to that of depth maps. Canny edges are inherently binary, and they hold much more fine grained detail about an object while trading off the ability to represent the relative depth of a pixel. The added figure 13 of page 18 showcases the results, where we find that our model generalizes well to canny edges. As mentioned above, this ability to generalize well stems from our depth disentanglement training, where information about the background is distilled to the model, even if the representation used doesn't hold inductive biases about the relative depth of an object. However, the inductive biases of each representation does come into play when generating images. In figure 13 (a) and (c), depth maps generate images that capture the sense of depth present in the conditioned images, while canny edges result in the generated images often looking flat. The opposite holds, for in figure 13 (b) and (d), it can be seen that canny edges generate better fidelity samples to that of depth maps when conditioned on flat, illustration based images. This leads to the conclusion that while DDT is successful in distilling information of what object to place in front of and cover, our model leverages the inductive biases of the representations used. If the representation inherently holds information about the relative depth of an image, DDT also distills this information into our model, resulting in the ability to compose images in a depth aware fashion.

We thank you for acknowledging the strengths of our paper in that:

• The methods we propose, DDT and soft guidance are novel and effective in enhancing the capabilities of diffusion models.
• The evaluations are comprehensive and quantitative results showcase better results than existing SOTA methods.

[W1-1] Type of conditions. Is the model capable of applying different types of conditions? [W1-2] Is the model capable of applying two different conditions simultaneously while recognizing the 3D relations?

[W1-1] Yes, our model indeed is capable of applying different types of conditions. We have added a section in our revised paper, section [A.4] Ablation Study of the appendix. We explore the usage of canny edges as an alternative to depth maps, due to the fact that while canny edges are spatial representations of an image, it trades off the ability to represent depth with more fine grained details. We include qualitative samples comparing generated images from canny edges and depth maps, in Figure 13. It can be seen that our model generalizes well to canny edges, a trait that stems from our depth disentanglement training where information about the salient object, and what's behind said object is distilled into our local fuser. However, canny edges hold inherently different inductive biases to that of depth maps. This can be seen playing a role in the generated samples, where canny edges yield better results for "flat" images (such as graphics), but often fail in capturing a sense of depth. The opposite holds for depth maps, so we find that special cases might call for variants of our model trained on different conditions. We have also included a quantitative evaluation in the same section, where we compare the canny edge variant of our model with other baseline models, and find the results to be comparable. [W1-2] Because our model generalizes well to different conditions, we find no reason for our model to fail in generalizing different conditions for the foreground and background. While due to limitations in our computational resources at the moment, we plan to explore this possibility, starting with canny edges for the foreground and depth maps for the background. This initial choice comes from our newly found insights while experimenting with the canny edge variant of our model, where we find that canny edges provide more control over the intricate details of an object. We hypothesize that leveraging depth maps as background representations with canny edges as foreground representations might give us the best of both worlds; fine grained control over the salient object while additionally providing a sense of depth.

[W2] Image triplets. There is no visualization of the prepared image triplets for training. I am curious regarding the quality of foreground image, background image and foreground mask. It's also especially important to analyze the inpainted background image and how it would negatively affect the training and final outcomes.

[W2] We kindly refer the reviewer to section [A.2] of the appendix of our paper, where we detail the process and provide visualizations of preparing the image triplets. We agree that the quality of the synthetic image triplets does influence the quality of the outcomes, especially the background image. Figure 9 of section [A.2], Details on CnC, of our appendix elucidates this factor about the image triplets, where we initially try out [1]LaMa, a widely utilized inpainting model. We show that the depth maps of the background images extracted by LaMa holds artifacts that don't provide information of what is behind the salient object, whereas background images extracted by SD don't. For depth disentanglement training to be fully effective, the background image has to distill information that is initially occluded by the salient object. If the quality of the background image were to be on par with the samples extracted by LaMa as in figure 9 (i.e. doesn't provide any information about the initially occluded background), it would hold the same effect of training our model with the same depth maps, as shown in figure 3 of our paper.

We also find the process of binary dilation to be crucial in inpainting with SD, because the masks $M$ extracted by the Salient Object Detection model we utilize are usually pixel-perfect. SD's inpainting module tends to leave certain edge artifacts when the masks just barely cover the targeted object, so we additionally use binary dilation to expand the mask. These extra steps while utilizing a much computationally demanding module (SD over LaMA) are taken to ensure the quality of our image triplets, for without it would negatively effect the quality of final outcomes, as you've pointed out.

[1]Suvorov, Roman, et al. "Resolution-robust large mask inpainting with fourier convolutions." Proceedings of the IEEE/CVF winter conference on applications of computer vision. 2022.

I appreciate the authors' detailed responses, which effectively addressed most of my preliminary concerns. Hence, I would like to raise my rating. The authors may want to carefully revise the manuscript and incorporate the materials in the rebuttal.