OmniBooth: Learning Latent Control for Image Synthesis with Multi-modal Instruction

We wish to convey our sincere appreciation to the reviewer for the thorough and insightful feedback provided. Below, we provide responses to the individual comments and questions raised by the reviewer:

(1) Comparison with MIGC++

We express our gratitude to the reviewer for their insightful references. Briefly, MIGC++ enhances the original MIGC by integrating image-level conditions. The MIGC framework utilizes a cross-attention network to facilitate controllable image generation. Based on our understanding, a characteristic of these methods is that, the cross attention is conducted in the global marching between all the image tokens and condition tokens, followed by using the layout mask to refine the attention score. This process has unnecessary computation, and needs to be computed instance-by-instance. In contrast, our method achieves unified multi-instance customization by employing a single Latent Control Signal for all instances, ensuring both computational efficiency and simplicity. We have added the reference of MIGC++ in our revised version.

(2) Counterfactual scenarios

Good suggestion! During rebuttal, we provide counterfactual generation results in Section B and Figure 9,10 of our paper. Please check it out. Our method faithfully generates a skateboard field engulfed in flames which rarely happens in the real world. In summary, these results not only validate the effectiveness of our approach but also underscore its ability to extend beyond the confines of the training data, thereby demonstrating its robust generalizability.

(3) Benefit of edge loss

As we have deleted the checkpoint trained without edge loss, it is difficult for us to provide a visual comparison. Here, we provide a theoretical analysis. In our framework, edge loss will enhance supervision over the edge area of each instance. This enhancement is expected to yield clearer object boundaries compared to models trained without such loss, and to ensure that instance contours are more accurately aligned with the provided masks.

(4) Bounding boxes condition

Our framework can be seamlessly adapted to box conditions by simply converting mask to box. Regarding overlapping instances, it is widely recognized that incorporating additional depth or 3D information can be beneficial. For instance, 3DIS [a] introduces a Depth-Driven Instance Synthesis framework that alleviates the issue of occlusion.

[a] 3DIS: Depth-Driven Decoupled Instance Synthesis for Text-to-Image Generation

Thank you for your reply, my main concern has been resolved

We would like to express our gratitude to the reviewer for their comprehensive feedback. Below, we provide responses to the individual comments and questions raised by the reviewer:

(1) Comparison with InstanceDiff

We first note that InstanceDiff employs a restricted benchmark, it resizing all images to 512x512 before evaluating perceptual task accuracy, which differs from the standard COCO benchmark. To address this issue, we re-ran the benchmark, aligning it with the original COCO metrics without resizing, and evaluated at the original resolution of COCO images, thereby ensuring a more equitable comparison.

Furthermore, we note that perception accuracy does not reflect generation quality. As shown in Figure 3 of our paper, the images generated by InstanceDiff contain human instances with unsatisfactory topology, potentially due to overfitting to the input layout guidance.

(2) Details

1. We utilize the COCO dataset, which comprises 220,000 annotated images.
2. As stated in Line 356 of our paper, for each image reference, we perform random horizontal flips and random brightness adjustments as data augmentation to simulate multi-view conditions. Consequently, the instance-image inputs do not perfectly align with the masks during training. This is intentional, as we aim to learn transferability, ensuring that the input image reference and mask do not need to align, thereby avoiding overfitting.
3. Yes, users are required to provide certain masks for controllable generation. The pipeline can be found in Figure 1 of our paper, where we leverage a white woman and a red bag for personalization.

(3) IP-Adapter results

Thank you for your suggestion. IP-Adapter has achieved a DINO score of 0.667, a CLIP-I score of 0.813, and a CLIP-T score of 0.289. In comparison with IP-Adapter, our method has demonstrated a better improvement in both DINO and CLIP-T scores, although the CLIP-I score is slightly lower. Dreambooth-LoRA is great, but we do not find quantitative results of it.

(4) Reference

The paper "Be Yourself" offers a versatile framework for multi-subject generation. We will add it to our revised version.

(5) OOD scenarios

As suggested by the reviewer, we provide the following results during rebuttal:

1. Non-realistic image styles. We provide counterfactual and OOD scenarios generation results in Section B and Figure 9 of our paper. In Figure 9 top row, we exhibit a non-realistic generation case, emulating the aesthetic of Hogwarts, which never happened in the real world.
2. Dog flying in the sky. We display a surprising result in Figure 10 that a dog is flying high in the sky. We achieve this by altering the background instance (all instances except the dog) to the sky. Our method accurately generates these OOD scenarios of a dog flying in the sky.
3. Rectangular mask for a person. Please check the results we provided in Figure 10 that we use a mask of cat to represent a human. Our method successfully warps the human body to conform to the contour of the mask.

In summary, these results not only validate the effectiveness of our approach but also underscore its ability to extend beyond the confines of the training data, thereby demonstrating its robust generalizability.

(6) Mask distributions

In our supplementary video, we provide results for generation using user-drawn masks. Notably, the distinct disparities in mask distribution do not impede the quality of the generated images. For missing or noisy masks, they have widely existed in the COCO dataset. For instance, all the black regions in Figure 3 of our paper denote areas that are missing. Despite these challenges, our approach effectively completes the missing regions by leveraging global prompts or scene context. Consequently, the distribution issues of masks become less significant in the context of our method's performance.

Thank you for your response. I still have a few comments:

1. I understand that certain metrics are not optimal for capturing the true quality of image generation tasks. Many papers address this issue by conducting user studies to quantitatively assess differences in quality. Since qualitative examples alone can be misleading, such a study could help strengthen readers' confidence in the superiority of your method.

2.2 Thank you for providing additional details. Regarding augmentations: are the horizontal flips applied to both the image and the mask jointly? If so, the object would still perfectly match the mask even after the augmentation.

2.3 My question pertains to Figure 5, where you personalize subjects from the DreamBooth dataset, which does not provide input masks. How do you derive the mask in this case? Additionally, how do you compare to methods like ELITE or IP-Adapter that do not use a mask as input?

5.1 Thank you for the additional example. I was specifically thinking about styles that deviate further from photorealistic images, such as sketches, comic books, or animations. Would your method be able to generate such styles (as SDXL can), or is this a limitation of your approach?

5.3 Thank you for sharing the additional example, this is indeed an interesting result.

I tend to keep my score and recommend the paper for acceptance.

We would like to thank the reviewer for their positive and detailed feedback. Below, we provide responses to the individual comments and questions raised by the reviewer:

(1) Box condition or mask condition

From our technical university, our framework can be seamlessly applied to box control. We opt for mask conditioning as it allows users to freely manipulate the object's contour, thereby providing finer controllability. In contrast, box conditioning is more straightforward but offers only coarse geometry guidance, resulting in the random generation of object shapes. Both box conditioning and mask conditioning are applicable within our framework, which can be freely adapted based on downstream requirements.

(2) Regarding reference net in Animate anyone

We thank the reviewer for the valuable reference. In "Animate Anyone," the reference net utilizes a concatenation approach to fuse the condition and noisy latent. Essentially, our framework can be seamlessly adapted to the reference net in "Animate Anyone." We can concatenate our Latent Control Signal with the noisy latent along the width dimension, then facilitate interaction between the condition and noisy latent through self-attention. One limitation of this approach is the quadratic increase in computational complexity, which is equivalent to doubling the resolution of the generated image, potentially posing challenges for high-resolution image generation.

We would like to thank the reviewer for the insightful and detailed feedback. Below, we reply to individual comments and questions raised by the reviewer:

(1) The reliance on foundation model

At first, it is believed that latent diffusion, ControlNet, and DINO are widely utilized foundational models or frameworks in the community. Numerous research works are based on them. As they contain powerful pretrained prior in computer vision, it is beneficial and essential to leverage these foundational models to attain enhanced performance and controllability. These would hardly be achieved if we were training from scratch.

Furthermore, it is non-trivial to achieve a unified image generation with multi-modal instruction. Our proposed Latent Control Signal represents a novel generalization of spatial control from the RGB-image domain to the latent domain. It effectively and comprehensively extracts features from both DINO image prior and CLIP text prior, a capability that was previously unattainable. Finally, our method enables highly expressive instance-level customization with intriguing results that have achieved consensus among the reviewers.

(2) Ablate ratio of global embedding

We appreciate the author's suggestion. In our paper, we have already conducted the experiment and provided the results in Appendix Section E. Overall, we found that there is an optimal mixing ratio between spatial embedding and global embedding.

(3) Complexity

As a unified framework that enables controllable image synthesis with multi-modal instruction, our method eliminates the need for a separate training stage in a multi-task manner. Moreover, our training strategy is universal and can be applied across a variety of tasks and models.

(4) Relevant work of UniControl and Uni-ControlNet

We concur with the reviewer's observation that there exists a body of pertinent literature in the realm of unified and multi-condition controllable image generation. However, in the context of UniControl and Uni-ControlNet, these frameworks are constrained as they come to leveraging multiple spatial-layout images as conditioning inputs. They can be difficult to achieve instance-level customization through the use of instance prompts or image references. From this perspective, our OmniBooth model attains a higher degree of controllability. Nonetheless, we will add these references in our revised version.

(5) Up and down arrows

Thank you for your valuable advice! All metrics in Table 2 are such that higher values are indicative of better performance.

(6) Regarding Resolution

As a characteristic of controllable generation with layout control, the smaller the object, the greater the difficulty in generation. In the context of our framework, we exclude instances with fewer than 50 pixels during training, as these types of objects are exceedingly challenging to learn and can impede the model's performance. Notably, despite this exclusion, our framework demonstrates the ability to generate high-quality images with precise control of small objects, as exemplified by the depiction of a white woman and a red bag in Figure 1.