Adversarial Supervision Makes Layout-to-Image Diffusion Models Thrive

We genuinely thank the reviewer for positively assessing our work and providing constructive feedback. We would like to provide detailed answers in the following to further clarify the remaining concerns:

(W.1) As mentioned in the Failure Cases, when editing the attribute of one object, it could affect the other objects as well. It is claimed to be inherited from Stable Diffusion.

We believe that the attribute/concept binding issue, i.e., editing the attribute of one object could affect the other objects as well, is an inherited problem from Stable Diffusion (SD), as also been reported in prior SD based text-to-image works [1,2,3,4]. Using words to describe detailed changes at specific locations can be lengthy and inaccurate, therefore realizing such edit via text can be difficult. SD still cannot accomplish the task perfectly, either ignoring the text or mapping the text description to different objects. With the further scaling of the denoising UNet, e.g., SDXL [5], such issue might be mitigated but still exists as mentioned in their limitation section. One hypothesis for the binding issue lies in the pretrained text encoders, which tend to compress information into a fixed number of tokens. Employing inference time optimization [2,3] could resolve the problem to a certain extent. Nevertheless, we believe this is an interesting yet open research question.

[1] Feng, Weixi, et al. "Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis." ICLR. 2022.

[2] Chefer, Hila, et al. "Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models." ACM Transactions on Graphics (TOG) 42.4 (2023): 1-10.

[3] Li, Yumeng, et al. "Divide & bind your attention for improved generative semantic nursing." arXiv preprint arXiv:2307.10864 (2023).

[4] Rassin, Royi, et al. "Linguistic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignment." arXiv preprint arXiv:2306.08877 (2023).

[5] Podell, Dustin, et al. "Sdxl: Improving latent diffusion models for high-resolution image synthesis." arXiv preprint arXiv:2307.01952 (2023).

(W.2) Despite thorough ablation tests, the paper does not give any insights on the experiments. The text in the experiments only describes the results instead of analyzing the phenomenon. Suggestion to add more analysis and insights of why it can work.

We thank the reviewer for pointing this out to help us further improve the manuscript quality. We will reiterate some of the insights presented in the Introduction and Method sections, and connect it better with the experiments. To provide a short summary of the important insights: ALDM improves upon the other methods from the training pipeline: the adversarial supervision explicitly leverages the label map condition; and the unrolling strategy can consider more sampling steps during the learning process, bridging the gap between training and the inference time sampling. The combination of both enables the model to well comply with the condition consistently over a time window, leading to the improvement on the alignment in the final result.

(Q.1) When using a frozen segmenter, mIoU is higher and FID is also higher. Why does it happen, any insight?

When using a frozen segmenter, the diversity of the generated images is very limited, as shown in Fig 6. in the Appendix. The generator (i.e., the diffusing UNet) tends to learn a mean mode to cheat the discriminator (a frozen segmenter in this case), leading to the mode collapse issue. Such mean class mode is essentially easier for the segmenter to classify, yielding high mIoU. Since the FID metric assesses both visual quality and diversity, such phenomenon is reflected in the high FID score. We have clarified this better in the revision as well.

We sincerely thank Reviewer d5qa's acknowledging the paper is well-written, the topic is interesting and challenging. We believe there is a misunderstanding on the motivation of our approach, and thus would like to reiterate the motivation, our technical contributions and placement of our approach in the scope of the semantic image synthesis work, both diffusion-based and GAN-based. We believe we point out an important issue in the current diffusion model pipeline and the proposed training strategy has proven its effectiveness on different diffusion model based approaches. We next would like to address individual concerns:

(1) The technical contributions of this manuscript are limited.

To the best of our knowledge, our adversarial supervision with the multistep unrolling strategy has not been proposed before for improving layout-to-image diffusion models. Our approach is indeed inspired by OASIS, which we have adequately acknowledged in the paper, i.e., cited it in multiple sections (introduction, related work, method, etc.) in the main paper. However, incorporating the broad concept of discriminator loss into a completely different type of generative model, i.e., diffusion model, is by no means trivial.

The traditional diffusion model suffers from poor alignment due to the suboptimal training objective without explicit supervision based on the label map, and its sampling characteristics also make consistent adherence to the conditional layout with time difficult. Little attention has been paid to improve the original diffusion reconstruction loss. Therefore, the proposed adversarial supervision and multistep unrolling is a novel training design for diffusion models, to explicitly encourage consistent alignment to the label map over a time window, closing the gap with inference time sampling. We believe our work focuses on the important yet underexplored aspects of diffusion models, i.e., the training strategy, and we demonstrate the effectiveness of our proposal on different recent diffusion model-based methods, e.g., OFT (NeurIPS'23), T2I-Adapter (ArXiv'23-Mar) and SOTA ControlNet (ICCV'23 Best Paper) (see Table 1 and below). Especially, the multistep unrolling is tailored for diffusion models, considering its sampling characteristic, which is significantly different from GANs.

Regarding the computing overhead, as mentioned in the "Implementation details", we only apply the unrolling every 8 optimization steps, such that the additional overhead is manageable. It is important to mention that multistep unrolling is a training strategy that can largely improve the alignment with the label map, coming at no additional cost at inference time, which is more important for most applications.

(2) The results (while improving over the diffusion baselines) are behind OASIS, a GAN from 2020.

Our goal is to improve text-to-image (T2I) diffusion models for semantic image synthesis. We agree with the reviewer that this class of generative models underperforms GANs in terms of alignment, which we have shown in Table 8 in the Appendix. However, there are many benefits of using large-scale T2I diffusion models for this task, such as powerful text controllability. Our ALDM can naturally synthesize novel samples, which can be used for downstream applications such as improved domain generalization (see Table 4 and Fig. 5). In contrast, GAN alone can only synthesize samples resembling the trained domain. We believe our proposed training strategies can inspire the community and can be combined with advancing future diffusion model architecture designs, which are orthogonal to our work, leading to better results in the future.

We sincerely thank the reviewer for the overall positive assessment and provided insightful feedback. We have revised the manuscript based on your constructive comments. In what follows, we provide our response to individual questions.

(Q. 1) Train a diffusion model from scratch for the task of L2I and T2I simultaneously. Recent segmentation models can produce the segmentation maps needed to augment the dataset (e.g. LAION-5B).

We agree that training a diffusion model for L2I and T2I jointly is a promising idea, and the recent segmentation models such as SAM have demonstrated impressive zero-shot performance, and in many cases can provide reliable label maps for training. Unfortunately, in our lab we have very limited computational resources, thus large-scale training, e.g., 256 A100 GPUs with 150000 hours reported for Stable Diffusion, is far beyond our computing capability. In contrast, fine-tuning only requires 2 A100 GPUs, which is affordable for us. As not having enough resources to train Stable Diffusion from scratch is not an uncommon status for many labs in the world, developing fine-tuning techniques is valuable as well. Nevertheless, we thank the reviewer's valuable suggestion, and we have incorporated this into the newly added "Future Work" discussion in Appendix E.

(Q. 2 & W. 1) (W.1) It's unclear if careful tuning of a frozen segmentation network's impact on the total loss wouldn't be competitive to the proposed adversarial approach. (Q. 2) for the case of a frozen pre-trained segmenter: did you train a diffusion model plus an additional loss for the segmentation model?

Our initial assumption was the same, however, we couldn't manage to resolve the issue after tuning the hyperparameters. As shown below, when decreasing the weighting $\lambda$ of the frozen segmenter loss, the mIoU drops heavily while the FID has minor improvement. The training loss is the same as formulated in Eq. (7), which is a combination of diffusion MSE loss and the segmenter loss. The only difference is for our proposed ALDM the segmenter as a discriminator is also updated with the diffusion model instead of being frozen. In the adversarial game, it is important that the discriminator and generator strike a balance to continuously improve each other. While using a frozen segmenter, the discriminator loses the chance to update itself, and the powerful generator (Stable Diffusion UNet) can freely find a cheating path, leading to the mode collapse issue, as observed in GANs [1]. The mean mode yields limited diversity thus high FID, and it's easier for the segmenter to classify the class mean mode thus the high mIoU.

[1] Thanh-Tung, Hoang, and Truyen Tran. "Catastrophic forgetting and mode collapse in GANs." 2020 International Joint Conference on Neural Networks (IJCNN). IEEE, 2020.

We thank the reviewer for the overall positive assessment, especially acknowledging the simplicity and effectiveness of our method. In what follows, we address the individual concerns in detail:

(1) The qualitative results seem to be much better than baseline methods, are they cherry-picked?

We selected representative examples to illustrate the problems in existing methods in the teaser figure, i.e., Fig 1. For other figures, we did not curate the results. We thank the reviewer to help us further improving the manuscript quality, and we have provided more comparisons in the Appendix. Besides, the quality comparison of random samples can also be seen in the extensive quantitative evaluation (e.g., Table 1), in which our method outperforms the other competitors.

(2) What is the difference between Control-Net + Adv Supervision and multi-step rolling in Table 1 compared with ALDM?

There is no difference between Control-Net + Adv Supervision + multi-step rolling and ALDM in other tables. In general, ALDM (adversarial supervision plus multi-step unrolling) can be seen as an improved training pipeline for L2I diffusion models, which can be combined with different adaptation architecture designs as shown in Table 1. In other tables, we use ControlNet as the default model combined with adversarial supervision and multistep unrolling, to represent ALDM. We thank the reviewer for pointing this out, and we have clarified accordingly in the revision (on Page 7 after Table 1).

(3) Can you show some outputs of the discriminator?

We have added some discriminator predictions in Fig. 15. The discriminator takes the input of the decoder, which actually turns the latents into the pixel images, as mentioned in the "Implementation details". From Fig. 15 the discriminator can categorize some regions as "fake" class (in black), meanwhile, it is also fooled by some areas, where the discriminator produces reasonable segmentation predictions matched with ground truth labels. Therefore, the discriminator can provide useful feedback to the generator (diffusion model) that it should produce realistic results meanwhile complying with the given label map.

We genuinely thank the reviewer for the prompt response. We are glad that our answers have improved your confidence that our work is above the acceptance bar.

Please let us know if there are any new concerns we could address to further improve the score during the subsequent discussion. We remain committed to addressing any further questions to ensure the continued improvement of our manuscript.