Universal Guidance for Diffusion Models

We thank the reviewers for the effort put into reviewing this paper, and we appreciate the valuable insight on related works.

Comparison with other works

Compared to our algorithm, DPS [R2] attempted to address guided generation using general guidance functions, but they proposed to apply the gradient of the $\ell_{p}$ loss between the ground-truth condition and the output of the guidance function given a predicted clean image. It is unclear how DPS can handle conditions that cannot be compared using $\ell_{p}$ loss, such as bounding box locations, from both method perspectives. On the other hand, our forward guidance using no step-wise refinement can be viewed as a generalization of DPS that accepts any type of condition and guidance function. Please refer to Figure 10 and Section E in the appendix for comparison between whether the guided generation uses step-wise refinement or not. We also remark that, from the method perspective, we make novel technical contributions by proposing backward guidance and a step-wise refinement algorithm to further improve the results. We also empirically demonstrate the success of our algorithm on more complex guidance functions. Please also check our additional comparison with LGD, another recent work on guided image generation, in the general response. The results validate the contribution and novelty of our algorithm, especially on widely-encountered guidance functions in real world.

For comparison with FreeDoM, due to a highly unusual issue, we believe the discussion should not be visible to the public. We communicate the discussion with AC via a private post and please directly contact the AC for the discussion.

Different guidance networks for a specific task

In principle, our algorithm is not restrictive to the guidance function used. So we believe it is totally possible to experiment with different segmentation networks and simply pick the best one for empirical use.

Reference

[R1] "FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model," ICCV 2023.

[R2] Chung, Hyungjin, Jeongsol Kim, Michael T. Mccann, Marc L. Klasky, and Jong Chul Ye. "Diffusion posterior sampling for general noisy inverse problems." arXiv preprint arXiv:2209.14687 (2022).

We thank the reviewer for the effort put into reviewing this paper, and we appreciate the valuable reference the reviewer brings up. While the papers mentioned by the reviewers have explored various ways to perform loss-guided generation on diffusion models, our work exhibits critical differences and novel contributions compared to these prior arts. We discuss the difference individually for each paper mentioned by the reviewer.

Comparison with PGDM

PGDM [Song et al’22] assumes the existence of a pseudo-inverse function given the guidance function, and it is unclear in the paper how their algorithm can be extended to general guidance functions. In contrast, our proposed universal guidance combines forward and backward guidance to achieve guided image generation using any guidance function, where backward guidance is a more general algorithm that reduces to PGDM if the pseudo inverse function is known.

Comparison with DPS

DPS [Chung et al’22] attempted to address guided generation using general guidance functions, but they proposed to apply the gradient of the $\ell_{p}$ loss between the ground-truth condition and the output of the guidance function given a predicted clean image. It is unclear how DPS can handle conditions that cannot be compared using $\ell_{p}$ loss, such as bounding box locations, from both method and empirical perspectives. On the other hand, our forward guidance can be viewed as a generalization that accepts any type of condition and guidance function. We also propose backward guidance and a step-wise refinement algorithm to further improve the results, and demonstrate the generation quality on complex tasks such as segmentation map guidance and object location guidance.

Comparison with [Song et al’23]

The method proposed in [Song et al’23], which we abbreviate as LGD, also addresses the issue of DPS and can handle general guidance function. They further proposed to compute the guidance gradient based on the loss average over different noisy samples of the predicted clean image. Different from their work, we propose a universal refinement algorithm that refines the sampling in each diffusion step by reinjecting suitable Gaussian noise. On top of that, we also propose backward guidance that further enhances the match between the generated images and the condition. Please also check our additional comparison with LGD in the general response, which validates the contribution and novelty of our algorithm, especially on widely-encountered guidance functions empirically.

Summary

In summary, our algorithm offers novel technical contributions that are significantly different from prior works by combining forward guidance, backward guidance and step-wise refinement. Our paper also offers novel empirical contributions by showing the success on more sophisticated and empirically applicable guidance functions such as segmentation and object detection networks. We hope the discussion and the experiment address the reviewer's concern about novelty and contribution, and hope that the reviewer will raise the final rating of the paper.

Reference

[Song et al’22] Song, Jiaming, Arash Vahdat, Morteza Mardani, and Jan Kautz. "Pseudoinverse-guided diffusion models for inverse problems." In International Conference on Learning Representations. 2022.

[Chung et al’22] Chung, Hyungjin, Jeongsol Kim, Michael T. Mccann, Marc L. Klasky, and Jong Chul Ye. "Diffusion posterior sampling for general noisy inverse problems." arXiv preprint arXiv:2209.14687 (2022).

[Song et al’23] Song, Jiaming, Qinsheng Zhang, Hongxu Yin, Morteza Mardani, Ming-Yu Liu, Jan Kautz, Yongxin Chen, and Arash Vahdat. "Loss-Guided Diffusion Models for Plug-and-Play Controllable Generation." (2023).

We thank the reviewer for the effort put into reviewing this paper, and we appreciate the insightful feedback on further ablation studies. Please check the experiments below for each individual question raised.

Ablation between our algorithm and directly replacing fcl and lce with off-the-shelf guidance (Eq. 5).

As suggested by reviewers, we also present 8 more combinations of text and bounding boxes. We summarize the quantitative results in the table below, and present the qualitative results in Figure 1 of the anonymized rebutall figures collection. The value is obtained by averaging over 10 random generations. We note that the "text" in the following table refers to a combination of object locations and text phrases, and please see the figure for details of each combination. Both quantitative and qualitative results demonstrate that using our proposed forward guidance with Eq. 6 leads to a much better match with the external guidance compared to Eq. 5 for off-the-shelf guidance functions.

Segmentation guidance on ImageNet diffusion model

As suggested by the reviewer, we also present the generation results on ImageNet diffusion model using segmentation guidance in Figure 2 and Figure 3 of the anonymized rebuttal figures collection. In particular, Figure 2 shows images generated with various segmentation maps, demonstrating that our algorithm is effective regardless of the underlying diffusion model. Figure 3 includes comparison between generations using ImageNet diffusion model and Stable Diffusion with the same segmentation mask.

Generations using objects other than dog as conditions

In Figure 1 of the anonymized rebuttal figures collection, we show generations with various combinations of dogs and cats in different positions. The results indicate that our algorithm can generate images beyond the "dog category", and we hope this alleviates the reviewer's concern about the diversity of generated images.

We thank the reviewer for the effort put into reviewing the paper and providing valuable feedback.

Would the inaccurate predicted clean image affect the performance?

Indeed, as the reviewer mentioned, we use predicted clean images to compute the guidance signal and perform guided image generation. We argue that inaccurate clean image prediction has limited impact due to the step-by-step nature of image sampling for diffusion model. Specifically, impacts of guidance using gradient signal is averaged out across different sampling step. In the earlier sampling steps, the guidance signal provides coarse adjustment to the image based on comparatively inaccurate clean image predictions. As the sampling step progresses, the clean image prediction becomes more accurate and the corresponding guidance signal further refines the generated image. The success of our algorithm also validates the reasoning.

Discussion of some previous works

We thank the reviewer for providing pointers to these valuable references. Collectively, these reference papers belong to a family of algorithms that freezes the unconditional diffusion backbone and train auxiliary models for controlled image generation. This family of algorithms has a different goal in mind compared to our proposed Universal Guidance, where we aim for an algorithm that does not require re-training at all for different conditional generation tasks. Nevertheless, we believe this family of algorithms is an interesting middle ground between our algorithm and full-blown conditional training, and presents exciting directions for further research. We will include the related discussion upon further revision of the paper.