Manifold Preserving Guided Diffusion

Your feedback for our work is greatly appreciated. In the following, we will address your questions and concerns in detail.

1. Broader Applications and Repetitive Qualitative Showcase (Weakness 1): Thank you for your suggestion. To further demonstrate the applicability of our method, we have added another experiment where we use a pre-trained CLIP model and text prompt to guide human face generation with pixel-space CelebA-HQ model. In Section E.1 and Figure 9 in the appendix, we show examples of samples guided by the text prompt and our method, in comparison with unconditional DDIM samples of the same random seed. Our method is able to create images that follow the text description provided while maintaining high fidelity. We have also updated our manuscript to include more qualitative examples for all experiments.

2. Better Results for Style Guidance Stable Diffusion Generation (Weakness 2): Based on Reviewer Yt4g’s suggestion, we were able to further improve the performance of our method on style guidance Stable Diffusion generation, and we have included new qualitative showcases in Figures 20 and 21 and quantitative results in Tables 2 and 4 in our paper.

3. User Study (Weakness 3): Thank you for your suggestion. We have conducted a user study for the style-guided Stable Diffusion generation task on Amazon Mechanical Turk to compare our method (MPGD-LDM) and two baselines (FreeDoM and LGD-MC). Details about the experiment are described in the appendix.

   The results are shown in Table 5 in the appendix as well as below. Users’ responses regarding Style and Text generally align with the style score and CLIP score in Table 2. In terms of overall user preference, our method outperformed both methods, which suggests that our method finds a better sweet spot balancing the text prompt condition and the style guidance.

4. Typos (Weakness 4): Thank you for pointing out the typos. We have corrected these errors in the revised manuscript.

Thank you for your thorough and insightful review. As the rebuttal period is coming to its end, we are reaching out to see if you have any additional questions or concerns that we can address. Thank you and looking forward to your response.

3. Comparison with DDNM (Weakness 3): Thank you for your suggestion. We have added the comparison experiment with DDNM on the FFHQ super-resolution task in Section E.2 in our paper. But before we dive into the discussion about the experiment, we would like to emphasize that DDNM is designed for only solving linear inverse problems, and it requires direct access to the operation matrix, its pseudo-inverse/SVD and the noise scale for the measurement, which are not parts of the assumptions that we have in our problem setting. As a result, DDNM is not applicable to the general setting of paper. Nevertheless, we also think it is valuable to better position our paper in the literature of linear inverse problem solving, and therefore we conducted the experiments described below.

   To make a fair comparison, we use the simplified version of DDNM with no time traveling and the same unconditional diffusion model pre-trained by the authors of DPS, which is a smaller model compared to the one DDNM used in their paper. Due to code availability, we use average pooling as the interpolation method which is different from our original experiment setting. Figure 11 demonstrates the qualitative results, Figure 12 shows the enlarged details and Figure 10 shows the quantitative results. As we can observe in the figures, the simplified version of DDNM can achieve an equally fast sampling speed compared to our method and can also obtain similar guidance quality. However, the images generated from DDNM exhibit various artifacts, such as high-frequency circular patterns and overly smoothed generation. These artifacts prevent DDNM from maintaining high fidelity while our method can generate more realistic details.

   That being said, we would like to point out that the generated images from DDNM tend to maintain better shapes whereas our method hallucinates small details more than DDNM. Therefore, when we compare PSNR values (Table 3), we can observe that DDNM is significantly better than our method. We also acknowledge that in the original paper of DDNM, in order to solve the noisy inverse problems, the authors suggest using multi-step time traveling to improve the performance which we did not deploy in order to make a fair comparison in terms of run time. Therefore, we believe that DDNM still has certain advantages over our method in terms of solving specific inverse problems. We think incorporating the consistency constraint in DDNM with our method can potentially achieve the best of both worlds, and will be an interesting future work direction.

4. Multi-step Optimization (Weakness 4): The effect of different numbers of optimization is indeed an interesting topic, and we found that the answer to this question actually depends on the tasks to which the users apply our method. Let’s assume we use time traveling as our optimization algorithm. Then, we can view it as a way to perform more Langevin steps in one diffusion step. Therefore, it has a higher chance of finding the optima, especially when the optima is further away from the original point. For example, suppose we want to sample “a headshot of a man” with CLIP guidance and pixel space CelebA-HQ model; then, if our unconditional DDIM sample is a headshot of a woman, then performing a multi-step optimization is going to be more beneficial since the samples that fit the prompt will be further away from the original unconditional sample. On the other hand, tasks like generating “a headshot of a person wearing red lipstick” only require small changes, and therefore, can be achieved in higher quality with fewer steps (see Figure 13). Generally speaking, the more challenging the tasks are (i.e. the more deviation required to get to the optima in expectation), the more beneficial multi-step optimization will be.

   That being said, we do observe in practice that a large number of steps does not always benefit the generation. For example, we can start to observe unnatural artifacts appearing in the background of the image when sampling with 7 and 15 steps in the “red lipstick” experiment. In fact, we hypothesize that with a step size that is not infinitesimally small, asymptotically infinite-step optimization may lead to significant deviation from the data distribution. In addition, we can use different optimization algorithms such as nonlinear conjugate gradient and/or apply multi-step optimization to a selective subset of steps, following FreeDoM, and we also observe that the step sizes need to be adjusted accordingly for different numbers of steps. The asymptotic behavior of the multi-step optimization and how to select these hyper-parameters will be an interesting future work. We have included the discussion we have in this response in Section E.3 of our paper as well.

We appreciate your valuable insights and acknowledgement of our efforts. Below, we respond to your questions and discuss the issues you have raised.

1. Reason for Speedup (Weakness 1): The reason for the speed-up is that our proposed MPGD does not require backpropagation through the diffusion model, which is generally known to be time-consuming. To be more specific, in DPS based methods, the guidance is calculated as $\nabla_{x_t} L(x_{0|t};y) = \nabla_{x_{0|t}} L(x_{0|t};y) \frac{\partial x_{0|t}}{\partial x_t}$ where $\frac{\partial x_{0|t}}{\partial x_t}$ requires backpropagation through the diffusion model. On the other hand, in MPGD, we only calculate the gradient $\nabla_{x_{0|t}} L(x_{0|t};y)$ with respect to $x_{0|t}$. Althoughin MPGD-AE and MPGD-Z, evaluating the autoencoder incurs additional costs, it is still faster than computing the gradients for the diffusion model itself. We have briefly discussed this reason in Section 4.2.1, and modified the writings in the experiment discussions to emphasize this reason.

2. Linear Manifold Hypothesis (Weakness 2): In response to your question about the linear manifold hypothesis in our theorem 2 analysis, it's important to clarify that our analysis of "the gradient of the auto-encoder lying on the tangent space of the manifold" does not actually depend on the linear manifold hypothesis. Proposition 1, which was initially proposed by [Chung et al. 2022], is based on this hypothesis.

   While we acknowledge that the linear manifold hypothesis may not be entirely realistic for all complex real image datasets, it is often observed that this assumption holds locally. A practical example of this can be seen in image interpolation. Consider interpolating between two headshots of the same person, where one image features black hair and the other blond hair. The interpolation can result in another plausible headshot of the same person with brown hair. This example illustrates that even though the overall image manifold might not be linear, local linearity can be a reasonable approximation for certain kinds of image transformations.

   Many prior works in the literature also make similar assumptions and implications. For example, the classic facial recognition paper [Turk et al., 1991] has discussed the validity of linear manifold hypothesis for specific subsets of image data such as well-aligned human face images. Hence, we believe this assumption can be made for the purposes of our analysis and is also meaningful for practical applications of our proposed methods.

[Chung et al., 2022] Improving diffusion models for inverse problems using manifold constraints. NeurIPS 2022.

[Turk et al., 1991] “Eigenfaces for Recognition”, Journal of cognitive neuroscience.

Thank you for your thorough and insightful review. As the rebuttal period is coming to its end, we are reaching out to see if you have any additional questions or concerns that we can address. Thank you and looking forward to your response.

Thank you for your constructive feedback. We would like to address your questions and concerns below.

1. More General Guidance Loss w/o Reference Images (Weakness 1 & Question 2): Thank you for bringing up this question. We would like to first clarify that our method is NOT limited to tasks with reference images, and therefore, we really appreciate you asking this question because although two of our original choices of tasks (FaceID guidance and style guidance generation) are not inverse problems, your question helped us realize that the previous example tasks we chose happen to all involve reference images. As a result, we have added an additional experiment where we use the CLIP model and text prompt to guide the pixel-space CelebA-HQ model. In Section E.1 and Figure 9 in the appendix, we show examples of samples guided by the text prompt and our method in comparison with unconditional DDIM samples of the same random seed. Our method is able to create images that follow the text description provided while maintaining high fidelity using an unconditionally pre-trained diffusion model.

2. Diversity of the Generated Images (Weakness 2): We apologize for the confusion caused by the presentation of this figure. Firstly, we would like to respectfully point out that the DDIM samples are the ones that showcase “the original p(x) that the model learned”. Therefore, in Figure 6 we can observe that our method is actually closer to the DDIM samples (and therefore the original p(x) that the model learned) regarding the color, structure, and other fine-grained details in the images. In fact, this is precisely because our method performs on the tangent spaces of the original DDIM clean data estimations, and therefore, is able to stay close to the original distribution while performing guided generation. We have also confirmed these qualitative observations with quantitative analysis in Table 1. The reason why these images seem to have strange color schemes is because the DDIM trajectories happen to produce these samples. We can also observe that FreeDoM, which is one of our baselines, generates similar images.

   That being said, to demonstrate our method’s generation diversity, we have included Figure 16 and an additional Figure 17 in the appendix for the pixel space FaceID guidance generation experiment. In these figures, we demonstrate that images generated by our method show diversity in various aspects, such as color, style, and facial expression.

3. Related Works (Weakness 3): Thank you for your feedback. We initially structured our paper to address the most relevant prior works and their limitations in Sections 2 and 3 before introducing our method in order to adhere to the page constraints while reserving an in-depth exploration of further related literature for the appendix. Based on your suggestion, we have now added Section 2.4 in our main paper to briefly discuss more related works while keeping the detailed discussion in the appendix.

4. Specifications of loss functions used in experiments (Question 1): We apologize for the confusion. Regarding the style-guided generation task, we recognized that our definition of the loss function was not as clear as it should have been. To rectify this, we have provided additional details in Appendix D for further clarification. To summarize, the settings for the super-resolution task are the same as those used in DPS [Chung et al., 2023], and the settings for the FaceID guidance generation and text-to-image style guidance generation are the same as those employed in FreeDoM [Yu et al., 2023].

[Chung et al., 2023] “Diffusion posterior sampling for general noisy inverse problems”, In Proc. ICLR, 2023.

[Yu et al., 2023] “FreeDoM: Training-free energy-guided conditional diffusion model”, In Proc. CVPR, 2023.

Thank you for your constructive feedback and your recognition of our work. We would like to answer your questions and address your concerns below.

1. FID (Weakness 1): We agree that it would be helpful to include FID since it is a more popular metric in the community. However, we would like to emphasize that for FID to be reliable, it would require at least 10k samples to evaluate for each experiment [Bińkowski et al., 2018]. Given the limited timeframe of the rebuttal period and the required sampling time for all the methods compared, it is not very feasible for us to produce a large enough number of samples to make proper FID comparisons, and we respectfully propose deferring this extensive analysis to future work, to ensure thorough and accurate evaluation.

2. Additional Failure Cases (Weakness 2): Based on your comment, we have added some noteworthy failure cases from the images generated in the style transfer task with StableDiffusion and have included a discussion in the Appendix F.2. In summary, there exist some instances where the expected style is not reflected in the generated images due to the design of the loss function or the relationship between the provided prompt and the reference style.

3. Comparison with DDIM (Question 1): We apologize for the confusion. In the context of our paper, DDIM performs unconditional sampling from the learned diffusion model whereas our method receives extra guidance from an external loss function provided by the users to perform conditional generation. In Figure 2, we illustrate the relationship between DDIM and our method, and we also include comparison experiments between DDIM and our method in the FaceID guidance generation and style guidance Stable Diffusion generation experiments. This comparison demonstrates how our method modifies the original unconditional DDIM sampling trajectory to generate conditional samples with respect to the loss guidance while keeping the samples on the correct manifolds, hence maintaining high sample quality.

4. Effect of Classifier-Free Guidance Scale (Question 2): Thank you for this question. In the appendix E.4 and the table below, we have included an additional quantitative comparison in style guided Stable Diffusion generation with various classifier-free guidance scales (CFG). This analysis actually helped us to produce better results with our method in this experiment, and we have also modified our main paper to include the updated results with further hyperparameter tuning. It's worth noting that the CFG scale has a positive impact on the style score, and strong CFG scales appear to help decrease the loss function. However, although in vanilla text-to-image generation tasks a larger CFG scale tends to lead to a higher CLIP score [Nichol et al., 2022], since the distribution we aim to sample from is also guided by another external loss function, we do not observe the same trend in our setting. In practice, we would suggest our users to adjust this hyperparameter to suit their preference of tradeoff between the style guidance and the text prompt condition.

[Nichol et al., , 2022] “GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models”, Proceedings of the ICML, 2022.

Thank you for your detailed response.

• Comparison with DDIM: Apologies for stating Question 1 so poorly. I meant to ask about the consistency of guidance when using the proposed method versus DDIM. In any case, this can now be inferred by examining the added failure cases.
• Effect of Classifier-Free Guidance Scale: I'm happy the analysis led to better results, and with the analysis itself.

Although FID evaluations are still missing, I find my concerns addressed properly and I maintain my score.

Thank you for your thorough and insightful review. As the rebuttal period is coming to its end, we are reaching out to see if you have any additional questions or concerns that we can address. Thank you and looking forward to your response.