Momentum-driven Noise-free Guided Conditional Sampling for Denoising Diffusion Probabilistic Models

Thanks for your constructive feedback. We highly value your comments and address them as follows:

Q1: The relation and difference between our work and the recent work Boosting Diffusion Models with an Adaptive Momentum Sampler

The recent work titled Boosting Diffusion Models with an Adaptive Momentum Sampler leverages the momentum algorithm to ascertain the prior denoising direction from preceding timesteps in the context of an unconditional denoising process. Our work, however, takes a different path. We use the momentum algorithm to enhance the consistency and robustness of the guidance gradient during the conditional sampling process. Despite the similar functions of the first- and second-order momentum in smoothing the gradient direction and adaptively assigning the update step size in both studies, our motivation for using momentum algorithm and the problem we seek to solve diverge from the aforementioned work.

Q2: Since the semantic and layout is constructed when t is large, and the gradient compression is larger there, it may destroy the semantic matching between the final generated image and the condition.

Thanks for this valuable questions. We think for tasks that requires much stricter guidance conditions, such as segmentation map, the problem of semantic and layout mismatching may happen due to the misalignment between the unconditional denoising process and the given conditions. For such kind of tasks, we think excluding the gradient of the intial steps from the suppression scheme, or increasing the number of denoising steps can help to alleviate the problem.

Q6: In Section 5.2, there is no description about what is $I_{cs}$.

Thank you for highlighting this oversight. We have now updated Section 5.2 to include the necessary explanation: `The differences between the content image $I_c$ and the stylized image $I_{cs}$ is expressed as'.

Q7: In Section 5.3, the ablation results on style transfer task indicate that "Raw+Filtering & Suppression" can achieve the best results on $L_{content}$. However, it's results on $L_{style}$ is only the third best.

In Section 4.3, we provide an overarching insight that suppressing the early gradient can enhance the sampling quality. We employ the simplest linear scheme to demonstrate this effect. However, we acknowledge that more effective suppression schemes may exist. For instance, we discovered that the suppression scheme $\tilde{g_t} = \frac{\sqrt{\bar\alpha_t}}{\sqrt{1-\bar\alpha_t}}\bar{g_t}$ yields superior results for the style-transfer task, achieving both the lowest content error ($L_{content} = 4.69$) and the most optimal style error ($L_{style} = 1.46$). Moreover, we can readily strike a balance between the content error and style error by modulating the ratio of $\lambda_c:\lambda_s$.

Thanks for your thoughtful reviews. We highly value your comments and address them as follows:

Q1: The experiment of style transfer task has no relationship with this topic. Therefore, I think the author better give an explanation about why conducting this part of experiments. Q2: Lacking the results of other works on improving clean-estimation guidance in style transfer task.

We appreciate the reviewer's query about the inclusion of the style transfer task in our paper. While we understand the concern, the purpose of incorporating this experiment was two-fold. The first is that previous methods based on noised samples are not suitable for conditional generation tasks on unlabeled datasets, such as arbitrary style transfer. By achieving arbitrary style transfer, we underscore the potential and flexibility offered by our clean sample-based method. Secondly, the effectiveness of our proposed method is further demonstrated through comparisons and ablation studies on style-transfer task. For a more comprehensive analysis, we include a comparison with the results from FreeDoM as follows. To ensure a fair comparison, we set the total sampling steps for FreeDoM to be 1110 steps, incorporating its time-travel strategy, while our method uses 1000 ddpm steps for sampling. We find that results from FreeDoM contains dotted noise and shows a inferior sampling quality. As the results show, our method outperforms FreeDoM on both $L_{content}$ and $L_{style}$. The qualitative comparison will be updated later.

Q3: The authors claim in Section 4.2 that the clean classifier+guidance function proves to be less reliable than the noise-robust classifier. Could the authors give a more direct proof like Figure 2?

In Appendix.C.2, we have supplemented our discussion with several sample sequnences of clean-estimations obtained during the denoising process in Figure 10. Upon examining these figures, it becomes evident that the clean-estimation is unstable during the denoising process(eg. in the red boxes). This instability subsequently results in an inconsistent guidance gradient throughout the denoising process.

Q4(1): The authors do not clarify which table is the results of this experiment.

Thanks for pointing out this issue. We have made the necessary addition in the PDF, referencing `As shown in Tab. 1' in Section 5.1.

Q4(2): The authors mention FreeDoM and Universal Guidance in the paper, but only provide the results of ADM+FreeDoM.

We have add the following comparison with DiT + FreeDoM here. We only provide the DiT + FreeDoM results here because FreeDoM is a follow-up work of Universal Guidance which shares a similar time-travel strategy and shows superior results. To ensure a fair comparison with our method, we have calibrated the total sampling steps for FreeDoM to closely align with ours. Our method yields superior outcomes on ADM model and style-transfer task, while demonstrates comparable results on DiT model. Given that DiT is a latent diffusion model, the guidance gradient $\frac{\partial \mathcal{P}}{\partial x_t}$ may exhibit a different pattern from ADM model. It may requires a futher tuning of our method's parameters.

Q5: The authors is recommended to prove the results of other methods on image generation task for comparison.

Thank you for your suggestion. To the best of our knowledge, we have compared our method with a variety of clean sample-based guidance approaches. However, further comparisons with noised sample-based methods might not provide a fair evaluation, given the inherent differences between these two kinds of methodologies. Clean sample-based methods normally yield inferior results than the noised sample-based methods, while they offer substantial flexibility in the choice of guidance functions and eliminate the additional training cost for noise-finetuning.

Thanks for your valuable feedback. We highly value your comments and address them as follows:

Q1: In Fig. 2b, the trends of filtered gradient and unprocessed gradient are almost the same rather than a gradual stabilization process. The simple method proposed in this paper is not the most effective solution to prevent noise in the gradient.

Figure 2b illustrates the degree of deviation between the current gradient direction and that of the previous timestep. A smaller value on the graph signifies a greater deviation. As depicted in the graph, the filtered gradient presents a substantially more consistent direction. Notably, the significant deviations at certain timesteps (for instance, t=650, t=940) are markedly reduced. Moreover, the experimental results outlined in Table 1 further reinforce the effectiveness of our approach. It is clear that our method significantly enhances the performance of the clean-guided DDPM, surpassing other methods by a considerable margin, thereby establishing itself as the most effective solution to date.

Q2: In Table 1 in Section 5.1, I note that the proposed method shows the limited improvement compared with `raw clean guidance', e.g., DiT in sFID, Prec and Rec, and ADM in Prec and Rec.

FID is a well-acknowledged and widely-used metric for evaluating image generation models, with sFID, Precision, and Recall typically serving as auxiliary metrics for a more comprehensive evaluation (Li et al., 2022b, Peebles & Xie, 2022). We wish to underscore that our method exhibits marked improvements on the ADM model for both FID and sFID metrics. Additionally, it demonstrates substantial enhancement on the DiT model for FID. These results collectively attest to the effectiveness of our proposed method.

[1] Peebles, W. and Xie, S., 2023. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 4195-4205).

[2] Zheng, G., Li, S., Wang, H., Yao, T., Chen, Y., Ding, S. and Li, X., 2022, October. Entropy-driven sampling and training scheme for conditional diffusion generation. In European Conference on Computer Vision (pp. 754-769). Cham: Springer Nature Switzerland.

Q3(1): In Table 1 in Section 5.1, I note that the authors only compared clean estimation-based conditional without comparing noise-guidance-based methods.

We did not compare with noise-guidance-based methods because our proposed method is specifically focused on enhancing clean-estimation-based methods. Notably, our approach empowers clean-estimation guidance to surpass the performance of noise classifier guidance in terms of both FID (4.20 vs. 4.59) and sFID (5.17 vs. 5.25), while maintaining comparable levels of precision and recall.

Q3(2): In figures 5, 6, and 7 of the qualitative results in the appendix, I can't see any comparisons with other baselines. Please add some experiments to show its ability.

We have add a qualitative comparison in Fig. 5 in Appendix.A.3.

Q3(3): What is the specific sampling cost in the experiments? Please add experiments to compare clean-estimation guidance techniques.

We have add the following comparison on the sampling cost in Appendix.A.5. The inference time is averaged on 10 denoising process running on an NVIDIA A100 GPU.