Variational Control for Guidance in Diffusion Models

We thank the reviewer for their encouraging feedback about experiment design and readability. We address specific questions below:

Of these three, #2 complicates things and makes the method non-linear and more complicated for optimization. Therefore, an ablation that shows the effect of excluding #2 is necessary. Perhaps #1 would be sufficient to regularize the guidance. Furthyeremore, for linear inverse problems, ut will have a closed form solution of #2 is omitted.

We would like to thank the reviewer for their insight. We agree that such an ablation could be useful to study the impact of different terms in the loss function in Eq. 17. Empirically, we find the choice of weighting to be largely task-specific (see Tables 7 and 8).

We thank the reviewer for their feedback. Please find our responses below.

The evaluation tasks make sense but the comparisons [...] lack [...] more recent methods. . Please refer to Essential References Not Discussed.[...] authors did not discuss or compare, such as [1-8].

This was also a common point from Reviewers DxJn, 113Q so we address it jointly here.

Re. Generalization to other tasks. Our framework indeed generalizes to non-linear tasks such as those pointed out by the reviewers. We demonstrate this by simply adapting the terminal cost in our method for Style and text-conditioned guidance. In more detail:

Style Guidance: Following the experimental setup in MPGD [He et al.], we evaluate our method on Style Guidance using Stable Diffusion (see Figure 1). We further quantitatively compare our method on 1k (reference images, prompts) with FreeDOM [Yu et al.] and MPGD in Table 1. Our quantitative and qualitative results suggest that NDTM outperforms competing baselines on prompt (CLIP score) and style adherence (Style Score) while exhibiting better perceptual quality.

Text Guidance with CLIP: Following the experimental setup in MPGD, we present qualitative results on text-conditional generation on the CelebA (see Figure 2. NDTM can be used to generate samples that adhere well to simple and complex prompts.

Therefore, we emphasize that our method can be applied to more general guidance scenarios. We will include additional qualitative results for these tasks with their experimental settings in a revised version of our paper. Note that we selected the best hyperparameters for the selected baselines, just like for all other experiments.

Re. Limited Empirical Comparisons. Based on the suggestion from Reviewer 113Q, we compare with two additional baselines, MPGD and RB-Modulation [Rout et al.], on the non-linear deblur and 4x superresolution tasks for the FFHQ and ImageNet datasets. NDTM outperforms these additional baselines on these tasks. See Table 2.

We also thank the reviewer for the interesting pointers, which we will include in the related work section for broader context. However, they primarily focus on discrete diffusion models or finetuning with human preference data—directions that differ from the scope and goals of our paper. In detail:

Re. Discrete Diffusion Models. Our work focuses on continuous state space diffusion models (see Section 2) and assumes a differentiable terminal cost (see line 120, Right Column in the main text). We think that optimal control provides an interesting approach to guiding discrete diffusion models and leave this open to future work.

Re. Finetuning with human preference data. In this work, we only work with training-free methods for conditional generation tasks and not with methods that finetune diffusion models with additional human preference data. Therefore, comparisons with baselines like Diffusion-DPO are outside the scope of this work.

The paper does not provide code for verification

We will make our code publicly available upon acceptance. Please refer to the pseudocode in Algorithm 1 in the main text and the corresponding hyperparameters in Tables 4,5 and 6 in the Appendix.

For the comparison with DPS [...] should also be selected from a space.

We tuned all baselines (including DPS) for the best performance for all tasks and datasets (see Appendix B.2).

While the paper discusses non-linear control [...] of non-linear dynamics.

We respectfully disagree, as there seems to be a misunderstanding: We both consider non-linear inverse problems and allow for non-linear control. In detail, while we do update the state $x_t$ additively with the control $u_t$, the updated state ($x_t + \gamma u_t$) is directly injected in the diffusion denoiser (modeled as a non-linear neural network). Therefore, the guidance process indeed depends non-linearly on the control. We acknowledge that to update the state $x_t$ with the control $u_t$, there could be multiple ways to achieve this. However, we don’t explore the full space of such possible combinations due to space and time constraints which we also highlight in line 175, Right column.

The method assumes differentiable objectives [...] generative tasks.

We clearly state our assumptions of a differentiable terminal cost in Line 120, Right Column in the main text. Therefore, while the argument that our method may not be ideal for all types of generative tasks is correct, we never claim this in our work. We agree that extending our method to discrete diffusion models and discrete terminal costs can be an interesting direction for further work and will clarify this point in more detail in Section 6 of our revised paper.

The paper lacks Impact Statements, which is required.

We have provided the impact statement at line 435 (Right Side) in the main text.

We thank the reviewer for their detailed and helpful feedback. Please see our response below.

I do find the statement [...] the authors claim in the introduction [...] that prior works which are based on optimal control [...] (all of which are widely studied [1,2,3,4])

Our framework indeed generalizes to non-linear tasks like Style and text guidance. Due to space constraints, please see our response to Reviewer UY5V (first point).

Moreover, the authors claim to be able to [...] However, the only blind inverse [...] to be inaccurate and not well justified by the evidence provided.

For the blind deblurring task considered in this work, the forward model y = k * x + n is non-linear since both the kernel k and the underlying signal x are unknown here, making the inverse problem non-linear and challenging. Therefore, we disagree that it is equivalent to solving linear deblurring. Moreover, as highlighted in our common response, we already show that our method can generalize to tasks like Style Guidance, providing further evidence in favor of its generalizability.

The authors have [...] proposed mostly valid evaluation criteria for the problem at hand [...] performance gain based on FID results in their abstract

We agree that 1000 samples may produce a biased FID. However, we evaluate all baselines similarly, so we expect similar improvements in FID for more samples. In addition, the reported FID strongly correlates with other more stable metrics, such as LPIPS and KID. Lastly, our choice of metrics and evaluation protocol is primarily based on prior work in the literature like DPS [Chung et al.] and RedDiff [Mardani et al.] and is therefore justified, as also noted by the reviewer.

The authors choose addition as the aggregation function [...] further justification and clarification on the assumptions when choosing the aggregation function of the control.

We do regularize u_t via the guidance scale $\gamma$, compare Eq. 17 in the paper. More specifically, the first term in Eq. 17 regularizes the magnitude of u_t, while the second term encourages a u_t which does not change the magnitude of noise in the guided signal x_t. This brings the guided trajectory close to the unguided trajectory and close to the usual input to the learnt score. Regarding the image manifold, since u_t is only added inside a score function, the terminal cost cannot make the trajectory leave the span of the score function. More importantly, superior empirical results across a range of linear and non-linear inverse problems justify our choice of aggregation function. Lastly, we did not explicitly explore specifying terminal costs that are incompatible with the diffusion prior. We think that this is an interesting future direction and will add this to the Limitations in Section 6 of the revised paper. We hope this clarifies any concerns about our choice of aggregation function.

The authors mentioned that the Tweedie’s estimate in DPS causes high computational requirement [...] exempt from the same limitation.

We think that DPS converges worse because the overall framework assumes that the gradient of the terminal cost is accurate at every step, but it is actually approximated by Tweedie’s estimate. We think that our framework is able to deal with this approximation to the terminal cost better through the additional flexibility in the guidance. We propose rephrasing the corresponding section for the final manuscript to reflect this intuition.

I have reviewed the appendix [...] So far the qualitative comparison is not comprehensive.

We include some more qualitative results on FFHQ here. In light of the new results on style guidance we will update the Appendix with more experimental details for these tasks, and additional qualitative results.

The most prominent concern I have about this paper is the lack of comparison with relevant literature [...] performance than the baselines selected by the authors.

Thanks for your feedback and for suggesting other competitive baselines. Based on the reviewers' suggestion, we compare with two additional baselines, MPGD and RB-Modulation, on the non-linear deblur and 4x superresolution tasks for the FFHQ and ImageNet datasets. NDTM mostly outperforms these additional baselines on these tasks. Due to space constraints, please see our response to Reviewer UY5V (Limited Empirical Comparisons).

The authors choose T=200,N=15 as the hyperparameters [...] Can the author explain why they can achieve this speedup?

The configuration T=200, N=15 is used only for the blind inverse problem, which is inherently more challenging and benefits from additional optimization. However, for all other tasks, we use a more efficient configuration (See Table 8), carefully balancing performance and runtime. The runtimes reported in Table 9 correspond to the superresolution task, where T=50, N=5. This setup allows our method to be over 14x faster than DPS on that task.

We thank the reviewer for their feedback. We address specific concerns as follows in the order of sections:

The method might be too expensive. The running time cost should be provided

Please compare the running time of the algorithm with other CFG methods.

Runtime is often faster than the closest competitor. See the runtimes in Table 2 in the main text and Table 9 in Appendix C.2, where we list runtimes once per hyperparameter configuration. In detail, our method is about one order of magnitude faster than DPS, the closest competitor for superresolution and random inpainting. While our method is indeed slower than for example Red-diff or DDRM, it always improves the sample quality. Note that we did optimize all hyperparameters for sample quality for all methods. We agree that reducing the sampling time is a crucial improvement for guiding diffusion sampling, but this limitation applies to all high-quality competitors.

What is the value of N in Algorithm 1?

We provide complete details of hyperparameters for our method and baselines in Tables 4,5 and 6 in the Appendix. To summarize, we set N=50 for linear and non-linear inverse problems and N=200 for blind inverse problems.

The evaluation does not have generation task for example label-condition generation, text2image generation. This hinders the evaluation of the work. [...] Please include these tasks in the paper.

Our framework indeed generalizes to non-linear tasks such as those pointed out by the reviewer. We demonstrate this by simply adapting the form of the terminal cost for Style Guidance generation and text-conditioned generation.

Style Guidance Generation: Following the experimental setup in MPGD [He et al.], we evaluate our method on Style Guidance using Stable Diffusion 1.4. Please refer to our qualitative results at this URL: https://imgur.com/a/wEDqoaR. We further quantitatively evaluate our method on this task on 1000 (reference image, prompt) pairs and compare with FreeDOM [Yu et al.] and MPGD in the accompanying Table 1. Our quantitative and qualitative results suggest that our NDTM can be applied to this task and outperforms competing baselines on prompt (CLIP score) and style adherence (Style Score) while exhibiting better visual perceptual quality.

Text Guidance with CLIP: Following the experimental setup in MPGD [1], we further present qualitative results on text-conditional generation using a pretrained diffusion model on the CelebA-HQ dataset, which can be accessed using this link: https://imgur.com/a/5jlnDLo. Similar to StyleGuidance, NDTM can be used to generate samples that adhere well to not only simple prompts (like Blond hair) but also more complex prompts such as “a person with blonde hair and a goatee”.

Therefore, we emphasize that our method can be applied to more general guidance scenarios. For brevity, we will include additional qualitative results for these tasks with their experimental settings (like the form of the terminal cost) in a revised version of our paper. Note that we selected the best hyperparameters for the selected baselines, just like for all other experiments.

We also urge the reviewer to check out our response Reviewer UY5V (see point Re. Limited Empirical Comparisons) which presents additional empirical comparisons with additional baselines on 4x super-resolution and non-linear deblur tasks

There are recent advances in classifier-free guidance but there is no discussion or comparison in the main paper. [see also] Essential References Not Discussed

We thank the reviewer for the interesting pointers, which we will include in a dedicated section on classifier-free guidance in the related work section. Note that the two approaches are complementary: Classifier-free guidance is based on retraining a diffusion model using example training data, training-free guidance is based on a pretrained model and a cost function. We think an extensive evaluation of different paradigms is beyond the scope of this work.