TFG: Unified Training-Free Guidance for Diffusion Models

We sincerely thank you for your insightful and constructive review, and we are so honored that you believe our work is novel, effective, comprehensive, and offers commendable contributions. We are delighted to address your concerns and questions below.

1. The existence of a broader spectrum of works and the overreach of the title.

The field of training-free guidance encompasses …a wide array of studies beyond those specifically addressed in this paper (e.g. [1]-[3]). Could you elaborate on how these works relate to the scope and contributions of your paper?

Thanks for this valuable suggestion. The three works you mentioned are training-free methods specifically for text-to-image tasks (compositional generation [1], layout control [2], spatial control [3]). [1] and [2] exploited the cross-attention mechanism in Stable Diffusion UNet to incorporate guidance targets (noun phrases [1], layout condition [2]), while [3] leverages the diffusion features of the guidance image (with DDIM inversion) to guide the diffusion process.

Unlike these works, which design task-specific guiding strategies, our paper focuses on general training-free guidance methods that can be applied to any diffusion model for any objective function. Basically, our work doesn’t focus on the design of the objective function for each task but on the universal methodology of guidance. That said, the improvement in architecture, diffusion models, and guidance targets is “orthogonal” to our work. We would like to discuss these works in our revised manuscript to clarify our scope and contributions.

2. The rationale for focusing specifically on TFG.

Your paper operates under the assumption that the forward model is known, leveraging training-free guidance for solving inverse problems. Given that knowing the forward model allows for the generation of a large number of samples to train a conditional diffusion model at a relatively low computational cost (e.g., 10 A100 GPU hours) and training-based approaches are much better than training-free ones, could you discuss the significant motivations of adopting a training-free approach in this context?

We sincerely thank the question that allows us to discuss the motivation of the paper further. In lots of cases when the conditions are complex, e.g. requires an image of a special dog species in certain scenarios or a molecule that has certain polarizability and synthesizability, they are extremely rare (with probability less than, e.g., $10^{-6}$) across the unconditional distribution. As such, “sampling a large number of targets for training a time-dependent classifier” itself is already impossible. However, training-free methods can effectively increase the sample rate from $10^{-6}$ to $10^{-2}$, making it possible to leverage training-based methods. In fact, we believe that one motivation of TFG is exactly to bridge towards the scenarios you describe. We will explicitly add this discussion in the paper.

3. Further elaboration of the theoretical part of the work.

The theoretical foundation of your paper seems to rest significantly on Lemma 4.1, which revisits the variance of MMSE estimator of the signal corrupted by Gaussian noise (e.g., (2.8) in [4]). This formula is widely adopted in diffusion papers (e.g., [5]). Given its established nature and previous applications, could you elaborate on how this lemma specifically contributes to the novel aspects of your framework?

We want to highlight that lemma 4.1 is simply used to illustrate the effect of “variance guidance” and is not used to prove our major theorem 3.2. Despite its wide use in previous works, here we use the lemma to point out that $\Delta _t$ in Line 7 is to control the second-order information, pointing to the difference between $\Delta_t$ and $\Delta_0$ from a theoretical perspective.

Concerning Lemma 4.1, it appears there is no assurance that the generated image accurately follows the conditional distribution, nor is there a guarantee that the loss decreases at each iteration as suggested by equation (7). Could you provide further clarification or additional theoretical support to address these concerns?

We agree that a theoretical guarantee on loss is important, but to the best of our knowledge, none of the existing training-free methods can prove that the generated sample follows the correct distribution as desired. This is not an issue of Lemma 4.1, but we believe that one crucial future direction is to provide a global-level guarantee for training-free guidance. The intrinsic difficulty here is to analyze the difference between a training-based classifier $f(x,t)$ and standard classifier $f(x)$, which are complex to be quantitatively captured.

The concept of "time-travel" being an Ornstein-Uhlenbeck process is intriguing but lacks a detailed derivation. Could you expand on how the theory of the Ornstein-Uhlenbeck process quantifies the benefits of time-travel in your framework?

We are delighted to further explain this. For each recurrent step, we compute $x_{t-1} = u(x_t)$ where $u$ corresponds to Line 6-9 and then add a Gaussian noise back in Line 10 to obtain updated $x_t$. Together, the process becomes

which is a typical OU process. This implies that when $N_{recur}$ goes to infinity, $x_t$ will converge to a certain distribution that is hard to compute analytically.

The selection of hyperparameters seems not to be grounded in theory. Could you discuss the rationale behind these choices and any potential theoretical support?

Our beam-search hyperparameter selection strategy depends on an intuition (assumption) that when fixing all other parameters, the loss function of the remaining parameter is a decrease-then-increase function, i.e., as the parameter increases from 0 to infinity, the loss first goes down (or never goes down) and then increases. This is observed and summarized from experiments, and under such an assumption, it can be proved that the (near-)optimal hyperparameter can be found with sufficient search step and small enough stepping size (multiplying factor). We would add this to our paper as suggested.

4. The coverage of the proposed space in existing literature.

The paper posits that the study of training-free methods can be encapsulated within the hyperparameter space of your framework. However, specific hyperparameter settings critical for the performance in existing works, such as the step sizes used in Face generation (at.sqrt()) and Style transfer ((correction * correction).mean().sqrt().item() * unconditional_guidance_scale / (norm_grad * norm_grad).mean().sqrt().item() * 0.2) in FreeDoM, are not explicitly covered. Could you address the omission of these settings and their impact on the comprehensiveness of your study?

We are delighted to explain this further. Theoretically speaking, TFG aims to provide a general framework that unifies different sub-hyperparameter spaces, and the practical implementation tricks are omitted (as this trick only appears in FreeDoM code, not its original paper). That said, our framework can naturally represent this nuanced setting. For example, if we want the gradient to be $\frac{\nabla g}{\| \nabla g\|}$ for a given target function $g$, we could simply set the $f$ in our algorithm as $\exp\{\int \frac{\mathrm dg}{\| \nabla g\|}\}$, in which case the gradient of $f$ becomes exactly what we want.

In practice, however, we observe that with the aid of hyperparameter searching, the output quality of our FreeDoM algorithm is comparable to the original implementation. Because of this, and to be consistent with the paper, we simply focus on the setting without normalization. Overall, we believe that such normalization should be similar to normalizing $\mathbf \rho$.

The hyperparameter search settings for the baselines in your comparative analysis are not disclosed, raising questions about the fairness and validity of the comparisons. Could you provide more details on these settings to ensure a transparent and equitable comparison?

We apologize for not providing a more comprehensive discussion beyond what was presented in Appendix E. We'd like to offer additional details here. For all methods discussed in our paper, we employed an identical beam search strategy with three beam search trials and a maximum of 7 steps. For each search, we start with initial parameters (identical across all methods) and double parameters to see if performance improved. Then, we keep the top 3 performances in the beam list after each step until the list becomes unchanged or the maximum step is reached. This beam search program is method-agnostic and compatible with all methods, ensuring fair and objective comparisons. To promote transparency and facilitate future research, we will open-source our code, all beam search runs, the best parameters for each algorithm, and their corresponding performance results.

Again, we sincerely appreciate your helpful review. Please let us know whether there is any additional concern that we could address to improve your evaluation of our work.

I appreciate the authors' effort in addressing my concerns, which has led me to moderately increase the evaluation score.

Regarding Question 2, I have experimented with several methods to fine-tune a network trained on clean images for use with noisy images [A, B]. My findings indicate that the training can be completed in one A100 hour and the performance is quite satisfactory. Could you please provide further clarification on this point?

For Question 3, you assert that Theorem 3.2 represents the main theoretical contribution. However, it appears to be straightforward as the proposed method is a combination of these methods. Could you explain why the proof is non-trivial?

In relation to Question 5, I would be interested in a more detailed discussion on the practical implications of the connections between the Ornstein-Uhlenbeck process and time travel. Specifically, how do these connections yield new guarantees or insights that could potentially enhance the methodology of time travel?

[A] More Control for Free! Image Synthesis with Semantic Diffusion Guidance

[B] Towards practical plug-and-play diffusion models

Thank you for your detailed response. I have adjusted my score to an 8. In particular, I believe the reproducible benchmark used in this paper will influence the trajectory of future research in training-free guidance. To ensure a comprehensive comparison, I strongly recommend that you include motion diffusion guidance from the LGD paper (a new application related to diffusion planning) and phase retrieval from the DPS paper (a nonlinear non-NN case) as additional benchmarks in the final manuscript. I have not listed this as a weakness, as I understand that these experiments may not be feasible within the rebuttal period.

We sincerely thank you for your insightful and constructive review, and we are honored that you think our work is non-trivial, novel, organized, comprehensive, substantive in experiments, and presents a valuable contribution. We are happy to address your concerns:

The authors claim all existing training-free approaches fit into their framework which is unlikely. Many training-free methods have been proposed and the authors could better place their framework in the context of recent literature. Section 3 Figure 1's demonstration that training-based conditional generation outperforms training-free is unsurprising, given that training-free methods are approximations.

Our algorithm is based on a comprehensive contemporary literature review of more than ten papers (see [1]-[10] below) and on a collection of all of their algorithms (that directly or indirectly use the algorithms we unify), trying to provide an elegant and unified way to study the problem. That said, we completely agree with your suggestion that we never fit all methods, and we are more than happy to explicitly clarify this and discuss our framework in the context of TFG literature.

The purpose of Figure 1 is to emphasize our observation that, unlike what previous works have claimed, training-free guidance is far from being addressed, and all existing methods could even fail on a very easy task. In other words, it helps demonstrate our motivation, rather than to help compare training-free methods with training-based methods. We will clarify this.

While hyperparameter search was helpful, the unified design space discussion in Section 4.1 led to limited theoretical insight.

Thanks for the suggestion! Currently, we organize Section 4.1 in a way that is “less theoretical” and “more intuitive” to help readers better understand each part of the parameter space without being overwhelmed by theory concepts. That said, the study of space involves Ornstein–Uhlenbeck process, Tweedie’s formula (second order), and probability convergence. We will try to extend our theoretical insights and put more fine-grained discussions in the appendix to ensure both intuition and theory are clearly conveyed to readers.

The hyperparameter setting strategy… could use more detailed explanation (see questions 2-4 below).

We answer each question separately below.

The benchmark Section 5 decides to not compare generation fidelity in favor of comparing the best algorithm in terms of generation validity. However, throughout the rest of the paper (e.g. figure 1, 2, and 3) tradeoffs between fidelity and validity are emphasized. Looking at Table 2, TFG is sometimes but not always the best algorithm in terms of both fidelity and validity simultaneously. The focus on only validity seems unjustified here.

We thank you for allowing us to further explain this. We completely agree that there is a trade-off between fidelity and validity, and depending on the user’s requirements, different metrics should be emphasized. The reason why we majorly compare validity is that for each algorithm during beam search, the metric we use to select the best run is exactly the validity (in a held-out set, of course). It’s important that the selection metric and the evaluation metric are the same to avoid unfair and tricky comparisons.

On the other hand, users could also arbitrarily combine fidelity and validity into a new “metric” and conduct beam searches via this metric if they prefer. Our framework does not have any restriction on this, and, unsurprisingly, TFG will still outperform existing methods. We thank you for your question, and we would like to clarify this in the paper.

Additionally, regarding to the questions,

Why does increasing $N_{recur}$ and $N_{iter}$ eventually hurt performance?

In fact, this phenomenon has been pointed out in previous works (e.g., MPGD) as well, and its underlying theory remains unclear. One observation is that if $N$ is too large, the generated images tend to be “valid” but highly unrealistic, possibly due to the amount of injected noise being too large.

In section 4.2, does the "structure analysis" that "increase" is best hold more generally on other tasks than label guidance?

Yes. Generally speaking, we find that “increase” works best (or with a negligible gap) among all of the tasks we consider in the paper.

The number of Monte Carlo samples in the Implicit Dynamic is set to 1 based on Table 1, but that choice needs more justification. Does more samples than 4 help? Why is implicit dynamic helpful with few samples? This seems contrary to the original motivation of estimating an average.

We find that more than $4$ of sample size does not help with sample quality because it reduces the stochasticity of the dynamics (think about changing the Gaussian noise in an SDE to, e.g., its expectation). We are delighted to provide more justification by adding a mathematical explanation of the role that the Gaussian noise is playing in the appendix and explaining why more samples are not beneficial.

In section 4.2, the beam search in "Searching strategy" could use more detail. What sample sizes are used and how did that enable quick search? Also, what metric is used when deciding the top K candidates in this search?

As mentioned in line 286-287, all searches are run with 1/8 of the test sample size and a maximum search step of 6. The test sample size for each task can be found in Appendix D. To conduct exhaustive grid search, we need to run more than $125$ experiments, which are much slower than our beam search strategy. The metric used when deciding the top K candidates is validity.

Again, we sincerely appreciate your helpful review. Please let us know whether there is any additional concern that we could address to improve your evaluation of our work.

[1] Controllable Music Production with Diffusion Models and Guidance Gradients.

[2] A Framework for Conditional Diffusion Modelling with Applications in Protein Design and Inverse Problems.

[3] Solving Audio Inverse Problems with a Diffusion Model.

[4] Diffusion Models for Audio Restoration.

[5] Vrdmg: Vocal Restoration via Diffusion Posterior Sampling with Multiple Guidance.

[6] Motion Guidance: Diffusion-Based Image Editing with Differentiable Motion Estimators.

[7] Training-free Multi-objective Diffusion Model for 3D Molecule Generation.

[8] Control3diff: Learning Controllable 3D Diffusion Models from Single-view Images.

[9] Steered Diffusion: A Generalized Framework for Plug-and-Play Conditional Image Synthesis.

[10] Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning.

We sincerely thanks for your insightful and constructive review, and we are more than honored that you think our work is interesting, novel, comprehensive, and will be helpful for the community. Regarding your concerns,

While I appreciate the comprehensiveness of the experiments on the proposed benchmark, I think the author fails to provide an informative analysis of the results. Currently, the results simply show that by optimizing the hyperparameters in the unified framework, we obtain better performance, which is a natural outcome since the existing methods are included in the framework. From a research perspective, I can think of several questions worth investigating, such as whether the optimal hyperparameters vary between tasks and, when using fixed models, how the optimal hyperparameters are affected by the target objective function. With this investigation, it would be best to reach some general conclusions to guide users in tuning the hyperparameters in practice, rather than just relying on grid search.

We agree the importance of more fine-grained analysis, and we are delighted to investigate more on our experimental results as you suggested. Specifically, we first explicitly present the optimal hyper-parameters of TFG and other existing methods for all tasks in the uploaded PDF (Table 2). We list several observations below.

1. Overall, optimal parameters vary widely between problems and datasets. For example, even with the same model and objective (e.g., label classifier on ImageNet or CIFAR10), the best hyperparameters vary widely from target to target. This highlights the importance of hyperparameter search.
2. The improvement of TFG over existing methods depends heavily on the difference between the optimal parameters and the subspaces of existing methods. See the next questions for a more detailed analysis.
3. Also note that in our paper, we didn’t do an exhaustive grid search, but instead perform an efficient beam search (line 254-264) to find the optimal hyper-parameters. All searches are run with 1/8 of the test sample size and a maximum search step of 6 (line 287). Our experimental results demonstrate that this is sufficient to find near optimal parameters (check out Table 1 in our uploaded PDF).

It seems that for around half of the tasks, the improvement is only marginal, while for a few other settings, the improvement is significant. I think this phenomenon is worth further investigation to understand in what scenarios TFG will provide improvements.

Thanks for the advice! We conducted a detailed investigation into the phenomenon and found that the improvements are highly related to the differences in the optimal hyperparameters between TFG and existing methods. For example, the $\bar\rho$ for UGD is the same as TFG for gender-age guidance task, where TFG only has 0.133% validity improvement over UGD. On the contrary, their values differ on the fine-grained classification task, and TFG has an 18.7% validity improvement over UGD. Overall, this depends on whether the optimal parameter lies in the subspace that existing methods can find. We will add this analysis to our revised manuscript.

In addition, regarding to the questions,

Why does the highest MO energy in Table 3 show a negative improvement? MPGD should be a special case of TFG. Is this because the grid-searched hyperparameters for TFG are not optimal?

We appreciate the reviewer for carefully examining our paper, and we want to point out that the reason TFG could occasionally have slightly worse performance in practice is due to the beam search computation limit we currently pose. More specifically, we allow TFG to search at most six steps (for all hyper-parameters) and all other methods for seven steps (in their subspaces). For the MO energy task, the searched parameter for MPGD is that $\bar \mu = 0.016$ (this is the only parameter that we need to search for MPGD), where the best (and last step) of TFG is that $\bar \mu = 0.004$ (because it uses one step to double another parameter). If we allocate more computational budget for the beam search steps, TFG will outperform MPGD on this target (in fact, eight steps suffice).

Can the author comment on other lines of work [1,2,3,4], specifically direct optimization approaches, on training-free optimization of diffusion models with a target objective? The task setting is the same as training-free guidance. Although it is evident that the direct optimization approaches are slower than training-free guidance, it is not clear how their performance compares.

Thanks for the valuable suggestion. We review all papers to understand the direct noise optimization approach (DNO), and we believe that this DNO has a different motivation than training-free guidance. Specifically, DNO is not only slow but also GPU-memory intensive, as gradients have to be propagated through the entire ODE process for multiple times until convergence. This makes it hard to implement and study within a short amount of time. We would like to leave the comparison to future studies, and we sincerely hope you can understand the difficulty.

Again, we sincerely thanks for the helpful review you provide. Please let us know whether there is any additional concern that we could address to help improve your evaluation to our work.

Dear Reviewer J9FB,

The deadline of the discussion period is soon approaching. We wonder whether our answers to your questions have addressed your concerns. If there are any additional discussion points or questions, we are happy to discuss.

We look forward to your comments. Thank you again for your time.

Best,

Authors of Paper8134

Thank you for the insightful and constructive review, and we are delighted that you think our work will greatly benefit the diffusion community. Regarding to the concerning questions:

Algorithm 1 seems to be one of the most important parts of this paper, but it lacks a detailed explanation. An explanation of Algorithm 1 along with the corresponding hyperparameters from Definition 3.1 in Section 3.1 would greatly aid understanding.

We agree that more explanation w.r.t. Alg 1 and the hyperparameter space will help with the understanding. We will incorporate the explanation of the relation between Alg 1 and Def 3.1: generally speaking, Alg 1 contains three operations (Mean Guidance, Variance Guidance, and Implicit Dynamics), and each operation in the algorithm separately control part of the hyper-parameter in Def 3.1. We will explicitly point how this relations, and how each operation in the algorithm affects the parameter space, i.e., what part of the space is controlled.

In Algorithm 1, the iteration part in line 8 could be explicitly described using a for loop in the algorithm, rather than as a comment.

We agree that explicitly writing down the for loop will make it more clear. Thanks for the suggestion.

I believe that Theorem 3.2 and its proof are not well formulated mathematically. If the authors want to formalize it as a theorem, they need to show mathematically that the same generated distribution can be produced. In my opinion, an explanation at an analytical level would be acceptable.

We really appreciate the suggestion, and we believe that keeping it as a theorem would still be helpful. That said, we will make the definition and proof more mathematically formal. Specifically, we will explicitly define the parameter space of each of the existing algorithms, and pointing out that each instantiation in their distribution corresponds to which parameter value in our parameter space.

Again, we sincerely thanks for the helpful review you provide. Please let us know whether there is any additional concern that we could address to help improve your evaluation to our work.

Dear Reviewer DCpp,

The deadline of the discussion period is soon approaching. We wonder whether our answers to your questions have addressed your concerns. If there are any additional discussion points or questions, we are happy to discuss.

We look forward to your comments. Thank you again for your time.

Best,

Authors of Paper8134