No Training, No Problem: Rethinking Classifier-Free Guidance for Diffusion Models

We thank the reviewer for the positive feedback on the clarity, novelty, and strong experimental evaluation of our paper. Below, we address the concerns and questions raised, and we hope that our responses satisfactorily address the reviewer’s comments.

Theory behind TSG

We agree with the reviewer that the simple linear approximation often does not hold in practice. Therefore, the Langevin dynamics interpretation serves merely as an intuitive explanation for why TSG may work under idealized conditions. We believe that, in reality, TSG's influence on sampling is non-linear and involves more changes to the sampling trajectory.

Comment about the effectiveness of TSG

We partially agree with the reviewer that TSG performs error correction by filtering overly noisy and soft sampling trajectories. However, we also observed that TSG can alter the overall structure of the generated image compared to the initial unguided sampling, though not as significantly as CFG. Therefore, we conclude that the effectiveness of TSG arises from a combination of error correction and guidance.

TSG for diffusion models without class labels

We tested TSG on Stable Diffusion, which integrates conditioning into the diffusion model via cross-attention instead of adding the condition to the time-step embedding. In this case, TSG still performs effectively, as shown in Table 3. Thus, we conclude that the effectiveness of TSG is not limited by this assumption. We would be happy to evaluate the UViT model and consider including this benchmark in the final version of the paper.

NFE and samplers for the FID

We will include these details in the final version of the paper. Our experiments used multiple samplers and NFEs for each model, demonstrating that ICG/TSG are effective across various samplers and NFEs in practice.

Formulation of the guidance

There are two equivalent formulations for CFG. The difference depends on whether we consider $w=0$ or $w=1$ for the unguided case. More specifically,

Hence, if we define $w' = w - 1$, the CFG formulation in the paper becomes equivalent to what the reviewer mentioned in the comment.

Undefined $\beta_t$

We thank the reviewer for this suggestion. We will include a description of this parameter in the final version, i.e., it controls the influence of noise during the sampling process.

Question about changes in the score estimation

We noted that, in practice, the estimation of the score function varies slightly depending on the random condition used as input to the network. We conducted a follow-up experiment to assess how much the sampling distribution changes when we alter the random seed for selecting $\hat{y}$ in ICG—while keeping other random factors, such as the initial noise $z_1$, exactly the same. The results are provided below.

We observe that the final FID is relatively insensitive to this variation. Therefore, we conclude that ICG is not sensitive to this design choice. Please also refer to the variance analysis section of our response to Reviewer 8vmw for a more detailed answer to this question.

thanks for the reply, I have raised my rating. Please also add the details in the future version of the paper

We appreciate the reviewer’s positive feedback on our paper, highlighting its clear writing, strong motivation, potential significant contributions, and robust experimental results. Below, we provide responses to the questions and comments raised and remain open to further discussion.

Clarification on the theory of ICG

We thank the reviewer for highlighting this potential area of confusion regarding the theoretical justification for ICG. We would like to clarify this point and are open to further discussion. First, we acknowledge that there is a gap between the idealized theoretical scenario described in Equation 7 and the reality of practical implementation. Specifically, Equation 7 assumes access to exact distributions. In this ideal case, if we have a conditional distribution $p(z_t | y)$ and choose a random vector $\\hat{y}$ independent of $z_t$, we would have $p(z_t | \\hat{y}) = p(z_t)$ by definition. However, when these distributions are implicitly estimated via score-based models such as conditional denoisers, the output of the denoiser, $D_{\\theta}(z_t, t, \\hat{y}) = \\mathbb{E}[x | z_t, \\hat{y}]$, is necessarily a function of the input arguments. We believe that it is here that the confusion in our claim is introduced. Note that while the output of the conditional denoiser (an estimate of the clean data $x$) is technically a function of the (random) condition, and is therefore not independent of it, our analysis in (7) relies only on the independence of the noisy input ($z_t$) and the condition.

With real-world, limited-capacity denoisers, we concede that it will generally be the case that $D_{\\theta}(z_t, t, \\hat{y}) \\neq D_{\\theta}(z_t, t, y_{\\text{null}})$. That ICG works despite this difference offers insight into whether the exact unconditional component is necessary for performing CFG during sampling. Our experimental results show that by using simple formulations for each model (e.g., randomly sampling a class label), we can replicate CFG’s behavior without explicitly needing the exact unconditional score function. Even so, consistent with (15), we can interpret each random-conditional score as an unbiased estimate of the unconditional score. The results of ICG suggest that while we are not necessarily following CFG’s prescribed direction exactly at each step, both methods yield similar target distributions. However, ICG has the advantage of a simplified training process and potentially faster convergence, as shown in Figure 3, by removing the CFG label-dropout objective.

The theoretical insights provided in the text and appendix are meant to offer intuition on why ICG may work in practice, without claiming this is the exact process occurring with practical denoisers. We believe we have collected considerable experimental evidence to validate the concept across various models and datasets. Nevertheless, we are grateful that the reviewer has identified an area in need of clarification, and we will modify Section 4 accordingly.

Empirical evidence supporting TSG

TSG is designed as a guidance method with minimal assumptions about the network architecture, making it applicable to all diffusion models (unlike SAG for instance, which relies on self-attention layers). Nonetheless, we conducted experiments comparing TSG with SAG and PAG using Stable Diffusion. The results provided below show that TSG outperforms both methods.

[1] Hong, Susung, Gyuseong Lee, Wooseok Jang and Seung Wook Kim. “Improving Sample Quality of Diffusion Models Using Self-Attention Guidance.” 2023 IEEE/CVF International Conference on Computer Vision (ICCV) (2022): 7428-7437.

[2] Ahn, Donghoon, Hyoungwon Cho, Jaewon Min, Wooseok Jang, Jungwoo Kim, SeonHwa Kim, Hyun Hee Park, Kyong Hwan Jin and Seungryong Kim. “Self-Rectifying Diffusion Sampling with Perturbed-Attention Guidance.” ArXiv abs/2403.17377 (2024): n. pag.

Baselines for ICG

ICG aims at replicating the behavior of CFG, which is why CFG is used as the ground truth for comparison. For this reason, the method is compared exclusively with CFG. If the reviewer has a specific benchmark they believe ICG should be evaluated against, we would be happy to include it in the final version.

Distribution for $\hat{y}$

Our goal is to identify independent conditions that can be fed into the model. For class labels and text tokens, this involves using random class labels or random text tokens, which can be generated through random integer sampling. However, we believe this is not the only approach for selecting random conditions, as demonstrated in our ablation study using a Gaussian noise vector. Exploring the optimal distribution for $\hat{y}$ represents an interesting direction for future work.

Hyperparameters used in the paper

Since ICG/TSG are inference methods, the hyperparameters were selected by generating a small set of images and using visual inspection. We did not conduct tuning on a specific dataset or perform a grid search. It is quite possible that better hyperparameters exist for each model.

Code availability

Unfortunately, due to internal copyright policies, we cannot share the full code for the paper. However, beyond the implementation details already provided, we will revisit Section E in the final version of the paper to ensure reproducibility.

Comparing ICG with CADS

CADS is a method to enhance the diversity of CFG, serving as an addition to regular CFG, whereas ICG is intended to replicate the base CFG behavior. Therefore, a direct comparison between CADS and ICG is not possible, as CADS is not designed to function independently without CFG. However, we can include the comparison between CFG + CADS and ICG + CADS as shown below, where we note that ICG + CADS slightly outperform CFG + CADS in terms of FID.

Connection between TSG and Langevin dynamics

This part aims to show that if we use an ODE as the diffusion solver and use TSG-guided output instead of the output of the diffusion model, under the idealized linearity assumption, the ODE update step will be transformed to an SDE. Since SDE-steps have been seen to improve the sampling quality due to error correction, we concluded that TSG-steps are to some extent following the same logic (at least under this idealized assumption). However, we agree that in practice, TSG is doing non-linear corrections to the trajectories, and its effect is closer to CFG rather than changing the ODE to SDE. The DiT model uses an SDE sampler (DDPM), and we still observed significant improvements by using TSG on top of this sampler.

Question about $s$ and $\alpha$ in TSG

$s$ is intended to scale the noise such that its variance aligns with the scale of the time-step embedding. We observed that deviating too far from the time-step embedding disrupts the diffusion model's ability to perform denoising correctly. Meanwhile, $\alpha$ is designed to progressively reduce the perturbation to zero as denoising advances, limiting guidance in the final steps. The parameters were selected through visual inspection and were not tuned on any specific dataset.

Contrast in ICG samples

Since ICG updates differ slightly from CFG at each step, the effect of the guidance weight also varies. The figures were generated using the same guidance scale for both ICG and CFG. By slightly lowering the guidance scale for ICG, we can achieve the same contrast and saturation levels across both images, as ICG updates are observed to be slightly stronger than those of CFG.

Explanation for Figure 3

We do not claim that ICG is better than CFG. In Figure 3, our goal is to demonstrate that the training budget allocated to the CFG objective could be better utilized to train the conditional model itself. The figure shows that applying ICG to a purely conditional model results in better FID at each checkpoint compared to a model that dedicates 10% of its training to the CFG label-dropout objective. This suggests that focusing all training resources on the conditional model and using ICG at inference can achieve CFG-like benefits without any training requirements. We observed no sensitivity of these results to the hyperparameters, with the performance improvement coming from the ICG model being effectively trained more due to not including the CFG objective.

Question about Figure 5

For Figure 5, we evaluated ICG on image-conditioned models, specifically for pose-to-image and depth-to-image generation. We chose ControlNet as a good open-source model for these tasks. Our intention is not to suggest ignoring text conditions for this specific model, but rather to demonstrate that the effect of ICG is not limited to class labels or text conditions. In this context, ControlNet without text serves as an example of a pure image-to-image model, accommodating scenarios where such models may not have access to text conditions.

Minor comments

We will review the manuscript thoroughly and incorporate the necessary revisions in the final version to address the reviewer's comments.

We thank the reviewer for providing valuable and constructive additional feedback on the paper. Below, we address each individual comment and question in detail. Additionally, we invite the reviewer to refer to the global comment for further information.

Comment on hyperparameter selection

We are happy to include our rule of thumb for selecting hyperparameters in the final version of the paper. The noise scale $s$ was chosen by observing the standard deviation of the time step embedding $t_{emb}$ and selecting values around it. For instance, choosing $s = 2 \times \text{std}(t_{emb})$. The parameter $\alpha$ controls how the perturbation should change over the denoising process. We observed that values within the range $[0.25, 1.5]$ generally work well across different settings. Higher values of $\alpha$ result in less perturbation, keeping the results closer to the base model.

As shown in the ablation studies, excessive perturbation may reduce FID, whereas insufficient perturbation often yields only minor improvements over the baseline. For each model, the exact hyperparameter values were selected by generating a small batch of images under various parameter settings and performing a quick visual inspection.

Comment on ICG theory

We are grateful to the reviewer for pressing us on this point, as further analysis has revealed what we believe to be a much more satisfying theoretical explanation for ICG’s success in providing the benefits of CFG without an unconditional model. Please see our general response above under “Bounding the Error on the Unconditional Score Approximation.”

The reviewer correctly points out that independence is a property of the distribution $p$ and not the choice of condition. It might be helpful to mention our original intuition, which takes some inspiration from permutation testing for independence. Ignoring the noisy $z$ variables for the moment, consider the distribution $p(x,y)$ representing the original data and their corresponding conditions. This is represented implicitly in the paired data $\{(x_i,y_i)\}$. In permutation testing, one shuffles the data to form new pairs $\{(x_i,y_j)\}$ for $i \neq j$ in order to break the dependencies. Such permutations create a new distribution $\tilde{p}(x,y) = p(x)p(y)$ in which $x$ and $y$ are independent by design. This motivated our approach to ICG, and when we empirically found that it works across a variety of models and datasets, the original intuition continued to make sense. However, the actual situation turns out to be more nuanced, which we address in the above-referenced global comment.

Comment on ICG bias

If one samples a condition from $p(y|z)$, then the resulting conditional score is indeed an unbiased estimate of the unconditional score. A second condition in which the random-conditional score is an unbiased estimate of the score is when $z$ and $y$ are independent. But as we acknowledge in the point directly above and address in our global comment on the score estimate, neither condition exactly applies in our case. As a result, our referring to the random-conditional score in our original rebuttal remarks as an unbiased estimate of the unconditional score is true only in idealized circumstances that do not necessarily apply in practice. We thank the reviewer for challenging this point. Please also see our global comment above.

We thank the reviewer for finding the paper well-written, with extensive experiments and good ablations. We hope that the comments below satisfactorily address the reviewer’s questions and concerns.

We would like to begin with the main question the reviewer poses, namely whether the unconditional score is necessary for obtaining the quality-boosting benefits of CFG. We believe that our work demonstrates that the unconditional score is not necessary. We have demonstrated that independent condition guidance (ICG) is sufficient for achieving CFG-like benefits across various models and datasets without ever training an unconditional model in any form.

We have also provided detailed responses to each question and comment below, and we welcome further discussion.

Variance analysis

We thank the reviewer for raising this question. We address it in two parts:

1. How much each intermediate sampling step $z_t$ differs between ICG and CFG.
2. How much the final sampled distribution is affected by these differences.

For the first part, we define the relative error for each trajectory as $e_t = \\frac{\\Vert z_t^{\\mathrm{ICG}} - z_t^{\\mathrm{CFG}}\\Vert}{\\Vert z_t^{\\mathrm{CFG}} \\Vert}.$

We computed $e_t$ for different random class labels as our choice of $\\hat{y}$ in ICG using the DiT model and observed that it is on average below 0.0025 (or 0.25%) across all time steps.

For the second question, we measured how different random seeds for choosing $\\hat{y}$ in ICG affect the final distribution, using CFG as the baseline and 10,000 generated samples. The results are as follows:

The results above indicate that the ICG trajectory is very close to the CFG sampling trajectory, and that slight variations between CFG and ICG at each step do not lead to any noticeable change in the target distribution.

ICG analysis based on a mixture of Gaussians

If we assume a Gaussian distribution for each class, with

then after the forward diffusion process, the conditional distributions become:

and the unconditional probability is equal to (assuming uniform prior for each class)

The conditional score functions in this case are given by:

The unconditional score can be expressed as:

In this case, the CFG update direction will simplify to

Next, let’s compute the ICG update. For this analysis, we assume that the random label $\\hat{y}$ is drawn from a Bernoulli distribution, i.e., $p(\\hat{y} = 0) = p_0$. In expectation, the ICG estimate of the unconditional score is equal to

This gives us

This analysis shows that in theory, there is a certain choice for $p(\\hat{y} = 0)$ such that the update direction of ICG and CFG exactly match in expectation. In every case, however, the expected update is always in the same direction as the CFG update rule.

In real-world problems, finding the categorical distribution that exactly matches the magnitude of the CFG direction is not straightforward. However, we experimentally showed that using a simple uniform distribution for $p(\\hat{y})$ is good enough to simulate the behavior of CFG. We will include a more detailed and generalized version of this analysis in the final version of the paper.

Question about the number of steps for ICG

ICG requires the same number of steps as CFG. For Table 1 and 2, we tested the models for several NFEs and ICG results were similar or better than CFG across all of our experiments.

We appreciate the reviewer’s time in considering our rebuttal and engaging in the discussion. Please find our responses below.

Comparison with Autoguidance

Autoguidance [1] is a concurrent work that leverages a second, intentionally inferior diffusion model to guide sampling in order to address some of CFG’s drawbacks, such as low sample diversity. By contrast, ICG simplifies the application of CFG by eliminating its dependency on an unconditional model but still shows the same quality-boosting benefits. While autoguidance, like our method, does not require an unconditional model, it does require training a smaller version of the conditional model for part of the original training process, making it unsuitable for direct application to pre-trained models, unlike ICG. For text-to-image models, the autoguidance paper suggests that the benefits of CFG and autoguidance are complementary, leading the authors to combine both methods for potentially improved text-image alignment. In this context, ICG could serve as a replacement for the CFG component in such combinations. Also, as we show in the paper, ICG is compatible with diversity-boosting techniques such as CADS.

FID in Table 2

The FID reported in the EDM2 paper was achieved through an extensive grid search across various combinations of guidance weights and EMA profiles for both conditional and unconditional models. Due to computational constraints, performing an equivalent grid search for ICG was not feasible. Instead, we compared the models using a different guidance scale (not optimized for FID) to ensure a fair evaluation. It is important to note, however, that both models achieve identical FD_Dino scores, which have been shown to correlate more closely with human judgment. We conclude that it is possible to identify specific parameters for ICG (in terms of EMA profiles + guidance scales) that replicate the FID reported in the original paper.

Variance analysis and the support of the distributions

The analysis was performed using a batch size of 32, with 10 different random seeds for choosing the random condition in ICG. We can perform the same analysis and compute the error for the score estimation defined as

In this case, the same computation gives us around 0.049 error for $e_t$ averaged over time steps. Note that the primary aspects of interest when sampling from diffusion models are the final modeled distribution and the sampling trajectories rather than the precise intermediate model predictions. Our empirical results show that slight variations in model predictions can still lead to the same final target distribution, as evidenced by almost the same FID values achieved by ICG and CFG.

We can also follow the error analysis using the GMM model. In this case, the error term for each prediction is given by:

Accordingly, the average error (in terms of norm) w.r.t. $p(\\hat{y})$ will be equal to:

Recall that to match the expectation, one should choose

which gives us

In this idealized formulation, if the distributions are relatively disjoint, the expected error term vanishes, as it depends on the product $p(z_t \\mid y=0) p(z_t \\mid y=1)$. We hope this explanation clarifies the reviewers' concerns regarding disjoint class distributions. Given the time constraints for the rebuttal, we would be happy to include a more detailed numerical error analysis (using both images and 2D distributions) in the final version of the paper.

We thank the reviewer for taking the time to consider our response. Below, we provide a detailed reply to the specific comments raised, while also referring to the global comment above for additional context.

Comment about bias in the score estimation

The intuition of the reviewer is correct that ICG is only unbiased under some idealized circumstances. We have revisited our analysis of the unconditional score estimator and have what we believe is a more satisfying theoretical perspective on it in our global remark under “Bounding the Error on the Unconditional Score Approximation.” We thank the reviewer for pressing this point, and we will include this bias analysis in the final version of the paper.

Comment about FD_DINOv2

We will revise the CFG FID results for EDM2 to ensure a fair comparison. The EDM2 authors’ hyperparameter search had results that are both method (EDM, CFG) and metric (FID, FD_DINOv2) specific. The choices that optimize one metric will not necessarily optimize the other, nor will they transfer across methods. The default settings in the EDM2 codebase allow us to reproduce the best FD_DINOv2 results reported in the original paper using CFG, which we are able to slightly improve upon using ICG with no separate hyperparameter search. Thus, the non-optimized setting we mentioned in our previous response only applies to FID.

In this case, the mean and std functions do not include a batch dimension and instead focus on the scales along the embedding dimension (as each embedding is a $1 \times D$ vector). We did not find this to be an issue in practice.

I appreciate the analysis provided by the authors and their thorough responses. I will raise my score after an answer to the following questions. I appreciate the authors patience

While the mixture of Gaussians is a toy example, the disjoint support can easily arise solving inverse problems, or image-to-image generation, where $p(z_t | y) \approx 0$ for $y$ sampled independently, can the authors run the following experiment:

1. Take N conditions y where $y$ is an image vector, the authors can use their existing experimental setup for this as well.
2. For each t in $[T, \dots, 1, 0]$, compute the norm of the scores $s_\theta(z_t, y)$ and $s_\theta(z_t, \widehat{y})$.
3. report the average norm for each t for both scores. The reason I am asking for this last experiment is that I suspect that the norm of $s_\theta(z_t, \widehat{y})$ is near zero, which would imply ICG is equivalent to running scaled conditional guidance for some settings.

Finally, can the authors provide some thoughts on the following questions:

1. If we are doing text to image generation, does ICG require us to keep a bank of text prompts for generation?
2. If we are doing image-to-image generation, does ICG require a bank of images for generation?
3. From their analysis, it seems that the authors would recommend sampling from a Normal distribution around $N(encoder(y), \sigma^2 I_d)$ or should we compute a global mean to be user for any conditioning variable $y$ as the paper suggested?

As it stands, there are other approaches which propose alternatives to using a marginal score for guidance and have a clear algorithm and implementation. I think if the authors provide some analysis on different approaches for sampling $y$ and some analysis of the error, it would make the paper much stronger.

We believe that we can address both of the reviewer’s questions by describing our procedure for the requested experiment. In the pose-to-image model, the pose is represented as a stick-figure image, which is a high-dimensional vector. This corresponds to the true condition $y$. We compute the (scalar) mean and standard deviation of the “pixel values” of $y$. To form $\\hat{y}$, we sample a Gaussian with the same mean and standard deviation to fill out a vector of the corresponding size. Each dimension of this vector is independently sampled from a Gaussian, so we are not perturbing $y$. This process is repeated at each sampling time step to produce a new $\hat{y}$ drawn from the same Gaussian. The plots we presented are the result of repeating this experiment over 16 pose conditions and averaging the norms of the scores calculated in each step.

Thank you for adding the requested plot.

1. how are the authors computing these scores? can they provide some details on how the curves were generated?
2. If the conditioning vector $y$ is not a scalar, but an image or text so a high-dimensional vector, so the method would use a Gaussian perturbation of $y_{\text{embed}}$ when there is a batch size of 1? I am surprised that is any different from conditional guidance.

For illustration, let's say that a single condition $y$ is a $32 \\times 32$ "image." We reduce these 1024 numbers down to a scalar mean and a scalar standard deviation. (That is, these two statistics are computed across the 1024 dimensions of $y$ and not over a batch of conditions.) Let's say the mean and standard deviation are 0.24 and 2, respectively. We then sample 1024 i.i.d. samples from $\\mathcal{N}(0.24,2^2)$ to fill out the $32 \\times 32$ random condition $\\hat{y}$.

Since in this example the two scalar parameters of the Gaussian used to sample $\\hat{y}$ are derived from the 1024-dimensional $y$, the reviewer is correct that $\\hat{y}$ is technically dependent on $y$ (and, by extension, $z\_t$), albeit very weakly. However, this procedure is done only to conveniently set the scale for sampling $\\hat{y}$ to keep it in the range of values the model saw during training. It is not actually a requirement of our method. One can also sample $\\hat{y}$ from a Gaussian with no reference to $y$ whatsoever as long as the range of values is reasonable.

We are glad that the reviewer's impression of the paper remains positive, and we appreciate the level of interest shown to what makes it work so well. We assure the reviewer that these subtler points will be revisited and clarified in the final version of the paper.