Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models

Thank you for the review. We will next address the explicit questions:

“Does the optimal interval vary with the model's performance? The results in Table 1 imply this, as the optimal intervals for EDM2 and DiT differ, and the optimal guidance weight for DiT is higher. If the optimal interval varies with model performance, the proposed method will require an exhaustive hyperparameter search for each diffusion model, which is inconvenient.”

Yes, based on our experiments with EDM2 and DiT the optimal guidance interval appears to be model specific, and it probably depends on the model’s performance as well. Finding the optimal sampling hyperparameters requires only model evaluations, not re-training, but it can indeed become somewhat expensive if done in a naive fashion. However, there are several ways to reduce the cost.

First, the upper and lower guidance limits can be determined separately, without the need for a two-dimensional search. This happens by first establishing the optimal upper limit by keeping the lower limit as zero, which is very close to optimal except for the computational cost (see Figure 4, right). Once the optimal upper limit is known, the lower limit can be determined afterwards. Second, the lower limit seems to affect the output in a monotonic fashion, making binary search applicable there. Finally, one can reduce the sample size of FID evaluation from 50k to, say, 5k, at least for an initial run, which accelerates the process by 10x.

“Will guidance at high noise levels be detrimental for conditions denser than the class and text conditions considered in this paper? Examples include InstructPix2Pix [A] using image conditions, ControlNet [B] using various spatial conditions, and Stable Video Diffusion [C] generating videos from images. Additionally, a text-to-image model based on the T5 encoder (DeepFloyd IF [D]), which provides richer text representation than the CLIP text encoder of Stable Diffusion XL, is also an example.”

Thus far, we have not experimented with conditioning signals other than class and text conditioning using CLIP. It is likely that major changes to the conditioning scheme will impact the optimal guidance interval.

“Looking at Figures 1 and 2, the guidance scale at each interval significantly impacts the ODE trajectory. Do the authors think this tendency would also be observed in flow matching generative models [E] (or rectified flows [F]) that learn straight trajectories, compared to previous VP and VE diffusion models?”

As discussed in [1], the trajectories generated by flow matching (FM) for a white noise latent follow the score function of progressively less noisy distributions (Eq. 7 in [1]), similar to diffusion. Thus, guidance has a similar statistical interpretation and behavior in both diffusion and flow matching.

The trajectories in FM are straight in the sense that they don't take entirely unnecessary detours, but they are still necessarily curved for any non-trivial data. This can be seen, e.g., in the first rectified flows of Figs. 3 and 4 of [2]. The EDM sampler used in our experiments is similarly designed to avoid unnecessary trajectory curvature to allow efficient sampling. We therefore expect that our findings translate to flow matching.

That said, FM allows for curious variants where the latent distribution is, e.g., another set of images instead of the typical Gaussian noise. To our knowledge, all state of the art applications of FM use noise latents, but the behavior of CFG in these more exotic settings seems to be a largely unexplored topic, and it is unclear if it (or guidance interval) would be applicable at all. Similarly, iterated reflows [2] would potentially behave differently.

[1] Zheng et al.: Guided Flows for Generative Modeling and Decision Making. Arxiv preprint, 2023.
[2] Liu et al.: Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow. In Proc. ICLR, 2023.

Thank you for the review. We will next address concerns and the explicit questions:

“The main weakness of the work is a lack of discussion of existing works that address the issues of high-guidance scales.”

Thank you for providing pointers to additional relevant previous work. We are glad to cite them. All of these papers propose guidance schedules that are continuous at high noise levels to enhance the results of their (unrelated) main methods, without evaluating the effects of these in isolation (apart from [1]). While our guidance interval can be seen as a step function schedule, this schedule was not advocated in any of these works. The closest to our method is Dynamic CFG [1] that modulates guidance weight linearly in an interval that mostly covers high noise levels. In contrast, we found guidance to be strictly detrimental in these regions, and disabling it there leads to our state-of-the-art FIDs.

In our initial experiments, we tried a linear schedule before finding it to be clearly worse than the simpler guidance interval. With EDM2-S, we obtained FID 2.02 at best with a linear schedule, whereas guidance interval achieves FID 1.68. Therefore, we focused the paper specifically on studying the guidance interval and its effects on the resulting image distribution. We also provide novel data confirming that guidance is harmful at high noise levels through our toy example and experiments with EDM2 and SD-XL.

[1] Sadat et al.: CADS: Unleashing the Diversity of Diffusion Models through Condition-Annealed Sampling. In Proc. ICLR 2024.

“1. Is Figure 3 compiled with a fixed EMA profile for the EDM2 model or is the EMA profile also optimized along with the guidance parameters?”

We use a fixed EMA profile in Figure 3. For EDM2-S, we use EMA lengths 2.5% and 8.5% respectively for FID and FD_DINOv2, and for EDM2-XXL, we use EMA lengths 1.5% for both FID and FD_DINOv2. To determine these, we ran a sweep over the EMA lengths and found that the EMA length that works best with our guidance interval also works best with CFG.

“2. Can you explain lines 112-118 in more detail? Does that mean you always use the low and high guidance scales based on the sampling discretization? This part was not very clear to me.”

This is correct. This part means that we either enable or disable CFG at the noise levels given by the noise level discretization of the sampler.

“3. Can you also plot Precision/Recall curves for the method to show the impact of interval guidance on quality and diversity in isolation?”

We are happy to add a plot with precision and recall curves. They clearly show that our approach allows increasing precision with a smaller penalty in recall in comparison to CFG.

“4. One major advantage of high classifier-free guidance is image-text alignment. Does the proposed method have a noticeable impact in this area? For example, in Figure 7, it seems that some parts of the prompt are more strongly emphasized by the normal CFG (e.g., the sky is orange).”

We have not performed a quantitative experiment using text-image alignment metrics, but generally, good text-image alignment seems to require roughly similar (i.e., quite high) guidance weights for both traditional CFG and our method. Prompts that benefit from added saturation and bright colors may work better with traditional CFG, as the artifacts that our method aims to remove may end up strengthening the alignment.

“5. Does the optimal guidance interval change when modifying the sampling algorithm, or are the thresholds mainly model dependent?”

The guidance interval is not overly sensitive to the sampler parameters, at least as long as the sampling algorithm remains the same (see 3rd paragraph of 4.2). We did not experiment with different sampling algorithms (DDIM, Euler, etc.) because the current state-of-the-art results are obtained with a 2nd order Heun sampler with 32 step deterministic steps. We expect that the optimal guidance interval depends mainly on the model.

Respectfully, this terse review appears to present a personal preference, but makes no factually supported arguments to rebut. CFG is a crucial but poorly understood component of diffusion image generators, and despite its apparent simplicity, our technique extracts its key benefits while significantly suppressing its downsides.