Smoothed Energy Guidance: Guiding Diffusion Models with Reduced Energy Curvature of Attention

We would first like to thank the reviewer for acknowledging the strength of our approach in its versatility. We demonstrate SEG's effectiveness in both unconditional and conditional settings, including text-conditional generation and ControlNet conditioning. This flexibility allows SEG to improve image quality across various generation tasks without requiring task-specific modifications. Additionally, we appreciate the reviewer's thoughtful comments and questions. We would like to address the main points raised.

Criteria for assessing image quality

While we guide readers more towards qualitative assessment, we do employ multiple approaches to evaluate image quality. Quantitatively, we use FID and CLIP scores to measure sample quality. To assess unintended side effects, we utilize LPIPS scores to quantify deviations from unguided images. Qualitatively, we present extensive visual comparisons (e.g., Figs. 2-5 and 7-10 in the main paper) that demonstrate improvements in definition, expression, sharpness of details, realism of textures, and overall composition. We believe this multi-faceted approach provides a comprehensive evaluation of our method's effectiveness. For overall improvement, we have also included uncurated samples from the Vanilla SDXL model, both without and with SEG, in Fig. 2 of the attached PDF.

Besides, as mentioned in a concurrent work [A], the FD-DINOv2 metric is another means to calculate Fréchet distances and is well-aligned with human perception. For reference, in the table below, we present FD-DINOv2 scores calculated using 50k samples from the EDM2-S model trained on ImageNet-64 to assess fidelity. We also include uncurated qualitative samples from this model in Fig. 3 of the attached PDF. This corroborates how the structure and quality of samples change, as well as the generality of our methods. Model | FD-DINOv2$\downarrow$ --- | --- No guidance | 95.1915 SEG ($\sigma\to\infty$) | 47.4733

[A] Karras, Tero, et al. "Analyzing and improving the training dynamics of diffusion models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Relationship between energy curvature reduction and guidance

We appreciate the suggestion to clarify this relationship. On a high level, CFG uses the difference between the prediction based on the sharper conditional distribution and the prediction based on the smoother unconditional distribution to guide the sampling process. By analogy, SEG reduces the curvature of the energy landscape underlying self-attention (Theorem 3.1). This creates a smoother landscape for a minimization step of attention during sampling, analogous to how classifier-free guidance uses the difference between conditional and unconditional distributions. By using the "blunter" prediction from this smoother landscape as negative guidance, SEG enhances sample quality without relying on external conditions or special training.

From a probabilistic perspective, this process can be thought of as maximizing the likelihood of the attention weights in terms of the Boltzmann distribution conditioned on a given configuration, i.e., the feature map. Blurring the attention weights diminishes this likelihood, as shown in Lemma 3.2, and also reduces the curvature of the distribution, as shown in Theorem 3.1.

General response and additional figures

We respectfully refer the reviewer to our general response and additional figures provided above. This material addresses key points raised in the initial review and highlights the strengths of our paper, as noted by other reviewers. Additionally, we have included new figures and results that we believe may address your concerns.

We hope these clarifications address the reviewer's concerns and highlight the strengths and contributions of our work. We're happy to provide any additional information or clarifications if needed.

Dear Reviewer cNNp,

Thank you again for your time and effort in reviewing our manuscript. We have posted our response addressing your concerns and suggestions.

If you have any additional questions or require further clarification, we are happy to discuss them. We eagerly await your valuable feedback.

Best regards,

Authors of Submission #4721

We sincerely appreciate your acknowledgment of our approach's strengths, particularly its elegant idea and thorough qualitative and quantitative evaluations. Thank you for your careful suggestions. We'd like to address the concerns and questions you've raised:

Improving conditional generation

We'd like to highlight that our method improves conditional generation even without CFG, thanks to the smoothed energy curvature. When used in its conditional version as a replacement for CFG, rather than being used in combination with CFG, it reduces the number of function evaluations by one. We conducted experiments with the class-conditional EDM2-S model trained on ImageNet-64. For reference, the table below presents FD-DINOv2 scores calculated from 50k samples using this model to assess fidelity. We've also included uncurated qualitative samples in Fig. 3 of the attached PDF. | Model | FD-DINOv2$\downarrow$ | |-------|-----------| | No guidance | 95.1915 | | SEG ($\sigma \to \infty$) | 47.4733 |

It's important to note that our method doesn't propose to blur the output directly, and Gaussian blur on score prediction itself reduces the noise level. Instead, SEG applies Gaussian blur to attention weights. This process still requires a partial forward pass, even when reusing features before the attention weights for efficiency, which incurs slightly more overhead than using the same prediction twice.

In addition, when it comes to general text-to-image generation like Stable Diffusion, the interface between the given caption and generated image is mostly the cross-attention. However, even though our method can be applied to cross-attention, and even temporal attention (in text-to-video generation, which indeed works to improve consistency), we deal only with self-attention in this paper. We believe the ideas you suggested are promising future directions, though, and those directions will be emphasized in our revised version.

Pipeline figure

We've included a draft of the pipeline figure (Fig. 4 in the attached PDF) and will incorporate this in the revision. This includes where the Gaussian blur with $\sigma$ is applied, the original and blurred attention weights, the associated energy landscape, and how the score predictions are obtained. We also include in Fig. 4(c) how the linear extrapolation between those predictions in SEG works.

Adding intuition to Lemma 3.1

While those familiar with Gaussian blur in the computer vision community may intuitively know this fact, we thought a clearer explanation was needed. However, we also find it intuitive that Gaussian blur preserves the mean while decreasing variance, since applying Gaussian blur to images causes the pixels to converge towards a similar value, reducing variance. Therefore, we will consider moving the formal proof to the appendix and instead include this intuition in the main text.

Definition of FID

We understand your point about FID. While it doesn't directly measure image quality, we use the metric as a proxy for realism and quality because it measures the distance to the real image-text distribution of the COCO dataset. We'll clarify this in the final revision.

Other concerns and suggestions

We acknowledge the typo in line 211 and confirm that SAG and PAG are indeed training- and condition-free. Also, we've incorporated most of your suggestions to improve the paper's clarity in our revised manuscript.

General response and additional figures

We respectfully refer the reviewer to our general response and additional figures provided above. This material addresses key points raised in the initial review and highlights the strengths of our paper, as noted by other reviewers. Additionally, we have included new figures and results that we believe may address your concerns.

Thank you again for your valuable feedback. We believe these changes will help us fine-tune the quality and clarity of our paper.

Dear Reviewer CTSN,

Thank you again for your time and effort in reviewing our manuscript. We have posted our response addressing your concerns and suggestions.

If you have any additional questions or require further clarification, we are happy to discuss them. We eagerly await your valuable feedback.

Best regards,

Authors of Submission #4721

We appreciate your thoughtful review of our paper. Below is our response to your review:

When sigma approaches 0

SEG with $\sigma\to 0$ is equivalent to the original sampling process. This doesn't necessarily mean $\\tilde{s}\_\\theta(x, t)$ goes to 0, but rather that the Gaussian kernel becomes a Dirac delta kernel, implying a single value of 1 in the center after discretization and normalization. Filtering with this kernel is an identity operation, so $s\_\\theta(x, t) = \\tilde{s}\_\\theta(x, t)$ as the filtering has no effect. Replacing $\\tilde{s}\_\\theta(x, t)$ with $s\_\\theta(x, t)$ in Eq. 7 yields: $dx=[f(x,t)-g(t)^2(\gamma_\mathrm{seg}s_\theta (x, t)-(\gamma_\mathrm{seg}-1)s_\theta(x, t))]dt+g(t)d\bar{\omega}=[f(x, t)-g(t)^2(s_\theta (x, t))]dt+g(t)d\bar{\omega},$ since the gamma $\gamma_\mathrm{seg} {s}_\theta (x, t)$ terms cancel out each other.

Explanation on claims

We thank the reviewer for raising this point. Although we already have the basic explanation on claims in the Appendix, we would like to clarify more.

Let $\mathbf{a}=(a_1,\ldots,a_n)$ denote the attention values before the softmax operation, and let $\tilde{\mathbf{a}} = (\tilde{a}_1,\ldots,\tilde{a}_n)$ denote the attention values after applying the 2D Gaussian blur. Let $H$ denote the Hessian of the original energy, i.e., the derivative of the negative softmax, and $\tilde{{H}}$ denote the Hessian of the underlying energy associated with the blurred weights.

The elements in the $i$-th row and $j$-th column of the Hessian matrices are given by:

$$h\_{ij}=(\xi(\mathbf{a})_i-\delta\_{ij})\xi(\mathbf{a})_j,\quad\tilde {h}\_{ij}=(\xi(\tilde{\mathbf{a}})_i-\delta\_{ij})\xi(\tilde{\mathbf{a}})_j b\_{ij},$$

respectively, where $b\_{ij}$ are the elements of the Toeplitz matrix corresponding to the Gaussian blur, and $\delta\_{ij}$ denotes the Kronecker delta.

Assuming $\\xi(\\tilde{a})\_i \\xi(\\tilde{a})\_j \\approx 0$ and $\\xi(a)\_i \\xi(a)\_j \\approx 0$ for all $i$ and $j$, which is a reasonable assumption when the number of tokens is large and the softmax values get small, the non-diagonal elements of the Hessians approximate to 0 and the diagonal elements dominate. The determinants of the Hessian matrices are approximated as: $|\\det(H)|\\approx\\prod\_{i=1}^n\\xi(a)\_i,\\quad |\\det(\\tilde{H})| \\approx \\prod\_{i=1}^n \\xi(\\tilde{a})\_i b\_{ii}$

We have the following inequality: $\\prod\_{i=1}^n\\xi(\\tilde{a})\_i b\_{ii}<\\prod\_{i=1}^n\\xi(\\tilde{a})\_i=(e^{\\sum\_{j=1}^n \\tilde{a}\_j}) / (\\sum\_{j=1}^n e^{\\tilde{a}\_j})^n\\leq (e^{\\sum\_{j=1}^n a\_j})/(\\sum\_{j=1}^ne^{a\_j})^n=\\prod\_{i=1}^n\\xi(a)\_i,$ where the first inequality follows from the property of the Gaussian blur kernel, $0 \\leq b\_{ii} < 1$, and the second inequality is derived from Lemmas 3.1 and 3.2, which demonstrate the mean-preserving property and the decrease in the lse value when applying a blur. The monotonicity of the logarithm function implies that the denominator involving the blurred attention weights is smaller. Eventually, we obtain the following inequality: $|\\det(\\tilde{H})|<|\\det(H)|$ This implies that the updated value is derived with attenuated Gaussian curvature ($K=\\det(\\tilde{H})$ in our case), of the energy function underlying the blurred softmax operation compared to that of the original softmax operation.

Why it does not present perceptual metric for Vanilla SDXL

This is because the LPIPS metric is calculated with Vanilla SDXL (line 229), to measure the side effects (how much the sampling process with guidance deviates from those of the original sampling process). You can recognize the perceptual distance as zero, since it does not differ from the original sampling process.

Discussion on parameters

In Fig. 1 of the attached PDF, we present samples with controlled $\sigma$ and $\gamma_\text{seg}$, which support our experiment in Sec. 5.5 and Fig. 6 and our claim in the main paper. Contributing to saturation, large $\gamma_\text{seg}$ values linearly push the pixel values far from the original prediction.

Additional details of quantitative evaluations

While we guide readers more towards qualitative assessment, we employ multiple approaches to evaluate image quality. Quantitatively, we use FID and CLIP scores to measure fidelity of samples. To assess unintended side effects, we utilize LPIPS scores to quantify deviations from unguided images. Qualitatively, we present extensive visual comparisons (e.g., Figures 2-5, 7-10) that demonstrate improvements in definition, expression, sharpness of details, realism of textures, and overall composition. This approach demonstrates our paper's main contribution, as Reviewer CTSN has mentioned.

Besides, as mentioned in a concurrent work [A], the FD-DINOv2 metric is another means to calculate Fréchet distances and is well-aligned with human perception. For reference, in the table below, we present FD-DINOv2 calculated using 50k samples from the EDM2-S model trained on ImageNet-64 to assess fidelity. We also include uncurated qualitative samples from this model in Fig. 3 of the attached PDF. | Model | FD-DINOv2$\downarrow$ | |-|-| | No guidance | 95.1915 | | SEG ($\sigma \to \infty$) | 47.4733 |

[A] Karras et al. "Analyzing and improving the training dynamics of diffusion models." CVPR 2024.

Computational cost

Our NFE is the same as other methods, SAG and PAG, which is 50 per sample. The contribution part in complexity is the blurring operation, which incurs quadratic cost in the number of tokens. The query blurring instead blurs the query matrix rather than the attention map (shown as equivalent in lines 170-176), so it avoids the quadratic complexity and it's our contribution that makes the blurring process feasible in the high resolution situation.

General response and additional figures

We respectfully refer the reviewer to our general response and additional figures provided above.

Dear Reviewer 9qC6,

Thank you again for your time and effort in reviewing our manuscript. We have posted our response addressing your concerns and suggestions.

If you have any additional questions or require further clarification, we are happy to discuss them. We eagerly await your valuable feedback.

Best regards,

Authors of Submission #4721

We appreciate your valuable suggestions and feedback. Here, we provide further explanations to address your questions.

Basically, our main theoretical results do not only apply to SDXL but also to different backbones across various conditions equipped with self-attention mechanisms. In addition, we report experiments using the conditional EDM2-S model [A] trained on ImageNet-64. Fig. 3 in the attached PDF showcases uncurated qualitative samples from this model. We also present FD-DINOv2 scores calculated using 50k samples to assess fidelity to the real image data, which is well-aligned with human perception [A]. These results demonstrate that our method generalizes well to a different backbone, showing significant improvements in sample quality.

[A] Karras et al. "Analyzing and improving the training dynamics of diffusion models." CVPR 2024.

Additionally, as per the reviewer's suggestion, we conducted experiments using various filter kernels and calculated the scores using 3k samples with the same random seeds for the EDM2-S model:

$\sigma_1$ and $\sigma_2$ used in the bilateral filter denote the parameters controlling the spatial extent of the filter and the influence of intensity differences, respectively.

In this experiment, SEG using the Gaussian filter outperforms those using different filters. While the other filters enhance the score, SEG is the approach that benefits from our theoretical grounding, allowing us to gradually reduce the energy curvature by increasing a single parameter, $\sigma$, which is the main contribution of our paper.

Thank you again for your thoughtful and valuable questions. We hope this clarification addresses your remaining concerns.

We appreciate your thoughtful review of our paper and the recognition of SEG's strengths. We'd like to address the concerns and questions raised:

How the model knows what to generate and why the style is different

First, we'd like to note that our method claims an inference time boost like other guidance techniques, e.g., CFG, by modulating and utilizing the modeled distribution. On a high level, samples from the blunter distribution have poor definition, expression, sharpness of details, realism of textures, and overall composition. When we use these as the negative prediction of the score for guidance, we gain samples without such properties.

Concretely, we can explain how SEG guides the sampling process by drawing an analogy to CFG and from a probabilistic perspective:

a) Similar to how CFG uses the difference between predictions based on sharper conditional and smoother unconditional distributions, SEG reduces the curvature of the energy landscape underlying self-attention (Theorem 3.1). This creates a smoother landscape for attention minimization during sampling, enhancing sample quality without relying on external conditions or special training.

b) From a probabilistic perspective, the energy is associated with the likelihood of the attention weights in terms of the Boltzmann distribution conditioned on a given configuration, i.e., the feature map. Blurring the attention weights diminishes this likelihood (Lemma 3.2) and reduces the distribution's curvature (Theorem 3.1).

The style changes with increasing $\sigma$ you mention seem like changes in the realism of textures or colors, and this is also an effect of using a blunter distribution as negative guidance. Still, our method does not change vanilla SDXL more than PAG or SAG in terms of qualitative and quantitative metrics while achieving a significant quality boost. For example, PAG significantly changes the color, style, and structure of the original SDXL output, as shown in Figs. 5, 7, and 8.

Discussion on parameters

In Fig. 1 of the attached PDF, we demonstrate experiments with controlled $\sigma$ and $\gamma_\mathrm{seg}$, which support our quantitative experiment and claim in Sec. 5.5 and Fig. 6 in the main paper. Larger $\gamma_\mathrm{seg}$ values can cause side effects such as saturation by pushing pixels linearly, potentially moving them out of the manifold, as can be inferred from Fig. 4(c) of the attached PDF. In contrast, Gaussian blur with $\sigma$ naturally smoothens the energy in diffusion models and yields the score prediction which is more likely to exist on the manifold, causing benign predictions even if $\sigma$ gets infinitely big.

Also, we chose optimal $\sigma$ values ($10$ and $\infty$) based on perceptual quality (in Figs. 2, 3, and 4) and metrics (FID and CLIP Score in Table 2). We found that in most cases, to generate realistic photos with better structure and composition, $\sigma \to \infty$ is the best choice, while for generating aesthetic images, $\sigma=10$ works well.

Application scenarios

SEG's main applications include enhancing both conditional and unconditional image generation across various domains. It has many applications in large-scale diffusion models trained on text-image pairs for unconditional and general generation. ControlNet, which uses conditions other than text, is one example. SEG can also be used for inverse problems.

Reliance on baseline models

While SEG does rely on the baseline model, this is inherent to all guidance methods, which aim to enhance or control the generation process during inference. We have clarified this limitation in the paper. Still, it can be seen as a strength that our theoretical results are general to all models and that SEG is used in both unconditional image generation and conditional generation with various types of input, as acknowledged by Reviewers 2oF8 and cNNp.

Abbreviations

We apologize for the labeling error in Fig. 5. "PEG" should indeed be "SEG". We will correct this in the final revision and ensure consistent use of abbreviations throughout.

General response and additional figures

We respectfully refer the reviewer to our general response and additional figures provided above. This material addresses key points raised in the initial review and highlights the strengths of our paper, as noted by other reviewers. Additionally, we have included new figures and results that we believe may address your concerns.

Dear Reviewer Rhhf,

Thank you again for your time and effort in reviewing our manuscript. We have posted our response addressing your concerns and suggestions.

If you have any additional questions or require further clarification, we are happy to discuss them. We eagerly await your valuable feedback.

Best regards,

Authors of Submission #4721

We would first like to thank the reviewer for acknowledging our visualization with various conditions as insightful and our paper as very interesting. We'd like to address the concerns and questions raised:

Additional details of quantitative evaluations

While we guide readers more towards qualitative assessment, we do employ multiple approaches to evaluate image quality. Quantitatively, we use FID and CLIP scores to measure fidelity of samples. To assess unintended side effects, we utilize LPIPS scores to quantify deviations from unguided images. Qualitatively, we present extensive visual comparisons that demonstrate improvements in definition, expression, sharpness of details, realism of textures, and overall composition. As Reviewer CTSN has mentioned, we believe this approach provides a comprehensive evaluation of our method's effectiveness and demonstrate our paper's point on the empirical side.

Besides, as mentioned in a concurrent work [A], the FD-DINOv2 metric is another means to calculate Fréchet distances and is well-aligned with human perception. For reference, in the table below, we present FD-DINOv2 scores calculated using 50k samples from the EDM2-S model trained on ImageNet-64 to assess fidelity. We also include uncurated qualitative samples from this model in Fig. 3 of the attached PDF. This corroborates how the structure and quality of samples change, as well as the generality of our methods. | Model | FD-DINOv2$\downarrow$ | | --- | --- | | No guidance | 95.1915 | | SEG | 47.4733

[A] Karras, Tero, et al. "Analyzing and improving the training dynamics of diffusion models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Compare against more recent works (e.g. Imagic, LEdits++, collaborative diffusion)

We respectfully would like to emphasize that our method differs from recent text-based editing works like Imagic, LEdits++, and collaborative diffusion in its generality and goal. Our approach is more general, aiming to improve quality for diffusion models equipped with self-attention, regardless of the presence of text conditions.

However, we recognize that these text-based editing methods are orthogonal to our method and could be potential applications of our approach. In the revised manuscript, we may include a discussion of the potential for synergistic applications in the future direction part.

Address the question about advantages over recent works

We appreciate the opportunity to clarify this point. While our method is complementary to those text-based editing works, our method features several characteristics:

1. Generality: Unlike text-based editing methods, our approach can improve image quality without requiring text prompts, making it applicable to a broader range of scenarios. SEG demonstrates effectiveness in both conditional and unconditional image generation scenarios, making it a flexible solution (Reviewers 2oF8 and cNNp).

2. Inference-time quality enhancement: SEG improves the overall definition, expression, sharpness of details, realism of textures, and overall composition of generated images across various conditions, e.g., no condition, text, Canny, or depth map (Reviewers 2oF8 and Rhhf). Text-based editing methods the reviewer mentioned do not target and induce quality improvement.
3. Novel approach: Our method, Smoothed Energy Guidance (SEG), presents an innovative, training- and condition-free approach to image generation using diffusion models. It offers an interesting alternative to classifier-free guidance (CFG) by leveraging the self-attention mechanism (Reviewers 2oF8, Rhhf, 9qC6, and CTSN).
4. Theoretical foundation: The paper provides a solid theoretical grounding, using an energy-based perspective and the concept of smooth energy landscapes to improve image generation (Reviewers Rhhf and CTSN).

General response and additional figures

We respectfully refer the reviewer to our general response and additional figures provided above. This material addresses key points raised in the initial review and highlights the strengths of our paper, as noted by other reviewers. Additionally, we have included new figures and results that we believe may address your concerns.

We will elaborate on these points in the revised manuscript to clearly articulate the unique benefits of our approach.

Dear Reviewer 2oF8,

Thank you again for your time and effort in reviewing our manuscript. We have posted our response addressing your concerns and suggestions.

If you have any additional questions or require further clarification, we are happy to discuss them. We eagerly await your valuable feedback.

Best regards,

Authors of Submission #4721

Thank you for your valuable feedback. We appreciate your suggestions, but we believe there are some misunderstandings that we'd like to address.

1. Regarding comparisons with state-of-the-art methods, we have already included experiments using ControlNet in our main paper (Figures 4, 9, and 10). These comparisons directly address your concern about contextualizing our contributions against powerful approaches.

2. In addition, we deliberately did not compare our method with text-based image editing approaches like LEdit++ because they do not align with our research goals. Our focus is on different editing paradigms, making such comparisons inappropriate as baselines for our work. This point is also addressed in the rebuttal.

As we are at the end of the reviewer-author discussion period, we respectfully ask that you consider the information already provided in our paper and previous responses. We have addressed many of these points earlier and believe our work stands on its own merits within its intended scope. We hope this clarifies our position and demonstrates that we have indeed addressed many of your concerns within the constraints of our research focus. We urge you to reconsider your evaluation in light of this information.

Again, thank you very much for your thoughtful and valuable questions.

Best regards,

Authors of Submission #4721