Normalized Attention Guidance: Universal Negative Guidance for Diffusion Models

We thank for acknowledging the paper’s clear presentation and comprehensive evaluation of NAG. Below we address the two points with new ablations and expanded applicability experiments.

As shown in Figure 8, the effect of normalization does not appear to be significant. Additionally, the experiment "w/o Norm & with Refine" is missing.

Thank you for catching the missing ablation. We have now added the “w/o Norm” ablative study alongside the Full NAG:

$\phi = 4$

$\phi = 8$

Removing normalization degrades both fidelity metrics (FID, PFID), and leads to even larger drops at higher scales. The degraded fidelity indicates that the guidance becomes unstable and disrupts the feature flow through the network. Normalization therefore acts as a critical safeguard by bounding feature magnitudes.

While NAG operates effectively in attention feature space, its applicability may be limited to models or architectures that leverage attention mechanisms.

Actually NAG is not inherently tied to attention mechanism. Because it exploits the semantic difference between two conditions, the same guidance algorithm can be applied to any conditioning module.

We examine two conditioning mechanisms in SDXL: cross‑attention and adaptive normalization.

• Adaptive‑Norm Only: applying NAG just in adaptive normalization layer already yields consistent gains.
• Attention + Adaptive‑Norm: There is no obvious improvement compared to attention only NAG, implying that guidance in adaptive normalization contributes minimal additional benefit over attention space NAG.

We focused our paper on attention layers because it is the dominant conditioning mechanism in today’s generative models, but these results clearly demonstrate that NAG can be ported to different conditioning mechanisms without modification of the core algorithm.

Thank you for recognizing NAG’s broad applicability and for these insightful questions.

Q1: Why does modulating self-attention outputs preserve generation quality and not corrupt the learned data manifold?

To clarify, NAG operates on cross-attention or multimodal DiT (MMDiT) layers, not on self-attention layers; the notation $Z$ in our paper denotes the cross-attention features (Equation 4).

As detailed in our response to Reviewer oLD6, NAG can be viewed as a variant of CFG’s implicit classifier gradient applied at intermediate cross-attention layers rather than only at the final output. NAG performs a series of small, implicit classifier‑guided steps where the positive and negative features remain semantically aligned, preventing runaway divergence from the manifold. The subsequent normalization and refinement bounds feature magnitudes near the model’s learned feature space, alleviating the risk of manifold corruption.

Moreover, these modifications are continuously smoothed by the model’s linear projections, LayerNorm, nonlinear activations, and other layers before the next attention operation. These architectural “filters” mix, re-center, and softly cap value spikes, preventing outlier attention features from cascading into large distributional shifts. distributional shifts.

Q2: Why specifically L1 normalization? Do you have any ablation studies comparing other normalization variants?

Thank you for pointing this out! We have run ablations comparing L1 against L2 and max-norm, but did not include them in the submission; we will add these results and accompanying discussion in the paper.

We find that swapping to L2 yields visually similar results but with slightly worse metrics, while max‑norm will soften fine details. Overall, any norm scales the attention features, providing constraints without changing semantic direction, so NAG’s performance is not really sensitive to the exact choice.

L1 was chosen because it conceptually best preserves proportional relationships and subtle semantics (lines 167 - 170), provides the best quantitative results, and it’s the most efficient to compute.

Q3: If NAG still requires CFG, what fundamental classes of problems does it solve that CFG alone cannot? If there's no such distinction, why not study the prediction of classifier-free guidance and then modulate attention layers within it?

We apologize for the lack of clarity. NAG works completely independently under few‑step sampling (Figure 6, Table 5), where CFG collapses (Left of Figure 3). In essence, NAG serves as a standalone alternative to CFG for negative guidance in few-step sampling.

Under multi-step sampling, NAG can also be used in conjunction with CFG. When paired with CFG, NAG further enhances qualitative and quantitative results (Figure 7, Table 6). This dual capability highlights NAG’s universality.

We appreciate the suggestion to illustrate CFG failures in Figure 1. And we will correct the notation around $R[i]$ to ensure clarity.

Q1: Any performance gaps when the guidance scale $\phi < 1$ (i.e., doing interpolation)? From Figure 2, it seems the gap only exists when the scale is large, while one can still have effective results by using the baseline NASA method by simply setting this hyperparameter properly.

For the UNet SDXL‑DMD2 model, we include an extended comparison against NASA in Appendix F (lines 530–541), covering both quantitative metrics and a user study, which illustrate improvements of NAG over NASA across all evaluated criteria.

Additionally, we evaluate DiT Flux‑Schnell under small guidance scales here. Importantly, $0<\phi<1$ does not strictly correspond to interpolation—it simply applies a lighter guidance strength. In the experiments, NASA at low $\phi$ does improve CLIP score but degrades FID and ImageReward, and crash in $\phi = 0.5$. In contrast, NAG demonstrates stable enhancements and achieves the best overall metrics on $\phi = 4$, boosting both alignment and image quality. As a result, NAG is robust to a wide range of $\phi$, significantly reducing tuning overhead.

Metrics reported for Baseline / NAG.

Q2: Have you experimented with the proposed method on stronger few-step generation DiT and UNet backbones to show consistent performance enhancement to support the claim that NAG can be uniformly applied?

Beyond the experiments in Section 5.1 of the main paper, we have evaluated NAG on a wide range of few‑step backbones in Appendix D (lines 524–529). Overall, we include SOTA models from academia and the community, such as DiT architecture Flux, SD3.5-Turbo (SIGGRAPH Asia 2024) and SANA-Sprint (ICCV 2025), as well as UNet architecture NitroFusion (CVPR 2025), DMD2 (NeurIPS 2024) and HyperSD (NeurIPS 2024). On average, we can observe +0.5 CLIP, −1.5 FID, -2.5 PFID, and +0.1 ImageReward gains.

Additionally, in the rebuttal to the Reviewer oLD6, we demonstrate improved performance, -1.3 FID and +20 IS, for the newly released single-step class‑conditional MeanFlow.

Q3: Can you provide analyses of the hyperparameters additionally introduced by NAG?

We include an ablation study on hyperparameters in Appendix F (lines 542–552). In particular, Figure 13 visually analyzes the performance sensitivity of normalization threshold $\tau$ and refinement factor $\alpha$. We hope this analysis, along with visual results, clarifies NAG’s sensitivity to hyperparameter choices.

Q4: Can you provide additional analysis of the proposed two stabilization mechanisms to demonstrate the technical contribution in this paper in addition to NASA?

Unlike NASA, which directly subtracts negative attention maps, NAG applies extrapolation grounded in the implicit classifier view (please refer the Q1 of reviewer oLD6 for more details). This yields more stable editing in attention space. The subsequent normalization and refinement further reinforce this stability.

These operations share two critical properties:

1. Firm Nonexpansiveness: Each update can only bring features closer together, never amplifying deviations beyond the model’s learned manifold.
2. Direction Preservation: By projecting onto norm‑balls or taking convex combinations, the original edit direction remains intact, ensuring semantic coherence.

While these properties guarantee features stay near the learned manifold, they do not ensure that NAG's attention space edits will always yield a stable guidance signal.

Empirically, however, our ablations confirm that both normalization and refinement are indispensable for stability. In practice, NAG remains a simple yet powerful mechanism for robust and controllable negative guidance across architectures and sampling regimes.

We believe NAG is just the beginning, and encourage future research to explore alternative attention space constraint formulations to achieve more stable guidance.

We’re grateful for the thorough reviews and recognition of our strong empirical generalization.

Q1: Beyond geometric interpretation, can authors provide a formal theoretical justification or probabilistic interpretation for extrapolating directly in attention feature space? Is there any formal rationale for why this manipulation maintains semantic coherence and stability?

Below, we provide a theoretical interpretation for NAG attention space extrapolation and will add it to the paper.

Implicit Classifier Gradient

For a pretrained diffusion network $G_\theta$, and timestep $t$ we denote the conditional and unconditional noise predictions as

$\hat\epsilon = G_\theta(x_t, c),\qquad\hat\epsilon^\varnothing =G_\theta(x_t, \varnothing).$

By Bayes’ rule, the implicit classifier over $x_t$ can be defined as

$p^i(c\mid x_t) \propto \frac{p(x_t\mid c)}{p(x_t\mid \varnothing)}.$

Its gradient can be approximated with

$\nabla_{x_t}\log p^i(c\mid x_t)= \nabla_{x_t}\log p(x_t\mid c) - \nabla_{x_t}\log p(x_t\mid \varnothing) \approx -\bigl(\hat\epsilon - \hat\epsilon^{\varnothing}\bigr).$

CFG then performs a heuristic gradient ascent step, leading to the guidance scaled by $\phi$

$\tilde\epsilon= \hat\epsilon + \phi \bigl(\hat\epsilon - \hat\epsilon^\varnothing\bigr).$

Generalizing to Intermediate Features

Rather than only adjusting the final noise estimate, we can view each intermediate feature map $F_l$ of layer $l$ as also carrying implicit classifier gradient. Denote

$F_l = G_\theta^l(x_t, c),\qquad F_l^{\varnothing} = G_\theta^l(x_t, \varnothing).$

Analogously, we approximate

$\nabla_{x_t}\log p^i(c\mid x_t)\approx -\bigl(F_l - F_l^{\varnothing}\bigr),$

so an ascent step in feature space yields

$\widetilde F_l= F_l + \phi \bigl(F_l - F_l^{\varnothing}\bigr).$

Negative Guidance

To steer generation away from unwanted attributes, we replace the null condition with negative condition $c^-$. Given

$F_l^+ = G_\theta^l(x_t, c^+),\qquad F_l^- = G_\theta^l(x_t, c^-),$

the unified negative guidance becomes

$\widetilde F_l = F_l^+ + \phi \bigl(F_l^+ - F_l^-\bigr).$

This directs $F_l$ both toward the positive semantics and away from negative features in one update.

Layer‑Wise Updates for Stability

CFG and NAG can be viewed as two special cases of where to apply the unified guidance.

• CFG (output‑space) applies a single global adjustment at the final layer. Under aggressive few‑step sampling, the positive and negative predictions have already drifted apart, causing severe artifacts.
• NAG (attention‑space) instead interleaves guidance steps in attention layers. Because $F_l^+$ and $F_l^-$ remain closer in feature space early on, each update is well‑conditioned and prevents runaway divergence. The cumulative effect of many modest, layer‑wise adjustments retains semantic coherence and provides stable negative guidance even with very few denoising steps.

By viewing each attention map as contributing an implicit gradient, NAG generalizes CFG’s single output guidance into a sequence of feature space guidance.

Q2: Considering PLADIS (ICCV 2025) already introduced attention-space extrapolation, how precisely does NAG differ methodologically from PLADIS?

We appreciate the pointer to PLADIS (published in March 2025) and we will include it and clarify the distinctions in our related work.

While PLADIS also operates in the cross‑attention domain, it contrasts dense versus sparse attention maps extracted from a single prompt. PLADIS treats the dense map as a perturbation of its sparse counterpart, in a manner reminiscent of Perturbed‑Attention Guidance (PAG).

By contrast, NAG explicitly exploits the semantic difference between a positive prompt and a negative prompt, providing targeted suppression of undesired attributes during generation, a capacity that PLADIS and PAG do not support. When a negative prompt is absent, NAG reduces to the perturbation‑style mechanism by treating the null condition as the denser attention reference. This dual capability sits naturally within our unified negative‑guidance framework.

Q3: What specific hyperparameter values (particularly the refinement blending factor α) are used across experiments, and why were these particular choices made? How sensitive is performance to variations in α?

All default NAG hyperparameters, including the refinement blending factor $\alpha$ and normalization threshold $\tau$, are listed in Appendix C (lines 521–523). Defaults $\tau = 2.5$, $\alpha \in [0.125, 0.5]$ vary by model family. In Appendix F (lines 542–552), we present an ablation study over $\alpha$ and $\tau$, demonstrating that performance remains stable for values of $\alpha$ below $0.5$ and $\tau$ around $2.5$.

Although a full grid search across every model family would be computationally prohibitive, we conducted smaller‑scale explorations to identify an effective range, then chose the defaults based on visual effect. In practice, users may begin with these settings and adjust them as needed to suit personal preferences.

Q4: What is the practical advantage of employing the additional complexity of normalization and refinement mechanisms to enable higher guidance scales?

High guidance scales often introduce visual artifacts or even collapse, limiting how aggressively users can steer generation. NAG’s normalization and refinement steps prevent these artifacts. As a result, users can apply much more aggressive negative guidance without risk, achieving robust suppression of unwanted attributes.

Q5: How does NAG perform independently without CFG or PAG?

NAG was designed to work as a standalone negative guidance in few‑step (1–8 steps) sampling where CFG and PAG fail (lines 54-55).

In our few‑step experiments (Section 5.1) on both UNet and DiT backbones, applying NAG by itself (without CFG or PAG) demonstrates substantial gains in CLIP alignment, FID/PFID, and ImageReward compared to the baseline. In multi‑step regimes (Section 5.3), NAG can be combined with CFG or PAG for even stronger guidance, but on few‑step models NAG is the only method that delivers universal negative guidance.

Q6: While NAG is designed explicitly for negative prompting, can the authors extend or generalize the NAG approach to standard (positive-only) text-to-image generation scenarios, without explicitly provided negative prompts?

Text-to-Image Flux‑Schnell on COCO‑5K

Good point! NAG can be applied with positive‑only generation simply by using an empty negative prompt (i.e. "" instead of "Low resolution, blurry"). Even with the empty negative prompt, NAG boosts CLIP and ImageReward, confirming enhanced alignment and perceptual quality.

Class-conditional MeanFlow‑B/4 on ImageNet

We also evaluate NAG for class‑conditional generation on ImageNet using the single‑step MeanFlow‑B/4 [1] model. By viewing the null class as the negative condition, NAG reduces FID and significantly improves IS.

These results confirm that NAG seamlessly supports positive‑only guidance.

[1] Mean Flows for One-step Generative Modeling. arXiv 2025

Additional Quantitative Metrics

Thanks you for the suggestions. We provide the specialized quantitative metrics HPSv2 and PickScore for few‑step models.

In experiments, HPSV2 exhibits consistent gains with NAG across all models, while PickScore shows mixed results. Across the three primary human‑preference metrics (ImageReward in Table 1, HPSV2, and PickScore), NAG produces reliable improvements in most of them (both ImageReward and HPSV2).

Metrics reported for Baseline / NAG.