Angle Domain Guidance: Latent Diffusion Requires Rotation Rather Than Extrapolation

Thank you for your positive comments. We provide our responses below.

1. Clarification on Computational Costs of ADG

Reviewer Concern: The reviewer mentions that computational costs associated with ADG, particularly in high-dimensional spaces, are not discussed in detail.

Response: We appreciate the reviewer’s observation. ADG introduces negligible computational overhead compared to standard sampling procedures. It does not require additional neural network evaluations; instead, its overhead consists solely of basic vector operations (e.g., normalization and angle clipping).

To quantify this:

• On SD3.5-large (latent dimensionality ≈ $2 \times 10^5$), the additional operations per inner loop cost around $1 \times 10^6$ FLOPs.
• In contrast, one full sampling step requires approximately $9 \times 10^{13}$ FLOPs (measured using the thop package).
• Empirical measurements over 100 generations on an A100 GPU show:
  • CFG: Avg. generation time = 6.74s
  • ADG: Avg. generation time = 6.72s

This confirms that ADG does not increase runtime, and minor variation may be attributed to system noise.

2. Behavior of CFG in Pixel-Space Diffusion Models

Reviewer Concern: The reviewer asks whether similar trends of norm amplification and image degradation occur in pixel-space diffusion models.

Response: Yes, similar degradation patterns are observed in pixel-space diffusion models under high guidance weights. As in latent models, CFG-induced norm amplification pushes pixel values toward extreme ranges, leading to oversaturation and unnatural contrast. visual example

These observations reinforce our hypothesis that norm amplification is a core issue, not limited to latent spaces.

3. Justification for Assuming Spherical Gaussian Latent Space

Reviewer Concern: The reviewer requests further elaboration or visualization to validate the assumption that the latent space follows a spherical Gaussian distribution.

Response: During the pretraining of VAEs used in latent diffusion models, the latent space is regularized to approximate a standard multivariate Gaussian distribution through a Kullback-Leibler (KL) divergence loss term. We acknowledge that Gaussian priors in VAEs are often idealized, and real-world latent spaces may deviate from this assumption due to model imperfections or data complexity. However, this Gaussian prior remains a practical and effective approximation that helps explain why directional information carries more semantic meaning than magnitude information.

Samples drawn from a high-dimensional Gaussian distribution tend to concentrate around a thin spherical shell, where the angular relationship between latent variables retains critical semantic information. Consequently, emphasizing angular alignment, as done in Angle-Domain Guidance (ADG), effectively preserves text-image consistency while mitigating the undesirable effects of norm amplification. This is also one of the reasons why we emphasize that, although ADG is theoretically inspired and has some theoretical guarantees, it remains a heuristic algorithm.

4. Choice of ADG over Spherical Linear Interpolation (SLERP)

Reviewer Concern: The reviewer suggests that using spherical linear interpolation (SLERP) could be a natural alternative to ADG for performing angular guidance and inquires why it was not considered.

Response: While SLERP is an elegant mathematical alternative, it is not suitable for high guidance weights in our setting. The reasons are as follows:

• SLERP inherently performs interpolation, not extrapolation. In cases where the guidance weight $\omega > 1$, SLERP extrapolates beyond the two endpoints, leading to uncontrolled norm growth.
• Our proposed ADG method, by contrast, constrains the norm of the generated sample under high guidance weights, as demonstrated by Proposition 4.1.
• Consider a simple example where:

$$\hat x_{0, c} = [1, 0]^\top, \quad \hat x_{0, \emptyset} = [0.5, 0.01]^\top.$$

Using SLERP with $\omega = 5$, the result is:

$$\text{slerp}(\hat x_{0, \emptyset}, \hat x_{0, c}, 5) \approx [15.9, -1.2]^\top,$$

which exhibits extreme sensitivity when the vectors are nearly aligned. In contrast, ADG mitigates this instability by ensuring angular alignment while maintaining controlled magnitude growth. We chose ADG over SLERP due to its norm control under high guidance weights, which is crucial for maintaining image fidelity.

Once again, thank you for your constructive feedback and for considering our paper for acceptance. We will revise our paper accordingly.

Thank you for your positive comments. We provide our responses below.

1. Performance of Higher CFG Values

Reviewer Comment:

I am curious about the performance of a higher CFG, e.g., larger than 20. This would further show the effectiveness of ADG.

Response:
We appreciate this insightful suggestion. We have conducted additional experiments with higher guidance weights, specifically at weight = 20. The results demonstrate that ADG maintains stable performance even under extreme guidance conditions, whereas CFG suffers from severe color distortions and semantic misalignment.These results highlight the robustness of ADG at high guidance levels. We will include qualitative visualizations at weight = 20 in the supplementary material. View Images

2. Evaluation with Complex Text Prompts

Reviewer Comment:

Since sdv3.5 is pretrained on text-to-image generation, it's better to show more results containing complex text prompts as guidance (instead of simple prompts that describe a single object). This would further show how text alignment becomes when we increase the guidance scale.

Response:
Thank you for this helpful suggestion. To evaluate ADG under more complex prompts, we curated 500 complex text prompts using GPT. Our experiments show that ADG significantly outperforms CFG under these conditions. For guidance weight = 8:

• ADG achieves a CLIP score of 0.355 and an IR score of 1.566
• CFG scores lower with a CLIP score of 0.338 and an IR score of 0.766

These results confirm that ADG preserves semantic alignment and image quality even with complex instructions. We will include detailed metrics and visual comparisons in the supplementary material. View Images

3. Experiments on State-of-the-Art Models (e.g., Flux)

Reviewer Comment:

More experiments on state-of-the-art models (e.g., Flux) would make the claim more solid.

Response:
Thank you for raising this important point. We initially considered incorporating results from state-of-the-art open-source models, including Stable Diffusion 3.5 large and Flux.1 [dev]. However, during our experiments, we discovered that Flux.1 [dev] applies guidance weight as a model input rather than through the conventional CFG mechanism.

Upon further investigation of the Flux architecture and related blog posts, we learned that Flux.1 [dev] is derived via guidance distillation from Flux.1 [pro], which is unfortunately not open-source. Since access to Flux.1 [pro] is required to modify the guidance mechanism, it was impossible for us to conduct experiments on the Flux family.

Once again, thank you for your constructive feedback and for considering our paper for acceptance. We will revise our paper accordingly.

Thank you for your positive comments. We provide our responses below.

1. Plot norm against guidance weight for ADG

Reviewer Comment:

To further strengthen this claim, I encourage the authors to plot norm against guidance weight (similar to Figure 2a) to find out whether ADG can empirically control sample norm as suggested by the theoretical result.

Response:
We appreciate this suggestion. We have added a new plot to Figure 2a, showing the norm against guidance weight for ADG.
View Figure 2a

The experimental results indicate that ADG effectively controls sample norms, preventing norm amplification even at high guidance weights.

2. Experimental results for flow-based models

Reviewer Comment:

However, no experimental results are provided for flow-based models.

Response:
Thank you for highlighting this point. We would like to clarify that our primary results were obtained on SDv3.5, and Stable Diffusion series starting from version 3 employs flow-based models[a].

To avoid any ambiguity, we will emphasize this point more explicitly in the camera-ready version.

3. CFG with normalization as a potential baseline

Reviewer Comment:

One potential baseline could be CFG with normalization. That is, performing CFG at each sampling step, followed by normalizing and re-scaling the estimated x_0's, similar to Algorithm 6. This could be a simpler remedy to the norm amplification issue compared to ADG.

Response:
This is a good suggestion. Since our motivation stems from the observation that angular information in the x_0 domain better aligns with semantics, while norm amplification under high CFG weights degrades sample quality, we designed ADG to enhance angular consistency. The method proposed by the reviewer is a simple way to control sample norms.

We implemented and tested this potential baseline (CFG and normalization in the x_0 domain). Experimental results show that although this method performs slightly worse than ADG, it significantly outperforms the original CFG. Our analysis suggests that the angle between the predicted x_0 and the unconditional prediction under this baseline is smaller than the angle between the conditional prediction and the unconditional prediction in ADG, limiting semantic alignment despite effective norm control.
View comparison

4. Explanation of non-commutativity and Equation 10

Reviewer Comment:

I encourage the authors to expand on what they mean by "non-commutativity of the tilting process with the forward process" (L126-127) and briefly explain why Equation 10 holds.

Response:
Due to space constraints, we could not elaborate on this point in the main text. However, as noted in [c], non-commutativity means that the weighted gradient used in CFG does not align with the true gradient of the tilted target distribution for t>0.

To illustrate this, consider a simple example where:

• The unconditional distribution is modeled as a Gaussian with zero mean and variance 10.
• The conditional distribution is another Gaussian with zero mean and unit variance.

When applying classifier-free guidance (CFG) with a guidance weight of 2, the tilted distribution becomes a Gaussian with:

• Zero mean.
• A variance of 10/19.

After applying the forward process, which introduces Gaussian noise over time (considering a Variance Exploding SDE, i.e., VE-SDE), the resulting distribution is convolved with a Gaussian kernel whose variance increases with time.
To simplify the computation, assume that the time-dependent Gaussian kernel at a particular time t* is modeled as a Gaussian with variance 1.

In this setting:

• The gradient of the CFG-modified distribution is a weighted sum of the gradients of the conditional and unconditional distributions, which results in a gradient proportional to -10/11 * x.
• However, this gradient does not match the gradient of the tilted distribution obtained after applying the forward process, which is proportional to -19/29 * x, highlighting the non-commutativity between the tilting process and the forward process.

Once again, thank you for your constructive feedback and for considering our paper for acceptance. We will revise our paper accordingly.

References

• [a] Stable Diffusion 3
• [b] Exploring diffusion and flow matching under generator matching. arXiv
• [c] What does guidance do? A fine-grained analysis in a simple setting. NeurIPS 2024.

We sincerely thank the reviewer for their thoughtful comments. While the overall evaluation was critical, your constructive feedback is highly valuable and will help us improve both the clarity and impact of our work. Below, we provide detailed responses to your key concerns.

1. Comparison with Adaptive Projected Guidance

Reviewer Concern:
The reviewer notes the absence of discussion on Adaptive Projected Guidance (APG) [ICLR 2025], suggests high similarity with our proposed ADG method, and questions whether ADG offers substantive novelty. 

Response:
We appreciate the reviewer for pointing out this important and timely work. APG appeared on arXiv in October 2024, shortly before our submission, and was therefore not included in our original analysis. We will include appropriate citations and a dedicated comparison section in the final version.

To clarify the distinctions between ADG and APG, we summarize the differences below: 

1. Attribution of Image Degradation:

• APG attributes image oversaturation and degradation to the parallel component of the difference vector $\Delta \hat x_0$ relative to $\hat x_{0, c}$, denoted as $\Delta \hat x_0^\parallel$.
• In contrast, we attribute oversaturation and distortion to the large norm of $\hat x_{0, CFG}$:

$$\hat x_{0, CFG} = \hat x_{0, c} + (\omega - 1) \Delta \hat x_0.$$

Removing the parallel component slows norm growth but fails to address norm amplification. Additional experiments show that normalizing $\hat x_{0, CFG}$ mitigates image degradation more effectively.
Visual example  2. Algorithmic Difference:

• APG modifies $\Delta \hat x_0$ in CFG:

$$\Delta \hat x_{0, APG} = \eta \Delta \hat x_0^\parallel + \Delta \hat x_0^\perp, \quad \hat x_{0, APG} = \hat x_{0, c} + \omega \Delta \hat x_{0, APG},$$

where $\eta < 1$. However, APG retains linear enhancement, leading to norm growth at high guidance weights.

• ADG, in contrast, performs angular-domain updates, effectively constraining norm growth (Proposition 4.1).
  ADG vs. APG Visual 

3. Empirical Performance and Stability:

Under high guidance weights, ADG successfully mitigates oversaturation and artifacts, whereas APG-generated images exhibit significant oversaturation, which reduces text-image alignment. visual examples
ADG outperforms CFG and CFG++ in both alignment (CLIP) and human preference metrics (ImageReward), especially under high guidance weights. quantitative results



2. Clarification on Theoretical Derivations and Algorithm Motivation

Reviewer Concern:
The reviewer seeks clarification on how the theoretical derivation leads to the conclusion that "CFG leads to excessively large norms of features." Additionally, they express confusion about the connection between the theoretical derivations and the proposed method. The reviewer also suggests illustrating the mathematical derivations with concrete generation examples where possible.
 Response:
Theorem 3.2 shows that linear extrapolation in CFG shifts samples toward the outer regions of the unconditional distribution, increasing the norms of the latent variables (features).
 The paper's logical flow is:

1. CFG Limitation: Norm amplification and anomalous diffusion for surface-class samples (Theorem 3.2 and Theorem 3.3).
2. Source of Norm Amplification: Linear enhancement of $\hat{x}_0$ leads to larger norms.
3. ADG Motivation: Angular-domain updates mitigate norm amplification (Proposition 4.1).

Fig. 3 illustrates this phenomenon with a Gaussian mixture model. The green, blue, and orange clusters represent surface classes, while the red cluster denotes non-surface classes. Higher guidance weights push surface-class samples away from the unconditional distribution, amplifying norms.




3. Clarification on “Magnitude Differences Are Secondary” Statement

Reviewer Concern:
The reviewer finds the statement "The focus on magnitude differences is secondary, ..." ambiguous. 

Response:
We appreciate the reviewer's feedback. The intended insight is:


• Linear enhancement methods (e.g., CFG) modify both the magnitude and direction of $\hat{x}_0$.
• While magnitude enhancement may improve semantic alignment slightly, it becomes detrimental at high guidance weights due to excessive norm amplification, resulting in image degradation.

Revised Statement:

"Linear enhancement methods simultaneously modify the norm and direction of $\hat{x}_0$. While norm adjustments may marginally improve semantic alignment, excessive norm amplification at high guidance weights becomes detrimental, resulting in oversaturation and distortion."

We appreciate your feedback and will revise the paper to clarify contributions and address concerns.