Token Perturbation Guidance for Diffusion Models

We sincerely appreciate the reviewer’s thoughtful and valuable feedback. We have addressed all comments in detail below, and we would welcome any further discussion.

Performance of TPG for conditional generation

We would like to emphasize that CFG is specifically designed and trained to improve prompt alignment in conditional generation. As such, surpassing its performance in this setting is inherently challenging for general, training-free methods, such as TPG. Nevertheless, unlike CFG, which is limited to conditional generation, TPG shows superior performance over other methods in unconditional generation and significantly narrows the gap between perturbation-based guidance methods and CFG. As noted in our response to Reviewer 83Sp, this gap is nearly eliminated in the case of Stable Diffusion 3, further highlighting the strengths and generality of our pipeline. Regarding computational efficiency, as the reviewer pointed out, there is no significant difference between TPG and CFG: both require two forward passes through the diffusion model to generate guidance signals. Moreover, since the shuffling operation used in TPG is extremely lightweight, the runtime for both methods is practically identical, resulting in negligible computational overhead from TPG.

Permutation vs. Shuffling

We apologize for any confusion caused. In our paper, the terms “shuffling” and “permutation” refer precisely to the same operation (i.e., randomly changing the order of tokens). We will revise the paper to consistently use a single term and clarify this explicitly in the definitions. Thank you for pointing this out.

Similarity between TPG and CFG

While TPG and CFG exhibit some similar behaviors in the properties of their update terms (as illustrated in Fig. 2 and Fig. 3), TPG should not be viewed as a special case of CFG. The similarities primarily arise from their common ability to effectively steer predictions toward higher-quality samples. However, the underlying mechanisms of TPG and CFG differ fundamentally: CFG explicitly leverages conditional information (e.g., text prompts), whereas TPG introduces perturbations through random permutations (shuffling) in the token space. Consequently, TPG does not require any specific training and naturally extends to unconditional generation scenarios.

Combining TPG and CFG

We thank the reviewer for this insightful question. Indeed, TPG and CFG can be effectively combined, as they operate through complementary mechanisms. Following this suggestion, we conducted an additional experiment to investigate the combined effect of TPG and CFG. Our findings confirm that integrating CFG with TPG further enhances overall performance. However, it is important to note that combining these methods requires an additional forward pass through the model, resulting in improved performance at the expense of increased computational overhead. Further improvements may be possible by optimizing the weighting scheme used for combining the predictions from CFG and TPG.

Table 1: Quantitative comparison of TPG and the combined TPG + CFG approach for conditional generation using SDXL, evaluated over 5k generated samples.

We appreciate the reviewer’s recognition of the strengths of our method and are pleased that the paper was found to be well organized and easy to follow. Our detailed response is provided below, and we remain open to further discussion and questions.

Comparison between TPG and CFG

It is important to emphasize that CFG is specifically designed and trained to enhance alignment with prompts in conditional generation. As such, surpassing its performance in this setting is inherently challenging for general, training-free methods like TPG. Nevertheless, unlike CFG, which is restricted to conditional generation, TPG demonstrates superior performance over other methods in unconditional generation and significantly narrows the gap between perturbation-based guidance methods and CFG. Please also note that as mentioned in our response to Reviewer 83Sp, this gap is nearly eliminated in the case of Stable Diffusion 3, further highlighting the strengths of our pipeline.

Combining TPG with CFG

Thank you for the insightful suggestion regarding mixed strategies. Indeed, combining TPG with CFG is feasible, as they operate through complementary mechanisms and influence the model in distinct ways. Following the reviewer’s suggestion, we conducted additional experiments to evaluate the combined effect of TPG and CFG. Our results show that this combination leads to improved performance, and further gains may be achievable by optimizing the weighting scheme used for combining the predictions from CFG and TPG.

Table 1: Quantitative comparison of TPG and the combined TPG + CFG approach for conditional generation using SDXL, evaluated over 5k generated samples.

Impact of larger guidance scale on TPG

Empirically, we found that a guidance scale of 3.0 yields the best results and have therefore adopted it as the default value for TPG. Additionally, we conducted comprehensive evaluations across a wide range of guidance scales to investigate the effect of larger values, as suggested by the reviewer. As shown below, increasing the guidance scale beyond 3.0 did not lead to improved performance. Please also note that larger guidance scales lead to comparable performance to our optimal scale, demonstrating the robustness of TPG with respect to the guidance scale.

Table 2: Quantitative comparison across different guidance scales (σ) for unconditional generation using SDXL, evaluated over 5k generated samples.

Analysis of the Behavior of TPG and CFG

We thank the reviewer for this question. As discussed in the paper, our primary motivation came from the observation that existing guidance perturbation methods struggle to mimic the behavior of CFG, particularly during the early denoising steps. This limits their effectiveness and results in lower image quality and weaker prompt alignment across various settings. Unlike prior approaches that perturb weights or attention modules, we chose to operate in token space by introducing guidance signals through shuffling, which disrupts local patterns without altering the global statistics of the features. That said, we acknowledge that our findings are mainly empirical, and a rigorous theoretical analysis of TPG would be a valuable direction for future work.

We appreciate the reviewer’s thoughtful comments and their recognition of TPG’s strengths. Please find our detailed responses below, and we would be happy to engage in further discussion if needed.

Why is orthogonality desirable?

We thank the reviewer for raising this important point and would like to provide additional clarification, which we will also clearly highlight in the revised paper. The concept of orthogonality originates from the observations made by [1]. They demonstrated that the CFG update term can be decomposed into parallel and orthogonal components. Importantly, the parallel component primarily contributes to oversaturation, whereas the orthogonal component enhances image quality. Inspired by this insight, we analyzed the orthogonality of guidance vectors relative to the ground truth noise. Our observations indicate that both TPG and CFG maintain a near-zero cosine similarity to the ground truth noise during initial and intermediate sampling steps, demonstrating similar behavior. In contrast, SEG and PAG exhibit distinctly different alignment patterns. We discuss these results further in Section 4 of our paper, and additional analyses can be provided upon request.

How to perform the perturbation

We apply TPG to all time steps and down-sampling layers in the U-Net architecture, except for the initial layers, which we empirically found to degrade performance when perturbed. As such, these initial layers are excluded. To maintain consistency across architectures, we shuffle all input tokens before applying positional encoding in the transformer blocks. We will provide further details and explicitly clarify these design choices in the final version of the paper. Additionally, we have conducted an ablation study on layer selection for SDXL in the unconditional generation setting, with the results reported below.

Table 1: Quantitative comparison of shuffling applied to down, mid, and up layers for unconditional generation using SDXL, evaluated over 5k generated samples.

Question about evaluation and model selection

We acknowledge the reviewer’s concern regarding our evaluation and would like to provide further clarification. Since most prior works, such as PAG and SEG, are specifically designed for U-Net–based diffusion models, we prioritized our experiments on SDXL and SD2, both well-established models, to ensure a fair experimental setup and comparability with prior work. In response to the reviewer’s suggestion, we have also evaluated the performance of TPG on Stable Diffusion 3, which employs a completely different training setup and a transformer-based architecture. The results of this comparison are reported below and will be included in the paper. These results confirm the robustness and generality of TPG, showing that its effectiveness extends beyond any particular model or architecture.

Table 2: Quantitative comparison of Vanilla SD3, TPG, and PAG for unconditional generation, evaluated over 5k generated samples.

Table 3: Quantitative comparison of Vanilla SD3, CFG, TPG, and PAG for conditional generation, evaluated over 5k generated samples.

Unconditional generation results in Table 2

We thank the reviewer for raising this point. We have noticed that the mention of "conditional generation" in Table 2 is a typo; the results shown correspond to unconditional generation using SD2.1. Since unconditional generation is generally a more challenging task, our intention was to highlight the effectiveness of TPG in this setting. We will correct this in the revised version of the paper and also include results for conditional generation with SD2.1 for completeness.

Exploring other perturbations

Following the reviewer’s suggestion, we conducted an additional experiment to explore perturbations that are not norm-preserving. One such perturbation is applying a Gaussian blur kernel to the input tokens, which we refer to as Token Blurring. As shown in the following table, Token Blurring demonstrates suboptimal performance compared to Token Shuffling in TPG. We present the results of this perturbation below, and we will include these findings in the paper.

Table 4: Quantitative comparison of Token Shuffling (ours), Token Blurring, and Vanilla for unconditional generation using SDXL, evaluated over 5k generated samples.

---

[1] Sadat S, Hilliges O, Weber RM. Eliminating oversaturation and artifacts of high guidance scales in diffusion models. In The Thirteenth International Conference on Learning Representations 2024 Oct 3.

We would first like to thank the reviewer for finding our proposed method to be an interesting idea and for acknowledging the well-organized and clear writing of our paper. We would like to address the concerns and questions raised as follows:

Comparison with AutoGuidance

AutoGuidance [1] uses a smaller/weaker version of the same conditional model for guidance. This requires training a separate model with reduced capacity and shorter training time, as well as accessing two models during sampling. In contrast, our method involves no additional training or architectural modifications. Since a weaker version of Stable Diffusion is not publicly available, a direct comparison between our method and AutoGuidance is not feasible. Training such a model would exceed our computational resources, requiring several weeks on multiple high-end GPUs. On the other hand, our method is training-free and can be readily applied on top of any pretrained model, which we consider an important advantage of TPG over AutoGuidance.

Simplicity of TPG

We acknowledge the simplicity of our proposed method, as the reviewer noted. However, we view this straightforwardness as a key strength rather than a limitation. TPG offers a simple and easy-to-implement adaptation for existing diffusion models that is condition-agnostic and requires no additional training. It consistently improves generation quality across a range of setups, including both conditional and unconditional generation. Notably, we believe that one of the main reasons for the popularity and widespread adoption of classifier-free guidance is its ease of use. While CFG is limited to conditional generation and requires a specific training setup, TPG can be seen as a similarly practical approach that extends these benefits to a broader class of diffusion models.

Analysis on why TPG works

As discussed in the paper, our main motivation came from the observation that other guidance perturbation methods fail to mimic the behavior of CFG, especially during the early denoising steps. This limits their effectiveness and results in lower image quality and weaker prompt alignment across various setups. Unlike previous approaches that operate in weight space or attention modules, we chose to operate in token space by forming guidance signals through shuffling, which disrupts local patterns without altering the global statistics of the features. One of our most important and interesting findings is that both TPG and CFG maintain a near-zero cosine similarity with the ground truth noise during the initial and intermediate steps. This aligns with the concept of orthogonality, previously introduced in [2]. It is believed that orthogonality enables more effective steering of the predicted direction, and generally, the orthogonal component contributes to improved image quality. We will clarify this point further in the final version of the paper.

Concerns about marginal improvements

We have shown that TPG not only narrows the gap between CFG and other training-free methods in conditional generation but also achieves state-of-the-art performance in the unconditional setting. As illustrated in Figure 4, other methods often fail to produce semantically coherent images, whereas TPG consistently generates results with higher visual quality and semantic consistency. This is further supported by our quantitative evaluations, where TPG outperforms other training-free guidance baselines by a noticeable margin. For example, Table 2 shows TPG achieves an FID of 69.31 compared to 124.04 for vanilla SDXL, which is nearly a 2$\times$ improvement. Taken together, we consider the improvements achieved by using TPG to be both fundamental and significant.

Compatibility with ViT-based models

We would like to emphasize that our method is not limited by model architecture and is compatible with ViT-based models (e.g., Stable Diffusion 3) as well. While previous perturbation-based guidance methods (e.g., PAG and SEG) were specifically designed for the U-Net architecture and may not generalize well to other backbones, we found that this limitation does not apply to TPG. To demonstrate this, we conducted an experiment comparing TPG with other baselines using Stable Diffusion 3, and report results for both conditional and unconditional generation below.

Table 1: Quantitative comparison of Vanilla SD3, TPG, and PAG for unconditional generation, evaluated over 5k generated samples.

Table 2: Quantitative comparison of Vanilla SD3, CFG, TPG, and PAG for conditional generation, evaluated over 5k generated samples.

This experiment shows that TPG not only achieves the highest quality in unconditional generation, but also performs nearly on par with CFG in the conditional setting. These results highlight the robustness of our method and confirm that TPG is not restricted to a single architecture. We will include this experiment in the final version of the paper.

Ablation on the guidance scale used in TPG

We assume the reviewer’s question refers to the guidance scale used in TPG, as our pipeline does not include any perturbation parameter. To address this, we have conducted a quantitative comparison across different guidance scales, and the results are given below:

Table 3: Quantitative comparison across different guidance scales (σ) for unconditional generation using SDXL, evaluated over 5k generated samples.

We will include this experiment as an ablation study in the final paper. Although a scale of 3 yields the best performance, TPG is robust to this choice, with other values also achieving reasonable results relative to the baseline ($\sigma=0$).

Quantifying the benefits of early-to-middle denoising steps

To the best of our knowledge, there is no established metric that directly quantifies the impact of early-to-middle denoising steps in the sampling process. However, we believe these steps play a crucial role in shaping the semantics of the generated images. As such, improvements in FID and CLIP scores may be interpreted as indirect evidence of effective denoising in early sampling stages. As shown in Tables 1 and 2, our method achieves the best FID and CLIP scores in both unconditional and conditional generation settings compared with other perturbation-based guidance methods. This likely reflects the benefits of enhanced early-to-middle denoising and aligns with the behavior observed in CFG. Additional results are provided in the Appendix and more can be added upon request if needed.

Ablation study for layer selection

Following prior works such as SEG and PAG for U-Net-based models, we tested perturbing the downsampling, mid, and upsampling layers separately. Our initial experiments showed that TPG is most effective when the perturbation is applied only to the downsampling layers (i.e., the encoder part of the U-Net). We also experimented with combinations of down, mid, and up layers, but these did not lead to improvements and, in some cases, resulted in degraded performance. In response to the reviewer’s question, we have conducted an ablation study on layer selection for SDXL in the unconditional generation setting. The results are provided below and will be included in the final version of the paper.

Table 4: Quantitative comparison of shuffling applied to down, mid, and up layers for unconditional generation using SDXL, evaluated over 5k generated samples.

---

[1] Karras T, Aittala M, Kynkäänniemi T, Lehtinen J, Aila T, Laine S. Guiding a diffusion model with a bad version of itself. Advances in Neural Information Processing Systems. 2024 Dec 16;37:52996-3021.
[2] Sadat S, Hilliges O, Weber RM. Eliminating oversaturation and artifacts of high guidance scales in diffusion models. In The Thirteenth International Conference on Learning Representations 2024 Oct 3.