RepLDM: Reprogramming Pretrained Latent Diffusion Models for High-Quality, High-Efficiency, High-Resolution Image Generation

We appreciate your kind words and recognition of the significance of our contributions. We did our best to address your questions as follows.

W1: On the validation of PFSA’s effectiveness

We fully understand your concerns about the effectiveness of PFSA. To address them, we have conducted the following efforts:

• In Section A of the supplementary material, we provide extensive analysis of the underlying mechanism of PFSA, which we elaborate on in our responses to Q3 and Q4.
• We incorporate PFSA into ControlNet (see Section C of the appendix), demonstrating its applicability under different generation conditions.
• We apply PFSA to various high-resolution training-free generation frameworks, including DemoFusion and HiDiffusion (Section D of the appendix), and observe consistent improvements, highlighting its generalizability.
• We also integrate PFSA into Stable Diffusion 2.1, showing that PFSA is compatible with different versions of SD models.
• Finally, we commit that, if the paper is accepted, we will release the RepLDM codebase promptly to facilitate reproducibility and further research within the community.

W2: On the detailed explanation of global structural consistency

Great suggestion. In this paper, global structural consistency refers to the plausibility of the overall layout and the realism of object structures in an image. Specifically, a reasonable layout should follow logical spatial arrangements—for example, the sky should be above the ground. Realistic object structures should align with common sense, such as a cat having four legs instead of five. We will incorporate this explanation in the next version of the paper.

W3: On the explanation of the effectiveness of Attention Guidance

To illustrate how Attention Guidance enhances the global structural consistency of latent representations, we conducted extensive experiments.

1. As shown in Fig. 12 of the supplementary material, we compute the deviation between the mean of each token and the overall mean across all tokens, and visualize this deviation. In columns (A) and (B) of Fig. 12, it can be observed that PFSA introduces pronounced stripe patterns in the token deviations, suggesting that it performs a clustering effect on tokens. Columns (C) and (D) present the heat maps of the deviation of tokens' mean, where it becomes evident that PFSA effectively aggregates semantically similar tokens, thereby enhancing the structural integrity and contours of the image.
2. To provide a more intuitive explanation, we conduct additional experiments to illustrate the effect of Attention Guidance. Specifically, given a clean RGB image $x$, we obtain its latent representation $z$ through VAE encoding. We then apply varying levels of noise to $z$: $z_t = \alpha_t \cdot \epsilon + (1 - \alpha_t) \cdot z$, where $t \in$ {$1,\ldots,9$}, $\alpha_t = 0.1\cdot t$, and $\epsilon \sim \mathcal{N}(0, I)$ is standard Gaussian noise. For $t=9,\ldots,1$, we compute $z_t^{PFSA}=PFSA(z_t)$ and $z_t^{AG} = \gamma_t \cdot z_t^{PFSA} + (1 - \gamma_t) \cdot z_t$, where $\gamma_9=1$ and gradually decays to $\gamma_1=0$ following the cosine annealing schedule. This process reflects the gradually diminishing effect of Attention Guidance (AG) during generation. We then decode $z_t$，$z_t^{PFSA}$, and $z_t^{AG}$ back to the RGB space, obtain $x_t$，$x_t^{PFSA}$, and $x_t^{AG}$, and compute their SSIM with respect to the original image $x$. The results are shown in the table below:

It can be observed that directly applying PFSA improves SSIM in the early steps of denoising, but leads to a drop in SSIM in the middle and later steps. This explains why directly using PFSA could result in unstable denoising, as noted in Section 3.2. By incorporating the annealed strategy of Attention Guidance, we can leverage PFSA to promote structural recovery while avoiding numerical instability, thereby improving the visual quality of the generated images. Qualitative results demonstrate that Attention Guidance enables the more rapid removal of noise and the earlier unveiling of image structure. Due to policy constraints, we will provide the visual results in the next version of the paper. The results show that the smoothing effect of PFSA can suppress part of the noise, thereby enhancing structural similarity.

A similar phenomenon can also be observed in a recent work, FreeU [Ref1], where image quality is improved by enhancing low-frequency components while suppressing high-frequency components. Interestingly, despite the suppression of high-frequency components, the generated images exhibit richer colors and details (as shown in their Figs. 10, 17, and 18). To sum up, these findings may suggest that reinforcing structural information during generation provides the model with a stronger prior for synthesizing the remaining fine details, thereby improving the overall visual quality of the images.

[Ref1] FreeU: Free Lunch in Diffusion U-Net. CVPR2024.

W4: On the numerical analysis of PFSA

To understand the effect of PFSA on the distribution of latent representations, we conducted a numerical analysis comparing latent features with and without the application of PFSA. Specifically, in Fig. 13 of Section A in the supplementary material, we compute the standard deviation and the Fourier amplitude spectrum of the latent representations with and without applying PFSA at different denoising timesteps. We observe that during the early and middle stages of denoising, PFSA increases the variance of the latent representations and amplifies the high-frequency components. In contrast, during the later stages, PFSA suppresses high-frequency components and reduces the standard deviation. This partially explains why attention guidance is more effective when applied during the early stages of denoising, but becomes less impactful in the later stages (as discussed in Fig. 9 of Section 5.1).

W5: On the further visualizations for better understanding PFSA

According to your suggestion, we conducted an additional visualization experiment: We randomly selected a latent token as the query and visualized its attention weights as a heatmap over the generated RGB image to highlight the regions most related to the query token. The results show that even without the linear projection layers for queries and keys used in conventional self-attention, the query token can still identify the key tokens that are most semantically relevant to the query token. This explains why PFSA is able to cluster semantically related tokens and enhance global structural consistency, even in the absence of learnable linear layers.

Unfortunately, we were informed via email that external links are not permitted in the rebuttal, and we are therefore temporarily unable to show these visualizations. However, we will incorporate the experiment in the next version of our paper.

W6: On the efficiency comparison across different models

Great suggestion. To the best of our knowledge, existing training-free high-resolution generation frameworks [Ref1,Ref2,Ref3] evaluate efficiency solely based on inference time. This is primarily because these frameworks utilize the exact same pretrained model, ensuring identical parameters across comparisons. Nevertheless, we agree with your point that a more comprehensive set of metrics should be used to reflect efficiency. Following your suggestion, we have collected the NFE and FLOPs of different models when generating a single image at 2048 × 2048 and 4096 × 4096 resolutions. As all models employ training-free methods and have identical parameter counts, we do not report the number of parameters.

Table 1: Inference cost of generating a 2048 × 2048 Image for different models.

Table 2: Inference cost of generating a 4096 × 4096 Image for different models.

Tables 1 and 2 show that RepLDM significantly reduces the NFE and FLOPs required for inference by decreasing the number of denoising steps at high resolutions, thereby substantially reducing the time needed to generate high-resolution images.

[Ref1] AccDiffusion: An Accurate Method for Higher-Resolution Image Generation. ECCV 2024.
[Ref2] HiDiffusion: Unlocking Higher-Resolution Creativity and Efficiency in Pretrained Diffusion Models. ECCV2024.
[Ref3] DemoFusion: Democratising High-Resolution Image Generation With No . CVPR 2024.

Thank you very much for your positive comments and efforts in reviewing our manuscript. We try our best to address your questions as follows.

W1: On the efficiency of ultra-high-resolution image generation

For training-free image generation at ultra-high resolutions, the inference time is expected to increase further, a challenge commonly encountered by existing methods. However, our RepLDM still achieves a significant reduction in overall generation time compared to other frameworks, such as DemoFusion, despite employing a progressive upsampling sub-stage. This is because our RepLDM integrates an upsampling sub-stage within the entire denoising process over T timesteps, whereas in existing methods, the upsampling sub-stage is applied after each denoising process. We effectively reduce the number of denoising steps required at high resolutions, which are the primary source of computational overhead, as illustrated in Fig. 21 of the supplementary material.

W2: On the error analysis of the quantitative metrics

Thank you for the constructive suggestion. Due to the high computational cost of high-resolution image generation, repeating experiments to compute standard deviations is expensive. As a result, most prior works do not perform error analysis [Ref1-Ref6]. To achieve a more rigorous evaluation, we conducted additional experiments to analyze the variability of the results. Specifically, for each experiment, we performed multiple generations (3 times) with different random seeds, computed the evaluation metrics for each generation, and reported the mean and standard deviation. Due to limitations in time and computational resources, we first performed this error analysis for RepLDM at a resolution of $2048 \times 2048$. We will extend this analysis to other resolutions and baseline models, and include the results in the next version of the paper. The results are shown in the table below:

Table 1: Quantitative comparison results. The best results are marked in bold, and the second best results are marked by $\underline{\text{underline}}$.

The results show that FID, FID_c, and CLIP exhibit relatively small variance, while IS shows comparatively larger fluctuations. This may be attributed to the fact that IS is not well-suited for evaluation on datasets other than ImageNet [Ref7]. In our future work, we will adopt more reliable metrics, such as KID, to replace IS for more accurate evaluation [Ref8].

[Ref1] DemoFusion: Democratising High-Resolution Image Generation With No. CVPR2024.
[Ref2] AccDiffusion: An Accurate Method for Higher-Resolution Image Generation. ECCV2024.
[Ref3] FouriScale: A Frequency Perspective on Training-Free High-Resolution Image Synthesis. ECCV2024.
[Ref4] ScaleCrafter: Tuning-free Higher-Resolution Visual Generation with Diffusion Models. ICLR2024.
[Ref5] Training-free Diffusion Model Adaptation for Variable-Sized Text-to-Image Synthesis. NeurIPS2023.
[Ref6] HiDiffusion: Unlocking Higher-Resolution Creativity and Efficiency in Pretrained Diffusion Models. ECCV2024.
[Ref7] A Note on the Inception Score. ICML2018.
[Ref8] Demystifying MMD GANs. ICLR2018.

Q1: On the efficacy of PFSA

Thank you for your suggestion. We would like to note that PFSA itself does not contain any parameters, and thus its efficacy is not affected by the input. Notably, PFSA influences the efficacy of Attention Guidance through the parameter $\gamma$, thereby altering the generation results. Therefore, $\gamma$ is a key parameter that significantly impacts generation quality. Furthermore, for evaluation metrics, while the SSIM metric is frequently employed in image super-resolution, our study primarily focuses on text-to-image generation, where ground truth images are unavailable. In this paper, we utilize commonly adopted metrics in generation tasks—FID, IS, and CLIP—to comprehensively assess image quality, consistent with previous methods. In addition, considering that our work targets high-resolution image generation, we follow prior works [Ref1,Ref2] and compute FID and IS on cropped patches of high-resolution images, denoted as $\text{FID}_c$ and $\text{IS}_c$. The results in Table 1 demonstrate the robustness of our RepLDM. In Section 5.1, we conduct both qualitative and quantitative ablation studies on $\gamma$, and provide a more comprehensive analysis in Section F.1 of the supplementary material. Our ablation results suggest that, within a certain range, larger values of $\gamma$ tend to introduce more fine-grained details and lead to better visual quality in the generated results.

[Ref1] DemoFusion: Democratising High-Resolution Image Generation With No $$$. CVPR2024.
[Ref2] AccDiffusion: An Accurate Method for Higher-Resolution Image Generation. ECCV2024.

Q2: On the computational complexity analysis of PFSA

Great suggestion. We would like to note that PFSA is only applied during the first stage of generation, and therefore does not introduce any computational or memory bottlenecks. Assume we have a high-resolution image $x$ with a resolution of $H \times W \times C$. we encode the image $x$ into latent space and obtain latent representation $z \in \mathbb{R}^{h \times w \times c}$. Before feeding $z$ into PFSA, we reshape it to a $(hw) \times c$ matrice. The computation of PFSA follows a formulation similar to that of self-attention:
$\text{PFSA}(z)=\text{Softmax}(\frac{zz^T}{\sqrt{c}})z$
Thus, the computational complexity of PFSA is $O((hw)^2c)$. Taking SDXL as an example, the training resolution is $H=1024$, $W=1024$. After VAE encoding, $c=4$, $h=H/8=128$, $w=W/8=128$. For each denoising step, the FLOPs of PFSA is approximately $2 \times (h \times w)^2 \times c$, which is around 2.15 GFLOPs—negligible compared to the FLOPs of the denoising network (several TFLOPs per step).

Q3: On the pixel space upsampling scheme

(a) The proposed RepLDM avoids artifacts caused by upsampling in the latent space by decoding latent representations into the pixel space for upsampling. In practice, we use bicubic interpolation to perform this upsampling, which does not rely on a convolutional kernel of fixed size. Alternatively, a super-resolution network can be employed to accomplish this upsampling step, potentially leading to better generation quality.
(b) We believe your proposed approach is insightful and highly feasible. We have previously considered a similar idea—training a neural network to directly perform upsampling in the latent space—which may align with your suggestion of using a blind model to mitigate the blurring artifacts caused by latent-space interpolation. We find your suggestion very constructive. To train such a blind model, a sufficiently large and high-quality high-resolution dataset would be required. Due to the limited time during the rebuttal phase, we will leave this as a promising direction for future exploration.

Dear Reviewer ZoSx,

Thank you once again for your comprehensive and thoughtful feedback on our submission. As the discussion period nears its end, we are eager to know if our additional results and clarifications have adequately addressed your questions. We would sincerely appreciate any further perspectives or discussions you might have at this stage. Thank you for your time and engagement!

Best regards,

Authors of Paper 15032

Thank you very much for your insightful comments and suggestions. We try our best to address your questions as follows.

W1: On the definitions of training resolution and high resolution

Great suggestion. To improve clarity, we will refine the annotation of the abbreviations HR and TR, and include the following explanation as a footnote in the next revision: In this paper, training resolution refers to the resolution used during model training; high resolution refers to a resolution that significantly exceeds the training resolution, to the extent that the model cannot directly generate satisfactory results at that resolution.

W2: On the efficiency analysis of intermediate encoding-decoding operations

To evaluate the computational cost introduced by multiple encoding and decoding operations, we analyze the time consumption of each component in DemoFusion and RepLDM when generating images at the resolution of $4096 \times 4096$. The results are shown in the table below:

Table 1: The time consumption of DemoFusion when generating 4096×4096 resolution images.

Table 2: The time consumption of RepLDM when generating 4096×4096 resolution images. The intermediate encoding/decoding operations are highlighted in $\underline{\text{underline}}$.

As shown in Table 1, denoising at high resolutions is a time-consuming process. DemoFusion requires substantial generation time because it performs the full denoising process at high resolutions. Table 2 shows that RepLDM significantly accelerates generation by substantially reducing the number of denoising steps at high resolutions. This is because RepLDM performs pixel-space upsampling through multiple rounds of encoding and decoding, producing high-quality low-resolution images that serve as better initialization. As a result, RepLDM can significantly reduce the number of sampling steps required for high-resolution generation, thereby accelerating the process. Moreover, Table 2 shows that the additional overhead from multiple intermediate encoding and decoding operations is relatively minor compared to the total generation cost.

Q1: On the analysis of the effectiveness of Attention Guidance

Very insightful comment. To provide an intuitive explanation, we conduct additional experiments to illustrate the effect of Attention Guidance (AG). Specifically, given a clean RGB image $x$, we obtain its latent representation $z$ through VAE encoding. We then apply varying levels of noise to $z$: $z_t = \alpha_t \cdot \epsilon + (1 - \alpha_t) \cdot z$, where $t \in$ {$1,\ldots,9$}, $\alpha_t = 0.1\cdot t$, and $\epsilon \sim \mathcal{N}(0, I)$ is standard Gaussian noise. For $t=9,\ldots,1$, we compute $z_t^{PFSA}=PFSA(z_t)$ and $z_t^{AG} = \gamma_t \cdot z_t^{PFSA} + (1 - \gamma_t) \cdot z_t$, where $\gamma_9=1$ and gradually decays to $\gamma_1=0$ following the cosine annealing schedule. This operation reflects the progressively diminishing influence of Attention Guidance during the generation process. We then decode $z_t$，$z_t^{PFSA}$, and $z_t^{AG}$ back to the RGB space, obtain $x_t$, $x_t^{PFSA}$, and $x_t^{AG}$, and compute their SSIM with respect to the original image $x$. The results are shown in the table below:

Table 3: The SSIM values of $x_t$, $x_t^{PFSA}$, and $x_t^{AG}$ with respect to the original image $x$. The best results are highlighted in bold.

It can be observed that directly applying PFSA improves SSIM in the early steps of denoising, but leads to a drop in SSIM in the middle and later steps. This explains why directly using PFSA could result in unstable denoising, as noted in Section 3.2. By incorporating the annealed strategy of Attention Guidance, we can leverage PFSA to promote structural recovery while avoiding numerical instability, thereby improving the visual quality of the generated images. Qualitative results demonstrate that Attention Guidance enables the more rapid removal of noise and the earlier unveiling of image structure. Due to policy constraints, we will provide the visual results in the next version of the paper. The results show that the smoothing effect of PFSA can suppress part of the noise, thereby enhancing structural similarity.

A similar phenomenon can also be observed in a recent work, FreeU [Ref1], where image quality is improved by enhancing low-frequency components while suppressing high-frequency components. Interestingly, despite the suppression of high-frequency components, the generated images exhibit richer colors and details (as shown in their Figs. 10, 17, and 18). To sum up, these findings may suggest that reinforcing structural information during generation provides the model with a stronger prior for synthesizing the remaining fine details, thereby improving the overall visual quality of the images.

[Ref1] FreeU: Free Lunch in Diffusion U-Net. CVPR2024.

Q2: On the similar quantitative ablation results of Attention Guidance

Thank you for your suggestion. We would like to point out that FID and IS quantify the statistical differences between two distributions [Ref1,Ref2,Ref3]. Since Attention Guidance mainly enhances visual quality by modifying the mid- and high-frequency components while preserving the low-frequency structure of the image, it has limited impact on the overall distributional statistics. Consequently, the resulting quantitative metrics remain similar.

[Ref1] A Note on the Inception Score. ICML2018.
[Ref2] Improved Techniques for Training GANs. NeurIPS2016.
[Ref3] Gans trained by a two time-scale update rule converge to a local nash equilibrium. NeurIPS2017.

Q3: On the evaluation metrics and datasets

During our experiments, Laion-5B was removed from Huggingface due to potential content risks. We note that the SAM dataset is of high resolution (average: $3300 \times 4950$). Although the released version is a downsampled version with the shorter side resized to 1500, the resolution still remains significantly higher than that of many existing vision datasets (e.g., COCO) [Ref1]. Therefore, we chose the SAM dataset as a substitute. For the evaluation metrics and the number of generated samples used to evaluate performance, we follow the settings used in several related works, where 1k prompts are used to generate $4096 \times 4096$ images for evaluation [Ref2]. This choice is made due to the extremely high inference cost associated with generating high-resolution images.

We highly appreciate your constructive suggestions regarding the experiments. For a more rigorous comparison, we conducted additional experiments:

• We recently discovered the availability of high-resolution Laion images on Huggingface and downloaded 20K images to serve as the benchmark.
• To enable more stable evaluation, we doubled the number of prompts used to generate $4096 \times 4096$ resolution images. Due to the substantial time required to generate 4096 × 4096 resolution images and current hardware constraints, we will provide the relevant experimental results in the next version of the paper.
• For RepLDM, we performed multiple generations (3 times) at 2048×2048 resolution using different random seeds and computed the mean and standard deviation. Due to time and hardware constraints, we will conduct the same variance analysis for other models as well.
• Regarding evaluation metrics, since Inception Score (IS) may lead to high variances beyond ImageNet, we follow prior work and adopt Kernel Inception Distance (KID) for more accurate evaluation [Ref6,Ref7].

Due to time and hardware constraints, we have temporarily compared our RepLDM with the top three baselines from Table 1 of the paper: DemoFusion, AccDiffusion, and HiDiffusion. We will continue to extend the experiments and include the results and analyses in the next version of the paper. The results are shown in the table below:

Table 4: Quantitative comparison results at $2048 \times 2048$ resolution. The best results are marked in bold.

Table 4 shows that RepLDM still achieves state-of-the-art performance on the Laion dataset, further highlighting its strong generalization capability.

[Ref1] Segment Anything. ICCV2023.
[Ref2] DemoFusion: Democratising High-Resolution Image Generation With No . CVPR2024.
[Ref3] A Note on the Inception Score. ICML2018.
[Ref4] ScaleCrafter: Tuning-free Higher-Resolution Visual Generation with Diffusion Models. ICLR2024.
[Ref5] Demystifying MMD GANs. ICLR2018.

We are grateful for your positive summary and constructive suggestions. We have tried our best to address your concerns as follows.

W1: On the definition of “parameter-free”

Good catch! In our paper, the term “parameter-free” in PFSA was intended to emphasize the absence of learnable parameters, in contrast to vanilla self-attention with learnable parameters. We appreciate your suggestion and will adopt a more accurate terminology in the next version by replacing Parameter-Free Self-Attention (PFSA) with Training-Free Self-Attention (TFSA).

W2: On the fundamental flaws of the base LDM

We would like to clarify this point from three perspectives.

• We would like to note that training-free frameworks are inherently limited by the capabilities of their underlying base models, which is a widely observed issue. To the best of our knowledge, existing training-free generation frameworks have not effectively addressed this limitation. Some efforts to address the limitations of base models, such as improving fine-grained text generation, rely on fine-tuning the base model with specific datasets [Ref1,Ref2,Ref3].
• Although fine-tuning the base model can alleviate some of its inherent limitations, it may come at the cost of degrading its generative quality. In comparison, the reprogramming framework better retains the base model’s original generation capacity.
• Our proposed RepLDM is a general framework for high-resolution generation. In Section C of the supplementary material, we demonstrate that RepLDM is also compatible with different generation conditions. This suggests that RepLDM can be applied to alternative, improved base models to circumvent certain limitations of the original base model.

[Ref1] AnyText: Multilingual Visual Text Generation And Editing. ICLR2024.
[Ref2] TextDiffuser-2: Unleashing the Power of Language Models for Text Rendering. ECCV2024.
[Ref3] TextDiffuser: Diffusion Models as Text Painters. NeurIPS 2023.

Q1: On the different PFSA calculation paradigm

For PFSA, our objective is to remove the learnable parameters from the Self-Attention mechanism, while maintaining its computational paradigm as unchanged as possible. Therefore, we opted to use symmetric $Q$ and $K$ matrices for computing PFSA. We appreciate your suggestion of using asymmetric $Q$ and $K$ matrices for PFSA computation, which we find to be a very interesting idea. To verify its effectiveness, we conducted additional experiments accordingly. Specifically, we apply a 2 $\times$ 2 pooling operation to downsample the $K$ and $V$ matrices. This ensures that the output of $\text{Softmax}(QK^T/\sqrt{d})V$ remains of shape $(hw) \times c$. We then use this new output tensor for guidance, while keeping all other hyperparameters unchanged. The results at 2048 $\times$ 2048 resolution are shown in the table below (the best results are marked in bold):

Quantitatively, all metrics, except for FID, show improvement. Qualitatively, compared to PFSA, asymmetric PFSA results in fewer details but demonstrates more advantages in structural adjustments. Due to conference policies, we are unable to provide access to these qualitative results at this time; however, we will include these analyses and experiments in the next version of the paper.

By the way, we have considered alternative attention paradigms for guidance, such as parameter-free linear attention mechanisms. However, our experiments showed that using a linear attention mechanism for guidance directly leads to meaningless noise or entirely black images.

Q2: On the difference between our PFSA and PFAM proposed by Wan et al.

We appreciate your recommendation and have read the paper. In the research conducted by Wan et al., they proposed a parameter-free attention mechanism (PFAM) to replace self-attention for more efficient image super-resolution. PFAM differs from the conventional Self-Attention in that it does not compute the similarity between tokens. Instead, the attention map in PFAM is obtained by applying the proposed symmetric activation function to the output of the convolutional layer. The output of PFAM is the result of an element-wise multiplication between the attention map and the features.

Overall, our PFSA differs from their attention mechanism (PFAM) in the following aspects:

• Task objective: PFSA is designed for generative tasks, aiming to improve the quality of high-resolution outputs. In contrast, PFAM focuses on super-resolution, where maintaining consistency between high- and low-resolution images is the primary goal.
• Modeling approach: PFSA retains the computational paradigm of self-attention and can be considered a specialized form of self-attention. In contrast, PFAM does not compute token similarity; instead, it focuses on the magnitude of tensor elements. Conceptually, its attention mechanism differs quite significantly from that of Self-Attention.
• Training strategy: PFSA operates without any training; although PFAM is parameter-free, it still requires joint training with convolutional layers to be effective.

Q3: On the stability analysis of PFSA and the role of $(1 - \gamma)z$

Very insightful comment. First, we would like to draw your attention to Section A of the supplementary material, where we provided a stability analysis of the PFSA. Specifically: (i) We compute the deviation of token means and observe that PFSA exhibits a clustering effect, bringing semantically related tokens closer together (Fig. 12). (ii) We compute the standard deviation of latent representations and their Fourier amplitude spectra, and find that PFSA is capable of modulating both the amplitudes across different frequency bands and the standard deviation of latent representations (Fig. 13).

These analyses suggest that PFSA may operate by modulating the distribution of latent representations. To make this effect more gradual and controlled, we introduce a guidance mechanism to adjust the representations progressively.

Your observation is highly insightful — the use of a linear combination with $(1 - \gamma)z$ serves to anchor the guided latent representations, helping to ensure that the latent representations remain on the data manifold. We will incorporate the explanation from the manifold perspective you mentioned in Section 3.2 of the next version of the paper.

Q4: On the sensibility analysis of annealing strategy

Great suggestion. Previous studies have shown that diffusion models first generate low-frequency structures and then generate fine details [Ref4,Ref5,Ref6]. Based on this observation, we introduce the annealing strategy to gradually decrease the structural guidance effect of PFSA. Considering that cosine annealing has demonstrated robustness in various applications (e.g., learning rate decay), we employ cosine annealing in RepLDM. We appreciate your constructive suggestion. Following your recommendation, we performed additional experiments using linear and exponential annealing strategies at 2048 $\times$ 2048 resolution to investigate the sensitivity of Attention Guidance to the choice of annealing strategy. The results are shown in the table below:

Quantitatively, comparable performance is observed across different annealing strategies. Qualitatively, the generated results also exhibited comparable improvements in structure and details. This suggests that Attention Guidance is not sensitive to a specific annealing strategy. Due to conference policy restrictions, we are unable to provide access to the qualitative analysis at this time, but we will include these analyses in the next version of the paper.

[Ref4] FreeEnhance: Tuning-Free Image Enhancement via Content-Consistent Noising-and-Denoising Process. ACM MM2024.
[Ref5] Relay Diffusion: Unifying diffusion process across resolutions for image synthesis. ICLR2024.
[Ref6] FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model.

Q5: On the generalization of hyperparameters to other pretrained LDMs

Great questions! We would like to draw your attention to Section E of Supplementary Material. To verify the generalization of the hyperparameters, we conducted experiments using the guidance hyperparameter settings of SDXL to Stable Diffusion 2.1. The results demonstrate that the hyperparameters used for SDXL are also applicable to SD2.1. Furthermore, we tested the same settings on ControlNet and other training-free generative frameworks (HiDiffusion, DemoFusion), with all results confirming the robustness of this hyperparameter set.

Thank you for the thorough and thoughtful rebuttal. Your clarifications on the “parameter-free” terminology, additional experiments exploring asymmetric PFSA, and the new annealing-schedule and stability analyses directly address my main questions. I appreciate the side-by-side comparisons (e.g., Table A-sym PFSA) and the evidence of hyper-parameter robustness across SD v2.1, ControlNet, DemoFusion, and HiDiffusion. All remaining issues are minor (terminology and small wording edits). I therefore maintain my overall positive assessment and still recommend acceptance. I look forward to seeing the updated manuscript incorporating the promised revisions and additional results. Thus, I will raise to accept.