About Supplementary Materials

Our full paper file exceeds 100MB. Due to upload size limitations, we have submitted a compressed version of the paper. To ensure clear visualization of the images, we have uploaded pages 18–27 of the appendix as supplementary materials. Please refer to these pages for further details!

Response to weaknesses

1. In fact, structural and object repetition issues are caused by the shift in the entropy of the attention maps in the Transformer layers during the early denoising stages, where low-frequency structures are generated first[1, 2, 3]. Previous works utilized the latent representations of low-resolution images as conditions to guide the denoising of high-resolution images, or attempted to correct the entropy to alleviate this issue. We discard these previous paradigms, and propose a novel PFSA serving to enhance the quality of low-resolution initialization images. Specifically, PFSA improves the quality of low-resolution images by clustering related tokens in the latent representation and adjusting the relative strength of the high-frequency and low-frequency components. Then, AP-LDM uses pixel-space interpolation to upsample the resulting low-resolution image, adds noise, and performs denoising to obtain a high-quality high-resolution image. In this process, AP-LDM avoids directly generating low-frequency structures in high-resolution images, thereby addressing the issue of object and structure repetition. For a detailed explanation of the PFSA mechanism, please refer to Appendix A.5 in the revised paper.

2. To lay the groundwork for future extension studies, we demonstrate the generalizability of attentive guidance in other generative frameworks in Appendix A.7 of the revised paper. Specifically, we apply attentive guidance to HiDiffusion and DemoFusion for ablation studies to verify its generalizability. Extensive quantitative and qualitative results demonstrate that attentive guidance significantly improves the performance of both HiDiffusion and DemoFusion. Notably, attentive guidance not only enhances image details but also effectively mitigates the structural collapse issues in HiDiffusion and the repetitive structure problems in DemoFusion. This further demonstrates the effectiveness and generalizability of attentive guidance.

Response to Q1 and Q2

Thank you for your valuable suggestions for enhancing our effectiveness. In Appendix A.6 of the revised paper, we provide extensive qualitative and quantitative comparisons of AP-LDM with HiDiffusion and the super-resolution method using BSRGAN. For the HiDiffusion with high hardware demanding, we can only generate images at 2048x2048 resolutions in our RTX 3090 GPUs instead of its required V100 GPUs. As shown in Table 8, the quantitative results indicate that AP-LDM outperforms HiDiffusion, while SDXL+BSRGAN achieves quantitative results similar to those of AP-LDM. In the qualitative analysis (Figure 17), we observed that HiDiffusion's forced resizing of UNet feature maps often leads to structural breakdowns in the generated images, with frequent occurrences of structural and texture distortions. We speculate that this is the main reason for HiDiffusion's suboptimal performance in the quantitative experiments. For the qualitative analysis of SDXL+BSRGAN (Figure 18), although using BSRGAN for super-resolution does not provide sufficient image details, it consistently preserves the structural integrity of the images. We speculate that this is the reason for SDXL+BSRGAN's good performance in the quantitative experiments. The authors of DemoFusion also compared their method with SDXL+BSRGAN, and the patterns we observed were consistent with their findings. Due to equipment limitations, we will later supplement the comparison results with HiDiffusion at other resolutions.

Reference

[1] FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model. ICCV 2023.
[2] Relay Diffusion: Unifying diffusion process across resolutions for image synthesis. ICLR 2024.
[3] Training-free Diffusion Model Adaptation for Variable-Sized Text-to-Image Synthesis. NeurIPS 2023.

Dear Reviewer UP1g,

Thank you once again for your comprehensive and thoughtful feedback on our submission. As the discussion period nears its end, we are eager to know if our additional results and clarifications have adequately addressed your questions. We would sincerely appreciate any further perspectives or discussions you might have at this stage. Thank you for your time and engagement!

Best regards,

Authors

We sincerely appreciate your time and constructive feedback. Below, we would like to address your concerns systematically and comprehensively.

Response

1. We would like to note that we encountered an OOM issue when conducting HiDiffusion experiments to generate 4096X4096 resolution images by strictly following their official implementation. We speculate that this may be related to some memory optimization tricks. We have tried some memory-saving tricks and are able to run it now. We will update the corresponding results soon. Hopefully, we can get the results before rebuttal period closes. Regardless of whether we can meet the deadline, we will certainly include the relevant results in the next version of the paper.

2. We would like to note that quantitatively evaluating the generative models is challenging. Though we can use metrics such as FID and IS, it is widely acknowledged that these metrics fail to comprehensively evaluate the performance of model’s generation. As a result, user studies are commonly employed to provide human-level evaluation with more intuition[1-7]. For example, in ScaleCrafter, they conducted both quantitative and user study analyses in comparison with the SD+SR approach. Their results show that, although ScaleCrafter performs worse than SD+SR on quantitative metrics, users significantly prefer the textures and details generated by ScaleCrafter. One important reason is that the goal of the SR model is to produce images consistent with the input, which limits its performance in high-resolution generation – needing more detail for true high-resolution visuals beyond simple smoothing[6-11]. For the rigor of the paper, following your suggestion, we are currently conducting a user study to compare SDXL+BSRGAN and AP-LDM. Due to time constraints, we will present the results in the next version of the paper.

References

[1] Are GANs Created Equal? A Large-Scale Study. NeurIPS 2018.
[2] High-Resolution Image Synthesis with Latent Diffusion Models. CVPR 2022.
[3] SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis. ICLR 2024.
[4] Adding Conditional Control to Text-to-Image Diffusion Models. ICCV 2023.
[5] AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning. ICLR 2024.
[6] ScaleCrafter: Tuning-free Higher-Resolution Visual Generation with Diffusion Models. ICLR 2024.
[7] Training-free Diffusion Model Adaptation for Variable-Sized Text-to-Image Synthesis. NeurIPS 2023.
[8] DemoFusion: Democratising High-Resolution Image Generation With No $$$. CVPR 2024.
[9] AccDiffusion: An Accurate Method for Higher-Resolution Image Generation. ECCV 2024.
[10] Make a Cheap Scaling: A Self-Cascade Diffusion Model for Higher-Resolution Adaptation. ECCV 2024.
[11] Any-Size-Diffusion: Toward Efficient Text-Driven Synthesis for Any-Size HD Images. AAAI 2024.

We maintained the same settings as in the experiments described in the main text and tested the performance of HiDiffusion in generating $4096\times 4096$ images, as shown in the table below (the best results are marked in bold, and the second-best results are marked in italics):

From the table, we can observe that HiDiffusion's performance drops when synthesizing higher-resolution images. We speculate that this is due to the following reasons:
(1) HiDiffusion directly performs upsampling and downsampling on the UNet feature maps, leading to significant distribution shifts;
(2) HiDiffusion uses windowing, which forces a change in the sequence length of the attention, resulting in a severe shift in the entropy of the attention map.

We hope our supplementary experiments address your concerns. If you have any further questions, please do not hesitate to let us know before the rebuttal period ends. Your feedbacks are highly appreciated and helpful.

(We are currently conducting experiments at 2048x4096 and 4096x2048 resolutions. Hopefully, we can update these results before rebuttal period closes.)

We maintained the same settings as in the experiments described in the main text and tested the performance of HiDiffusion in generating $2048\times 4096$ and $4096\times 2048$ images.

• The comparison of generation results at $2048\times 4096$ resolution is shown in the table below (the best results are marked in bold, and the second-best results are marked in italics):

• The comparison of generation results at $4096\times 2048$ resolution is shown in the table below (the best results are marked in bold, and the second-best results are marked in italics):

• From the results in the two tables above, we can see that HiDiffusion's performance drops at higher resolutions, which is consistent with the trend observed in the $4096\times 4096$ resolution results we reported earlier. We speculate that the reasons for the performance drop in these higher resolutions are the same:
  • HiDiffusion directly performs upsampling and downsampling on the UNet feature maps, leading to significant distribution shifts;
  • HiDiffusion uses windowing, which forces a change in the sequence length of the attention, resulting in a severe shift in the entropy of the attention map.

We hope that the complete results we have provided could address your concerns. We greatly appreciate the time and effort you have dedicated to reviewing our work. Your suggestions have been extremely helpful in improving our work.

About Supplementary Materials

Our full paper file exceeds 100MB. Due to upload size limitations, we have submitted a compressed version of the paper. To ensure clear visualization of the images, we have uploaded pages 18–27 of the appendix as supplementary materials. Please refer to these pages for further details!

Response to Weaknesses

1. Following your suggestion, we conduct further comparisons and ablation experiments using the SD2.1 pre-trained model, as presented in Appendix A.8 of the revised paper. The experimental results demonstrate the generalization capability of AP-LDM. In the future, we will use SD2.1 to reproduce more models and conduct a more comprehensive comparison.

2. In Table 9 of Appendix A.6.1 in the revised paper, we provide memory usage details for different models when generating images at various resolutions. Some models exhibit lower memory usage for high-resolution image generation compared to lower resolutions due to the use of a tiled decoder for high-resolution images. This indicates that the memory bottleneck during high-resolution image generation lies not in the denoising process but in the VAE decoding process. In AP-LDM, the pixel space upsampling process is also not the memory bottleneck; instead, the bottleneck is in the VAE encoding process. To address this, we employed a tiled encoder for pixel encoding. As a result, while AP-LDM does not offer additional memory savings compared to other methods, it does enable the synthesis of ultra-high-resolution images—such as 4096x7280 or even higher—on consumer-grade GPUs like the 3090, as shown in Figure 1.

3. The goal of attentive guidance is to enhance the structural consistency of the latent representations, directly operating on the representations obtained after each denoising step. In diffusion models, the latent representation at step t serves as the output of step t+1 and simultaneously as the input to step t-1. In other words, attentive guidance works on both the model input and the model output, which is why we did not perform ablation studies specifically for it.

4. Following your suggestion, we have enlarged all the red bounding boxes in the paper. Due to space constraints in the main text, we have included close-up views in the appendix. You can find the comparisons with detailed close-ups in Figures 17-22 of the appendix. Due to time constraints, we will provide more close-up comparison results with other models in the next version.

Dear Reviewer 9bp6,

Thank you once again for your comprehensive and thoughtful feedback on our submission. As the discussion period nears its end, we are eager to know if our additional results and clarifications have adequately addressed your questions. We would sincerely appreciate any further perspectives or discussions you might have at this stage. Thank you for your time and engagement!

Best regards,

Authors

About Supplementary Materials

Our full paper file exceeds 100MB. Due to upload size limitations, we have submitted a compressed version of the paper. To ensure clear visualization of the images, we have uploaded pages 18–27 of the appendix as supplementary materials. Please refer to these pages for further details!

Response to weaknesses

In the revised paper, Appendix A.5 provides a detailed explanation of how PFSA improves structural consistency. Specifically, PFSA can cluster related tokens in the latent representation, which improves the structural consistency of the latent representation. At the same time, during the early and mid-stages of denoising, PFSA amplifies the high-frequency components of the latent representation, which results in generated images with richer details and colors. Please refer more details in Appendix A.5.

Response to Questions

• Response to Q1: Thank you for your constructive feedback. In fact, we have considered using gradients to optimize the guidance scale. The idea was to freeze the parameters of SDXL and learn the guidance scale. Specifically, we can sample a time step t from {1, …, T} and sample an image x_0, then optimize the guidance scale by predicting the noise added to x_0. It is worth noting that attentive guidance improves the image by adjusting the latent representation across multiple consecutive time steps. Intuitively, the guidance scale should be optimized across different time steps simultaneously, but achieving this is quite challenging. Currently, we have not found a suitable framework for learning the guidance scale.

• Response to Q2: A lot of research has shown that diffusion models generate images by first producing the low-frequency components and then the high-frequency components[1, 2]. As the denoising process progresses, the low-frequency structural information in the latent representation gradually increases, and the semantics become more pronounced. When t becomes large, due to the noise present and the fact that the representation is in the latent space, the semantic information is difficult to interpret directly. In the revised paper, in Appendix A.5, we provide a detailed explanation of PFSA. Through experiments, we demonstrate that PFSA can cluster tokens in the latent representation, even when noise is present (Figure 15). Additionally, we performed a Fourier transform on the latent representations at different time steps (Figure 16). From the Fourier transform, it is evident that when PFSA is applied in the early denoising process, the resulting latent representation exhibits a significant frequency band difference from the original latent representation. Later in the denoising process, applying PFSA causes only minor changes in the high-frequency components. This indirectly suggests that the early latent representations differ semantically from the later ones.

• Response to Q3: For each denoising step, the time consumption on high-resolution image generation stage is several times longer than that on low-resolution image generation stage. Methods like DemoFusion require the entire denoising process to be applied to high-resolution image generation stage, making them extremely time-consuming. Although there are differences in detail between high and low-resolution images, the low-frequency structures are consistent. AP-LDM fully leverages this property, allowing the denoising process for high-resolution images to avoid starting from scratch and instead only require 5-15 steps, thus significantly accelerating the generation process. In the revised version of our paper, in Appendix A.4, we provide detailed information on the time consumption for denoising at different resolutions.

Reference

[1] FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model. ICCV 2023.
[2] Relay Diffusion: Unifying diffusion process across resolutions for image synthesis. ICLR 2024.

Dear Reviewer 38vj,

Thank you once again for your comprehensive and thoughtful feedback on our submission. As the discussion period nears its end, we are eager to know if our additional results and clarifications have adequately addressed your questions. We would sincerely appreciate any further perspectives or discussions you might have at this stage. Thank you for your time and engagement!

Best regards,

Authors

Dear Reviewer XiPQ,

Thank you once again for your comprehensive and thoughtful feedback on our submission. As the discussion period nears its end, we are eager to know if our additional results and clarifications have adequately addressed your questions. We would sincerely appreciate any further perspectives or discussions you might have at this stage. Thank you for your time and engagement!

Best regards,

Authors

About Supplementary Materials

Our full paper file exceeds 100MB. Due to upload size limitations, we have submitted a compressed version of the paper. To ensure clear visualization of the images, we have uploaded pages 18–27 of the appendix as supplementary materials. Please refer to these pages for further details!

Response to Weaknesses

1. Although hyperparameter selection is empirical as in the most previous methods, we conduct comprehensive ablation experiments to provide a specific range of parameter selection, demonstrating our generalization capacity across different prompts and frameworks.

   • We followed most of training-free high-resolution generation methods with wide recognition, such as DemoFusion, AccDiffusion, and HiDiffusion, to tune hyperparameter experimentally. We would like to highlight that the key lies not in whether parameters are tuned empirically, but in whether the parameters derived from ablation studies are applicable to most scenarios. In our experiments, we identified a suitable range of parameters for attentive guidance and progressive upsampling when using the SDXL model. Through extensive evaluations, we demonstrated that this hyperparameter range is effective across various resolutions. In the revised version of our paper, Appendix A.8 further shows that the hyperparameter range, suitable for SDXL, also applies to SD2.1. In Appendix A.7, we demonstrate that the hyperparameters for attentive guidance also perform well for AP-LDM and other generative frameworks (DemoFusion and HiDiffusion). These experiments highlight the generalizability of the hyperparameter selection derived through experimental exploration.
   • Since there are variations in individual preferences for image contrast and detail richness, we note that there is no absolute standard for selecting the optimal parameters of the attentive guidance and the progressive upsampling. We have provide a stable range of parameter choices and explained how each parameter influences the generation results, where the specific selection of parameters ultimately depends on individual preferences.

2. Existing training-free generative frameworks confront two main challenges: (1) difficulty in generating images with rich details and vibrant colors, and (2) excessively long generation times. To address these issues, we introduce two methods: (1) PFSA, which enhances the structural consistency of latent representations, improving the richness of details and colors in generated images; and (2) a progressive pixel-space upsampling framework, enabling the rapid generation of high-resolution images with globally consistent semantics.
   We emphasize that the issue of text generation errors, mentioned in the Limitation section, is not caused by our PFSA but rather stems from the inherent limitations of the pretrained model's generative capabilities. As shown in Figure 11, even when directly using SDXL for inference at its trained resolution, the generated images still exhibit such text errors. Since PFSA operates only during the low-resolution generation process, it introduces negligibly additional inference time. Even though our AP-LDM takes longer to generate ultra-high-resolution images, it is still faster than existing methods.

3. Regarding the inconsistency in naming in the paper, we have corrected the relevant typographical errors and performed a thorough review. We sincerely appreciate your feedback.

Response to the Questions:

Considering that our primary objective is fundamentally a HR image generation task rather than traditional super-resolution tasks, it is sufficient to maintain the consistency of the target subject without requiring perfect alignment of the details.
We present the intermediate results of the model's generation process. In Figure 14 of the revised paper, we showcase the intermediate outputs. The results in Figure 14 were obtained using a total of three sub-stages. As observed, the images across different stages exhibit high structural consistency, with only minor and reasonable differences in details.

Our full paper file exceeds 100MB. Due to upload size limitations, we have submitted a compressed version of the paper.

To ensure clear visualization of the images, we have uploaded pages 18–27 of the appendix as supplementary materials. Please refer to these pages for further details!