FreCaS: Efficient Higher-Resolution Image Generation via Frequency-aware Cascaded Sampling

We sincerely thank this reviewer for the recognition on the efficiency and generation of our FreCaS. Based on this reviewer's suggestion, we have conducted more experiments as well as a user study. Please find what below our itemized responses to this reviewer's questions.

Q1 Hyperparameter settings.

Thanks for the comment. Training-free based methods typically involve a few hyper-parameters to control the diffusion process. For example, DemoFusion relies on specific values of $\sigma_1$, $\sigma_2$, $\alpha_1$, $\alpha_2$, $\alpha_3$, and so on. To minimize the complexity, in FreCaS we provide two ways to set the hyperparameters:

• Default values for immediate use: For all hyperparameters, we provide default values for specific applications. We generally recommend setting $w_l$ to be consistent with the guidance scale, $w_h$ to 35, and $w_c$ to 0.6. These default values are effective across a wide range of scenarios.
• Ablation studies for parameter setting guidance: In the paper, we conduct extensive ablation studies to explore the significance and impact of each hyperparameter on image quality. Specifically, larger $w_h$ tends to generate higher-quality details but may introduce artifacts. $w_c$ is primarily used to balance detail generation and semantic structure. Through these experiments, we provide clear guidance for adjusting these parameters based on specific requirements.

While FreCaS does rely on hyperparameters, the provision of default values and the guidance for setting them ensure that the tuning process is manageable, which improves the practicality and usability of our method.

Q2 Performance limitation by pre-trained models.

Thanks for the comment. First of all, we agree with this reviewer that as a training-free method, FreCaS relies on the pre-trained models. A stronger pre-trained model will offer the adapted model a better foundation. However, we'd like to respectfully argue that the development of training-free methods is just to address the limitations of pre-trained models in various downstream applications. A well-designed training-free method, such as our FreCaS, can indeed improve much the performance of the pre-trained model. For example, both HiDiffusion and our FreCaS are effective in lifting SD2.1 and SDXL models in higher-resolution image generation.

On the other hand, FreCaS demonstrates superior generation capabilities compared to existing training-free high-resolution image generation methods. In particular, FreCaS focuses on improving the general sampling process of diffusion models, so that it can be easily applied to more advanced yet more complex pretrained models like SD3, where DemoFusion and HiDiffusion fail to perform effectively.

We hope our explanations can address this reviewer's concern.

Q3 Differences from FreeU and FreeEnhance.

Thanks for the comment. There are two main differences between FreCaS and FreeU and FreeEnhance.

First, FreeU and FreeEnhance focus on enhancing the visual quality of generated images at base training sizes. In contrast, FreCaS focuses on improving the efficiency and effectiveness of generating higher-resolution images than the training sizes.

Second, FreeU adjusts the weights of different frequency components of network features, and FreeEnhance adds lower-magnitude noise to the high-frequency component of latents. However, our FreCaS operates at the denoising scores by assigning higher CFG strength to higher frequency components.

Actually, our FreCaS is orthogonal and complementary to those two methods. For example, integrating FreCaS with FreeU results in an improvement of 1.05 on the FID metrics without introducing additional latency on 2048$\times$2048 image generations using SDXL.

We sincerely thank this reviewer for the recognition on the effectiveness of our work. We have conducted extensive additional experiments based on this reviewer's constructive suggestions. Please find what below our point-to-point responses to this reviewer's comments.

Q1 New insights of FreCaS.

Thanks for your constructive comments. While the coarse-to-fine and frequency domain enhancement approaches have been used in many existing works, we would like to clarify that our FreCaS is not a simple combination of existing coarse-to-fine approaches and frequency-domain operations, but it has significant differences from existing methods such as HiPrompt and ResMaster. In specific, FreCaS introduces several novel insights in efficient coarse-to-fine design and details enhancement in frequency domain.

First, FreCaS presents a novel efficient coarse-to-fine sampling framework. Current methods, such as DemoFusion and ResMaster, perform a complete diffusion process at each resolution. This leads to redundant computations in the early stages of high-resolution diffusion, as much of the information is already obtained in the low-resolution diffusion process. In contrast, FreCaS transitions the diffusion from low to high resolutions in just one process. Such a framework significantly reduces the computational overhead. For example, while ResMaster is about twice as fast as DemoFusion in generating 4096$\times$4096 images using SDXL, FreCaS achieves a 7.58$\times$ speedup, which is significantly faster while keeping the generation quality.

Second, our Frequency-Aware Classifier-Free Guidance (FA-CFG) strategy efficiently and effectively enhances details in the frequency domain. Existing methods, such as HiPrompt and ResMaster, focus on maintaining semantic consistency by aligning low-frequency components with global prompts or low-resolution reference. In contrast, FreCaS emphasizes on generating more details by assigning higher CFG strengths to high-frequency components. Our FA-CFG strategy is different yet complementary to HiPrompt and ResMaster, potentially offering additional performance benefits when used in conjunction with these methods. Unfortunately, HiPrompt and ResMaster do not release their codes for reproduction.

Additionally, FA-CFG allows for much higher CFG strengths (from 7.5 to 35) without introducing artifacts, which is a key challenge for using CFG strategy in image generation.

We will add the above explanations in the related work section of the revised manuscript. Hope our clarifications can address this reviewer's concerns.

Thank you for the explanation. The missing details have been well supplemented. I appreciate the efficiency of the proposed method, especially when compared with other training-free methods. However, there are some concerns regarding comparisons with training-based and SR-based methods. From Table 1, it is evident that SDXL+ESRGAN or SDXL+SUPIR outperform FreCaS in many metrics, with the efficiency of SDXL+ESRGAN being particularly noteworthy. This raises two possibilities:

1. The evaluation metrics used may not be reasonable, despite their wide use in previous work.
2. The SDXL+ESRGAN pipeline is already an effective and efficient solution, negating the need for more complex methods.

While I acknowledge that training-based methods may take more time to generate high-resolution results, I doubt that the results produced are inferior to those of training-free methods. Additionally, it is crucial to provide the stability and success rate of the proposed method. My experience with most training-free methods indicates that it usually takes numerous attempts to achieve a relatively good result, despite yielding favorable quantitative evaluation results.

We thank this reviewer for spending his/her precious time continuing to discuss with us. We truly appreciate it. However, we respectfully cannot agree with this reviewer's opinion in downgrading training-free based methods. While training-based methods have advantages on stability, our training-free based FreCaS method has clear advantages on generation quality. This reviewer argued that NR-IQA metrics are not perfect to evaluate image quality, however, the FID, IS and CLIP Score metrics are not perfect either. We feel it is somewhat unfair to emphasize some metrics while ignoring other metrics to judge the performance of a method.

Considering that none of the existing metrics is perfect to evaluate image generation methods, we agree with this reviewer that human subjective evaluation can be a relatively more reliable approach. Actually, we have conducted a user study in the past two days after we submitted the response to this reviewer. We just obtained the results. Please refer to Table 5 and our responses below for the details.

Q8 User studies.

We conducted a user study to explore the generated image quality and success rate of Pixart-Sigma, UltraPixel, and our FreCaS. The results are listed in the following Table 5.

For the study on generation quality, we randomly select 20 prompts from Laion5B and generate one image per method for each prompt, creating 20 sets of images (3 images per set). Five volunteers (3 males and 2 females) were invited to participate in the test. All the volunteers are not working in the area of image generation to avoid potential bias. Each time, the set of 3 images for the same prompt are presented to the volunteers in random order. The volunteers can view the images multiple times, and they are asked to select the image with the best quality from each set. There are 100 votes in total.

For the study on success rate, we randomly select 10 prompts and generate five images per prompt for each method, resulting in 50 images per method. We invited the same five volunteers as in the study of generation quality to judge whether the generated image is a success or failure. When making the decision, the volunteers are instructed to consider two factors. First, whether the image content is faithful to the description of the prompt. Second, whether the image quality is satisfactory. Only when both the two requirements are met, the generation is considered as a success. There are 250 judges for each method.

Tab. 5 User Studies on $\times 4$ Generation with SDXL.

As we can see from Table 5, our FreCaS outperforms significantly Pixart-Sigma and UltraPixel in terms of image generation quality, with 58% of the images being voted as the best. In terms of success rate, UltraPixel works the best, with 96 out of 250 images being marked as successful. Our FreCaS lags behind, with 68 successful cases. However, we argue that our FreCaS still generates more successful results than Pixart-Sigma (52 images), indicating that a well designed training-free method can surpass training-based methods.

From Table 5, we can also observe that none of the methods, including training-based and training-free ones, achieves a success rate higher than 40%. This implies that there are much space to improve. A hybrid approach that can take the advantages of both training-based and training-free methods can be a promising solution.

We sincerely thank this reviewer for the recognition on the efficiency of our work.

Q1/Q2 Missing FID scores in the main experiments and ablation study.

Thanks for the suggestion and we are sorry for the missing of FID scores. We further evaluate all the methods and each component of our method using the FID metric.

Table 1 presents the FID results of all competing methods. In this table, "DO" means "duplicated object", which indicates whether the method takes the duplicated object problem into consideration. "SpeedUP" denotes the efficiency speed-up over the DirectInference baseline. We can see that our FreCaS achieves the best FID scores in all the experiments, which aligns with the trends observed in other image quality metrics.

Table 2 presents the FID results of ablation studies to validate each component of FreCaS. One can see that our cascaded framework significantly improves the performance and efficiency over the baseline, reducing the FID score by 22.52 and decreasing the latency by 20.39 seconds. The high efficiency of our cascaded framework is evident, as it performs only one diffusion process across resolutions. However, existing methods often conduct an independent diffusion process at each resolution.

Additionally, our FA-CFG and CA-reuse strategies further improve the FID and IS scores, and show substantial improvements in $\text{FID}_p$ and $\text{IS}_p$, demonstrating their effectiveness in enhancing image details and semantic appearances. These strategies introduce minimal additional latency.

We have added the results of FID scores in the revised manuscript.

We sincerely thank this reviewer for the recognition on the effectiveness of our work. We have performed more experiments and comparisons based on this reviewer's suggestion. Please find our point-to-point responses in the following.

Q1 Missing FID scores.

Thanks for the suggestion and we are sorry for not presenting the FID scores in the submission. We have calculated the FID metrics in both the main experiments and the ablation studies. Please kindly refer to our responses to the Q1/Q2 of Reviewer z3AD for the details.

Q2 More 4K comparisons in realistic styles.

Thanks for the suggestion. As suggested, we have provided more 4K visual comparisons under realistic scenes in Fig. 7 of the updated supplementary file. As can be seen, our FreCaS consistently delivers better results in both image layout and semantic details. In addition, please refer to our responses to Q5 of reviewer Sncf for more visual results under real-world scenarios.

Q3 Differences from DemoFusion and FouriScale.

Thanks for the comment. We'd like to clarify that our FreCaS differs significantly from DemoFusion and FouriScale:

• DemoFusion uses a cascaded sampling framework that performs multiple full diffusion processes at different resolutions, leading to repetitive computations and significant computational overhead. For example, when generating a 4096$\times$4096 image, DemoFusion needs to perform multiple full diffusion processes at lower resolutions. In contrast, FreCaS conducts a single diffusion process across resolutions. This new cascaded framework significantly reduces the computational overhead and achieves a 7.58$\times$ speedup when generating 4096$\times$4096 images using SDXL.
• FouriScale does not use a cascaded framework. Instead, it generates higher-resolution images by explicitly enlarging the receptive field of convolutional layers, which is effective but limited to more advanced diffusion models like SD3. FreCaS, on the other hand, focuses on the general sampling process of diffusion models, so that it can be easily applied to other diffusion models. Additionally, FreCaS outperforms FouriScale in both image quality and inference speed when integrated with SDXL, highlighting the significant performance differences between the two methods.

Q4 Effectiveness of timestep shifting, FA-CFG and CA-reuse.

Thanks for the suggestion. To better verify the effectiveness of each component of FreCaS, we conducted more ablation studies on our proposed cascaded framework without timestep shifting, our proposed cascaded framework with timestep shifting, FA-CFG, and CA-reuse strategies. Please kindly note that the "timestep shifting" refers to the "F-step calculation" of this comment.

Tab. 1 Ablation studies on $2048\times 2048$ generation of SDXL. "CF w/o TS" denotes our proposed cascaded framework without timestep shifting. "CF" is our proposed cascaded framework.

Our cascaded framework significantly outperforms the baseline, with a decrease of 22.52 in the FID score and a reduction of 20.39 seconds in latency. This demonstrates the high efficiency of the new cascaded framework. When removing the timestep shifting, the results improve slightly on the FID and IS metrics but at the cost of a significant decrease in $**FID**_p$ and $**IS**_p$. This is because setting "F" equal to "L" shortens the diffusion process, which helps maintain the overall structure but limits the enhancement of details in expanded frequency bands. Our FA-CFG strategy improves both FID and IS scores and shows substantial improvement in $\text{FID}_p$, demonstrating its effectiveness in generating realistic image details. CA-reuse strategy further enhances $\text{IS}_p$, indicating its effectiveness in improving semantic appearance. Moreover, these strategies introduce minimal additional latency.

We hope our response can address this reviewer's concerns.