ClearSR: Latent Low-Resolution Image Embeddings Help Diffusion-Based Real-World Super Resolution Models See Clearer

Thank you for your valuable feedback. Our response is as follows:

Q1: Lacks more detailed comparisons, such as inference time, parameter count, and computational cost.

A1: Thank you for your thoughtful suggestion. We compare the complexity of our ClearSR with that of several SD-based Real-ISR methods (DiffBIR, PASD and SeeSR), including total parameters, trainable parameters, MACs, inference step, inference time and inference speed. All methods are tested on an A40 GPU. Although the additional layers increase the number of parameters and computational cost, we can see that our ClearSR has fewer total parameters, trainable parameters, and MACs compared to SeeSR. For inference speed, since the Diffusers library is optimized for the Classfier-Free Guidance (CFG), we disabled CFG during inference to achieve a fair comparison. Note that DiffBIR originally does not use CFG. In addition, we can also observe that our ClearSR performs well when the inference step is set to 20 (lower MACs and a reduced inference time). This further proves that our model has stronger generative capabilities, allowing it to recover good results even with fewer inference steps. We have added the complexity comparisons to Appendix E in the revision.

Quantitative comparison of ClearSR-s20 on DRealSR dataset is as below.

Q2: Missing some key details, like the number of inference steps, and Figure 10 doesn't provide the names of the comparison methods.

A2: Thank you for your reminder. The number of inference steps is 50. The comparison methods in Figure 10 are the same as those in Figure 6, listed from left to right as follows: Zoomed LR, Real-ESRGAN, ResShift, StableSR, SeeSR, ClearSR (Ours), HR. We have made this clear in the revision.

Q3: While the motivation is good, the novelty of the solution seems relatively weak.

A3: Thank you for recognizing the motivation of our paper. Our solution primarily focuses on making the model’s generation more consistent with the LR information. To achieve this, we use additional cross-attention layers to further constrain the control signal and design two modules to preserve detail and structural information. In terms of the solution, firstly, our solution is novel. To our knowledge, we are the first to highlight the importance of LR latent embedding. We efficiently utilize LR information by providing additional constraints in the latent space to obtain better control signals. Previous methods, such as PASD, did not optimize the control signal itself. SeeSR uses the semantic signals to improve the model's generative capability, which might lead to outputs that are inconsistent with the LR information. Moreover, our approach is simple and effective. In Figure 2, we show that our method can better extract LR information. In Figure 4, we can see that the output from DPM contains more high-frequency information which is helpful for reconstructing details while SPM mainly contains low-frequency information which preserves structural information. However, there are still some limitations in our approach. For instance, when constraining the control signal, we hope to design more efficient solutions. Additionally, in the decoupling of high-frequency and low-frequency information, it may be necessary to supply further processed LR information to the DPM and SPM. This will be part of our future work.

Q4: Selection of α and β Parameters: How were the values for α and β in the LSA strategy chosen? Did you perform a systematic parameter search or optimization? Are these parameters required to be tuned for different datasets or image types, and is there a way to automate their selection?

A4: Thank you for your valuable question.

1. "How were the values for α and β in the LSA strategy chosen? Did you perform a systematic parameter search or optimization?" Our LSA method is designed to balance fidelity and generation. However, the "optimal balance" between fidelity and generation is relatively subjective. In the paper, we tested multiple sets of α and β values and ultimately selected the ones that we believed were relatively well balanced in fidelity and generation.
2. "Are these parameters required to be tuned for different datasets or image types?" Since our model performs well under various degradation conditions, these parameters do not need to be tuned for different datasets or image types. As mentioned in the original paper, we used LoRA layers to fine-tune the VAE, enabling our model to adapt to severe degradation conditions. Please note that during inference, we also use the fine-tuned VAE to provide LR guidance. We will conduct an experiment to validate our model's ability to adapt to degradation. We conducted the experiment on the DRealSR dataset, where we added extra degradation to the LR images to simulate severe degradation conditions. Using the HR image as the reference, we calculated the PSNR, SSIM, and LPIPS before and after adding degradation:

Subsequently, we input the degraded images into SeeSR and ClearSR. The table below shows the metrics before and after adding degradation:

We further calculated the changes in these metrics before and after adding degradation:

We can see that the generative metric changes for SeeSR and ClearSR are relatively small. However, ClearSR shows a smaller decrease in PSNR and SSIM, indicating that it adapts better to severe degradation conditions. 3. "Is there a way to automate their selection?" Automatically selecting hyperparameters is a complex task that typically requires extensive engineering investigation to determine the final solution. Therefore, we did not report these results in the paper. However, we proposed a simple and effective strategy to validate the feasibility of automatic hyperparameter selection: For instance, by first calculating the MANIQA score of the LR image and then choosing different values of α and β based on the MANIQA score. Our strategy is demonstrated in the table below:

In the table below, we present the metrics using this strategy:

Although our LSA method performs similarly under various degradation conditions, meaning this strategy may not improve overall metrics, it does have specific effects. We present the metrics for each group of LR images below. We can see that for lower-quality LR images (Group 1, Group 2), our model tends to generate more details (as shown by improvements in NIQE and MANIQA). For higher-quality LR images (Group 3, Group 4), the model outputs results more consistent with the LR image (as shown by improvements in PSNR and SSIM). Overall, using this strategy enhances the visual quality of the model's output.

Q5: Implementation of LoRA Layers: How does the choice of LoRA rank (set to 16) impact model performance, and was this rank value optimized experimentally?

A5: Thank you for your question. We select the appropriate LoRA rank experimentally. The table below shows the ablation results for the VAE LoRA rank. As seen, reducing LoRA rank improves fidelity but has a negative impact on generation metrics. On the contrary, increasing LoRA rank has a negative impact on fidelity but improves generation metrics. To balance fidelity and generation, we finally set the VAE LoRA rank to 16.

The table below shows the ablation results for the UNet LoRA rank. We can see that both smaller LoRA rank and larger LoRA rank improve fidelity but have a negative impact on generation metrics. To balance fidelity and generation, we finally set the UNet LoRA rank to 16. We have added this ablation study to Appendix F in the revision.

Q6: Something about classfier-free guidance, cfg: During the inference stage, by adjusting the CFG value, RealSR methods based on pre-trained T2I diffusion models can also balance fidelity and perception. The authors did not report the CFG settings during inference, such as the CFG value and negative prompt. Additionally, the proposed LSA control method needs to be compared in detail with the CFG control method to highlight the differences.

A6: Thank you for your thoughtful suggestion. Similar to PASD and SeeSR, we also utilized Classifier-Free Guidance (CFG). The cfg scale is set to 7, the positive prompt is "clean, high-resolution, 8k, detailed, realistic", and the negative prompt is "dotted, noise, blur, lowres, smooth". Regarding the differences between LSA and CFG, our analysis is as follows:

1. In terms of approach: LSA directly adjusts the prediction through LR latent embeddings, while CFG controls generation by blending predictions from two different prompts.
2. In terms of computational cost: LSA has almost no extra computational cost, while CFG requires generating two predictions of two prompts, which doubles the computational cost.
3. In terms of adjustment range: LSA can enhance the fidelity by increasing α and can also enhance the generative capability of the model by increasing β, serving as a bidirectional adjustment strategy. With CFG, a given prompt set can only adjust in one direction. Additionally, setting the cfg scale too high often results in unnatural outputs, limiting its adjustment range.
4. In terms of the output quality: Using ClearSR settings as a baseline, we adjusted the model's output through both LSA and CFG, aiming for a PSNR of 29 for high fidelity and a PSNR of 27.5 for high generative capability. In the first comparison, results using LSA have advantages on generation metrics. In the second comparison, although the generative metrics were similar, an excessively high cfg scale led to unnatural outputs, reflected by a significant increase in LPIPS.

Overall, LSA outperforms CFG in terms of computational cost, adjustment range, and the output quality.

Thank you for your valuable feedback. Our response is as follows:

Q1: The authors need to compare the complexity of ClearSR with that of the other methods, including the model parameter counts, inference time, and inference timestep.

A1: Thank you for your thoughtful suggestion. We compare the complexity of our ClearSR with that of several SD-based Real-ISR methods (DiffBIR, PASD and SeeSR), including total parameters, trainable parameters, MACs, inference step, inference time and inference speed. All methods are tested on an A40 GPU. Although the additional layers increase the number of parameters and computational cost, we can see that our ClearSR has fewer total parameters, trainable parameters, and MACs compared to SeeSR. For inference speed, since the Diffusers library is optimized for the Classfier-Free Guidance (CFG), we disabled CFG during inference to achieve a fair comparison. Note that DiffBIR originally does not use CFG. In addition, we can also observe that our ClearSR performs well when the inference step is set to 20 (lower MACs and a reduced inference time). This further proves that our model has stronger generative capabilities, allowing it to recover good results even with fewer inference steps. We have added the complexity comparisons to Appendix E in the revision.

Quantitative comparison of ClearSR-s20 on DRealSR dataset is as below.

Q2: The authors should double-check the understanding of PASD in Line 190.

A2: Thank you for your reminder. In Line 190, “Similar to PASD (Yang et al., 2023), we let the LR image pass through the CLIP image encoder to obtain the image-level feature $\mathbf{p}$ and replace the null-text prompt in the UNet decoder”, which means the similar way to extract some high-level information using the pre-trained model. In contrast, PASD uses ResNet, YOLO, and BLIP to extract information, and then converts it into image-level features using the CLIP encoder. We directly use the CLIP encoder to process images, and then pass the feature output from the CLIP image encoder through two MLP layers to match the shape and adapt to the degradation of the LR. This feature is then used to replace the null-text prompt.

Q3: The authors should add a clearer description of the image-level feature $\mathbf{p}$ in Figure 3. How is $\mathbf{p}$ integrated into SD Unet, and what is its role in the framework?

A3: Thank you for your suggestion. As mentioned in A2, since $\mathbf{p}$ directly replaces the null-text prompt, it will interact with the UNet through the cross-attention layer, which originally interacted with the text embedding. In our framework, similar to PASD, $\mathbf{p}$ serves as additional high-level information to enhance the model's generative ability, but it may lead to a decrease in fidelity. We conduct the ablation study and the results are as follows:

Q4: The authors should explain more clearly in Table 2. Why does the SPM improve fidelity, while window-based cross-attention layers in DPM weaken fidelity? In addition, the ablation study that includes a model without DPM should also be provided for a more complete picture of each module's contribution.

A4: Thank you for your thoughtful question. In Table 2, with the addition of window-based cross-attention layers, although PSNR shows a slight decrease, SSIM still shows improvement. The improvement of SSIM indicates that the structural information of the LR is better preserved. Considering PSNR and SSIM together, fidelity has not been weakened. On the other hand, the window-based cross-attention layers in DPM contribute to the improvement in generative metrics, demonstrating the effectiveness of our method. We also appreciate your suggestion to add an ablation study. It can be observed that without DPM, the model's fidelity decreases significantly, further demonstrating the effectiveness of our method.

Thank you for your valuable feedback. Our response is as follows:

Q1: Some descriptions in the paper may lead to confusion. The authors classify detail information as high-frequency information and structural information as low-frequency information. However, edges can also represent structure and are actually considered high-frequency information. The authors should use more appropriate terminology to avoid ambiguity.

A1: Thank you for your thoughtful suggestion. Edges can indeed be considered as high-frequency information. In ClearSR, the DPM is proposed to preserve detailed information in the LR image, while the SPM is proposed to maintain the global structural information information. To avoid ambiguity, we believe that the term "Global Structure Preserving Module" more accurately conveys our model design than the "Structure Preserving Module" (SPM). We have corrected the name of the module and modified any descriptions that may cause confusion according to the suggestion in the revision.

Q2: The experimental results do not clearly demonstrate that the proposed method performs better in terms of fidelity (PSNR, SSIM, LPIPS, etc). In addition, similar approaches have also appeared in DiffBIR and PASD, and the author should provide a thorough comparison with the strategies proposed by these other methods.

A2: Thank you for your question. In Table 1 of the original paper, to balance fidelity and generation, we selected appropriate α and β values, which allowed ClearSR to outperform previous diffusion-based methods (DiffBIR, PASD, SeeSR) in both fidelity and generation. We also adjusted the LSA settings and introduced another version of ClearSR, ClearSR*, which surpasses diffusion-based methods (no generative priors) in terms of fidelity (PSNR, SSIM, LPIPS) and is closer to the fidelity of GAN-based methods. To more intuitively demonstrate the effectiveness of our method, we adjust the LSA settings and provide two additional versions, including ClearSR-a (α=0.05, β=0.01) and ClearSR-b (α=0.02, β=0.01). In these versions, ClearSR-a is compared with Real-ESRGAN and ResShift, while ClearSR-b is compared with SeeSR. We can see that ClearSR-a outperforms Real-ESRGAN and ResShift in terms of PSNR and MANIQA. Similarly, ClearSR-b performs better than SeeSR in terms of PSNR and MANIQA, which demonstrates that our LSA could perform better in terms of fidelity with appropriate α and β values.

We also provide a detailed comparison with the inference strategies proposed by DiffBIR and PASD. Firstly, the inference strategies proposed by DiffBIR and PASD can only enhance fidelity but cannot improve the generative capability. In contrast, our LSA method can enhance both fidelity and generation by adjusting α and β. We adjust the LSA settings and provide three additional versions, including ClearSR-c (α=0.01, β=0.00), ClearSR-d (α=0.00, β=0.01), and ClearSR-e (α=0.01, β=0.01). As shown in the table below, we can see that our LSA can enhance the fidelity by increasing α (ClearSR-c) and can also enhance the generative capability of the model by increasing β (ClearSR-d). With appropriate settings, both the fidelity and generative capability of the model can be enhanced simultaneously (ClearSR-e).

Furthermore, we apply the inference strategies proposed by DiffBIR and PASD to ClearSR. We also adjust the LSA settings and provide two additional versions, including ClearSR-f (α=0.01, β=0.01) and ClearSR-g (α=0.015, β=0.01). Specifically, ClearSR-f compares ClearSR with the DiffBIR strategy, and ClearSR-g compares ClearSR with the PASD strategy. Note that, since DiffBIR's method is based on MSE Guidance, it introduces additional computational cost. As a result, we use the standard MSE Guidance in ClearSR-f to achieve a fair comparison. As can be observed, when the generative metrics are similar, our method achieves higher fidelity. Moreover, since LSA can enhance both fidelity and generation by adjusting α and β separately, our method offers broader adaptability, further proving the effectiveness of our method.

Q3: The LR latent embedding, which is the output of the VAE encoder, has a size of 4x64x64, while the input image is 3x512x512. Compared to the original image, the LR embedding loses a significant amount of spatial information. Therefore, the LR latent embedding may not be suitable for supplementing detail and structural information.

A3: Thank you for your valuable question. The LR embeddings do lose some information, but a well-trained VAE is still capable of preserving rich LR information. We use the VAE from SD 2.1-base to reconstruct images from the DRealSR test sets.

We can see that the reconstruction results show great PSNR, SSIM, and LPIPS scores. These results are much better than the output of existing methods, demonstrating the potential of LR embeddings. Additionally, how to integrate LR information into the SD model is a challenge, often requiring alignment between the input information and the latent space of SD. From this perspective, VAE is a useful tool. After fine-tuning VAE with the LoRA layers, it can well map LR images to a proper latent space while retaining rich information.

Q4: Figure 2 shows that the proposed method has a low KL divergence value between the control signal and the low-resolution latent embedding. This suggests that the authors have introduced two modules to achieve a similar distribution between the LR latent embedding and the control signal. So why not use the LR latent embedding directly? Furthermore, from past work (DiffBIR, PASD, SeeSR), we know that the role of the control branch is primarily to remove degradation and bring it closer to the HR distribution. However, the method proposed by the authors results in the distribution of the control branch outputs being closer to the distribution of LR latent embedding, which is puzzling.

A4: We appreciate your insightful question. Although we use LR latent embedding to constrain the control signal, this does not mean that the LR latent embedding can directly serve as the control signal. The LR latent embedding serves as input to the control branch and is inherently incompatible with the SD UNet. Through the control branch, the LR latent embedding is converted into the control signal that is adapted to the UNet, which is the purpose of diffusion adapters like ControlNet. During the conversion process of the LR latent embedding, as you mentioned, one role of the control branch is to make the control signal closer to the HR distribution, which is essentially a generative process. Since ControlNet is initialized using the parameters of the UNet, this process is also influenced by the generative prior of the UNet. However, during this process, the UNet's generative priors often distort the LR information, causing deviations in the generated output. In ClearSR, we similarly aim for the control branch to possess strong generative capabilities, but we correct the generative direction of the generative priors by adding additional constraints, resulting in output more consistent with the LR image.

In Table 1 in the paper, it can be seen that our model exhibits stronger generative capabilities. Moreover, in Figure 2, we show that we successfully constrain the control signal in the latent space. These two results together demonstrate that our model achieves more "controllable" generation, and Figure 1 shows our model can generate results that are more consistent with the LR images.

Thank you for your reply.

In the response to Q2, I learned that the LSA setting is a key factor in balancing generative capability and fidelity. However, it still does not demonstrate the role of the detail preservation module in enhancing fidelity. The authors can remove the LSA setting to validate whether the proposed clearsr has an advantage over other methods in terms of fidelity. By the way, from Fig. 3 in the PASD paper (https://www.ecva.net/papers/eccv_2024/papers_ECCV/papers/01705.pdf), the inference strategies proposed by PASD is also helpful for generative capabilities.

In the response of Q3, how are the reconstruction results calculated? Are they computed between the output of VAE with LR as input and GT?

Thank you for your valuable feedback. Our response is as follows:

Q1: The author claims that ControlNet cannot preserve the LR information well in Figure 2. Is it because that ControlNet adds noise to the LR conditional during training and inference? Does the proposed approach also follows this setting as ControlNet? The authors should explicitly state whether they follow the same noise addition process as ControlNet, and if not, to explain how their approach differs.

A1: Thank you for your question. Firstly, according to the DiffBIR paper, there is an ablation study on adding noise to the LR condition. Using only the LR condition as input to the control branch results in improved fidelity but a decrease in generative metrics. However, in this experiment, the changes in both fidelity and generative metrics are quite slight, indicating that noise has a negligible impact on the control branch. This experiment demonstrates that noise is not the primary reason why ControlNet fails to preserve the LR information well. Additionally, adding noise to the LR condition does introduce some randomness. However, in DiffBIR, the noise is concatenated with the LR condition, which means that the impact of the noise is reduced. In ClearSR, our code is based on DiffBIR, so the noise addition process is the same as in DiffBIR. Furthermore, the noise addition process for ControlNet in Figure 2 of the ClearSR paper also employs the concatenation approach. We can see that even with the concatenation approach, the final control signal still cannot preserve the LR information well.

Q2: The additional modules introduced in this paper may also increase the cost of training and inference. Some evaluation on the complexity should be provided, e.g., parameters, flops and inference time. The authors may consider provide some numerical comparison with existing baselines.

A2: Thank you for your thoughtful suggestion. We compare the complexity of our ClearSR with that of several SD-based Real-ISR methods (DiffBIR, PASD and SeeSR), including total parameters, trainable parameters, MACs, inference step, inference time and inference speed. All methods are tested on an A40 GPU. Although the additional layers increase the number of parameters and computational cost, we can see that our ClearSR has fewer total parameters, trainable parameters, and MACs compared to SeeSR. For inference speed, since the Diffusers library is optimized for the Classfier-Free Guidance (CFG), we disabled CFG during inference to achieve a fair comparison. Note that DiffBIR originally does not use CFG. In addition, we can also observe that our ClearSR performs well when the inference step is set to 20 (lower MACs and a reduced inference time). This further proves that our model has stronger generative capabilities, allowing it to recover good results even with fewer inference steps. We have added the complexity comparisons to Appendix E in the revision.

Quantitative comparison of ClearSR-s20 on DRealSR dataset is as below.

Although our LSA method performs similarly under various degradation conditions, meaning this strategy may not improve overall metrics, it does have specific effects. We present the metrics for each group of LR images below. We can see that for lower-quality LR images (Group 1, Group 2), our model tends to generate more details (as shown by improvements in NIQE and MANIQA). For higher-quality LR images (Group 3, Group 4), the model outputs results more consistent with the LR image (as shown by improvements in PSNR and SSIM). Overall, using this strategy enhances the visual quality of the model's output.

Additionally, automatically selecting hyperparameters is a complex task that typically requires extensive engineering investigation to determine the final solution. Therefore, we did not report these results in the paper. However, we will continue to conduct related research to further enhance the visual quality of our model's output.

Q4: SUPIR has more powerful generative ability than the baselines in the paper. The authors may want to explain why SUPIR was not included as a baseline, or to consider adding it to their comparisons if feasible.

A4: Thank you for your suggestion. Although SUPIR demonstrates stronger generative capabilities, the comparison is unfair due to differences in the dataset and model size. We will explain these two aspects in detail.

In terms of datasets, SUPIR collected a private dataset of 20 million high-resolution, high-quality images for model training. However, ClearSR uses the same datasets as SeeSR and PASD, which consist of DIV2K, Flickr2K, DIV8K, OST, and the first 10K face images from FFHQ, totaling approximately 25,000 images. SUPIR's dataset is 800 times larger.

Regarding the model, SUPIR is based on SDXL. However, ClearSR is based on SD2.1-base and is only one-third the size of SDXL. Other baselines also typically use models of similar size, such as SeeSR using SD2-base and StableSR using SD2.1-base. Overall, comparisons with SUPIR are unfair in terms of both datasets and model size, and the dataset used by SUPIR is difficult to obtain. Therefore, we did not include SUPIR as a baseline but discussed it in the paper.

Q5: Why choosing window cross-attention rather than full-attention and how to decide the window size? The authors should provide empirical or theoretical justification for using window cross-attention, and explain how they determined the optimal window size.

A5: Thank you for your question. In Table 2, we demonstrate that not using window partition results in a decrease in fidelity. Additionally, we found that smaller window size improves fidelity but results in a reduction in generative ability. The results of the ablation study on window size are as follows:

We can see that increasing the window size leads to a decrease in fidelity, while decreasing the window size results in a reduction in the model's generative ability. To strike a balance between fidelity and generative ability, we finally chose a window size of 16 for ClearSR. We have added this ablation study to Appendix F in the revision.

Q3: The proposed Latent Space Adjustment strategy is somewhat tricky. How to choose ideal hyperparameters can be tough and case-by-case. Moreover, when the degradation is severe, adding LR guidance into the inference may leads to blurry outputs. The authors should consider providing guidelines or heuristics for choosing hyperparameters, and discussing how their method performs under severe degradation conditions and the quality of the guidance under such cases.

A3: Thank you for your valuable question. We will first answer the question about degradation and then discuss how to choose hyperparameters. Regarding the question of how ClearSR performs under severe degradation conditions, firstly, we used LoRA layers to fine-tune the VAE, enabling our model to adapt to severe degradation conditions. Please note that during inference, we also use the fine-tuned VAE to provide LR guidance. We conducted the experiment on the DRealSR dataset, where we added extra degradation to the LR images to simulate severe degradation conditions. Using the HR image as the reference, we calculated the PSNR, SSIM, and LPIPS before and after adding degradation:

Subsequently, we input the degraded images into SeeSR and ClearSR. The table below shows the metrics before and after adding degradation:

We further calculated the changes in these metrics before and after adding degradation:

We can see that the generative metric changes for SeeSR and ClearSR are relatively small. However, ClearSR shows a smaller decrease in PSNR and SSIM, indicating that it adapts better to severe degradation conditions. Next, we demonstrate the impact of different hyperparameters of LSA under severe degradation conditions, and compare these results with those under normal degradation conditions (before adding extra degradations). As seen, even in severe degradation conditions, increasing α still improves fidelity, and the effect is similar to that observed under normal degradation conditions. Therefore, LSA performs well under severe degradation conditions, and adding LR guidance into the inference under high degradation conditions does not lead to blurry outputs.

Then, regarding the choice of LSA hyperparameters:

1. Our LSA method is designed to balance fidelity and generation. However, the "optimal balance" between fidelity and generation is relatively subjective. In the paper, we tested multiple sets of α and β values and ultimately selected the ones that we believed were relatively well balanced in fidelity and generation.
2. As mentioned above, our ClearSR adapts well to severe degradation conditions. Using the default settings generally provides good results. Therefore, users do not need to select hyperparameters case-by-case. Moreover, LSA allows for a wide range of adjustments, so users can adjust the hyperparameters to their specific needs to achieve their desired results.
3. We can also use a simple strategy for automatic hyperparameter selection. For instance, by first calculating the MANIQA score of the LR image and then choosing different values of α and β based on the MANIQA score. Our strategy is demonstrated in the table below:

In the table below, we present the metrics using this strategy: