Solving Diffusion ODEs with Optimal Boundary Conditions for Better Image Super-Resolution

Thanks to author's response

However, I need to point out several issues:

1 The referential significance of NIQE is not substantial. Increasing the proportion of GAN loss easily diminishes it, but it leads to severe artifacts. I am more inclined towards the author's use of FID as an alternative.

2 DiffIR has reported comparisons on real-world super-resolution in the supplementary material and has open-sourced relevant code and models. You can directly compare them. Please provide information on parameter count, runtime, and performance comparsion.

3 From a time comparison perspective, it seems that your method takes 30 times longer than GAN methods, but the performance improvement is limited. This is difficult to accept in practical applications.

4 Both PSNR and LPIPS are crucial. A high LPIPS and low PSNR indicate significant distortion in images, perhaps making them appear better. However, the authenticity of super-resolution results is crucial in many scenarios.

I hope the author can demonstrate the significance of this article to me. Is it effective, fast, or does it achieve high metrics? I am willing to improve my rating based on such aspects.

W4. Please provide a comparison of the computational complexity and runtime for all the methods mentioned in the paper to show your effectiveness.

The comparisons of running time are given in the following tables. All the times are tested on one single RTX 2080Ti GPU.

For bicubic-SR, our baseline is SR3 and resolution of SR images is 256*256. The comparison is:

For real-SR, our baseline is StableSR and resolution of SR images is 512*512. The comparison is:

Our method could achieve better performances with fewer sampling steps, which means better efficiency compared with the official sampling methods of existing diffusion-based SR models. The comparison of running time is in proportion to the comparison of sampling steps, which has already been given in the paper. However, because of the characteristics of diffusion models, it is still extremely difficult to achieve shorter running time with diffusion models compared with GAN-based methods which only need one forward process.

W5. Tables 1 and 2 show that the PSNR is not particularly high, indicating that the network's fidelity is not good. Super-resolution tasks not only seek visual improvement but also place great importance on fidelity. Compared methods have better fidelity. Therefore, I suggest the authors work on improving both PSNR (fidelity) and LPIPS, as this would provide stronger evidence of the effectiveness of your method.

The authors understand that PSNR is a well-recognized measure for SR. However, in recent years, its drawbacks have been revealed due to its failure to consider sampling randomness in measurement and not robust to simple geometric transformation. Therefore, recent generative model-based SR methods (including GAN-based, flow-based, diffusion-based, and so on) [4, 5, 6, 7] focus on the perceptual quality of SR images, mainly measured by LPIPS. This protocol is also adopted in a recently published paper [8], confirming that worse PSNR but better LPIPS leads to better perceptual quality.

Furthermore, thank you for your valuable suggestion of improving both distortion and perception simultaneously. However, it is extremely challenging to achieve such a goal due to the widely known "Perception-Distortion Tradeoff [9]" which has been studied in depth. We will take your advice into consideration seriously in our future work.

We really appreciate your excellent review of our paper. If our responses do not address all of your concerns, you are welcome to ask us to give more detailed discussions. We are glad to answer your further questions.

[1] Wang, Xintao, et al. "Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data", ICCVW, 2021.

[2] Xia, Bin, et al. "Knowledge Distillation based Degradation Estimation for Blind Super-Resolution", ICLR, 2022.

[3] Xia, Bin, et al. "DiffIR: Efficient Diffusion Model for Image Restoration", ICCV, 2023.

[4] Ledig, C., et al. "Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network", CVPR, 2017.

[5] Wang, Xintao, et al. "ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks", ECCVW, 2018.

[6] Lugmayr, Andreas, et al. "SRFlow: Learning the Super-Resolution Space with Normalizing Flow", ECCV, 2020.

[7] Saharia, Chitwan, et al. "Image Super-Resolution via Iterative Refinement", IEEE TPAMI, 2022.

[8] Gao, Sicheng, et al. "Implicit Diffusion Models for Continuous Super-Resolution", CVPR, 2023.

[9] Blau, Yochai, and Tomer Michaeli. "The Perception-Distortion Tradeoff", CVPR, 2018.

Thank you so much for your time of giving thoughtful review to our paper. About your concerns, we give the responses below. We hope that these responses can solve your questions about our work.

W1. The methods compared in Table 2 are all outdated. It is necessary to compare them with some state-of-the-art real-world super-resolution tasks [1,2].

Thanks for the reminder. We have included the comparisons in our rebuttal with more clarifications.

First, the proposed method is a novel sampling method that aims at getting better SR results from existing diffusion-based SR models. Thus, the effectiveness of our method is more revealed by the performance gap of the model with/without our sampling technique, instead of the overall performance comparison, which is demonstrated in existing comparisons (i.e., the bottom 3 rows in Tab. 2).

Second, our comparisons to DASR (published in ECCV 2022) have shown the effectiveness of our method in both LPIPS and NIQE.

Third, we give the results of Real-ESRGAN [1] and KDSR-GAN [2] in the following table.

Although our method does not outperform them in LPIPS due to the performance gaps between different baselines, it can still generate SR results with significantly better NIQE.

W2. You need to compare with the state-of-the-art diffusion SR method [3], which also appears to have performed bicubic-SR and real SR tasks.

Thanks for the reminder. We will make efforts to add the related results in our updated version.

First, the proposed method is a novel sampling method that aims at getting better SR results from existing diffusion-based SR models, instead of a new diffusion-based SR model. Thus, the vital comparison is between our sampling method and previous sampling methods on the same diffusion-based SR model (i.e., the bottom 3 rows in Tab. 2), as we mentioned in the answer to your question 1.

Second, we leveraged StableSR (https://arxiv.org/abs/2305.07015) as our baseline, which was pre-printed in May 2023. DiffIR [3] (https://arxiv.org/abs/2303.09472) was pre-printed in March 2023. Thus, StableSR is also up-to-date and competitive compared with DiffIR.

Last, we are glad to include the related comparisons. However, in the original paper of DiffIR [3], the real-SR task presented in our paper is not involved (only including image inpainting, bicubic-SR, and motion deblurring). We need more time to train the DiffIR on the task of real-SR to adapt it to our setting and include the related results in the future.

W3. From the figures in the appendix, it seems that the visual improvement is not significant.

The visual improvements of SR results are relatively subjective. However, it is still easy to recognize obvious improvements in our visual results in the appendix.

In Fig. 5, our method generates more accurate and clearer crossties of the railway and windows of the building. In Fig. 6, our method reconstructs more sharp-edged floor tiles and wall bricks with fewer artifacts. In Fig. 7, our method generates more distinct handrails and columns.

In order to make the visual effects more distinctive, we will give residual images to reveal the differences in the camera-ready version. We will also consider conducting a user study to give a quantitative comparison of the results from different methods.

Thank you so much for your reply. We respond to your concerns below.

Q1. The author should revise their paper by presenting results on the best-performing DiffIR and comparing them with SOTA RealSR methods, such as real-esrgan and KDSR in order to demonstrate their real contribution to pushing the upper bounds in the field.

Thank you so much for your suggestion. We will add the results on DiffIR in the next version of our paper to demonstrate the superiority of the proposed method.

Q2. I want to know whether the author will release their codes and models for public application.

Our method employed several codebases because of the different tasks.

For bicubic-SR, we will release the code (including the diffusion model itself and the code of extracting the $\tilde{\mathbf{x}}_T$), the re-implemented SR3 model, and the $\tilde{\mathbf{x}}_T$ used in the corresponding experiments if our paper is accpted.

For StableSR and DiffIR, we directly leveraged their open-source codes (StableSR: https://github.com/IceClear/StableSR, DiffIR: https://github.com/Zj-BinXia/DiffIR) and the corresponding models. We really appreciate their contributions to the community. We will release the $\tilde{\mathbf{x}}_T$ used in the corresponding experiments and the codes of extracting the $\tilde{\mathbf{x}}_T$ if our paper is accpted.

Thank you again for your time in giving reviews and suggestions for our paper. All of your comments really assist us in demonstrating the superiority of our paper and make our paper more robust.

Q1. The paper only discusses the context of SR (and other low-level tasks in the Sec. 5). But it seems that the theory does not limit the relationship between guidance and generated results. Do the authors think that the method can be leveraged in more general generation tasks such like text-to-image generation?

Your suggestion of leveraging the method in other tasks is so valuable. As we have mentioned in our paper, the proposed method can also be used in other low-level vision tasks. However, maybe it is not suitable enough for high-level generation tasks. Because these tasks do not have unique ground truths, it is difficult to define the concept of "optimal sample" which was defined by the ground truth (to be more specific, HR images in our paper). For example, in the task of text-to-image generation, as you mentioned, there can be several images that are semantically aligned to the input text. In other words, there is no exact "ground truth" to an input text.

Thank you so much for your suggestion. We will consider your advice seriously in our future works of leveraging our method in more low-level vision tasks.

Q2. It seems that the improvement on StableSR (in the task of real-SR) is less than the improvement on SR3 (in the task of bicubic-SR). Could the authors discuss the reasons for this phenomenon?

We think that the reason to the performance gap on StableSR and SR3 is that StableSR builds diffusion model in the latent space, while SR3 builds diffusion model in the image space. However, we actually propose the concept of "optimal sample" in the image space (Eqn. 9). Thus, the diffusion process in the latent space affects the accuracy of the approximation of our method (Eqn. 15, 16, 18) slightly, causing less improvement of StableSR. However, our method still helps StableSR to sample better results than its official sampling method (DDPM-100) in shorter time.

Thank you again for your approval of our paper and your valuable review. You are welcome to ask us to give further responses to address your concerns.

[1] Ledig, C., et al. "Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network", CVPR, 2017.

[2] Wang, X., et al. "ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks", ECCVW, 2018.

[3] Ho, J., and Salimans, T.. "Classifier-Free Diffusion Guidance", NIPSW, 2021.

Thank you so much for your thoughtful review which helps us to improve the quality of our paper. For your questions, we have the following responses. We hope that they can help to address your concerns. Sincerely looking forward to your reply.

W1. The proposed method leverages LPIPS as the implementation of distance measurement function $M(·,·)$. Can we leverage pixel-level metrics like negative PSNR as $M$? The authors should give more discussions.

We can also use MSE (which is equivalent to negative PSNR) as the implementation and give results on bicubic-SR in the following table.

The results are slightly worse than the LPIPS-oriented $\tilde{\mathbf{x}}_T$ which is used in our paper. This is because MSE is not a less optimal metric to evaluate such distance. Lower MSE may not mean better image quality, which is a consensus in the domain of image SR [1, 2].

W2. The paper assumes that the model is well-trained. However, even a “well-trained” model cannot fit the real data distribution precisely. Thus, does such an assumption cause potential inaccuracy of the conclusion?

Indeed, the gap between the parameterized distribution of diffusion models and real data distribution will affect the accuracy of our analysis, and this is the reason that our $\tilde{\mathbf{x}}_T$ is an "approximately" optimal boundary condition. We have pointed it out in our paper (in the text description before the Eqn. 16). In order to mitigate the inaccuracy, we leverage a reference set of LR-HR image pairs to find the $\tilde{\mathbf{x}}_T$, instead of one single LR-HR pair.

W3. The paper claims that $p_\theta (\mathbf{y}|h_\theta (\mathbf{x}_T, \phi))$ is approximately uniform. However, what ensures that the model does not have biases when leveraging the “blank token”, which is essentially a placeholder token (that doesn't actually exist)?

As you mentioned, the blank token $\phi$ is only symbolic guidance, which means not give any guidance to the model when generating. When generating images from $\phi$, we are actually unconditionally generating images (such unconditional generating can be achieved by randomly dropping the LR images during training [3]). For $p(\mathbf{x}_0|\phi)$, a "well-trained" model means it should be close to the training data distribution $q(\mathbf{x}_0|\phi)$, which is uniform.

Furthermore, we give more discussion on our claim, "for a well-trained model, such probability (the probability of unconditionally generating an image which is the corresponding SR image of the LR image y) is also approximately uniform". For a "well-trained" model, such probability is close to the probability of "randomly sampling an image which is the corresponding SR image of a certain LR image $\mathbf{y}$ from the HR image dataset". The distribution of the latter one is consistent with sampling training data during the training process, which is uniform for different $\mathbf{y}$. Thus, such probability will also be uniform.

W4. It seems that the process of calculating the approximately optimal BC is time-consuming. How long does it take?

The time of calculating the approximately optimal BC is about 20 hours on a V100 GPU. As we have mentioned in our paper, we take $R=300$ and $K=1000$ when calculating the approximately optimal BC. In practice, we did not simply sample $R \cdot K=300,000$ SR images. We first test all the $K=1000$ $\mathbf{x}_T$ on a small reference set containing 10 LR-HR pairs and only retain the top 300 $\mathbf{x}_T$ as a pre-screening process. Then we leverage a larger reference set and repeat the filtering process. At last, the final reference set contains all the 300 reference pairs.

Such a time cost is a one-off. We just need to calculate the approximately optimal BC once for one diffusion-based SR model. This time cost is much lower than the time cost of training the model itself. Thus, such a process is not time-consuming.

Thank you so much for your reviews and comments of our paper. Your summary of our paper is precise and concise. About your concerns, we give responses below. We hope they can address your concerns.

W1. The experiments are insufficient. Although the proposed method has achieved good results on LPIPS, the PSNR values on multiple test sets are very low. The author did not discuss it in depth and did not show the results of SSIM.

First, the authors understand that PSNR and SSIM are well-recognized measures for SR. However, in recent years, their drawbacks have been revealed due to their failure to consider sampling randomness in measurement and not robust to simple geometric transformation. Therefore, recent generative model-based SR methods (including GAN-based, flow-based, diffusion-based, and so on) [1, 2, 3, 4] focus on the perceptual quality of SR images, mainly measured by LPIPS. This protocol is also adopted in a recently published paper [5], confirming that worse PSNR but better LPIPS can lead to better perceptual quality.

Second, we appreciate the reviewer's comments. The evaluations of SR are still open questions. We will thoroughly discuss and address these issues in our revised paper.

W2. The paper stated that the proposed method has fewer parameters and is more efficient than the GAN method, but did not show the corresponding comparison results. Such as comparison of specific parameters or running time.

Sorry for the confusion. We make the further clarification as follows.

First, our method is a sampling method of current diffusion-based SR models, instead of a new SR model. Thus, our method could be leveraged to sample better SR results from existing methods without any additional parameters.

Second, our method could achieve better performances with fewer sampling steps, which means better efficiency compared with the baseline diffusion-based SR model. The comparison of running time is in proportion to the comparison of sampling steps, which is already given in the paper. However, because of the characteristics of diffusion models, it is still extremely difficult to achieve faster running time compared with GAN-based methods which only need one forward process. The comparisons of running time among different sampling methods from one diffusion-based SR model are given in the following tables. All the times are tested on one single RTX 2080Ti GPU.

For bicubic-SR, the comparison on baseline SR3 is (resolution 256*256):

For real-SR, the comparison on baseline StableSR is (resolution 512*512):

Our statement refers to that our method is able to sample better SR results compared with GAN-based methods from current diffusion-based SR models, achieving better efficiency and effectiveness. Such diffusion-based SR models cannot outperform GAN-based methods with previous sampling methods. We apologize if our statement leads to ambiguity. We will refine the corresponding statement in the next edition of our paper to make it more precise.

Thank you again for your review of our paper. You are welcome to further ask us about your concerns which our responses do not fully address and ask us to give more detailed responses. Really looking forward to your reply.

[1] Ledig, C., et al. "Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network", CVPR, 2017.

[2] Wang, Xintao, et al. "ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks", ECCVW, 2018.

[3] Lugmayr, Andreas, et al. "SRFlow: Learning the Super-Resolution Space with Normalizing Flow", ECCV, 2020.

[4] Saharia, Chitwan, et al. "Image Super-Resolution via Iterative Refinement", IEEE TPAMI, 2022.

[5] Gao, Sicheng, et al. "Implicit Diffusion Models for Continuous Super-Resolution", CVPR, 2023.