Dissecting Arbitrary-scale Super-resolution Capability from Pre-trained Diffusion Generative Models

Q1: If I understand correctly, you are suggesting that we should include the performance results for non-integer scaling. We have included the results of our model as well as several baseline models. However, it is worth noting that most of these baselines only support integer-scale super-resolution. Therefore, we have only included two additional baselines, namely LIIF and ITSRN.

Q2: To evaluate our model, we followed the degradation pipeline designed by ESRGAN and used the corresponding degradation code. We applied various methods such as blur, JPEG compression, and noise to degrade the original image. We then injected noise and performed denoising to assess the recovery effect of our model. Additionally, we utilized the two-order pipeline from ESRGAN-pipe to degrade the image multiple times and finally downsampled it by a factor of 2. Interestingly, when using a single type of degradation, the model showed impressive image restoration capabilities. Among the different degradation methods, the model performed best in terms of recovering blurred images, but struggled the most with compression at a 4x scale. Furthermore, we observed that the visual degradation caused by a single degradation method, especially JPEG compression, was not very apparent. However, in real-world scenarios where images undergo multiple degradations using ESRGAN's pipeline, the model's performance significantly declined. Image restoration with multiple degradations remains a challenging task.

Q3: Following your suggestion, we have included the results of LPIPS and the inference time, which are presented below:

Q1: Thank you for your advice. We have included the results of ASSR baselines (LIIF, ITSRN). However, other baselines like SwinIR, ESRGAN, DDNM, etc., only support super-resolution with integer scales. Although we can compare them by first using an integer-scale model to upscale the images and then rescaling them to 256x256 using a method like Bicubic, we have been advised that this type of comparison is unfair.

Q2: In our arbitrary-scale super-resolution setting, we have a target resolution of 256x256. Our goal is to upscale any resolution under this setting to the target resolution. So, for an input of 100x100, we are actually conducting a 2.24x super-resolution. We consider this as an ASSR capacity, and we believe this setting can potentially be used in image and video applications. Since users prefer images and videos in fixed resolutions like 1080p, 2K, 4K, it may be a better choice to upscale an image with a resolution of 854x480 to 1920x1080 rather than upscaling it to a different scale with a similar resolution to 1920x1080.

Q3: When we mention using the interval, we mean that if the total diffusion sampling steps from noise $x_T$ to image $x_0$ is 50 (T=50), we inject 0.1*T = 5 steps of noise to obtain $\hat{x}_t$ (t=5) and then denoise $\hat{x}_t$ to get $x_0$.

Q4: As shown in Figure 3, the results obtained at different scales vary. For example, an acceptable result can be achieved at a noise level of 0.2 for a 2.5x magnification, but the 4.5x image still appears blurry at the same magnification. Therefore, we need to inject more noise. When the noise level reaches 0.3 or 0.4, the effect improves, but as we mentioned earlier, there may be slight differences in details compared to the original image, which we consider tolerable. The significant difference occurs when the noise level in the PRF exceeds the threshold of about 0.5-0.6, where a noticeable difference occurs. To strike a balance, our current strategy is to empirically estimate the appropriate recovery interval for each scale. As shown in Figure 4 and Figure 6, there is a suitable recovery interval for each scale. Within this interval, the model's recovery effect on the image does not vary significantly, so we choose the leftmost side of the interval as the noise level we inject.

Q5: The low-level fidelity measure and high-level signature are consistent with Figure 3. In this context, the term "high-level signature" refers to whether the image depicts the same person, while "low-level fidelity" refers to the clarity or blurriness of the image.

Q6: Thank you for pointing that out. In our statement, we actually meant to express the idea of "restoring a low-resolution image to a high-resolution image." We will carefully review our draft to avoid similar typos in the future.

Sorry for the Q4 and Q5:

For Q1 and Q2, we have revised our manuscript and change the claim from ASSR to TRSR (Target-resolution super-resolution).

For Q3, here is the table of noise level that we use empirically. As we have proved in the appendix, The noise level is related to the difference between HR and LR images. Currently, we do not have a mathmatical tool to model this difference under different scale. So we leave it as our future work.

For Q4, we forget to post the answer as reviewer [cxpS] also ask the same question. The result is shown in the comments Q5 in the reviewer [cxpS]. Current, it is still challenging to restore the image in real-world complicated degradation setting. However, what we want to focus on in this paper is the ability to recover HR from any-scale LR image, we will keep improve the model performance under complicated real-world setting.

For Q5, we have add the visual comparison in different scale in the supplementary file. Since images of different scales almost follow the same path to return HR images, there is no big visual change between different scale.

Q1: Upsampling LR images to match the target pixel size is a commonly used technique in diffusion-based SR methods. While this technology is not new, our main contribution lies in combining upsampling with noise injection to utilize diffusion model for ASSR tasks. We have also conducted a mathematical analysis to explain why this approach can be effective. We acknowledge that our previous wording may have caused some confusion as Reviewer [c56f] said and we will be more careful with our choice of words and expressions in our revised version.

Q2: The concern you raised regarding computational cost is a potential issue that exists in all current diffusion-based research. Our method naturally reduces some computational overhead by injecting noise only to a portion of the original image. Instead of starting from pure noise to generate the image, we begin the denoising process from the middle point, saving computational resources. Additionally, we have employed the DDIM sampler to reduce the total number of sampling steps to 50. Furthermore, consistency models suggest that one-step denoising can directly yield the original image from any intermediate result. Although we consider this as a potential solution for computational overhead, we leave it as future work.

Q3: Our codebase is built upon the original DDPM [1]. In this codebase, the base-channel for the U-Net is set to 64. However, the latest DDPM codebases have larger base-channels, such as 256 (DDRM, DDNM) and 320 (LDM). This difference in parameter settings creates a gap between our model and other DGM models. Due to limitations in computing resources and time, it is currently challenging for us to train with a higher base-channel. We did attempt to reduce the base-channel from 64 to 32 to alleviate training overhead, but this led to a significant decrease in model performance.

Q4: DDNM and DDRM are two recent studies that we consider relevant to our work. We acknowledge that StableSR is a novel diffusion-based SR model, and we have included its results for comparison. However, it is important to note that the experimental comparison may not be entirely fair, as StableSR can only upscale LR images to 512, while our experimental settings require a target pixel size of 256x256. Therefore, we had to downsample the StableSR image to match our target size. We provide this result as a reference, understanding that the comparison may not be entirely equitable.

Q5: As your suggested, we followed the degradation pipeline designed by ESRGAN and tested it using the corresponding degradation code on DIV2K. We applied Blur, JPEG compression, and noise methods to degrade the original image. We then injected noise and denoised the image to evaluate the recovery effect of the model. Additionally, we used the two-order pipe from ESRGAN-pipe to degrade the image multiple times and finally reduced the image by 2 times. Interestingly, when using only a single type of degradation, the images were still repaired effectively. Among the mentioned degradations, the model performed best in recovering blur, but struggled the most with compression at 4x scale. Furthermore, we observed that the degradation results of image processing with a single degradation method were not visually obvious, especially in the case of JPEG compression. However, when applying ESRGAN's pipeline to degrade images multiple times, the model's performance significantly declined. Therefore, image recovery with multiple degradations remains a challenging task.

Q6: Yes, all of our experiments were conducted using the same experimental settings.

Q7: In our experiments, we used the same compression method for the same compression ratio. Therefore, we assumed that all the images had the same blur level. We also used the same parameters to inject noise and denoise the images during the enhancement process. However, when entering the formula (19) that we derived, we realized that the decision of noise level is actually related to the difference between the high-resolution (HR) and low-resolution (LR) images. If the initial blur level of the image is different, the image with a higher degree of blur will experience a greater fidelity loss, requiring more noise injection. From this perspective, the degree of blur does play an important role in the trade-off. Unfortunately, we were unable to find suitable mathematical tools to establish a connection between the Fidelity term and the degree of blur in this paper.

[1] https://github.com/lucidrains/denoising-diffusion-pytorch

For Q1, I will revise the claim according to your advice, but I am not sure whether I can revise the title.

For Q4&Q5, I will add more visual comparisons in my revised version.

For Q6: If you mean our arxiv version, the difference is because we use different datasets for evaluation since in our last version we are suggested to use more common datasets for evaluation.

Thanks for the authors' response.

Q1: The claim of arbitrary-scale super-resolution (ASSR) is strange, as mentioned by reviewer oyvV, as this work generates the results with a fixed resolution. Generally, the ASSR works can generate results with any resolution, or any scale, rather than the fixed resolution. If this can be regarded as ASSR, nearly all of the restoration works can be regarded as so. Please revise it carefully.

Q2~Q3: It is well addressed.

Q4: You can give some visual comparisons in the main paper by submitting a revised version.

Q5: Please give the visual comparison in the main paper.

Q6: I mean why the given quantitative results are not consistent with the original paper, as they have the same experiment settings?

Q7: It is well addressed.

Q1: Thank you for your suggestion. Regrading of the “pioneering”, our main intention is to convey that the use of pre-trained DGM to implement downstream ASSR without distillation and fine-tuning is a novel method that offers a fresh perspective on DGM usage. We apologize for any misunderstanding caused by the term "pioneering technology." In our revised version, we will be more careful with our choice of words and expressions.

Q2: We agree with your idea that prior works should be properly cited. However, when designing Figure 1 and Figure 2, we did not refer to the design of Figure 2 in LDM. Figure 2(a) draws inspiration from the Figure 2 of consistency models [1], which we have cited in the references. Additionally, we have cited the LDM paper. Furthermore, we do not claim to be the first to discover this, as the Diffusion model has seen rapid development in the past two years, with numerous new papers being published daily. It is not surprising that many papers may have made similar observations. We have referenced all the papers we found during our research. However, due to the fast-paced nature of the latest research progress, it is possible that we may have missed some papers.

Q3: We have also attempted to apply our ideas to LDM. However, we encountered a problem when transplanting this solution to LDM. Most text-conditioning in LDM is latent-space based, which means that before conducting the Diffusion process, the image needs to be encoded by the VAE model. However, the compression of the VAE model presents a challenge. The VAE model acts as an amplifier, exacerbating the gap between the HR image and the LR image in the latent space, resulting in narrower PRF windows. Therefore, achieving the same effect as the native Diffusion model when using LDM becomes difficult.

Q4: Yes, as you said, we actually meant to express the idea of "restoring a low-resolution image to a high-resolution image." Thank you for pointing this mistake out.

[1] Song Y, Dhariwal P, Chen M, et al. Consistency models[J]. 2023.