One-Step Effective Diffusion Network for Real-World Image Super-Resolution

Q1. Comparison with DMD.

Thanks for the nice suggestion. While both OSEDiff and DMD draw upon the concept of variational distillation from ProlificDreamer, they differ significantly in several aspects. First, DMD is designed for text-to-image tasks, whereas OSEDiff is tailored for image restoration tasks, aiming to reconstruct an HQ image from its LQ counterpart. To achieve this goal, OSEDiff employs a trainable VAE encoder to remove degradation from LQ images, and it directly takes the LQ as input to the diffusion network to avoid the uncertainty caused by random noise sampling. Trainable LoRA layers are introduced to the diffusion UNet to adapt it to the restoration task. In contrast, DMD lacks these specific designs tailored for image restoration tasks. Furthermore, DMD requires full parameter fine-tuning of two SD models, while OSEDiff only needs to fine-tune a small number of LoRA parameters. This makes OSEDiff more memory-efficient and training-friendly. We will cite DMD and discuss its similarity and differences from OSEDiff in the revision.

Q2. Comparison of the total number of model parameters and FLOPs.

Thanks for the suggestion. The table below compares the total number of model parameters and FLOPs of different methods.

We see that those pre-trained SD model based methods (i.e., StableSR, DiffBIR, SeeSR, PASD and OSEDiff) have a similar total number of parameters. SinSR employs the diffusion model trained in ResShift, which is much smaller than the pre-trained SD model. However, the generalization capability of ResShift and SinSR is much lower than that of the SD based methods.

In terms of FLOPs, the multi-step diffusion-based models have significantly higher FLOPs than single-step methods (e.g., SinSR and OSEDiff) because they run the SD UNet for multiple times. The input resolution of SinSR's VAE decoder is twice of that of OSEDiff's VAE decoder, as SinSR uses an f4 VAE and OSEDiff uses an f8 VAE. Overall, OSEDiff and SinSR have similar FLOPs. We will add these analysis in the revision.

Table: Complexity comparison among different methods. All methods are tested with an input image of size 512×512 on an A100 GPU.

Q3. Robust text prompts from LQ.

Sorry that we did not make it clear. We adopt the DAPE module from SeeSR, which has proven to be robust to the degradation of LQ images for extracting tag-style text prompts. The DAPE module is trained to generate correct semantic prompts even when the input images are severely degraded.

Q4. Limitations in fidelity.

Compared with traditional image restoration methods, the recent methods that leverage pre-trained SD priors can produce perceptually much more realistic results. As a compromise, some generated details may not be faithful enough to the input LQ image. This is an inherent limitation of such generative prior based methods, as noted by this reviewer.

Compared with other methods along this line, however, the proposed OSEDiff has achieved a much better balance between fidelity and creativity. First, it takes the LQ image as input without any random noise and performs only one-step diffusion. This significantly improves the stability of image synthesis using diffusion priors, reducing much the possibility of generating false details. On the other hand, OSEDiff utilizes VSD loss to regularize the generator network, ensuring that it can preserve the generative capacity of the pre-trained SD model. As can be seen from the experiments in the main paper, OSEDiff achieves state-of-the-art results in both fidelity and perceptual metrics among the SD based methods. In future research, for image scenes or regions that require high-fidelity, such as face and text, we can apply a higher weight of fidelity loss. For scenes like greenery and hair where the perceptual quality is more important, we can adaptively apply a higher weight of VSD loss to enhance expressiveness.

Q1. Unclear motivation.

Thanks for the comments. The goal of this work is to develop an efficient and effective Real-ISR method by using the pre-trained SD prior. In the research and development of Real-ISR methods, how to construct LQ-HQ training pairs is a critical issue. Therefore, we spend a certain amount of space to introduce this problem so that readers can have a more complete understanding of Real-ISR. Based on this reviewer's comments, we will compress this part and make the introduction of LQ-HQ pair more concise.

Q2. The benefits of reduced uncertainty.

Thanks for your suggestion. Previous diffusion-based Real-ISR methods apply several denoising steps to generate HQ latents from noisy inputs. However, this process introduces unwanted randomness in the Real-ISR outputs, causing variations with different noise samples. OSEDiff uses LQ image as input to the diffusion network without random noise sampling, enabling a deterministic mapping from LQ to the HQ image. Meanwhile, OSEDiff leverages the VSD loss to enhance the generation capability of the LoRA finetuned network.

As suggested by this reviewer, in Figure 2 of the enclosed PDF file we provide visual comparison of the Real-ISR results of OSEDiff, StableSR and SeeSR. For StableSR and SeeSR, we show their results with two different noise samples. One can see clearly the difference between the two outputs caused by the randomness of noise. For example, the details in StableSR-1 are overly generated, while the result of StableSR-2 is smooth. In contrast, OSEDiff does not involve noise in the input, and it achieves stable Real-ISR result. We will add the visual evidence and analysis in the revision.

Q3. Visualization of network outputs.

Thanks for the good suggestion. As suggested by this reviewer, we visualize the distributions of outputs of OSEDiff with and without VSD loss using t-SNE on the RealSR dataset. The distribution of pre-trained SD model is plotted as a reference. We use LLaVa1.5 to generate the captions for the RealSR dataset, and use them as prompts for pre-trained SD, with the inference steps set to 50. As shown in Figure 3 of the enclosed PDF file, the output distribution of OSEDiff with VSD loss is much closer to the pre-trained SD than the OSEDiff without VSD loss. This clearly validates that the VSD loss aligns the distribution of diffusion network outputs with that of pre-trained SD. We will add this part in the revision.

Q4. Why use CLIPIQA for Real-ISR evaluation.

We follow previous works (StableSR, DiffBIR, and SeeSR) to use CLIPIQA as an evaluation metric for fair comparison. The visual examples in Figures 1, 4, and 5 the enclosed PDF file demonstrate that higher CLIPIQA scores generally correspond to better visual quality of the images. However, we agree with this reviewer that highly reliable no-reference IQA remains an unsolved problem, especially for real-world images, and it needs more efforts from the community to find a more robust metric.

Q5. Details about trainable LoRA module.

Thanks for the suggestion. We finetuned all layers except the normalization layers using LoRA. We empirically found that fine-tuning the normalization layers causes the model to crash, so we frozen them. We will add more detailed descriptions in the revision.

Q6. Null prompts vs. Tag prompts.

Though the results with null prompts score higher on some reference metrics (e.g., PSNR and DISTS) than those with tag-style prompts extracted by DAPE, their visual quality is inferior. This is mainly because the commonly used metrics such as PSNR cannot faithfully reflect the visual quality of images. As shown in Figure 4 of the enclosed PDF file, the results using tag-style prompts have richer leaf vein textures, clearer lines, and less noise. Therefore, we choose tag prompts extracted by DAPE. We will further clarify this in the revision.

Q1. Comparison with SinSR.

The improvements of OSEDiff over SinSR mainly come from the pre-trained SD model and the VSD loss. The SD model, pre-trained on large-scale data, contains rich prior knowledge of natural images, enabling effective one-step generation. Additionally, we finetune the multi-step SD model into a single-step network using LoRA and VSD loss. It ensures the one step generation capacity of OSEDiff for restoration tasks while avoiding the lengthy inference time of multiple steps.

Q2. Total params rather than training params.

Thanks for the suggestion. The table below show the total number of parameters, FLOPs and the inference time of the competing methods. Please refer to our responses to Q2 of Reviewer KDhM for more discussions. We will add this table and the associated discussions in the revision.

Table: Complexity comparison among different methods. All methods are tested with an input image of size 512×512 on an A100 GPU.

Q3. Why is the PSNR lower when using VSD Loss compared to not using it, and more visualization results.

In the objective function of a general learning-based image restoration framework (refer to Eq. (1) in the main paper), there are two major components: a fidelity term and a regularization term. The fidelity term is usually measured and constrained by $L_2$ or $L_1$ norms, which is friendly to the PSNR metric, while the regularization term is designed based on the employed prior knowledge of natural images. In our work, the VSD loss, which aligns the distribution of network outputs to that of SD prior distribution (i.e., natural image prior distribution), is used as the regularization term. With VSD, the fidelity will be traded-off a little but the perceptual quality will be much enhanced (due to the alignment with natural image prior distribution). Without VSD, the network will focus on optimizing the fidelity term so that the PSNR metric can be improved; however, the perceptual quality will be decreased. Visual examples can be found in Figure 5 of the enclosed PDF file. Without the VSD loss, the PSNR of the restoration results is higher, but they lack semantic details.

Thank you very much for pointing out this problem. We carefully re-examined OSEDiff's inference code and found that it is a Python decorator function that consumes much time. This function is just to calculate the inference memory usage without affecting anything on the restoration process. We therefore removed this function and reassessed the inference time of different modules for both OSEDiff and SinSR. The results are shown in the table below:

We can see that the overall time consumption of OSEDiff is 0.116s, which is actually less than SinSR (0.130s). Though the DAPE module in OSEDiff costs additional time, the VAE Encoder, UNet, and VAE Decoder modules of OSEDiff consume less time than those of SinSR. Note that although SinSR's UNet has lower FLOPs (202G) than OSEDiff's UNet (355G), its inference time is nearly doubled because SinSR's UNet employs Swin Transformer blocks. The frequent window partitioning operations elevate memory access and data movement costs, resulting in increased latency. Similar observations can be seen for OSEDiff's DAPE module, which also uses a Swin Transformer backbone. Its FLOPs is less than 1/3 of that of UNet, yet its time consumption is comparable to UNet.

The time consumption of SinSR reported in our main paper is 0.16s, which includes the time for converting the model output tensor to an image. We will change it to 0.130s in the revision for fair comparison.

Again, we sincerely thank this reviewer for the careful reading of our paper and indicating this problem. We will correct this issue in the revision.

Q1. Multiple inference steps for OSEDiff.

Please kindly note that OSEDiff is specifically designed for one-step diffusion for Real-ISR. Unlike previous multi-step methods (e.g., StableSR, PASD, SeeSR), which are all based on ControlNet by using noise as input and LQ image as control signal, OSEDiff directly uses LQ image as input (without random noise) and apply LoRA to finetune the diffusion network for producing HQ output. In its current network design, OSEDiff cannot be used for multiple denoising steps. Nonetheless, we believe that for the Real-ISR task, OSEDiff is much preferred as it achieves faithful and perceptually realistic results while significantly reducing the complexity. If we wish to extend OSEDiff to a multi-step framework for stronger generative capability, we could model multi-step diffusion in the residual space, similar to ResShift. We will explore this possibility in the future.

Q2. Details about regularizer networks.

Thanks for pointing out this problem. We will add more details on the regularizer networks in the revision. To model the distribution of natural and restored images, two diffusion models are needed to compute the KL-divergence via variational score distillation (VSD): one for the natural image distribution and another for the restored image distribution. Since the pre-trained SD model can effectively represent the natural image distribution, one regularizer network can be the frozen SD U-net. On the other hand, the U-net can be fine-tuned to model the distribution of restored images. Therefore, we fine-tune the SD U-net as another regularizer network in a similar way to that we fine-tune the SD as the generator network. However, they are trained differently: the generator is trained on $t=999$ to perform super-resolution, while the regularizer is trained on $t \in (\{0, \dots,999\})$ to align with the restored image distribution via VSD loss.

Q3. LQ as condition.

Sorry for the confusion caused. Actually, different from previous methods such as StableSR, PASD and SeeSR, our proposed OSEDiff does not use the LQ image as a condition but directly as the input to the UNet. First, we upsample the LQ image to the target size. Then, we input it into the encoder to obtain the latent feature, which is passed into the UNet to obtain the refined feature. Finally, the refined feature goes through the decoder to output the restored HQ image. For example, for $\times$4 SR on a 128$\times$128 LQ image, we first use a bicubic interpolator to upsample it to 512$\times$512. Then, we pass it to OSEDiff to get a 512$\times$512 HQ image. We will clarify this in the revised manuscript.

Q4. Finetune VAE encoder.

We apologize for any lack of clarity in our manuscript. The VAE encoder is set to be trainable to remove degradation. In other words, we fine-tune the encoder with LoRA so that it can also serve as a degradation removal network to pre-process the input LQ image.

To better illustrate the role of fine-tuning the encoder, we compare the performance of OSEDiff on the RealSR dataset with and without fine-tuning the VAE encoder. The results are shown in the table below. We can see that fine-tuning the encoder significantly improves the no-reference metrics. Although fine-tuning the encoder leads to slightly worse full-reference metrics such as PSNR and LPIPS, these metrics do not necessarily indicate better visual quality. In Figure 1 of the enclosed PDF file, we visualize the results of OSEDiff with and without encoder fine-tuning. One can see that fixing the VAE encoder may introduce some artifacts, which can be caused by the severe degradation in the LQ input. We will add the above discussions in the revision.

Table: Comparison of OSEDiff with and without fine-tuning the VAE encoder on the RealSR dataset.

Q5. LR inputs of different resolutions.

Yes, OSEDiff can handle inputs of different resolutions in a similar way to other diffusion based super-resolution methods. First, the LR image (with the size of H$\times$W) is upsampled to an HR image $Y_{bicubic}$ (with the size of rH$\times$rW) using bicubic interpolation, where $r$ is the SR factor. If the short side of $Y_{bicubic}$ is less than 512, we resize its short side to 512 before feeding it to OSEDiff. Finally, the output is resized back to rH$\times$rW to obtain $Y_{osediff}$. In other cases, $Y_{bicubic}$ (rH$\times$rW) is directly fed into OSEDiff to get $Y_{osediff}$ (rH$\times$rW). For the $\times$4 SR task, if LR is 32$\times$32, we first upsample LR to 512$\times$512 using bicubic interpolation, enhance it to 512$\times$512 by OSEDiff, and then resize the output to 256$\times$256.