One Step Diffusion-based Super-Resolution with Time-Aware Distillation

Table 7: Complexity comparison among different SD-based SR methods. All methods are tested on the ×4 (128→512) SR tasks, and the inference time is measured on an V100 GPU.

Q8: Are there any limit conditions for using the method? The author should discuss and analyze the limitations of the proposed method. It is recommended to add a discussion of a discussion of potential limitations or where the proposed method might not perform as well.

A8: Thank you for your suggestion. Although our single-step method demonstrates strong performance, it shares a common limitation with current single-step distillation methods: increasing the number of inference steps alone does not yield better performance. Thus, developing a distillation method that matches the performance of state-of-the-art single-step approaches while enabling additional inference steps to enhance performance is a key area of our ongoing research.

References

[1]Yin, T., Gharbi, M., Zhang, R., Shechtman, E., Durand, F., Freeman, W. T., & Park, T. (2024). One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 6613-6623).

[2]Hertz, A., Aberman, K., & Cohen-Or, D. (2023). Delta denoising score. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 2328-2337).

Table 4: Quantitative results of different SR methods. The best and second best results are highlighted in bold and italic. ∗ indicates that the result was obtained by replicating the method in the paper.

Table 5: Generative performance on unconditional CIFAR-10. The best results are highlighted in bold.

Q6: In the experimental section, the authors compare many GAN and transformer-related methods. However, the proposed method is a diffusion model and should be compared with the most relevant diffusion models to validate its efficiency, especially accelerated diffusion models, including OSEDiff[3], DPM++[4], Unipc[5], etc.

A6: Thank you for your suggestion. Since OSEDiff is an SD-based SR method, we compared our approach to OSEDiff while distilling the SD-based SR model SeeSR. This ensures a fair comparison, as both methods were trained on the same dataset. As shown in Tables 1, 2, and 3, our method outperforms OSEDiff across most evaluation metrics.

In response to the reviewers' suggestions, we have also incorporated the designed sampler methods, Unipc[5] and DPM++[4], into Tables 1, 2, and 3. (Note that we did not apply these samplers to ResShift, as ResShift modifies the standard Markov chain, creating challenges for its adaptation to these samplers.) Despite this, the results clearly demonstrate that our method significantly outperforms methods employing these samplers.

Q7: The authors claim that the method is designed to accomplish effective and efficient image super-resolution, but did not include a complexity comparison of the different methods (including parameters, sampling steps, running time, MACs, etc.), which is crucial for diffusion models. Please provide a Table to compare these computational complexity metrics with the key baselines.

A7: Based on the reviewers' feedback, we have included a complexity comparison between TAD-SR and baseline methods, as presented in Tables 6 and 7. Table 6 focuses on comparisons with GAN-based methods and diffusion-based super-resolution methods trained from scratch. The results demonstrate that TAD-SR accelerates the teacher model, ResShift, to a single inference step, improving its speed by approximately tenfold. Table 7 highlights a comparison of inference time with SD-based super-resolution methods, revealing that our method's inference delay is only 7.6% of the teacher model, SeeSR.

Table 6: Complexity comparison among different SR methods. All methods are tested on the ×4 (64→256) SR tasks, and the inference time is measured on an A100 GPU.

Thank you for your comments and feedback. We address your concerns here.

Q1: It is confusing which is the final output of the model when inference, z_0^{stu} or z ̂_0^{stu}? It is not clearly indicated in Figure 4. Please explicitly state in the text and figure.

A1: Thank you for pointing out this issue. $z_0^{stu}$ is the final output of the student model. $\hat{z}_0^{stu}$ represents the clean value predicted by the teacher model after re-adding noise to the output of the student model. This value is used to calculate the loss. We will revise the paper and the images in the manuscript to enhance clarity and make them easier to understand.

Q2: The authors should clarify if the teacher model is used at all during inference, or if it is only used during training. If I understand correctly, only the student model samples one step, and then the teacher model is used later to sample multiple steps to get the final clean latent, so the model performance relies heavily on the performance of the teacher model, and is not exactly efficient.

A2: Thank you for your suggestion. As the reviewer understands, the teacher model is only used during the training. Additionally, we not only leveraged the knowledge from the teacher model but also incorporated the ground truth (GT) into the distillation framework through adversarial learning to provide additional supervision for the model. Therefore, the performance of our method is not solely dependent on the teacher model’s performance.

Q3: What is the purpose of setting the weighting function (ω = 1/CS )? Please provide intuition for why this weighting function was chosen, and what effect it has on the training process or results.

A3: Apologies for the confusion. What we intended to convey is that our score distillation loss is averaged over both spatial and channel dimensions, which facilitates model optimization [1][2]. However, there was an error in the formula expression, and we will correct this in the next version of the manuscript.

Q4: In order to eliminate the dependence of the proposed method on the teacher model of ResShift, the relevant ablation experiments should be conducted by replacing the different teacher models to validate the effectiveness of the proposed method.

A4: Thank you for your suggestion. We have included the results of distilling the SD-based SR method SeeSR into a single step using TAD-SR. The quantitative and qualitative experimental results are presented in Tables 1, 2, and 3. As shown in the tables, our proposed distillation method demonstrates strong generalization capabilities, effectively distilling different teacher models into a single step and generating promising results.

Table 1: Quantitative comparison with state of the arts on RealSR dataset. Following the experimental setup of SeeSR, the LR images in the RealSR dataset were center-cropped to 128 $\times$ 128. The best and second best results are highlighted in bold and italic.

Thank you for your response. We will address your remaining concerns as follows.

Q1: The explanation that the proposed model relies heavily on the teacher model.

A1: First, we would like to clarify that during inference, only the student model performs single-step sampling to generate samples, while the teacher model supervises the student by generating samples through multi-step sampling during training.
Second, the knowledge distillation technique aims to transfer knowledge from the teacher model to student model through training, meaning the performance of the student model is inevitably influenced by the teacher. However, to prevent the student model's performance from being entirely constrained by the teacher, we have incorporated ground truth into the distillation framework through adversarial learning, providing additional supervision. Experimental results demonstrate that our method even outperforms the teacher model on certain non-reference metrics. Furthermore, in response to the reviewer’s comments, we replaced the teacher model in our experiments. As shown in Tables 11, 12, and 13 of the paper, our method continues to generate high-quality images through single-step inference, clearly demonstrating its effectiveness.

Q2: The weighting function of HSD.

A2: Regarding the loss weight, we followed the approach used in DMD[1] and DDS[2], normalizing the loss across both spatial and channel dimensions($i.e.,\omega = 1/CS$). This normalization is commonly applied in prior model training, as it facilitates better model optimization. We have also provided results without weighting function $\omega$ for comparison, and the effectiveness of the weighting function is evident from Table 1.

Q3: Complexity comparison.

A3: Ultimately, we would like to emphasize that both Table 2 in the initial manuscript and Table 6 in the revised manuscript provide a comparison of the sampling steps, inference time, and parameter count between our method and other methods. Additionally, in response to the reviewer’s comments, we have included a comparison of FLOPs, with the results shown in Tables 2 and 3. Table 2 focuses on comparisons with diffusion-based super-resolution methods trained from scratch. Table 3 highlights a comparison of computational complexity with SD-based super-resolution methods.

Table 1: Ablation studies of the weighting function of HSD on RealSR and RealSet65 benchmarks. The best results are highlighted in bold.

Table 2: Complexity comparison among different SR methods. All methods are tested on the ×4 (64→256) SR tasks, and the inference time is measured on an A100 GPU.

Table 3: Complexity comparison among different SD-based SR methods. All methods are tested on the ×4 (128→512) SR tasks, and the inference time is measured on an V100 GPU.

[1]Yin, T., Gharbi, M., Zhang, R., Shechtman, E., Durand, F., Freeman, W. T., & Park, T. (2024). One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 6613-6623).

[2]Hertz, A., Aberman, K., & Cohen-Or, D. (2023). Delta denoising score. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 2328-2337).

Q5: The motivation is not clear. If the proposed method wants to achieve one-step SR, why it is important for student model to learn how to deal with the intermediate steps?

A5: Sorry, it may be that our description was not clear enough, which caused a misunderstanding for you. We will enhance the readability of the paper in the revised PDF. To clarify, our student model accepts a fixed time step $T$ to generate clean samples in a single step. The intermediate time steps we sample are used solely to calculate the loss. Specifically, we leverage the pre-trained diffusion model's ability to handle intermediate time steps to constrain the single-step output of the student model. Diffusion models typically predict low-frequency information in the early stages of denoising and high-frequency information in the later stages. Therefore, we add varying levels of noise to both the clean samples generated by the student model and the teacher model, then feed them into a pre-trained diffusion model for prediction. By calculating the distance between the two predicted values, we can constrain the samples generated by the student model to match the high-frequency or low-frequency information in the teacher model's generated samples.

Q6： Will increase the inference steps contribute to the improvement of the performance?

A6: This is really a good question! Normally, if only a single time step is sampled to train the student model, simply increasing the number of iterations during inference will not lead to any performance improvement. This is because the model has only learned the mapping from noisy data to clean data at that specific time step and lacks the ability to process noisy data at other intermediate time steps. This limitation is common to all single-step distillation methods. Thus, developing a distillation method that matches the performance of state-of-the-art single-step approaches while enabling additional inference steps to enhance performance is a key area of our ongoing research. We will include a discussion on this aspect in the revised PDF.

References

[1] Wang, J., Yue, Z., Zhou, S., Chan, K. C., & Loy, C. C. (2024). Exploiting diffusion prior for real-world image super-resolution. International Journal of Computer Vision, 1-21.

[2] Xie, R., Tai, Y., Zhao, C., Zhang, K., Zhang, Z., Zhou, J., ... & Yang, J. (2024). Addsr: Accelerating diffusion-based blind super-resolution with adversarial diffusion distillation. arXiv preprint arXiv:2404.01717.

[3] Wu, R., Yang, T., Sun, L., Zhang, Z., Li, S., & Zhang, L. (2024). Seesr: Towards semantics-aware real-world image super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 25456-25467).

[4] Sauer, A., Lorenz, D., Blattmann, A., & Rombach, R. (2025). Adversarial diffusion distillation. In European Conference on Computer Vision (pp. 87-103). Springer, Cham.

[5] Xu, Y., Zhao, Y., Xiao, Z., & Hou, T. (2024). Ufogen: You forward once large scale text-to-image generation via diffusion gans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 8196-8206).

Table 2: Quantitative results of different methods on the dataset of CelebA-Test. The best and second best results are highlighted in bold and italic. ∗ indicates that the result was obtained by replicating the method in the paper.

Table 3: Ablation studies of the proposed methods on ImageNet-Test benchmarks. The best results are highlighted in bold.

Q3: Although I understand that StableDiffusionXL also employs adversarial loss, it appears less elegant to me due to the inherent limitations of GANs.

A3: Recently, many diffusion-based methods [4][5] have begun integrating adversarial learning into the training process. Experimental results demonstrate that this approach can significantly enhance model performance, underscoring its potential value.

Q4: In addition to the difficulty of assessing performance without PSNR and SSIM, the reported improvements seem marginal compared to existing methods.

A4: In addition to PSNR and SSIM, our method demonstrates significant improvements over SinSR in other metrics. The table below lists the percentage improvements achieved by our method compared to SinSR.

Table4: Quantitative comparison with SinSR method in super-resolution tasks.

Thank you for your comments and feedback. We address your concerns here.

Q1: The organization of the paper needs improvement, as it is challenging to clearly understand the core idea. For instance, Fig. 2, which aims to illustrate the paper's motivation, has a caption that provides limited information.

A1: Thank you for your suggestion. We will carefully describe the details of this method in the revised manuscript to improve the readability and clarity of the paper.

Q2: The paper lacks essential metrics, such as PSNR and SSIM, to evaluate model fidelity. As shown in previous works, there is a trade-off between PSNR, SSIM, and CLIPIQA, MUSIQ. Reporting only LPIPS and non-reference IQA metrics is insufficient to demonstrate performance. Both the main results and ablation studies should include these metrics.

A2: Thank you for your suggestion. We have included PSNR and SSIM metrics in both our main experiments and ablation studies, as shown in Tables 1, 2, and 3. However, our experimental results, along with findings from previous studies, indicate that PSNR and SSIM do not always align with human perception or other indicators such as LPIPS, CLIPIQA, and MUSIQ. Specifically, when image quality improves, and these perceptual indicators yield higher values, PSNR and SSIM often decrease. Conversely, an increase in PSNR and SSIM typically corresponds to smoother and blurrier images. For instance, while methods such as LDM, ResShift, and DASR achieve higher PSNR and SSIM scores compared to others, the images they generate tend to appear smoother or blurrier (as shown in Figures 6 and 12). We infer that this discrepancy likely arises because PSNR and SSIM measure image differences in pixel space, whereas human perception and other metrics evaluate images based on perceptual quality. Therefore, we regard PSNR and SSIM as reference metrics rather than primary evaluation metrics in real-world super-resolution tasks, consistent with the conclusions of prior work [1][2][3].

Table 1: Quantitative results of different methods on the dataset of ImageNet-Test. The best and second best results are highlighted in bold and italic. ∗ indicates that the result was obtained by replicating the method in the paper.

Q3: Please compare the inference time of TAD-SR and baseline methods.

A3: Based on the reviewers' feedback, we have included a complexity comparison between TAD-SR and baseline methods, as presented in Tables 4 and 5. Table 4 focuses on comparisons with GAN-based methods and diffusion-based super-resolution methods trained from scratch. The results demonstrate that TAD-SR accelerates the teacher model, ResShift, to a single inference step, improving its speed by approximately tenfold. Table 5 highlights a comparison of inference time with SD-based super-resolution methods, revealing that our method's inference delay is only 7.6% of the teacher model, SeeSR.

Table 4: Complexity comparison among different SR methods. All methods are tested on the ×4 (64→256) SR tasks, and the inference time is measured on an A100 GPU.

Table 5: Complexity comparison among different SD-based SR methods. All methods are tested on the ×4 (128→512) SR tasks, and the inference time is measured on an V100 GPU.

Q4: In Fig. 10 and Fig. 12, TAD-SR’s results appear to contain many fragmented particles, which make the images look sharper at first glance; however, this is actually due to the addition of pseudo-textures or unnatural details. Could you explain the cause of this? For instance, could it be due to the adversarial loss?

A4: Upon careful examination of the images generated by our method and other super-resolution approaches, we observed that various methods may produce pseudo-textures in certain images to differing extents. We found that some of the unnatural textures generated by our method exhibit the same pattern as those produced by the teacher model, which we speculate may be due to the inherent properties of the diffusion model itself. Additionally, on real-world datasets, this phenomenon is likely attributed to the inconsistent degradation encountered by the training and testing models. While degradation during model training is artificially synthesized and may exhibit certain statistical features, real-world degradation is more complex and diverse, which could lead to the generation of pseudo-textures.

Q5: Following the concern raised in my 4th question, could you please provide more qualitative comparisons that contain fine details or small textures?

A5: Sure, we provide more qualitative comparisons that contain fine details in Figure 9 and Figure 15 of the revised PDF.

Table 3: Quantitative comparison with state of the arts on DIV2k-val dataset. The best and second best results are highlighted in bold and italic.

Q2: Since you claim that TAD-SR can achieve better reconstruction of high-frequency information, please present the spectrum images of the LR input, GT, baseline methods’ reconstruction, and TAD-SR’s reconstruction. Examine the differences in the high-frequency patterns around the periphery of the spectrum images.

A2: Thank you for your valuable suggestion. In Figure 10 of the appendix, we present the Fourier transform spectra of low-resolution (LR) images, ground truth (GT) images, and reconstructions from different super-resolution (SR) methods. From these spectra, it is evident that our method preserves more high-frequency information compared to other diffusion-based SR methods.

Thank you for providing valuable feedback on our paper despite your busy schedule. We address your concerns here.

Q1: Since this is a distillation method, please compare more diffusion-based distillation SR methods, like OSEDiff [1], quantitatively and qualitatively. (Why are the comparison with diffusion-based distillation SR methods missing in some tables and figures?)

A1: Thank you for pointing out this issue. We have compared our method with OSEDiff, with quantitative results presented in Tables 1, 2, and 3. In the main text, we primarily use the super-resolution model ResShift trained from scratch as the teacher, enabling a fair comparison with SinSR, which also distills ResShift. In the appendix, we employ the SD-based SR method SeeSR as the teacher and mainly compare our approach with other SD-based SR methods and SD-based distillation SR methods. Due to substantial differences in the datasets used to train ResShift and SeeSR, comparisons involving SD-based SR methods are omitted from certain charts in the main text.

Table 1: Quantitative comparison with state of the arts on RealSR dataset. Following the experimental setup of SeeSR, the LR images in the RealSR dataset were center-cropped to 128 $\times$ 128. The best and second best results are highlighted in bold and italic.

Table 2: Quantitative comparison with state of the arts on RealLR200 dataset dataset. The best and second best results are highlighted in bold and italic. Note that since the RealLR200 dataset lacks high-resolution images, we only computed non-reference metrics.

Thank you for your comments and feedback. We address your concerns here.

Q1: The evaluation is not comprehensive. Some image fidelity metrics are lacking, such as PSNR and SSIM on ImageNet-Test, where the competing methods ResShift and SinSR all reported.

A1: Thank you for your suggestion. We incorporate PSNR and SSIM metrics into the evaluation of the ImageNet dataset. However, we want to emphasize that these two metrics are secondary in real-world super-resolution tasks[1][2][3]. For instance, while methods such as LDM, ResShift, and DASR achieve higher PSNR and SSIM scores compared to others, the images they generate tend to appear smoother or blurrier (as shown in Figures 6 and 12). This discrepancy likely arises because PSNR and SSIM measure image differences in pixel space, whereas human and other metrics evaluate images based on perceptual quality. Therefore, PSNR and SSIM metrics should be considered as reference points only, which aligns with observations in previous studies[1][2][3].

Table 1: Quantitative results of different methods on the dataset of ImageNet-Test. The best and second best results are highlighted in bold and italic. ∗ indicates that the result was obtained by replicating the method in the paper.

Q2: The improvement over the previous single-step distillation method SinSR is minor. Considering that LPIPS—a crucial metric for perceptual quality—is very important, the increase from 0.221 to 0.227 represents a big drop in quality and is not slight.

A2: Thank you for your feedback. We replicate SinSR using the open-source code, and the experimental evaluation results are shown in the third-to-last row of Table 1. Compared to the replicated SinSR, our method improved the LPIPS metric, decreasing it from 0.231 to 0.227. Additionally, our method demonstrates significant improvements over SinSR in most metrics. The table below lists the percentage improvements achieved by our method compared to SinSR.

Table2: Quantitative comparison with SinSR method in super-resolution tasks.

Q2: It seems like the method requires teacher model to generate clean samples, which can be computationally expensive, even if you pre-compute the data off-line.

A2: The generation of samples by teacher models does incur additional computational costs; however, these costs remain within an acceptable range, particularly when generating samples offline. We compare our method with SinSR in terms of training time. As shown in Table 3, when generating clean samples online, our training time is only two hours longer than that of SinSR, and the distillation process can be completed within a day. Moreover, generating samples offline further reduces both training time and computational resource consumption. Additionally, we compare the GPU memory usage of our method during training between offline generation clean samples and online generation clean samples. The results show that the online generation of clean samples increases GPU memory usage by less than 5%, which is within an acceptable range. Furthermore, because SinSR requires learning a bidirectional mapping between noise and images, its GPU memory usage is higher than that of our method.

Table 3: A comparison of the training cost on 8 NVIDIA V100.

Q3: The background of SDS and how to reduce the bias is unclear to readers without prior knowledge.

A3: Thank you for your valuable suggestion. In the revised manuscript, we will include more background information on SDS and provide a clearer explanation of how we address deviations in SDS.

Thank you for your comments and feedback. We address your concerns here.

Q1: The biggest concern is insufficient baselines. The method compare against a large number of non-diffusion based methods or diffusion based iterative methods, but it lacks comparisons against the most closely related methods: other diffusion distillation algorithms. This method distill a pre-trained SR diffusion model into one step with some specific design for SR, but there are many distillation methods designed for general diffusion models, such as consistency model and the family of distribution matching distillation. The authors should run controlled experiment with the same teacher model with different algorithms to emphasize the relative advantage. For example, personally I found CM works well in distilling SR model into one step, and DMD and its variant can distilled the more complicated T2I model into one step. Their relative performance on SR diffusion is what we really care.

A1: Thank you for your valuable suggestion. We apply both consistency models and distribution matching distillation (DMD) to SR tasks for evaluation. Specifically, we employ consistency distillation under L2 loss and set the same boundary conditions as consistency models: $c_{skip}(t) = \frac{\sigma_{data}^2} {(\eta_t-\eta_0)^2 + \sigma_{data}^2}, c_{out}(t) = \frac{\sigma_{data}(\eta_t - \eta_0)}{\sqrt{\sigma_{data}^2+\eta_t^2}},$ which clearly satisfies $c_{skip}(0) = 1$ and $c_{out}(0) = 0$. For DMD, we alternately update the fake score network and generator, with the weights of the distribution matching distillation loss and regression loss set to 1.

The experimental results are presented in Table 1. As shown in the table, the high-resolution images generated by the model using consistency distillation are significantly inferior to those produced by other super-resolution methods across all metrics, which appears to contradict the reviewer's findings. We speculate that this discrepancy may be due to ResShift modifying the standard Markov chain of the diffusion model, making it difficult to apply consistency distillation directly. While applying DMD to super-resolution tasks has yielded promising results, it still falls short of our method. To further validate the effectiveness of our approach, we also transferred it to an unconditional generation task. The results of this evaluation on CIFAR-10 are presented in Table 2. As shown, our method achieves competitive performance, even in unconditional generation tasks, outperforming both consistency models and DMD.

Table 1: Quantitative results of different SR methods. The best and second best results are highlighted in bold and italic. ∗ indicates that the result was obtained by replicating the method in the paper.

Table 2: Generative performance on unconditional CIFAR-10. The best and second best results are highlighted in bold and italic.

We sincerely appreciate your valuable feedback and insightful suggestions, which have greatly helped us improve our manuscript. We have carefully addressed your concerns in our response and revised the manuscript accordingly. 

We understand that you have a busy schedule, but we would be grateful for any additional feedback or response you may have regarding our paper, as reviewer input is crucial for improving the quality and clarity of our work. Alternatively, if our revisions adequately address the issues raised, we kindly request a reconsideration of the score based on the clarifications and improvements made. 

Thank you once again for your time and consideration.

Best Wishes!

Authors of Submission 1713