Response to Reviewer hd5s (denoted as R4)

Q4-1 A similar GAN-based idea can also be found in Seaweed-APT [1], the author should consider making a discussion with this paper.

A4-1 Thank you for your suggestion. The key difference between D3SR and Seaweed-APT is that we explore the benefits brought by a larger-scale discriminator in training. Additionally, the discriminator in D3SR receives the results of latent forward diffusion as input. We will include the appropriate references in the main body of the paper.

Q4-2 The proposed EA-DISTS requires further decoding into the pixel space during training, which may introduce training bottleneck on the training efficiency and memory requirement.

A4-2 We highlight that EA-DISTS does not introduce too much training cost or memory consumption. It is worth noting that, besides EA-DISTS, the pixel-level reconstruction loss (e.g., L1/L2 loss) also relies on decoding latents into pixel space, which is exploited in almost all SR methods. In this sense, decoding latents into pixel space is a necessary step and the proposed EA-DISTS will not introduce too much extra training cost. In practice, the increased training cost and memory consumption is negilible and hard to be observed.

Q4-3 While the paper claims that using SDXL as a discriminator for adversarial training overcomes previous teacher-based distillation methods, it is noticeable that previous methods [31, 32, 33] all apply SD2.1 as a teacher model. It is unclear whether the good performance comes from a more powerful model, i.e., SDXL or from the adversarial training without the teacher model.

A4-3 We present the performance of using SD2.1 as the discriminator in our ablation study. The detailed metrics and comparisons with other methods are shown in the table below. Under the same pretrained model used as the discriminator, our method still outperforms other one-step diffusion SR methods. Furthermore, employing a larger discriminator (e.g., SDXL) leads to further performance improvements.

This demonstrates that adversarial training with a diffusion-based discriminator can overcome the limitations imposed by teacher models, and using a stronger model as the discriminator further enhances performance.

Q4-4 If possible, it is better to add a user study to enhance the comparison with human preference.

A4-4 Thank you for your suggestion. We have added a user study, and the results are shown in the table below. We selected five classic scene types: landscape, human, plant, animal, and architecture, with a total of 20 participants involved in the evaluation. Our method achieved a recognition rate of 60.63%, surpassing other methods.

I appreciate the response from the author. I have some minor questions that require further clarification:

1. The author claims that the main differences with Seaweed-APT lie in (1) we explore the benefits brought by a larger-scale discriminator in training: APT also uses a large discriminator. So the difference here is that the paper proposes to use a larger discriminator than generator? (2) Additionally, the discriminator in D3SR receives the results of latent forward diffusion as input.: Any particular reason and advantage of this operation compared with directly comparing between the fake and the real data for the discriminator?

2. It is a little unclear why pixel-level reconstruction loss (e.g., L1/L2 loss) also relies on decoding latents into pixel space, which is exploited in almost all SR methods in A4-2. Many SR methods like StableSR, DiffBIR, SUPIR, etc do not rely on decoding the latent features for loss calculation during the training of the diffusion model. I suggest the author to tone down this claim and add it in the revision as the cost of this decoding process is affordable under common training settings, e.g., 512x512 patches.

3. Any reason that the author omits the comparsion on CLIP-IQA, which is widely used by the main baselines of this paper, including SinSR, OSEDiff, ADDSR, etc?

Response to Reviewer Cmxu (denoted as R1)

Q1-1 The main difference between D3SR and DMD2 lies in the removal of the KL divergence penalty derived from real and fake score functions. As a result, the technical contribution appears incremental rather than fundamentally novel.

A1-1 There may be some misunderstanding about our contributions and we will clarify them in the following two points.

1. We highlight that our main contribution lies in exploring the effect of scaling up the discriminator model for one-step diffusion models, instead of removing the KL penalty. We demonstrate that using the same model as the discriminator and generator has its limitations, and employing a larger-scale discriminator can further enhance model performance.
2. Additionally, we found that the performance improvement brought by DMD's KL penalty is not as significant as that achieved through adversarial training and the training is computationally expensive. Thus, we directly remove it for efficient training. Without DMD's KL penalty, our improved performance merely comes from the proposed adversarial training strategy, which further shows the effectiveness of our method.

Q1-2 By initializing the discriminator with the SDXL encoder, D3SR implicitly incorporates high-resolution priors from SDXL. This makes it difficult to disentangle whether the performance gains are due to the proposed method itself or simply the use of a stronger pre-trained model. Moreover, prior distillation approaches could similarly benefit from incorporating more powerful diffusion models as guidance.

A1-2 We present the performance of using SD2.1 as the discriminator in our ablation study. The detailed metrics and comparisons with other methods are shown in the table below. Under the same pretrained discriminator backbone, our method still outperforms other one-step diffusion SR methods. This demonstrates the effectiveness of our proposed approach.

Q1-3 Although the authors claim improved visual quality, several examples in the supplementary material reveal potential issues such as hallucinated textures and color shifts . For instance: In Figure 2 (Case 2) and Figure 3 (Case 3, Case 6), unnatural or fabricated textures appear in the generated images. Color distortions are noticeable in Figure 2 (Case 5) and Figure 3 (Case 6). These artifacts may limit the practical applicability of the method, especially in scenarios where faithfulness to the input is critical.

A1-3 In real-world image super-resolution tasks, some scenes exhibit severe degradation, making it unrealistic to fully recover the LR observation back to the ground truth. Compared with other methods, our approach provides better clarity and realism.

In addition, we conducted a user study as shown in the table below. We selected five representative scene types: landscape, human, plant, animal, and architecture. A total of 20 participants were involved in the evaluation. Our method achieved a recognition rate of 60.63%, surpassing all other methods.

Thank you for your valuable comments! We would like to follow up on whether our response and the additional experiments have addressed your concerns. We look forward to your further feedback on these aspects, and we are happy to make any necessary adjustments if needed.

Thank you again for your time and thoughtful review!

Response to Reviewer Cj9F (denoted as R3)

Q3-1 The use of a diffusion model as a discriminator has been explored in prior work, such as in UFOGen (CVPR24). This paper's validation that a larger diffusion discriminator yields better performance is somewhat incremental and limited in novelty and significance.

A3-1

1. Our main contribution lies in exploring the impact of scaling up the discriminator model for one-step diffusion models. We demonstrate that using the same model as both the discriminator and the generator has its limitations, and employing a larger-scale discriminator can further enhance model performance. We highlight that, to our knowledge, it is the first work to investigate the impact of discriminator size on the training of one-step diffusion models. We believe that our proposed method is beneficial to many other research works and could be a popular training strategy.
2. Significance of performance improvement: We highlight that our improvement is competitive with recent works [29, 30, 31] and can be consistently observed across various benchmarks. In Table 1, our method yields the best or second-best in most of the perceptual metrics, including DISTS, NIQE, MUSIQ, etc. More critically, our model is much more efficient than the compared methods in terms of both inference speed and model size (see Table 3). We are convinced that these results are significant.

Q3-2 The paper lacks comparison experiments with the latest works, such as ADCSR and PiSA-SR, which were presented at CVPR 2025. This absence makes it difficult to fully assess the competitiveness of the proposed method in the context of the most recent advancements in the field.

A3-2 We have added comparisons with the latest methods, and the results are shown in the table below. Our method achieves the best performance on no-reference (NR) metrics. We highlight that these metrics are often highly related to the perceptual visual quality. Moreover, we also compare the visual results with these methods and observe that our method is able to produce significantly more details, together with obviously better visual quality. Nevertheless, due to the limitation of "not allowing to show images or links" in rebuttal, we will include these visual comparisons in the final version and cite these works.

RealSR

DIV2K-val

RealSet65

Q3-3 For the ablation studies on discriminator types, could the authors consider including some powerful visual feature extraction networks for comparison, such as DINO, in addition to CNN-based discriminators? This would provide a more comprehensive understanding of the impact of different types of discriminators on the performance of the proposed method.

Thank you for your suggestion. We have conducted additional experiments accordingly. Specifically, we use different versions of DINO-v2 models as the discriminator in D3SR, and the results are shown in the table below.

Consistent with the conclusions of our paper, as the discriminator model scales up, the performance of the generator gradually improves. The results using DINO as the discriminator are slightly inferior to those obtained using a pretrained diffusion model as the discriminator.

Thank you for your valuable comments! We would like to follow up on whether our response, along with our innovations, ablation studies, and comparison methods, has addressed your concerns. We look forward to your further feedback on these aspects and are happy to make any necessary adjustments if needed.

Thank you again for your time and thoughtful review!

Response to Reviewer CutL (denoted as R2)

Q2-1 The authors claim in the paper that the proposed Large Diffusion Discriminator is superior to the existing VSD method, but there is a lack of corresponding experimental verification in the ablation study. Does the advantage of Large Diffusion Discriminator mainly come from the use of a better model (i.e., SDXL)?

A2-1 We present the performance of using SD2.1 as the discriminator in our ablation study. The detailed metrics and comparisons with other methods are shown in the table below. OSEDiff is a representative method based on VSD. Under the same pretrained model used as the discriminator, our method outperforms other one-step diffusion SR approaches. Furthermore, employing a larger discriminator (e.g., SDXL) leads to further performance improvements.

These results demonstrate that the advantage of the Large Diffusion Discriminator stems not only from the stronger pretrained model but also from the adversarial training with a diffusion-based discriminator.

Q2-2 As shown in Tab.1, D3SR achieves lower PSNR/SSIM/LPIPS scores compared to other diffusion-based SISR models.

A2-2 In real-world image super-resolution (Real-ISR) tasks, some scenes exhibit severe degradation, making it impractical to fully reconstruct the ground truth from the LR observation. Therefore, for full-reference (FR) metrics, their reliability is often lower than no-reference (NR) metrics in Real-ISR scenarios.

NR metrics better reflect the clarity, realism, and human preference of the generated images. This has been demonstrated in prior works such as SUPIR [1] and TSD-SR [2].

[1] Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild. CVPR 2024
[2] TSD-SR: One-Step Diffusion with Target Score Distillation for Real-World Image Super-Resolution. CVPR 2025

Q2-3 The proposed methods and OSEDiff are both based on the Stable Diffusion model. Why is there such a big difference in the total parameter in Tab.2 (1.4\times10_3M vs. 966.3M)? What is the main reason for this? In addition, the total parameter of OSEDiff is also significantly different from the value reported in its official paper. An explanation for these inconsistencies would be helpful.

A2-3 Unlike OSEDiff, we remove modules such as the text encoder and DAPE, which are not essential for one-step diffusion SR. This simplification is also discussed in our main paper. As a result, our method has significantly fewer parameters compared to OSEDiff.

We use the thop library to compute the parameter counts, and the details of each component are listed below. The parameter sizes of UNet and VAE match those reported in the official Stable Diffusion release. The reason for the discrepancy with the numbers reported in the OSEDiff paper remains unclear.