Towards Realistic Data Generation for Real-World Super-Resolution

Thank you very much for recognizing our work: "novel generative model", "offers a significant advantage over traditional methods" and "superior performance". Next, we will address each of your concerns.

Minor Point: Thank you for your correction. We have updated this in the revised version at lines 86-87.

1: Differences to Syndiff

On the one hand, Syndiff is limited to generating degradation distributions similar to the training data, which restricts its ability to generalize to novel and diverse LR domains. This limitation makes it less suitable for handling various real-world LR scenarios. In contrast, our method adaptively extracts degradation representations from real LR images and generates corresponding paired data. This adaptability enhances the SR model's generalization across diverse scenarios, as demonstrated in Tables 1 and 3 of the main text.

On the other hand, Syndiff's generation process lacks explicit content and degradation conditions to control the output, often resulting in content loss and unrealistic degradation. Our method addresses these issues by extracting degradation representations from real LR images and content from HR images, using these as conditions to guide the diffusion process. This approach ensures better control over both content fidelity and degradation realism. As shown in the first row of Figure 4 in the main text, Syndiff-generated LR images suffer from color distortion, content loss, and unrealistic degradation. In contrast, our method produces images with significantly higher content fidelity and degradation characteristics that closely match real LR images, thereby demonstrating its superiority in both realism and accuracy.

2: Degradation modeling beyond RealESRGAN degradations

Our training process is conducted in two stages. In the initial pre-training stage, Real-ESRGAN is utilized to provide the degradation extractor with an initial capacity for degradation representation extraction. However, the degradation distribution learned at this stage lacks sufficient realism for diverse real-world scenarios. To address this, we fine-tune a subset of the degradation extractor's parameters using real LR images. This fine-tuning process adapts the learned degradation distribution to better align with real-world LR images, significantly enhancing the extractor's generalization ability. The effectiveness of this approach is supported by our experimental results. As shown in Table 3 of the main text, our method consistently generates more realistic LR images across various real-world settings, including RealSR, DRealSR, and Smartphone datasets. For instance, our generated data achieves 26.16 dB on RealSR, compared to 24.40 dB from Real-ESRGAN. This improved realism translates into superior SR model performance, as evidenced by the results in Tables 1 and 2. Additionally, we provide visualizations of synthesized LR images to further validate our method. As shown in Figure 3, LR images synthesized by Real-ESRGAN exhibit noticeable discrepancies compared to real LR images, whereas our method produces LR images with much higher similarity to real data. Figure 5 also validates that our method can extract different degradation representations from real-world LR images, enabling more accurate and realistic data generation. These findings demonstrate that our approach effectively models degradation distributions beyond the capabilities of Real-ESRGAN.

5: Comparison with bicubic LR images.

Following your suggestion, we incorporate bicubic degradation as a baseline and compare the SR performance. To ensure a fair comparison, we utilize the officially released pre-trained RRDB models trained on bicubic data and evaluate them on the RealSR and Smartphone datasets, as shown in Tables R3 and R4. The results clearly demonstrate that the RRDB model trained on our data achieves significantly better SR performance across various evaluation metrics, particularly in the perceptual-oriented LPIPS metric. This highlights the limitations of a single bicubic model, which struggles to handle the high complexity of real-world degradations [2,3]. Thank you for your valuable feedback. We have added this comparison to Appendix A.12 of the revised manuscript.

Table R3: Comparison with RRDB model trained on bicubic data on RealSR benchmark

Table R4: Comparison with RRDB model trained on bicubic data on Smartphone benchmark

[2] Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data

[3] Designing a Practical Degradation Model for Deep Blind Image Super-Resolution

6: The Adaptability of the Proposed Method

Our method adaptively generates accurate and realistic paired data tailored to the target LR scenario. To validate its effectiveness, we conduct experiments on three real-world benchmarks: RealSR, DRealSR, and Smartphone. For each benchmark, we extract degradation representations specific to the LR scenario and apply them to the data generation process. Compared to existing methods, our approach consistently achieves the highest-quality data synthesis, enabling SR models to deliver superior performance across all benchmarks, as shown in Tables 1, 2, and 3 in the main text. Additionally, to evaluate the generalization capability of our method on out-of-distribution (OOD) data, we test it on the Real Video SR dataset, as detailed in lines 423–431 and Table 5 of the main text. Our method also demonstrates superior performance on this dataset, further validating its adaptability and robustness across diverse real-world scenarios.

Thank you very much for your valuable suggestions. If you have any further suggestions, please do not hesitate to share them with us.

3: Performance comparison before and after fine-tuning for degradation and content encoders.

To enhance the generalization of E_deg and E_cont on unseen real-world distributions, we fine-tune a small subset of their parameters. Specifically, we only fine-tune the last layers of E_deg and E_cont, as depicted in Figure 2 of the main text. This design ensures that information cannot bypass the former layers directly to the final output, preserving the integrity of the learned representations. To further validate this point, we conduct additional experiments. As shown in Table R1, even without fine-tuning, our method generates highly realistic LR images and achieves superior performance compared to existing methods. After fine-tuning, the network's generalization ability in real-world scenarios is further enhanced, enabling more effective synthesis of LR images under diverse conditions. Moreover, as illustrated in Figure 5 of the main text, our method can generate LR images with varying degradation levels by extracting degradation representations from different LR images while preserving the content representation from an HR image. We greatly appreciate your valuable feedback and have added a detailed illustration in Appendix A.11 of the revised version.

Table R1: Performance Comparison of LR Synthesis on RealSR before and after Fine-tuning

4: Effectiveness of Pre-training E_deg and E_cont

Following your suggestion, we remove the pre-trained weights of E_deg and E_cont and train them jointly with DDPM from scratch. However, without pre-training, E_deg and E_cont fail to effectively distinguish between content and degradation. This limitation prevents the degradation extractor from accurately extracting degradation information from real LR images, leading to a significant decline in the accuracy of synthesized LR images. Similarly, the content extractor struggles to capture high-fidelity content representations, resulting in generated images with reduced fidelity, as demonstrated in Table R2. These results highlight the critical role of pre-training in establishing the initial capacity of E_deg and E_cont, enabling them to effectively separate and extract content and degradation information. We have added this comparison in Appendix A.11 of the revised manuscript.

Table R2: Performance comparison of LR synthesis on RealSR between with and without pre-training of E_deg and E_cont

Dear Reviewer X8hR,

Thank you very much for recognizing our work: "novel generative model", "offers a significant advantage over traditional methods" and "superior performance".

We would like to thank you again for the valuable time you devoted to reviewing our paper. Since the end of the discussion period is getting close and we have not heard back from you yet, we would appreciate it if you kindly let us know of any other concerns you may have and if we can be of any further assistance in clarifying them.

Thank you once again for your contribution to the development of our paper.

Authors

Dear Reviewer X8hR,

Thank you for your valuable contribution to the development of our paper. We are pleased to hear that most of your concerns have been addressed. As the end of the discussion period is approaching, we look forward to your response. We would appreciate it if you could let us know of any remaining concerns or if there is any further assistance we can provide to clarify them.

Best regards,

Authors

Thank you very much for recognizing our work: "facilitating the generation of realistic low-resolution images that more accurately replicate real-world degradation patterns", "enhancing data realism and adaptability across different models" and "highlighting its practical effectiveness for real-world applications".

Next, we will address each of your concerns.

1. Differences to method [A]

First, the data synthesis approach in [A] generates degradation distributions limited to the training data, often failing to generalize effectively to new and diverse LR domains. In contrast, our method extracts degradation representations directly from real LR images of new domains, enabling the production of accurate LR images and demonstrating significantly stronger generalization capabilities.

Second, the decoupling process in [A] is based on two key assumptions: a. The assumption that LR and HR images share the same content representation. However, in scenarios with high downsampling rates and severe degradations, the content of LR images can undergo substantial changes, violating this assumption. b. The assumption of a linear hypothesis in the representation space, expressed as $Z_d = Z_{lr} - Z_c$, where degradation representation $Z_d$ is obtained by subtracting the content representation $Z_c$ from the LR feature $Z_{lr}$. This hypothesis assumes a linear relationship in the latent space, while deep representations do not inherently satisfy such linearity.

In contrast, our method avoids these strict assumptions by employing carefully designed contrastive and reconstruction strategies to learn unique and complete degradation representations as well as pure content representations. This approach ensures greater practicality and robustness in real-world scenarios. As the code for [A] is unavailable and the rebuttal time is limited, we plan to reproduce this method and include comparative experiments in the final version of the paper.

2. Why is it necessary to fine-tune the partial parameters of the extractors?

After pre-training with well-designed strategies using synthetic LR images, our extractors effectively capture robust content and degradation representations. To further improve the generalization of the extractors on unseen real distributions, we fine-tune a subset of their parameters, as illustrated in Figure 2 and discussed in lines 213–215 of the main text. To validate the effectiveness of this fine-tuning, we conduct comparative experiments on the RealSR dataset, with and without fine-tuning, as presented in Table R1. The results clearly show that fine-tuning enhances the model's ability to capture real degradation and content features, resulting in more realistic synthesized LR images. We appreciate your valuable suggestion and have included this analysis in Appendix A.11 of the revised manuscript.

Table R1: Performance comparison of LR synthesis on RealSR between with and without fine-tuning

3. Performance evaluation on blurred LR scenarios.

Following your suggestion, we select blurred LR images from the popular REDS benchmark [1] to test and validate the improved generalization of our proposed method in more real-world scenarios. Specifically, we evaluate RRDB models pre-trained using synthesized data from different methods. The results, as shown in Table R2, demonstrate that our method consistently achieves the best performance on out-of-distribution data with blur degradation. Thank you for your valuable suggestion, and we have included this analysis in Appendix A.13 of the revised manuscript.

Table R2: Performance comparison on REDS.

[1] NTIRE 2019 Challenge on Video Deblurring and Super-Resolution: Dataset and Study

4. Performance comparison with different quantities of Real LR images

In the training process, considering that real LR images are relatively easy to collect, we utilize real LR images from public datasets and self-collected data to ensure that the generation model fully learns the degradation distribution of real-world scenarios. However, during testing, only a small number of real LR images captured in the test scenarios are required to generate realistic paired data. For instance, in RealSR, we use only 100 real LR images for data synthesis, which is relatively easy to obtain, and achieves high-quality results, as shown in Table 3 of the main text. To further evaluate the performance of the proposed method with fewer real LR images, we conduct comparative experiments by reducing the number of real LR images to 50 and testing the performance of the RRDB model, as presented in Table R3. The results show that, compared to existing methods, our approach significantly enhances the generalization ability of the RRDB model in real-world scenarios, achieving the best performance.

Table R3: Performance comparison with existing data synthesis methods

5. Comparison with contrastive learning method [D]

In DASR [D], the contrastive learning approach for training the degradation model constructs positive samples from different patches of the same image and negative samples from different images. However, the degradation between different patches of the same image is not always identical. For instance, in a defocused LR image, the defocused and non-defocused areas exhibit completely different degradations, and the degradation between different images may sometimes be similar. In contrast, our strategy constructs positive samples by keeping the degradation consistent while varying the content and generates negative samples by keeping the content consistent while varying the degradation, as described in Section 2.1 of the main text. This design ensures that the degradation extractor captures the complete and unique degradation distribution of the current LR image. To validate this, we replace the degradation model in [D] and train the diffusion model, then generate real LR images on the RealSR dataset. The results, shown in Table R4, demonstrate that our proposed degradation model produces more realistic LR images compared to [D]. Additional ablation studies on our contrastive learning strategy are presented in Appendix A.7. Thank you for your suggestion. We have included this comparison in Appendix A.14 of the revised version.

Table R4: Performance comparison with difference contrastive learning

6. Training Time

Our model contains only 73M parameters and requires approximately 768 V100 GPU hours (4 days on 8 V100 GPUs) for training. In comparison, Syndiff has 101M parameters and requires around 960 GPU V100 hours (5 days on 8 V100 GPUs). Additionally, unlike Syndiff, our method needs to be trained only once and can be adaptively applied to different real LR scenarios without requiring retraining, offering both efficiency and flexibility.

7. Related Papers

Thank you for your advice. We have included these related papers [A-D] in the revised version.

[A] Learning Many-to-Many Mapping for Unpaired Real-World Image Super-resolution and Downscaling, TPAMI 2024.

[B] Benchmarking Neural Network Robustness to Common Corruptions and Perturbations, ICLR 2019.

[C] Efficient Test-Time Adaptation for Super-Resolution with Second-Order Degradation and Reconstruction, NeurIPS 2023.

[D] Unsupervised Degradation Representation Learning for Blind Super-Resolution, CVPR 2021.

Thank you very much for your valuable suggestions. If you have any further suggestions, please do not hesitate to share them with us.

Thank you very much for recognizing our work: "provide a strong motivation", "a novel approach" and "the robustness of the proposed approach across a variety of settings".

Next, we will address each of your concerns.

1. Clarifying Real-World LR in Figure 2 (b) and Providing More Visual Comparisons on Real LR Images from the Internet

Real-world LR images originate from various sources, including photography (e.g., images captured by smartphones and DSLRs), compressed low-quality images from the Internet, and other scenarios. To evaluate the effectiveness of our method comprehensively, we test on widely used benchmarks such as RealSR and DRealSR, as well as our collected images captured by smartphones. As demonstrated in Tables 1, 2, and 3 of the main paper, our method consistently achieves superior results. Additionally, to assess the generalization capability of our method on Internet-derived LR images, we provide visualizations in Figure 10 (Appendix A.15). These examples show that our method reconstructs much clearer textures compared to other approaches. We greatly appreciate this valuable feedback and have incorporated a detailed description in Appendix A.15 of the revised manuscript.

2. Minor Presentation Details

a. Mismatch between Figure 2 and Eq. 5

Thank you for pointing this out. We have updated Figure 2 accordingly.

b. Inconsistent Colors in Figure 1

Thank you for your suggestion. We have aligned the colors of different existing methods and their corresponding cycles in Figure 1 to enhance the readability of the paper.

c. Minor Issue in Figure 1 (a)

We have aligned the representations of "G," "S," "N," and "K" in Figure 1. The updated Figure 1 clearly indicates these elements.

d. Explanation of X_{hr}

X_{hr} in Eq.5 refers to HR images. We have added the relevant description in the revised version at 194-196 lines.

3. Performance Comparison with Real Collection Data

Table R1: Performance comparison with existing data synthesis methods and real data collection

Following your suggestion, we further train the RRDB model using the real-collected training subset of RealSR and evaluate it on the test subset, as shown in Table R1. Since the training and test data in RealSR are captured using the same camera, they share a similar degradation distribution. Consequently, the model trained on real data achieves better performance compared to those trained on simulated data. However, collecting real-world data is often impractical, highlighting the importance of an accurate simulation method. Compared to other data synthesis approaches, our method is capable of generating realistic large-scale paired training data across diverse scenarios, enhancing the generalization capability of SR models. Furthermore, by incorporating our generated data into the real-collected training set, as shown in the last row of Table R1, the SR model's performance improves further, demonstrating the effectiveness of our approach. We appreciate your valuable suggestion and have added this description to Appendix A.10 of the revised manuscript.

If you have any suggestions, please do not hesitate to share them with us.

Thank you very much for recognizing our work: "improves the generalization and performance of SR models across various real-world benchmarks", "achieves competitive SR results" and "practical and advantageous in real-world applications". It is widely recognized that the quality of training data is crucial to model performance. To address the limitations of existing methods, we develop an innovative approach to produce large-scale, high-quality data that accurately simulate real-world degradations. This significantly improves the generalization capability of SR methods in real-world applications.

Next, we will address each of your concerns.

1. Why does the content extractor extract "pure" content information?

The ability of the content extractor to capture 'pure' content information is primarily driven by the network's supervision objectives. The input to the content extractor is an LR image with various degradations, while the target is an HR image free of these degradations. From the perspective of model optimization, degradation information does not contribute to minimizing the objective function, which focuses on reconstructing high-resolution content. For example, the MSE loss emphasizes pixel-level accuracy, steering the model away from learning degradation-related features. Consequently, the network learns to disregard degradation characteristics and instead prioritize extracting 'pure' content information. As illustrated in Figure 7 of the appendix, our content extractor effectively separates degradation from content and extracts 'pure' content information, recovering high-quality results.

2. Explanation of X_{hr}

In Equation (5), X_{hr} refers to high-resolution images. We have clarified this in the revised manuscript on lines 194–196, highlighted in blue. Thank you for this valuable suggestion, which has helped improve the clarity of our work.

3. Parameters comparison

Considering that BSRGAN and Real-ESRGAN are not deep-learning-based methods, following your suggestion, we compare the capacity and performance of deep learning-based Syndiff in Table R1. We see that our model not only has fewer parameters but also consistently achieves better performance on RealSR, DRealSR, and SmartPhone benchmarks. Thank you very much for your valuable suggestions. We have included this comparison in section A.9 of the appendix in the revised version.

Table R1: Parameters and Performance Comparison

If you have any further suggestions, please do not hesitate to share them with us.

Thank you for your response. We are pleased to hear that most of your concerns have been addressed. We would like to further clarify the remaining questions:

1. Function of the content extractor

We need to clarify the function of the content extractor within our proposed framework. As illustrated in Figure 1(c), during the data generation process (inference period), we utilize the content extractor exclusively to process high-resolution (HR) images (sourced from DIV2K) to enhance the fidelity of the generated LR images. The input images are guaranteed to be high-quality and free from out-of-domain (OOD) degradations. On the other hand, in the second training stage depicted in Figure 2, if we can collect diverse real-world noisy images (as you suggested in [1]) for training, our content extractor can be fine-tuned to become more robust in extracting relatively pure content information.

We acknowledge the challenge of extracting completely pure content information from various degraded images without training on such data. This is an area we plan to explore further, as it holds significant potential for improving our system's ability to process a more diverse range of images. We will cite [1] and add these discussions in the revised manuscript.

[1] Masked Image Training for Generalizable Deep Image Denoising, CVPR 2023

2. The decoupling of content and degradation extractors

During the pre-training stage, the content and degradation extractors are trained on synthesized paired data using carefully designed strategies to acquire the ability to extract content and degradation representations, respectively, as described in Section 2.1 of the main text. In the second fine-tuning phase, where only LR images are available, we freeze most layers of the content and degradation extractors, leaving only the last layer of each extractor trainable, as shown in Figure 2 and lines 212-214 of the main text. This ensures that the main functionalities of the extractors remain largely unchanged, preserving the decoupling property.

Table R1: Performance comparison of swapping degradation and content extractors.

To verify this, we swap the two extractors during data generation to test if they become homogenized. We compare the generated LR images with real LR images from the RealSR dataset, with the accuracy reported in Table R1 below. The results show that swapping the extractors leads to a complete collapse in model performance, indicating their distinct roles in data generation. In contrast, our proposed method achieves excellent data generation accuracy, validating the effectiveness of the decoupling approach. We will include these discussions in the revised manuscript to enhance the readability of the paper.

Thank you again for your valuable feedback.

Dear Reviewer qCkN,

Thank you for your valuable contribution to the development of our paper. We are pleased to hear that most of your concerns have been addressed. As the end of the discussion period is approaching, we look forward to your response. We would appreciate it if you could let us know of any remaining concerns or if there is any further assistance we can provide to clarify them.

Best regards,

Authors