SAN-Diff: Structure-aware noise for super-resolution diffusion model

Weaknesses:

1. Almost all of the quotes in the article are in the wrong format. Please refer to the difference between \citet and \citep.

   Thank you for your suggestion. We have already corrected this error in the revised version.

2. For the super-resolution task, it is suggested to provide general experiments with multiple scales, such as ×8.

   Our experimental setup is based on ESRGAN, SRDiff, and DiffBIR, which only tested the X4 setting in their experiments. Therefore, we report the comparison of our method with other models in the X4 setting in Table 2 of the paper. Additionally, in Appendix Table 7, we present the results of SRDiff trained by us and the results of our method in the X2 setting.

   The following are the results for the X8 setting.

   model	Urban100				BSDS100				Manga109				General100				DIV2K
   	PSNR-Y	SSIM	MANIQA↑	FID ↓	PSNR-Y	SSIM	MANIQA↑	FID ↓	PSNR-Y	SSIM	MANIQA↑	FID ↓	PSNR-Y	SSIM	MANIQA↑	FID ↓	PSNR-Y	SSIM	MANIQA↑	FID ↓
   SRDiff (X8)	20.95	0.5787	0.4375	46.71	22.74	0.5295	0.3294	155.69	23.10	0.7471	0.3919	24.79	24.93	0.6682	0.3756	105.39	24.54	0.6628	0.3533	8.3945
   SAN-Diff (X8)	21.36	0.5913	0.4188	43.75	23.29	0.5500	0.3253	154.13	23.27	0.7520	0.3959	25.12	25.34	0.6836	0.3648	102.17	25.18	0.6790	0.3430	8.5189

3. Only SRDiff is compared in the Table 1. NOT various diffusion-based image super-resolution methods.

   In Table 2, we report the results of two types of super-resolution models: GAN-based models (SRGAN, SFTGAN, ESRGAN, USRGAN, SPSR, BSRGAN) and diffusion-based models (LDM, StableSR, DiffBIR, SRDiff). We also conduct a fair comparison with our method across multiple metrics. Please refer to Table 2 for the detailed numerical results.

Questions:

1. Why not compare the proposed method with common baselines like EDSR and other widely used models?

   Thank you for providing this valuable comment. In general, image super-resolution approaches can be categorized into distance-based(like EDSR) and generative-based(like diffusion-based and GAN-base) methods. Since our method focuses on improving diffusion-based SR models, we primarily compare it with generative SR models (GAN-based and diffusion-based) rather than distance-based SR models like EDSR.

   As the official pre-trained weights for EDSR are no longer available, we used the pre-trained weights provided by BasicSR for testing. Below are the comparison results between our model and EDSR.

   model	Urban100				BSDS100				Manga109				General100				DIV2K
   	PSNR-Y	SSIM	MANIQA↑	FID ↓	PSNR-Y	SSIM	MANIQA↑	FID ↓	PSNR-Y	SSIM	MANIQA↑	FID ↓	PSNR-Y	SSIM	MANIQA↑	FID ↓	PSNR-Y	SSIM	MANIQA↑	FID ↓
   EDSR_L	24.43	0.7500	0.4576	5.86	26.34	0.7142	0.3231	91.85	23.99	0.8243	0.4256	5.84	27.22	0.7971	0.4204	54.61	30.75	0.8442	0.3555	0.3303
   EDSR_M	24.16	0.7374	0.4297	7.23	26.29	0.7099	0.3037	96.27	24.05	0.8214	0.4044	6.24	27.14	0.7944	0.3986	55.39	30.44	0.8366	0.3428	0.3456
   SAN-Diff	25.54	0.7721	0.6709	4.53	26.47	0.7003	0.6667	60.81	29.43	0.8899	0.6046	2.40	30.29	0.8353	0.6346	39.15	29.34	0.8108	0.5959	0.3809

2. MANIQA, which is a metric for no-reference image quality assessment, seems less common for this super-resolution task. Why not use the more popular LPIPS metric?

   Thank you for providing this valuable comment. We chose to use MANIQA instead of LPIPS to evaluate our method because we believe that MANIQA provides a more accurate assessment of super-resolution image quality. MANIQA evaluates image quality from multiple dimensions [1], including color consistency, structural details, and blurriness. In contrast, LPIPS primarily focuses on perceptual similarity between images. While LPIPS effectively measures differences in image features, it is less capable of analyzing overall image quality. Therefore, to accurately assess the differences in texture and structure between the reconstructed images and HR images, we opted to use MANIQA as a reference metric.

To provide a more comprehensive and objective evaluation of our method, we combined subjective metrics (LPIPS), objective metrics (PSNR), and the number of artifacts to create image, and you can find it in LPIPS.png or in Figure 9 at appendix, which illustrates a holistic assessment of the model's performance. These results highlight the outstanding performance of our model under combined evaluation criteria.

[1] Maniqa: Multi-dimension attention network for no-reference image quality assessment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.

For Weakness:

• What is the meaning of SAM-DiffSR in this comment? Do you mean SAN-Diff?
• For Table 1, could you give more comparisons of the effectiveness and efficiency about other recent diffusion-based methods? Note that SRDiff was published in 2022, which is not so new.

Dear TbBy,

Thank you for your constructive comments and valuable suggestions to improve this paper. ​If you have any more questions, we would be glad to discuss them with you.

Thank you very much.

Best regards, Author

Dear TbBy,

We apologize for the repeated reminder. It has been 6 days since we submitted our responses, but we have not yet received your feedback. We simply want to ensure that we have fully addressed all your concerns.

Your main concerns appear to relate to the values of the metrics reported and the comparisons of the effectiveness and efficiency about other recent diffusion-based methods. We have provided detailed responses to these questions in both our reply to you and the general response. May we kindly ask if you could spare some time to review our responses and share your feedback?

If there are any remaining points that require further clarification, please rest assured that we are committed to providing detailed answers to all your inquiries and addressing any concerns you may have. We value clear and open communication, and will make every effort to ensure that all aspects of the matter are fully explained to your satisfaction.

Thank you very much.

Best regards, Author

Thank you for your reply and constructive questions. There are a few foundational concepts that need clarification.

Current super-resolution (SR) models can be classified into two categories:

• Distance-based SR models: Examples include EDSR[3], SwinIR[4], HAT[5], and DRCT[1]. These typically use distance loss (e.g. $L_1$ pixel loss) to train the model.
• Generate-based SR models:
  • GAN-based models: Examples like ESRGAN[6], which usually combine distance loss with additional perception loss, and are trained using a generative adversarial approach.
  • Diffusion-based models: Examples include SRDiff[7], LDM[8], and StableSR[2], which generally use MSE loss and are trained through the denoising process.

The difference in training methods results in distinct advantages for distance-based and generate-based models in terms of image reconstruction. Distance-based models, due to the constraint of distance loss, tend to reconstruct images with more accurate structures but less defined textures. On the other hand, generate-based models, due to their different training processes and losses, tend to produce images with clearer textures but may exhibit distorted structures. Therefore, in the field of single image super-resolution (SISR), these two categories are typically compared separately.

For example, taking DRCT [1], the highest-scoring model on the Urban100 benchmark, it primarily compares models like EDSR[3], SwinIR[4], and HAT[5] in their paper, which are all distance-based SR models. In contrast, StableSR[2] compares models like BSRGAN[9] and LDM[8] in paper, which are generate-based SR models.

We would also like to reiterate that comparing our method with distance-based SR models on their stronger metrics (PSNR and SSIM) is not a fair comparison. The conventional and fair approach is to compare our method with generate-based SR models across all relevant metrics. Our goal is to improve the PSNR and SSIM scores of diffusion-based SR models, thereby enhancing the accuracy of structure and texture reconstruction. Our experimental results validate the effectiveness of our approach.

---

Regarding the discrepancy in metrics compared to the original papers, we would like to reiterate that all the data reported in our paper (whether for our method or others) can be consistently reproduced using open-source tools.

The field of super-resolution has a long development history, and over time, there have been numerous changes in the evaluation settings and scripts used to assess model performance. This makes it difficult to directly compare models using the data reported in original papers in an intuitive and fair manner. Therefore, we have chosen to use a unified open-source evaluation tool to fairly assess all methods. This allows readers to make an intuitive and accurate comparison between our method and the others without needing to switch between different testing settings.

---

[1] DRCT: Saving Image Super-resolution away from Information Bottleneck[J]. arXiv preprint arXiv:2404.00722, 2024.

[2] Exploiting diffusion prior for real-world image super-resolution[J]. International Journal of Computer Vision, 2024: 1-21.

[3] Enhanced deep residual networks for single image super-resolution[C]//Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2017: 136-144.

[4] Swinir: Image restoration using swin transformer[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2021: 1833-1844.

[5] Activating more pixels in image super-resolution transformer[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023: 22367-22377.

[6] Enhanced super-resolution generative adversarial networks[C]//Proceedings of the European conference on computer vision (ECCV) workshops. 2018: 0-0.

[7] Srdiff: Single image super-resolution with diffusion probabilistic models[J]. Neurocomputing, 2022, 479: 47-59.

[8] High-resolution image synthesis with latent diffusion models[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 10684-10695.

[9] Designing a practical degradation model for deep blind image super-resolution[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 4791-4800.

Weaknesses:

the limitations of relying on SAM as the sole provider of segmentation masks

Thank you for your valuable comments. Compared to the original diffusion model without structural guidance, masks generated by existing SAM models can improve performance, as demonstrated in our experimental results.

However, the performance of our model does depend on the quality of the segmentation masks, as they capture the structural information of the corresponding image. Our model benefits from SAM's fine-grained segmentation capability and its strong generalization ability across diverse objects and textures in the real world. Nevertheless, the performance of our model is also limited by the capabilities of the segmentation model itself. For instance, SAM may struggle to identify structures with low resolution in certain scenes. While the model can partially mitigate this issue by learning from a large amount of data during training, it is undeniable that higher segmentation precision (e.g., SAM2) and finer segmentation granularity would significantly enhance the performance of our approach. This is one of the potential limitations of our proposed framework, and we have included this discussion in the revised version of our paper in Appendix C.3.

Our experiments demonstrate that even though existing segmentation models may not perfectly distinguish every region in the real world, combining our method with a super-resolution model has substantially improved performance on super-resolution tasks. We believe that utilizing prior knowledge from segmentation models to enhance generative models is a promising avenue for further exploration.

Questions:

1. For the example shown in Fig 5,6, is it possible for the authors provide the segmentation masks using SAM?

   Figure 7 shows the segmentation results from SAM. However, it is important to emphasize that our method only uses the mask information during the training process, and no mask guidance is used during inference. It can serve as a reference for analyzing the super-resolution results.

2. In Figure 1, it is hard for me to think of why this improvement happens because of the contribution from SAM. I highly doubt that the image region you contoured can be identified as different “regions” segmented by SAM. How would the authors justify the contribution of SAM?

   We aim to augment diffusion-based image super-resolution (SR) models with fine-grained masks provided by SAM. In our paper, we compare two designs. The first involves using the information provided by SAM as a conditioning factor for the U-Net, following a commonly used approach in guided diffusion. The second design, structural noise modulation, is our heuristic approach, with the derivation provided in the supplementary materials. From the results in Figure 1(B), both qualitative and quantitative results demonstrate the effectiveness of incorporating segmentation information to enhance the model’s capabilities, as well as the fact that structural noise modulation yields better performance without adding extra inference cost.

   Furthermore, as you mentioned, the model learns information "statistically." Although there may be some regions where the segmentation model's performance is limited and it might fail to distinguish or incorrectly segment certain areas, the model learns from a large amount of data during training. The denoising model (U-Net) can "statistically" learn the necessary structure-level ability to distinguish different regions. Therefore, these issues do not affect the model to get ability to distinguish regions during training, and they do not prevent the model from reconstructing more accurate structures and textures during inference.

   A similar example can be seen in classification tasks using the ImageNet dataset. According to [1], ImageNet contains a significant number of mislabeled samples, samples with multiple labels, or samples that do not belong to any label. This is analogous to the regions SAM fails to correctly identify. These mislabeled data points are essentially "negative samples" in the training dataset. However, this does not hinder the training of classification models like ResNet or ViT, as the model learns "statistically" during training. It can self-correct learned errors in labels and predict the correct results during inference.

   [1] Northcutt, C. G., Athalye, A., & Mueller, J. (2021). Pervasive label errors in test sets destabilize machine learning benchmarks. arXiv preprint arXiv:2103.14749.

3. In Table 2, I see the proposed model performed the best. However, the FID score for SRDiff and SAN-Diff suddenly dropped a lot across different datasets (the last two rows), which causes my concern that if the authors didn’t choose good enough baselines. Please justify. I also would like to refer other reviewers’ comments.

   Our goal is to maintain the advantages of diffusion-based models in subjective metrics while reducing common issues such as distorted structures, misaligned textures, and bothersome artifacts. We aim to improve the accuracy of structural and texture reconstruction in the model. This goal is reflected in our results, where we significantly improve objective metrics (PSNR, SSIM) while maintaining similar subjective metric performance (as shown in the last two rows) compared to the baseline and other generative models. Additionally, subjective metrics, objective metrics, and the number of artifacts all reflect the accuracy and visual quality of the reconstructed images. To properly assess the quality of the reconstructed images, it is necessary to evaluate the model's performance based on all three types of metrics. In Figure 2 and Figure 8, we present images that highlight the excellent performance of our model based on this comprehensive evaluation.

Weaknesses:

1. Lack of Novelty in Methodology

   We sincerely appreciate your thoughtful comment. We would like to clarify that the Structural Position Encoding (SPE) module is a novel approach designed to encode structural position information within the masks generated by segmentation models. This module is introduced for the first time in our paper and is not based on any pre-existing method. Regarding the SAM model, we use it to extract structural information for structural noise modulation within our framework. This design is flexible and does not restrict the source of segmentation masks, meaning that segmentation models other than SAM can also be utilized. We have chosen SAM specifically due to its exceptional ability to generate fine-grained segmentation masks.

2. Limited Novelty in Designing Loss Function

   The key modification to the loss function lies not in the MSE metric itself, but in the object to which the distance is measured. In our proposed framework, we closely follow the DDPM setting [1] within the context of the noise modulation scenario, leading to the formulation of the modified loss function. By training with this modified loss, the denoising model is able to incorporate structural information, resulting in improved SR image generation. Our experimental results demonstrate the effectiveness of this design.

   [1] Denoising diffusion probabilistic models. Advances in neural information processing systems, 33, 6840-6851.

3. Limited Exploration of Alternative Modulation Strategies

   The primary aim of our work is to enable the model to achieve structure-level differentiation between regions. To this end, we use segmentation masks as the source of structural information for training, as they inherently contain rich structural details. While intermediate features may also carry abundant information, they are less directly related to image structure and more challenging to utilize effectively. Therefore, we have focused on a segmentation-mask-driven approach in this study. In future work, we plan to explore feature-based techniques to further enhance model performance.

4. No Discussion on the Time Complexity of SPE

   Generating segmentation masks on the DIV2K dataset using SAM takes approximately 4 hours on GPU, while synthesizing SPE masks with the SPE module requires only 15 minutes on CPU. These data do not need to be regenerated for every training session; instead, a single inference of the training dataset using the segmentation model can be performed beforehand, and the results can be reused across different training runs. Therefore, we consider the cost of this one-time preprocessing negligible compared to the total training time of approximately 50 hours.

5. Dependency on High-Resolution Training Data

   In the current literature on image super-resolution, the standard approach involves training super-resolution models using pairs of low- and high-resolution data. However, there has been little to no exploration of high-resolution image-free super-resolution tasks. We believe this is an interesting task and leave it for future research.

Questions:

1. Could the authors mention if there are any unique aspects in how SAM and SPE are integrated or applied that are tailored for this task?

   We leverage the powerful fine-grained segmentation capabilities of SAM to modulate the noise distribution across different regions. The fundamental concept behind the SPE module is to assign a unique value to each segmentation area. The segmentation mask generated by SAM comprises a series of 0-1 masks, where each mask corresponds to an area in the original image sharing the same semantic information. To achieve noise modulation based on mask information, we merge these masks using the SPE module. Inspired by successful practices in language models, we use RoPE to encode the masks at different positions, embedding the relative positional information of different segmentation areas into the noise.

2. Could the authors try using the other losses like structural loss instead of standard MSE loss for better results?

   Our modification to the loss function lies not in the MSE metric itself, but in the object to which the distance is measured. While adopting other metrics, such as MAE, may also be effective, this is beyond the scope of our paper’s focus. Furthermore, to ensure a fair comparison with existing works [1] that consistently use the MSE loss, we have adhered to the same configuration in our experiments.

   [1] Denoising diffusion probabilistic models. Advances in neural information processing systems, 33, 6840-6851.

3. Could the authors mention if they tried alternative feature extraction techniques like -depth maps, CNN features etc. instead of segmentation masks.

   Our goal is to enhance the accuracy of structural and texture reconstruction in diffusion models without compromising their advantages in subjective metrics. The most intuitive approach is to leverage the structure-level ability of segmentation models to distinguish different regions. We believe that combining depth maps or CNN features with our method could further improve the model's performance, and we look forward to related research in the future.

4. Could the authors provide concrete timing measurements or complexity analysis for the SPE module. This would help to understand the practical implications of using this approach.

   Synthesizing SPE masks on the DIV2K dataset using the SPE module takes only 15 minutes. Moreover, these masks do not need to be regenerated for each training session. A single inference using the segmentation model on the training dataset, followed by saving the results, allows these masks to be reused across different training runs.

Dear mefZ,

Thank you for your constructive comments and valuable suggestions to improve this paper. ​If you have any more questions, we would be glad to discuss them with you.

Thank you very much.

Best regards, Author

Dear mefZ,

We apologize for the repeated reminder. It has been 7 days since we submitted our responses, but we have not yet received your feedback. We simply want to ensure that we have fully addressed all your concerns.

Your main concerns appear to relate to our contributions and the dependence on data and computational complexity. We have provided detailed responses to these questions in both our reply to you and the general response. May we kindly ask if you could spare some time to review our responses and share your feedback?

If there are any remaining points that require further clarification, please rest assured that we are committed to providing detailed answers to all your inquiries and addressing any concerns you may have. We value clear and open communication, and will make every effort to ensure that all aspects of the matter are fully explained to your satisfaction.

Thank you very much.

Best regards, Author