UPS: Unified Projection Sharing for Lightweight Single-Image Super-resolution and Beyond

Q1. Discussion and analysis with attention sharing...

Thanks for your suggestion. ShareFormer [1] presents a local similarity map-sharing scheme between neighboring attention layers for lower latency. Thus, ShareFormer shares a static similarity map for neighboring attention layers while UPS calculates dynamic similarity maps with layer-refined features in a shared projection space.

Skip-Attention cuts off some intermediate attention layers to improve efficiency and performance for high-level tasks. LaViT [2] proposes a residual-based attention downsampling that fuses the initial calculated attention scores to guide the aggregation of the following layers, resulting in faster efficiency and improved classification accuracy.

Therefore, Skip-Attention [3] and LaViT follow the existing coupled optimization scheme (reduce some attention calculations), and UPS proposes a decoupled learning strategy to enhance performance. We will cite the insightful studies and add this discussion to our revised paper.

Q2. Report the computational quantities of different models

Thanks. We agree with you on this point and we will include the FLOPs (like Fig. 1(c) of our paper) of different CNNs and Transformers for a comprehensive comparison. Please also refer to the FLOPs results in Tab. 1, 2 (b), 3 and Table 4 of our response PDF file.

Q3. Inference efficiency of different models

Please see the first answer of the global response and more inference comparison in the Tab. 1. 2(b), 3, 4 in our response PDF file. We will provide all these discussions in our revised paper.

Q4. Experimental evaluation real-world SR.

Thanks for this good comments. Following SwinIR-GAN, our baseline model, we train UPS-GAN with the same configuration using the widely used BSRGAN degradation. We compare UPS-GAN with SwinIR-GAN, BSRGAN, RealSR, Reshift, and StableSR for real-SR task. We evaluate all real-world SR models using the same benchmarks (RealSRSet) and assessment metrics as SwinIR-GAN. As shown in Table 2a of the PDF file (also the Tab below), our proposed UPS-GAN outperforms other state-of-the-art GAN-based and even Diffusion-based methods (Reshift NeurIPS 2023[7] and StableSR IJCV 2024[8]) in terms of NIQE, NRQM, and PI metrics, achieving the best quantitative results (5.09/6.84/4.19). This confirms the effectiveness of UPS for real-world SR tasks.

Q5. The adopted similarity calculation method in UPS

As you pointed out, apart from the ReLUFormer discussed in our method section, we will cite and discuss the related works in our revised paper.

References

[1] ShareFormer: Share Attention for Efficient Image Restoration. arxiv 2023.

[2] You Only Need Less Attention at Each Stage in Vision Transformers. CVPR 2024.

[3] Skip-Attention: Improving Vision Transformers by Paying Less Attention. arxiv 2023.

[4] EfficientViT: Multi-Scale Linear Attention for High-Resolution Dense Prediction. ICCV 2023.

[5] Swin transformer v2: scaling up capacity and resolution. CVPR 2022.

[6] Swin2SR: SwinV2 Transformer for Compressed Image Super-Resolution and Restoration. ECCVW 2022.

[7] Yue, et al. ""Resshift: Efficient diffusion model for image super-resolution by residual shifting." NeurIPS, 2023"

[8] Wang et al. "Exploiting Diffusion Prior for Real-World Image Super-Resolution." IJCV, 2024.

Thank you for your additional comments. We're pleased that our previous response addressed some of your concerns. Here are our further responses to the issues you've raised:

1. As acknowledged by you and the other reviewers, UPS is a novel and effective lightweight SISR algorithm. Nonetheless, we explored the potential benefits of applying UPS to this new task. We ensured a fair evaluation by strictly following the same baseline (SwinIR, BSRGAN) and using consistent evaluation metrics, such as NIQE, NRQM, and PI, for all competing methods on the RealSRSet [1] benchmark.

2. In response to Q1, we have thoroughly examined the contributions of the suggested works and outlined the key differences between them and UPS. To summarize, UPS operates independently from these transformers. For instance, while ShareFormer generates a static similarity map for neighboring layers, UPS conducts dynamic similarity calculations within a shared projection space. Furthermore, unlike these three works, which adhere to coupled layer-specific similarity and feature extraction optimization, UPS introduces a decoupled approach for these aspects, specifically designed for lightweight SISR.

3. Due to the absence of ground truth data in the RealSRSet [1] benchmark, we cannot calculate reference-based metrics like PSNR, SSIM, and LPIPS. Instead, we present results using CLIPIQA and MUSIQ, which are widely accepted in diffusion-based models. Although UPS-GAN follows the GAN-based framework of our baseline models (SwinIR-GAN), it consistently achieves top-1 and top-2 performance when compared to other leading methods, as shown in the Table below.

1. Zhang et al. "Designing a Practical Degradation Model for Deep Blind Image Super-Resolution." ICCV, 2021.

Q1. Conduct experiments outside of the lightweight model...

Thanks for your thoughtful comments. As you appreciated the potential generalization of UPS on more image restoration tasks, e.g., JPEG compression removal and image de-noising tasks, we explore the potential application of UPS for common SISR and real-world SR tasks and more baseline frameworks (e.g., HAT/HAT-light, DRCT/DRCT-light) and report the results in Tab. 2 (a) and Tab. 3/4 of our global PDF response.

For both common and lightweight single image super-resolution (SISR), we incorporate UPS into other state-of-the-art SISR models, including SwinIR, HAT/HAT-light [4], and DRCT/DRCT-light [5], based on your suggestion. As shown in Tables 3 and 4 of our response file, we observe that UPS consistently improves the performance and inference efficiency of these models (HAT-UPS, DRCT-UPS, SwinIR-UPS, and their lightweight versions HAT-light-UPS, DRCT-light-UPS, and SwinIR-light-UPS). We will include these insightful experiments in our revised paper. We believe your insightful suggestion and our exploration enhance the wide-ranging value of UPS.

Additionally, for real-world super-resolution (SR), we follow our baseline model, SwinIR-GAN (trained using BSRGAN degradation), and train UPS-GAN. We evaluate all real-world SR models using the same benchmarks (RealSRSet) and evaluation metrics as SwinIR-GAN. As shown in Table 2a of the PDF file (also the Tab below), our proposed UPS-GAN outperforms other state-of-the-art GAN-based and even Diffusion-based methods (Reshift NeurIPS 2023[1] and StableSR IJCV 2024[2]) in terms of NIQE, NRQM, and PI metrics, achieving the best quantitative results (5.09/6.84/4.19). This confirms the effectiveness of UPS for real-world SR tasks.

Q2. Latency comparison

Please see the first answer of the global response. We will provide all these discussions in our revised paper.

Q3. Comparison with the SOTA model, e.g., Omni-SR

As discussed in our paper (Line 175), we use the widely used DIV2K for training. When compared with Omni-SR [6] (trained on the same DIV2K), UPS surpasses it in most cases.

To make a fair comparison with Omni-SR+ (trained on DF2K), we re-trained UPS+ from scratch on DF2K. We can see that UPS+ generally attains superior performance than Omni-SR+ in the below table. For instance, UPS+ achieves 0.27dB/0.20dB/0.29dB improvements than Omni-SR+ on Manga109 x{2,3,4}.

More results can be found in Tab. 1 of our response PDF. We will include this comparison in Tab. 1 of our revised paper and highlight the two different training setups to ensure a clear and fair comparison.

References

[1] Yue, et al. ""Resshift: Efficient diffusion model for image super-resolution by residual shifting." NeurIPS, 2023"

[2] Wang et al. "Exploiting Diffusion Prior for Real-World Image Super-Resolution." IJCV, 2024.

[3] Liang, et al. "Swinir: Image restoration using swin transformer." CVPR, 2021.

[4] Chen, et al. “Activating More Pixels in Image Super-Resolution Transformer.” CVPR, 2022.

[5] Hsu, et al. “DRCT: Saving Image Super-resolution away from Information Bottleneck.” arxiv, 2024.

[6] Wang, et al. “Omni Aggregation Networks for Lightweight Image Super-Resolution.” CVPR, 2023.

Dear Reviewer HE6k: We have made every effort to address all concerns and provide comprehensive evidence.

For Q1, we explored real-world SR and large models for SISR, and also integrated UPS into DRCT/DRCT-light and HAT/HAT-light to boost their performance.

For Q2, as acknowledged by all other reviewers, we provided inference efficiency for further comparison.

For Q3, we conducted a detailed comparison with Omni-SR and Omni-SR+, showing that UPS/UPS+ outperforms them under a fair setup (using DIV2K/DF2K for training).

Could you please let us know if our rebuttals and further responses have answered all your questions? We greatly appreciate it.

We sincerely thank you for your appreciation of the novelty and effectiveness of our UPS for lightweight SISR.

Q1. A typo: Line 49...

We appreciate your kind comment. We will carefully revise our paper to fix the typo.

Q2. Figure 2: the misplacement of $S_1$ and $S_i$

Thanks, we will correct the placement of the $S_1$ and $S_i$ accordingly.

Q3. The indetity mapping of $X_i$ and $V_i$

For the lightweight scenario, we aim to further reduce the computational cost and model size. Thus, we explore cutting off the linear mapping between $X_i$ and $V_i$, and our early experimental analysis (presented in Tab. 5-C of our PDF file) suggests such a design will not lead to a performance drop. We will add this discussion to our revised paper.

Q4. The denotation of $D$ in $Q_i^D$ in Eq. 6

$D$ means the projection dimension in our UPS. We are sorry for this typo. The correct Eq.6 should be:

where $T$ means the transpose operation for matrix multiplication. We will double-check all these equations to avoid any difficulties in understanding our work.

Q5. Quantitative results on the DIV2K dataset...

In this work, we adopt the DIV2K for training our models as indicated in Line 175 "we utilize the DIV2K image dataset for training". We will highlight this important training setting in our revised paper.

Thank you for your kind comments and appreciation of our work. We will do our best to improve the final version of our paper based on your suggestions.

Q1. In line 30, "Fig. 1a.(1-3) below shows over 0.95% (0.99%, 0.95%, 0.96%) ..."

Thanks a lot for pointing out this typo, actually it's a writing error. It should be "Fig. 1a.(1-3) below shows over 0.95 (0.99, 0.95, 0.96) ...". We will revise our manuscript to avoid misunderstanding.

Q2. Experimental results when the activation function is consistent with SwinIR-light

That is quite a profound question. We have provided the ablated analysis (Tab. 5-A) in the global response PDF file. Also, we report the specific quantitative results in the table below. As we can see, the main improvement comes from our UPS design instead of the ReLU activation. The performance gap between the two different activation choices is only 0.04dB, which represents 11% of the total improvement of 0.36dB. In other words, the 89% improvements come from the UPS design. We will include this ablation analysis in our revised paper.

Q3. To compare the inference time

Please see the first answer of the global response. We will provide all these discussions in our revised paper.

Q4. More effective optimization strategy may also enhance the performance

We agree with you on the point that an advanced training strategy may also boost the performance of entangled SISR models. As praised by you and other Reviewers, UPS provides a simple yet effective optimization scheme: without any special training strategies, e.g., taking additional training costs or carefully model finetuning, while yielding improved results.

We investigate two existing training strategies for the SISR task. RDSR [1] incorporates dropout techniques to achieve better testing results, while DRCT [2] employs a progressive training scheme that involves multi-stage training to enhance final performance. Here, we compare UPS with RDSR [1] and progressive training schemes in DRCT [2]. To do this, we re-train SwinIR-light using the above two training strategies. As shown in Table 5-B of our PDF file (also presented in the Tab. below), UPS delivers superior results compared to both of these optimization methods. Nevertheless, we hope our exploration will inspire future research to develop more effective algorithms to better address this challenge.

References

[1] Kong, et al. "Reflash dropout in image super-resolution." CVPR, 2022.

[2] Hsu, et al. "DRCT: Saving Image Super-resolution away from Information Bottleneck.", arxiv, 2024.

Thank you for your kind comments and for appreciating our work. We will include these valuable discussions in the final version of our paper.