UIR-LoRA: Achieving Universal Image Restoration through Multiple Low-Rank Adaptation

Thanks for your thorough review and valuable feedback.

1. Detail distortion arises from operations such as downsampling or pooling. Using skip connections has become a standard and commonly used method to address this issue, as seen in the bypass decoder in [R1], the skip connections in [R2] and [R3], and even tracing back to the original U-Net. Firstly, this issue is not our focus and we have never claimed skip-connections as a contribution in our paper. Secondly, skip-connections are already widely adopted in neural networks and we have provided the references that our paper follows. In these cases, we think our handling of this part is appropriate and will not cause misunderstandings for our method.
2. The original CLIP and DA-CLIP do not use text representations to distinguish degradation types during testing; instead, they use image features extracted by the image encoder to classify degradation types. We used the trained DA-CLIP [R4]. The fine-tuning process of CLIP is a contribution of DA-CLIP, and we have never claimed it as our contribution. DA-CLIP uses the extracted degradation representation as input to the attention part of the diffusion process. In contrast, we use the representation to calculate the similarity of each degradation type and then combine the LoRA of different degradations based on this similarity. Compared to DA-CLIP's implicit representation, our way is an explicit and clearly defined approach. Intuitively, explicit methods have stronger interpretability, and experimentally, Sections 4.3 and 4.4 validate the effectiveness of our approach.
3. SD-turbo is a one-step method. When using SD-turbo, we did not use ControlNet or concatenated operations; instead, we used the representation of the LQ image in latent space as input directly. This way can be found in [R2]. And we will add this detail in our paper.
4. When training and testing DiffBIR, we used its default parameter settings and default structure, which inherently does not include skip-connections. Our experiments were conducted under the same data and environment for comparison. Therefore, there is no issue of unfairness. The lower performance of DiffBIR is due to the need to use SwinIR for preprocessing. When the preprocessing network struggles to handle multiple degradations effectively, the performance metrics of DiffBIR will be lower. The overall parameter can be found in T1.

T1:

5. In the dataset, we test Restormer with the entire image whenever possible. However, for some images, their dimensions cannot be evenly divided by 2 during the forward process. In such cases, we use a tiled mode to test these images, as shown in the JPEG degradation image in Figure 3.
6. We used trained DA-CLIP, and except for the blur degradation, the accuracy for the other nine degradations in the test set was 100%, as shown in Table 4 of Daclip-IR. Therefore, in Section 4.5 of our paper, we applied a simple and effective modification to improve the prediction accuracy for motion blur.

R1: Zero-Reference Low-Light Enhancement via Physical Quadruple Priors. CVPR2024.

R2: One-Step Image Translation with Text-to-Image Models. Arxiv2024.

R3: Exploiting Diffusion Prior for Real-World Image Super-Resolution. IJCV2024

R4: Controlling vision-language models for multi-task image restoration.ICLR2024.

We are truly grateful for your positive feedback on our work.

1. ControlNet is used in DiffBIR, and it adds a single encoder to handle various degradations, but its performance is still limited by task conflict. However, LoRA can be applied to any layer of a pre-trained model with a small number of parameters and works very well. This is why we chose LoRA in our approach.
2. The “+” in L123 corresponds to the parameter merge notation. In practice, we use feature weighting. We will take your suggestion into account and make this notation more accurate.
3. Several pre-trained generative models, such as SD1.5, SDXL, SD-turbo, and others, can serve as the base model for our approach. Although using multi-step pre-trained models theoretically offers stronger generative capabilities, we chose the one-step SD-turbo considering efficiency.
4. In Eq. (4), we use the letter 'o', not the number '0'.
5. In Section 4.4, we used two datasets, REDS and LOLBlur. REDS contains real-world degradations, while LOLBlur, although synthetic, is simulated based on real imaging processes and its data is consistent with real scenes.
6. Thank you for the detailed review of our writing. We will correct this error.
7. In our experiments, we used the trained DA-CLIP. Our vocabulary bank includes ten types of trained degradations.

Thanks for your thorough review and valuable feedback.

1. Core technical contributions: The core idea of our method is to introduce the paradigm of multi-domain transfer learning into multi-task image restoration, which aims to address the issues of task conflict and feature sharing in multi-task image restoration. This paradigm has not yet been applied in the field of image restoration. Along this line, we focused more on the overall framework rather than technical differences because we believe that the multi-domain transfer learning perspective presented in this paper can inspire more multi-task image restoration methods in the future.
2. Method description: Many papers in this field use terms like 'all-in-one'[R1, R2] and 'universal'[R3]. Following these works, we also used the term 'universal.' As you mentioned, multi-task image restoration is more accurate and we will revise this in our paper.
3. Claim in introduction: Thank you for your suggestion. We will revise the claim in L29 about 'sharing all parameters.' Our intention was to describe how inappropriate parameter sharing can be one of the sources of task conflict, without affecting the validity of the problem itself or our method. Essentially, our method also addresses how to allocate shared versus task-specific parameters. However, the difference is that we provide such an allocation scheme from the perspective of multi-domain transfer learning.
4. Thank you for the detailed review of our writing. We will correct these errors and thoroughly check our paper to ensure there are no other mistakes.
5. Additional results: The “mixed degradation” column in Table 3 uses the LOLBlur data from Table 2. Each image in LOLBlur contains at least two types of degradation, so the results for K=2 (Top-2) and K=10 (All) are better than K=1 (Top-1), as shown in the “mixed degradation column” in Table 3. Additionally, we synthesized more mixed degradation data as the reviewer nZDS's request. The impact of the hyperparameter K on the results is shown in T1. When the input image has mixed degradations, a larger K will result in better restoration performance.

T1:

R1. All-in-one image restoration for unknown corruption. CVPR2022.

R2. Promptir: Prompting for all-in-one image restoration.NeurIPS2023.

R3. Selective Hourglass Mapping for Universal Image Restoration Based on Diffusion Model.CVPR2024.

I thank the authors for the response and address their reply point-by-point.

1. The author rebuttal states that the core technical contributions did not focus on 'technical differences'. For the reviewer, this remains a confusing strategy for the focus of technical contributions. The crux of my concern is that task conflicts and feature sharing are well understood concepts and the rebranding of method terminology ('multi-domain transfer learning') has not added significant additional problem insight, for me.

2. The authors seem to concede that 'multi-task image restoration' is an accurate description of the work. This appears to further blur any distinction on method terminology.

3. Assuming the introduction claim is appropriately amended, I can consider this concern well addressed.

4. I can consider this concern well addressed.

5. I can consider this concern somewhat addressed and appreciate the additional experimental work. The effect of K in mixed degradation scenarios appears fairly underwhelming (ie. are we near the noise level?). Further I remain unconvinced that PSNRs of circa 12--15 on unknown degradations ('smoky') and mixed scenarios ('Hazy-Blurry-Noisy') are evidencing what might be considered succesful and pragmatic method performance.

In sum, authors address a subset of my concerns however core issues remain problematic. I therefore do not increase my score.

Thanks for the response from the reviewer. But we cannot agree with the reviewer's response.

1. Core contribution: We respectfully disagree with the reviewer's view that this is a rebranding of method terminology. We introduced the "multi-domain transfer learning" paradigm primarily because pre-trained generative models inherently have shared image priors across different image restoration tasks, which current "all-in-one" or "multi-task" image restoration methods have neither proposed nor utilized. Additionally, using parameter-efficient adapters to capture the differences between various degradation restoration tasks and avoid task conflicts is also a novel approach, which has not been explored in the image restoration field. If the reviewer considers our method to be a "rebranding of method terminology," please provide the references on which this innovative criticism is based. Otherwise, this claim is not convincing.

2. Method description: "Multi-task image restoration" is a more appropriate term; however, aside from Daclip-IR, other papers with the same setting currently use the terms "all-in-one" or "universal." We have simply followed the common usage but are also considering adopting the more accurate term "multi-task image restoration," as suggested by the reviewer. The reviewer claims that we only trained on 10 types of degradation, but we also tested on mixed degradation and included experiments with unknown degradation during the rebuttal period. Whether in terms of the types of degradation or the complexity of degradation, we have already surpassed all-in-one methods like AirNet and PromptLR. We have covered the diversity of image restoration tasks as comprehensively as possible. From this perspective, why should it not be considered "universal"?

3. Additional results: First, please do not overlook the scenarios in which 𝐾 is used in our method. It addresses mixed degradation, which involves data distributions that were not encountered during training, and this is inherently a challenging problem. We are unsure where the reviewer’s term "near the noise level" comes from. Could you please provide more specific details? If the reviewer is referring to the data we tested, please note that our method shows improvements across four metrics and has been tested on multiple images, which avoids fluctuations based on a single image or a specific metric. Additionally, the data used in the "smoky" and "Hazy-Blurry-Noisy" experiments are also out-of-distribution. Existing image restoration methods generally struggle with this case, and even many methods have not considered such a case. Please also do not overlook the performance improvements of our method compared to other SOTA methods.

I thank the authors for their further clarifications and robust discussion.

1. The additional dialogue on distinction to previous work, with regard to incorporation-of-priors strategy, is helpful and further alleviates my concerns on this point.

2. I think we have (somewhat) converged here.

3. I agree with the authors that qualitative examples will likely further aid reader understanding on method sensitivity to this hyperparameter and would welcome such additions.

As previously noted, my remaining concerns can be considered smaller and I believe my rating, confidence scores now reflect this accurately.

Thanks for your patient discussion. We welcome the reviewer to discuss our paper based on fairness and objectivity.

1. Although IPT[R1] has pre-training and fine-tuning stages, it differs significantly from our method. IPT uses synthesized degraded image pairs and learns the mapping from degraded images to clear images during the pre-trained stage. This involves incorporating priors about the degradation process, which is different from the image distribution prior used in the generative models of our method. Additionally, during the fine-tuning stage, the shared parameters are still updated while our pre-trained generative model is frozen. This means that it is necessary to reload the pre-trained parameters in IPT and retrain the parameters when testing different degraded images. Our pre-trained generative model is frozen, meaning the shared parameters have the same weights when testing different degradation types. In contrast, IPT’s shared parameters have different weights during testing. With the fine-tuning stage and the need for explicit degradation categories, this method cannot even be classified as an all-in-one approach. This is why AirNet[R2] claims that they are the first to propose the all-in-one restoration task.

2. Thanks for the reviewer's suggestions. The core issue with using terms like "multi-task," "all-in-one," or "universal" is how to define the range of degradation types included in the term "all." Currently, this range is not clearly defined in existing papers. Based on the response, the reviewer agrees with this reason. While we believe "multi-task" is a suitable term in this field, the experiments we conducted also support our claim of universal image restoration. And considering the "reader expectation management" mentioned by the reviewer, we will revise this description as we promised in the original rebuttal.

3. If the reviewer thinks the performance improvement of our 𝐾 strategy is minimal, it will be better if they can refer to the ablation studies of existing restoration methods, such as HInet[R3] and Restormer[R4]. For example, in Restormer, each innovative module has an improvement in PSNR between 0.05 and 0.2. Additionally, the reviewer compares the results of 𝐾=1 and 𝐾=2. But the model with 𝐾=1 is already an improved version of our method, not the "baseline." Our model with 𝐾=1 can handle the primary degradation in the image, and increasing 𝐾 aims to address secondary degradations in case of mixed degradation. We will add a comparison of the processing results when K has different values in the paper if we have the opportunity to submit a camera-ready version.

R1. Pre-trained image processing transformer. CVPR 2021.

R2. All-in-one image restoration for unknown corruption. CVPR2022.

R3. Hinet: Half instance normalization network for image restoration. CVPR2021.

R4. Restormer: Efficient transformer for high-resolution image restoration. CVPR2022.

I thank the authors for their further thoughts and the discussion.

1. On core contribution - I appreciate the authors' reply around using shared image priors across different image restoration tasks. Apologies if the use of the term "rebranding" was clumsy, I intend to communicate that any distinction with a core strategy of augmenting a pre-trained model with task-specific components remains somewhat subtle for this reviewer. Multi-task methods have indeed made use of 'shared image priors' for some years (e.g. [R1]), for multiple image restoration tasks. I consider the novelty of core-strategy somewhat modest, however the combination of method components do indeed differ.

2. I agree that authors experimentally cover a diverse set of image restoration tasks. On "universal" terminology; this is largely a matter of taste. If we define the word as "applicable to all cases", then one might expect similar ID / OOD performance. The comment was originally designed to help the authors with reader expectation management. I appreciate that opinions may diverge on this.

3. My concern here is largely about the statistical power of the effect size relating to K. Additionally; if we are considering PSNR deltas on the order of 10^-2, are these differences qualitatively distinct in image space? meaningful? I find the evidence relating to 'choice of K' claims not overly convincing. I acknowledge that OOD scenarios are challenging (see previous comment) and that method performance improves c.f. alternatives however strong performance, in absolute terms, would seem to evade all methods under comparison.

In sum the authors and I seem unable to reach agreement on a subset of points, however these may not be pathological in nature. I'm happy to somewhat modify my score to reflect this.

References

R1. Chen et al. 2021. Pre-trained image processing transformer. CVPR 2021.

Thanks for your valuable suggestions and for acknowledging our work.

1. Why REDS and LOLBlur？ We used the REDS and LOLBlur mixed datasets because the mixed degradation scenarios in these datasets are common, whereas the mixing in MID6 is not commonly seen in real-world scenarios. Since MID6 has not released its data, we simulated the mixing of three types of degradations based on the imaging process and compared our method with several representative image restoration methods. The results are shown in T1.

T1:

2. Network weight: The network used in Table 2 was tested directly after being trained as shown in Table 1. This means that the networks for testing in Table 1 and Table 2 use exactly the same parameters and weights, which is intended to validate the performance of our method on mixed degradation data.

3. Unknown degradation: The REDS and LOLBlur datasets we used contain mixed degradations that are common in real-world scenarios and are also outside the distribution of the training set. From the results in Table 2, we can see the advantages of our method. Additionally, we tested the performance on the unknown degradation - smoky. The comparison with representative restoration methods is presented in T2, showing that our method has a performance improvement compared to Restormer, PromptIR, and Daclip-IR.

T2:

4. Table 3 explanation and additional experiment: When the image has only one type of deterioration, the “top-1” strategy and “top-2” strategy perform similarly, as indicated in the “Multiple Degradation” column of Table 3. However, when the degraded image has more than one type of degradations, the “all” strategy and the “top-2” strategy outperform the “top-1” strategy, as illustrated in the “Mixed Degradation” column of Table 3. We conduct the experiment on the data of “Hazy-Blurry-Noisy” like MID6, As shown in T3, since the images have three types of degradations, the “top-2”, “top-3”, and 'all' strategies are superior to the 'top-1' strategy.

T3:

R1: MID6: Multimodal Prompt Perceiver: Empower Adaptiveness, Generalizability and Fidelity for All-in-One Image Restoration, CVPR, 2024.

Dear Reviewer nZDS, I have noticed that you have not yet responded to the authors' rebuttal. I kindly urge you to engage in a discussion with the authors at your earliest convenience to help advance the review process.

Thanks for your thorough review and valuable feedback.

1. Motivation of the weighted sum: When the image has only one type of deterioration, the “top-1” strategy and “all” strategy perform similarly, as indicated in the “Multiple Degradation” column of Table 3. However, when the degraded image has more than one type of degradation, the “all” strategy outperforms the “top-1” strategy. Our motivation for utilizing weighted sum is to enable the model to perform better in cases of mixed degradation.
2. Why average scores? : It is difficult to accurately assess the quality of an image using the PSNR metric alone, as shown in Table 6. Therefore, we also employed SSIM, LPIPS, and FID metrics, as shown in Tables 7, 8, and 9. From the comparison results in Tables 6, 7, 8, and 9, we can see that for the blurry, raindrop, and rainy tasks, our method achieves the best score on at least one or more of the four metrics. Despite the LOL dataset including just 15 images, our metrics are the second best among all the compared methods. Additionally, our focus is on multi-degradation image restoration tasks and average metrics provide a concise and comprehensive measure of multi-task image restoration models' performance, which is a common practice in this field, as seen in Table 3 of DA-CLIP.
3. Additional comparison: We retrain the method of [R1] and test it using the multiple degradation dataset and the mixed degradation dataset. The results are displayed below. It is evident that our method outperforms R1 on the four metrics as well as on the average metric.

T1: Multiple Degradation:

T2: Mixed Degradation:

R1. Selective Hourglass Mapping for Universal Image Restoration Based on Diffusion Model, CVPR 2024