Controlling Vision-Language Models for Multi-Task Image Restoration

The authors are appreciated for providing clarifications and additional experiments regarding my concerns. They have addressed some of the questions, such as Q1. However, the responses to other critical inquiries remain unsatisfactory.

• Generalization ability. As pointed out by other reviewers, the proposed method and provided codes does not show superior performance with complex degradations, which might be the true home field of diffusion models. Although in general response, the author provides experiment of light rain as OOD experiment. It is still a quite simple case compared with complex degradations used in related works such as StableSR, DiffBIR etc.

• Motivation and Advantages of Utilizing VLM. The author responses that CLIP provides more accurate visual embeddings and stronger generalization ability. However, it is worth noting that other pretrained backbones, like DINO, may yield more robust visual embeddings. This explanation does not entirely justify the selection of CLIP in this context. Additionally, the learning pipeline of DA-CLIP may not necessitate linguistic components due to the simplicity of the degradation types involved. For instance, a similar classifier, such as AirNet, could potentially perform just as effectively. It is my belief that the advantages and potential versatility of language components remain underexplored in this study. It would be more intriguing if the method could generalize to more complex degradations, such as "snowy and rainy," as this could be the key to justifying the inclusion of a text encoder: the ability to specify various degradation types using natural language.

• The usage of diffusion model. While the diffusion model does lead to improved perceptual quality (measured by lpips and fid), the enhancement is marginal and not visually obvious. Even the disparities in results presented in Figure 8 are relatively minor, and there is no presentation of results for NAFNet+DACLIP. Considering that the diffusion model substantially increases computational cost and runtime compared to NAFNet, one must question why diffusion model is necessary here. While I acknowledge that NAFNet in conjunction with DACLIP is also your results, if the diffusion model does not exhibit superiority, it might be better to consider removing it from the paper.

In conclusion, the provided responses have not adequately addressed my primary concerns, and I maintain my original rating preference.

Thank you for your insightful review and valuable feedback. We address your questions in turn below:

Q1: There are no justifications about what is embedded in the HQ content embedding. It is better to provide some comparisons with the HQ content embedding and the original image embedding.

Thank you for this great suggestion. In Figure 9(c) and 9(d) of the revised manuscript, we have added the comparison of applying LQ content embeddings (the image embeddings obtained from the original CLIP) to the baseline model on both unified and degradation-specific image restoration tasks. Our DA-CLIP content embeddings clearly outperform the baseline of using original image embeddings. Moreover, the predicted HQ content embedding can also improve MSE-based approaches, please refer to the "General Response" (Table 2 and Table 3) for more details.

Q2: The effectiveness and necessity of the prompt learning module is not well discussed. It is better to provide some ablation studies to show the effectiveness of the prompt learning module compared with naive cross-attention.

Thanks for this nice suggestion. Since the prompt is applied to all blocks of our model, we cannot substitute it with cross-attention which is really computationally inefficient (see Section 3.2). Instead, we included a comparison of replacing the prompt module with a simple network in Figure 9(h) of the revised manuscript, and the result demonstrates the added benefit of the prompt learning module.

Q3: The performance compared with NAFNet+DA-CLIP is not superior.

We need to clarify that the NAFNet+DA-CLIP is also our method, which is proposed to prove that DA-CLIP can also improve the performance of vanilla MSE-based restoration approaches. For more details please refer to the "General Response" Table 2 and Table 3. Moreover, as we mentioned in the model evaluation section, this paper mainly focuses on perceptual metrics (which means realistic visual results) and our diffusion-based method performs the best in terms of LPIPS and FID. As further illustrated in the face inpainting case in Figure 8, our DA-CLIP generates correct details for the left eye region while other MSE-based methods can only produce blurry and smooth contents.

Q4: It is known that the FLOPS is not a good metric for the computation complexity, especially for diffusion models which require multiple iteration steps. The paper should provide the inference time for the proposed method, and better in the main paper.

Thanks for pointing this out. It is true that diffusion-based methods require more inference time for multiple iterations. We have now added the runtime comparison in the below table. Note that MAXIM is implemented with the JAX GPU version, thus its comparison may not be fair. We then add the PromptIR and NAFNet for further reference. For the diffusion baseline IR-SDE, our DA-CLIP only adds 0.06s over 100 iterations (the prompt module and cross-attention also add 0.3s). For the direct MSE-based baseline NAFNet, there is no obvious increase in inference time.

Table 4. Comparison of the number of parameters and model computational efficiency. The flops and inference time are computed on the inpainting dataset.

---

Dear Reviewer bHgA,

Thank you for your reply. We are pleased to know that our new experiments have addressed some of your questions. In the following we further address your remaining concerns:

Q(a): Generalization ability.

We would like to note that the blind approaches like StableSR and DiffBIR can recover images with mixed blur, compression, and noise because they are specifically trained on these mixed degradations, see the argument in [1, Fig.2]. It is undoubtedly interesting to create models to handle concurrent degradations like “rain + shadow”, “snow + low-light”, “haze + raindrop”, etc. However, mixing these degradations is hard since most of them are captured from specific scenes and camera devices, and manually synthesizing these degradations requires knowing the corresponding domain-specific knowledge (i.e., the true degradation function). We agree that our generalization ability thus is not perfect (but is still better than other all-in-one methods such as AirNet and PromptIR). Considering this, we have revised the description of our method from “universal” to “multi-task”, and we have also emphasized the limitations in the revised manuscript. The light rain experiment evidences that our method is robust to different degradation levels. In addition, we provided additional real-world restoration examples in Figure 16 for reference, which also illustrates that our method has better generalization ability than peer all-in-one approaches.

Q(b): Motivation and Advantages of Utilizing VLM.

Incorporating pretrained foundation models (e.g., CLIP, DINO) for low-level vision tasks such as image restoration is still an open problem. In this paper, we focus on the VLM for accurate semantic instructions which are also used in other text-guided image generation models like Stable Diffusion, ControlNet, Imagen, InstructPix2Pix, etc. However, our core component, the image controller, is applicable to any other backbones (e.g., DINO) that have a ViT-based image encoder, as shown in Figure 3(a). We appreciate the reviewer’s feedback and suggestions for more robust backbones and we will definitely test them in future work.

Moreover, we also appreciate the feedback on linguistic components which is really valuable and inspires our future work! As we explained in Q(a), our method is not able to properly process mixed degradations in the current work. We thus agree with the reviewer that the linguistic components are under-explored and we have then added it to our limitations. However, we would like to clarify that our method not only provides the ability of degradation classification but also predicts the clean content embedding which can further improve the performance. While other simple classifiers can hardly do this since there are no ground truth HQ embeddings in the dataset.

Q(c): The usage of diffusion model.

Image restoration methods are usually sensitive to image details and textures. It has been shown that MSE-based models tend to produce blurry results to get high peak signal-to-noise ratios (PSNR), see the arguments in [2, pp.1] and [3, pp.1]. Previous works [2,3] address this issue by incorporating adversarial networks, which are however notoriously unstable and inaccurate [4, pp.1]. Thus, most recent works tend to utilize diffusion models to generate realistic outputs [4,5,6].

The visual results for NAFNet+DACLIP are very similar to other MSE-based methods like AirNet and PromptIR. As suggested, we will add more visual comparisons to the Appendix in our revision. Moreover, we would like to clarify that our improvement in perceptual scores is significant compared to other MSE-methods. In particular, our method surpasses the MAXIM and PromptIR by ~47 and ~14 in terms of FID in Table 2(a) and Table 3, respectively (from 58.7 to 11.7 in Table 2(a), from 48.2 to 34.8 in Table 3). Apart from the inpainting case in Figure 8 (left eye region), Figure 5 (deraining) and Figure 15 (inpainting) also illustrate that MSE-based approaches tend to produce blurry and over-smooth visual results (in degraded areas) compared to our method.

Moreover, choosing a specific restoration model (diffusion- or MSE-based) is not our key component. In this paper, we would like to include both IR-SDE+DA-CLIP and NAFNet+DA-CLIP to demonstrate that our DA-CLIP can be integrated with various restoration models, to demonstrate the general applicability in that sense.

---

[1] DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior. ArXiv 2023.

[2] Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. CVPR 2017.

[3] Image-to-image translation with conditional adversarial networks. CVPR 2017.

[4] Image super-resolution via iterative refinement. TPAMI 2022.

[5] Denoising Diffusion Restoration Models. NeurIPS 2022.

[6] Image restoration with mean-reverting stochastic differential equations. ICML 2023.

Thank you for your comprehensive and insightful review. We address your questions in turn below:

Q1: It is questionable that the caption embedding can provide a high-quality image representation supervision for the content embedding. [...] It seems that $e_c^I$ mainly serves to provide semantic instruction for the restoration network, especially for diffusion-based models.

A key motivation for leveraging the pretrained VLM for image restoration is that VLMs like CLIP can accurately align image representations with corresponding caption embeddings. Note that the caption generated from an LQ image would be different from that of an HQ image. Such as the snow example shown in Figure 4, the caption from HQ image precisely describes the main content of the scene, which supervises our model to learn a snow-free HQ image representation from the LQ image. In addition, since we fixed the text encoder in training, matching the heavy snow scene to a snow-free caption embedding guarantees that the learned content embedding contains no snow information, which is called "clean" and "high-quality" in the context of image desnowing.

We have further illustrated this by applying two different image content embeddings (HQ embedding from DA-CLIP and LQ embedding from the original CLIP) to the baseline in Figures 9(c) and 9(d) of the revised submission. The results show that our DA-CLIP content embedding can provide more useful supervision to the learning process on both unified and degradation-specific restoration.

Moreover, the learned content embedding $e_c^I$ can also benefit MSE-based restoration approaches. As we illustrated in the "General Response" Table 2 and Table 3, both NAFNet and PromptIR can be further improved by adding our DA-CLIP content embeddings.

Q2: The used experimental setup is too simple to demonstrate the superiority of this complex method. It is not difficult for a unified network, e.g., a vanilla version Restormer, to deal with the all-in-one image restoration setting with a specific degradation level for each degradation type.

Although we can directly train a vanilla network across multiple degradations, its performance is inferior to approaches that learn discriminative degradation context explicitly [1,2]. As illustrated in the "General Response" Table 2, combining the vanilla model NAFNet with our DA-CLIP significantly outperforms the original NAFNet on the unified restoration task. In addition, as nicely suggested, we have added the Restormer experiment in Table 3 of the revised manuscript (also in the "General Response" Table 2). It is worth noting that PromptIR is built upon Restormer with additional degradation prompt modules, which largely improves the unified restoration results.

---

[1] All-in-one Image Restoration for Unknown Degradations Using Adaptive Discriminative Filters for Specific Degradations. CVPR 2023.

[2] PromptIR: Prompting for All-in-One Blind Image Restoration. NeurIPS 2023.

Thank you for providing valuable feedback for our paper. We address your concerns in turn below. We sincerely hope that this response addresses all the concerns and that the reviewer would consider re-evaluating our work. We look forward to and appreciate your further feedback, we would be happy to provide further clarifications if needed.

Q1: My main concern with this paper is its task setting. First, "Universal Image Restoration" is a term that is not so easily justified. This paper simply brings together ten different image restoration tasks, which is closer to "multi-task" than the so-called "universal". [...] For instance, an image with both rain streaks and subsequent compression artifacts cannot be accurately restored. This is not "universal".

Note that the "universal" means our method is a general framework in which the proposed DA-CLIP can be applied to improve both degradation-specific and unified image restoration models, as illustrated in Table 2 and Table 3 of the manuscript. We have rewritten the introduction to clarify this in the revised manuscript. It is also worth noting that all existing unified models have been evaluated only on three or four degradation types, while our method handles ten restoration tasks with various input sizes and complicated modalities (as shown in Figure 1). In this situation, learning discriminative degradation context explicitly has been proven a vital step for unified image restoration [1,2,3].

In addition, please note that none of the existing unified models trained on our dataset can restore images with multiple specific degradations (such as a scene that mixes shadows and raindrops, like the Raindrop example in Figure 8), since we have only one degradation type label in each training image pair. That is, our DA-CLIP shares this limitation with all other existing unified restoration models. However, it would indeed be possible to add some simple degradations like noise and blur to synthesize multi-degradation image pairs in training, potentially mitigating this issue. Thank you for your feedback, we would be very interested in creating more practical models that are able to fully restore images with mixed degradation types in our future work.

Q2: Moreover, this paper seems to ignore a host of more "universal" solutions, such as Real ESRGAN, BSRGAN, StableSR, DiffBIR, etc. Merging degradations to achieve better generalization is a new direction (which is not so new anymore). But this paper barely discusses whether these methods are "universal" or not.

We need to clarify that approaches like Real-ESRGAN, BSRGAN, StableSR, and DiffBIR are not "universal" since most of them mainly focus on the image super-resolution problem. They are indeed practical for some blind degradation conditions such as noise and blur, but can never recover images from more complicated scenarios such as shadow removal, deraining, inpainting, etc. Moreover, the goal of unified image restoration is not only to provide generalization ability but also to avoid repeated training of individual networks for specific degradation types [3]. As nicely suggested, in our revised manuscript, we have added a discussion on these blind image restoration methods in the related work section.

---

[1] Learning Weather-General and Weather-Specific Features for Image Restoration Under Multiple Adverse Weather Conditions. CVPR 2023.

[2] All-in-one Image Restoration for Unknown Degradations Using Adaptive Discriminative Filters for Specific Degradations. CVPR 2023.

[3] PromptIR: Prompting for All-in-One Blind Image Restoration. NeurIPS 2023.

Dear Reviewer hnHQ,

Thank you very much for your reply and we appreciate your comments on the definition of “universal” and suggestions for the justification.

1. We were not trying to oversell our method by using the word "universal". We were seeking a suitable term to inform that our method can benefit both degradation-specific and unified (all-in-one) image restorations. We have also emphasized the limitations (e.g., to deal with multiple degradations at the same time) in the resubmission. We will change the word “universal” in the manuscript to a neutral one, such as multi-task, as per your request, to avoid abusing the term.

2. We have also reclarified the difference to the suggested related works (e.g., Real-ESRGAN, BSRGAN, StableSR, DiffBIR). Specifically, these works do not make use of the semantic embeddings from the vision-language model, while on the other hand, the gist of our paper is to incorporate such semantic embeddings, and we have shown that it indeed improves the image restoration performance.

3. We agree with the reviewer that vanilla deep learning frameworks can be directly trained for unified image restoration. However, they are not specifically designed for multi-task learning. Hence, training these methods by directly mixing all degradations ignores learning accurate discriminative degradation contexts, see the arguments in [1, pp.3] and [2, pp.2]. Therefore, the purpose of DA-CLIP is to incorporate additional semantic instructions (degradation and HQ content embeddings) to steer vanilla networks for better performance (such as IR-SDE+DA-CLIP and NAFNet+DA-CLIP).

4. We definitely agree that "first work" is not a reason for acceptance, and for that we did not list it as a contribution either. As acknowledged by other reviewers, our proposed DA-CLIP informs the community an interesting and promising direction of combining pretrained vision-language models with image restoration frameworks. Our experiments and analysis have also empirically demonstrated the effectiveness of our method.

Best, Authors

---

[1] All-in-one Image Restoration for Unknown Degradations Using Adaptive Discriminative Filters for Specific Degradations. CVPR 2023.

[2] PromptIR: Prompting for All-in-One Blind Image Restoration. NeurIPS 2023.

Thank you for your review and comments, which accurately summarized our paper and our intended contributions. We provide our point-to-point response below:

Q1: In Figure 1, DA-CLIP achieves surprisingly high accuracy in ten degradation types. How are these experiments set up? In contrast, CLIP performs poorly in many types. What prompts do the authors use for classifying degradations in CLIP?

In our experiments, we use the ten degradation types as label texts, and the prompt is set to "a [degradation type] photo". Since we freeze the text encoder in training, the same text embedding would be used for both DA-CLIP and the original CLIP to match low-quality images. We provide additional experiments of directly retraining or finetuning CLIP on our dataset in Table 4 of the paper. Our DA-CLIP achieves a similar accuracy with CLIP + finetune, while it can also output clean content embedding to further improve downstream image restoration performance.

Q2: In Figure 6, PromptIR is comparable or even better than the proposed DA-CLIP in most tasks on fidelity metrics.

Note that PromptIR is a purely MSE-based approach that achieves high fidelity metric scores (PSNR and SSIM) but will produce perceptually unsatisfying results[1]. Our method, similar to other diffusion approaches[2], mainly focuses on generating realistic images and achieves the best perceptual performance (LPIPS and FID) in Figure 6 and Table 3 of the paper. As can be seen in the inpainting example in Figure 8, the face generated by PromptIR loses all details on the left eye region, while our method predicts more details and produces relatively good visual results. In addition, simply adding the HQ content embedding of DA-CLIP can further improve PromptIR across all metrics (please refer to the "General Response" Table 2 for more details).

Q3: In Table 2(c), the PSNR of DA-CLIP highly deviates from that of MAXIM.

Similar to PromptIR, MAXIM is also an MSE-based method and is designed for degradation-specific image restoration. Although it surpasses DA-CLIP by 2dB in terms of PSNR on the GoPro dataset, our method still obtains the best perceptual scores (0.058/6.15 in LPIPS/FID compared to 0.089/11.57) and produce visually better results. In the "General Response" Table 3, we have further added an ablation experiment of applying DA-CLIP to NAFNet on two single-degradation tasks: image deraining and low-light enhancement. The results show that DA-CLIP clearly improves the performance of NAFNet, illustrating the effectiveness of our method on degradation-specific restoration.

Q4: In addition, the results on task-specific restoration do not show a clear benefit of using a universal model for all tasks. It is believed that the merit of universal models is that different tasks can benefit each other, or at least be helpful in generalization to new domains.

We need to clarify that separately training models for each task often gives better restoration performance than the unified setting, as each model then only needs to focus on one degradation throughout the training. This phenomena has been discussed in more detail in recent previous work [3, 4]. The main motivation for the unified model is 1) to avoid repeated training of the same model for individual tasks and 2) to avoid the need to know the specific input degradation type in order to apply the relevant model (which is usually required by task-specific methods[3, 4, 5]). Thank you for this valuable feedback, we have now clarified this motivation in the revised introduction section.

---

[1] Photo-realistic single image super-resolution using a generative adversarial network. CVPR 2017.

[2] Image super-resolution via iterative refinement. TPAMI 2022.

[3] AirNet: All-in-One Image Restoration for Unknown Corruption. CVPR 2022.

[4] PromptIR: Prompting for All-in-One Blind Image Restoration. NeurIPS 2023.

[5] All in One Bad Weather Removal using Architectural Search. CVPR 2020.