AdaIR: Adaptive All-in-One Image Restoration via Frequency Mining and Modulation

R4.1:t-SNE results:

Following the suggestion, we have incorporated a new method [5] for t-SNE comparisons, as shown in Figure 2 of the paper. Our method is better at learning discriminative degradation contexts.

R4.2:Rationale of t-SNE:

The use of t-SNE is well-established in all-in-one algorithms [6,7] as a means of evaluating the ability to distinguish between different degradation types. While we acknowledge the existence of correlations between degradations, we believe this does not necessarily conflict with the goal of decoupling these degradations. Instead, the goal is to strike a balance; capturing these correlations while maintaining sufficient separation for effective restoration.

As shown in Figure 14, the t-SNE results for the five-task setting demonstrate this balance. The correlations among degradations are reflected in the proximity of clusters. For instance, the cluster for low-light image enhancement is closer to the dehazing cluster, as both degradations predominantly affect the low-frequency components of the image. This proximity effectively captures the underlying relationship between these tasks while ensuring that individual degradations remain distinguishable.

R4.3:Why UNet?:

Several recent all-in-one works (Art$_{\text{PromptIR}}$, PromptIR, TransWeather, DL, InstructIR) employ a hierarchical architecture. In our AdaIR, we also use UNet design and demonstrate how to simply add AFLB into the main network. This can easily be extended to other networks as well.

Regarding the use of diffusion models versus regression-based methods, the choice depends on the intent of image reconstruction. If the primary goal is to preserve the original content without introducing artifacts or hallucinations (as is crucial in image restoration tasks), a regression-based method like AdaIR is more suitable. Diffusion models, while powerful, are generative in nature and prone to hallucinating information, which poses challenges in scenarios where fidelity/faithfulness to the original content is critical.

R4.4:Differences with existing algorithms:

Ref.[1] introduces a Transformer-based architecture that leverages spectral multiplication for image deblurring. However, this method lacks mechanisms for interaction across different frequency bands, and its frequency quantization matrix remains static after training, limiting the model's adaptability to varying input characteristics. In contrast, Ref.[2] proposes a wavelet-based model for underwater image restoration, which explores frequency properties through frequency component swapping in the frequency domain. Despite its focus on frequency manipulation, the objectives and methodologies of Ref.[2] differ substantially from ours. Similarly, Ref.[3] employs global average pooling for feature separation but is not designed for all-in-one restoration scenarios. Its binary separation strategy is constrained to the lowest frequency and its complement, resulting in limited adaptability to diverse degradation types. Ref.[4] presents a CNN-based algorithm for general image restoration, utilizing adaptive convolution and spatial softmax operations. However, it does not comprehensively address the representation and interaction of a broad spectrum of frequency bands. When we replaced our frequency separation approach with the method from Ref.[4], the performance decreased by 0.99 dB in PSNR, further highlighting its limitations. In contrast, our approach performs frequency separation by applying adaptively generated masks to Fourier features, enabling comprehensive coverage across a wide range of frequency bands and ensuring adaptability to diverse degradation types. Additionally, we integrate tailored attention mechanisms to enhance frequency integration, effectively harnessing the unique properties of individual frequency components.

[1] Kong et al., Efficient Frequency Domain-Based Transformers for High-Quality Image Deblurring, CVPR, 2023.

[2] Zhao et al., Wavelet-based Fourier Information Interaction with Frequency Diffusion Adjustment for Underwater Image Restoration, CVPR 2024.

[3] Cui et al., Hybrid Frequency Modulation Network for Image Restoration, IJCAI 2024.

[4] Cui et al., Selective Frequency Network for Image Restoration, ICLR 2023.

[5] Xu et al., Unified-width adaptive dynamic network for all-in-one image restoration, arXiv 2024.

[6] Potlapalli et al., Promptir: Prompting for all-in-one image restoration, NeurIPS 2023.

[7] Ai et al., Multimodal Prompt Perceiver: Empower Adaptiveness Generalizability and Fidelity for All-in-One Image Restoration, CVPR 2024.

We extend our sincere gratitude to the reviewer for their valuable time and insightful feedback. We value your constructive feedback and hope that our responses have appropriately addressed all the concerns.

We really appreciate the valuable time to respond to our feedback based on the reviewer's comments. Further, we are happy to address any remaining concerns.

R3.1:Frequency domain methods:

We appreciate the reviewer pointing out this oversight. We have now included frequency-based methods [1,2] in the paper and have reworded the relevant text accordingly. While methods [1,2] utilize manual or non-adaptive approaches for feature separation and perform frequency interactions without accounting for the distinct characteristics of different frequency components, our model introduces adaptive frequency separation. This enables the integration of frequency components based on the inherent properties of individual frequency subbands. The adaptability of frequency learning remains a core innovation of our work. To highlight this distinction, we have revised and refined the statements in both the abstract and introduction sections to enhance clarity and emphasize the unique contributions of our approach.

R3.2:Reusage of existing techniques:

The primary goal of AdaIR is to tackle the challenge of adaptive all-in-one image restoration by introducing a frequency-guided learning framework. While AdaIR builds on foundational components from prior works (e.g., Transformers and attention mechanisms), its novel contributions lie in the macro-level integration of adaptive frequency learning through the proposed Adaptive Frequency Learning Block (AFLB). This block is specifically designed to decouple and modulate low- and high-frequency components in a way that is adaptive to the specific degradation present in the input image. Unlike prior frequency-based methods that employ manual or fixed approaches for feature separation, FMiM adaptively mines low- and high-frequency components through a mask generation block that dynamically adjusts to the spectral properties of the degradation. This adaptability ensures that frequency components are extracted in a degradation-specific manner. FMoM facilitates cross-band interaction by leveraging tailored attention mechanisms (H-L and L-H units) to exchange complementary information between frequency bands. This ensures that both global and local degradations are effectively addressed—a feature not seen in prior frequency-based models.

Our objective is to develop a plug-in module tailored for all-in-one image restoration algorithms. To this end, we select the widely adopted Restormer as the baseline model. Additionally, to simplify the implementation of our mechanism and avoid extensive architecture tuning, we incorporate a CBAM-inspired attention mechanism. Despite the simplicity of design choices, our method achieves a remarkable performance improvement of 1.94 dB in PSNR compared to the baseline model.

Our goal in this work was not to focus on introducing a multitude of fine-grained or incremental changes but rather to identify and implement simple yet key modifications. These targeted contributions allowed us to address the global objective of frequency-guided image restoration while ensuring the model remains computationally efficient and adaptable across multiple tasks.

R3.3:Cite Restormer and CBAM works:

In our initial submission, we credited Restormer while describing the overall pipeline in Section 3.1 and CBAM when comparing this mechanism with alternatives in Figure 8 and Appendix B (Design Choices of FMoM). Following the reviewer’s suggestion, we have added additional references for these two works in Figure 3 and the relevant sections of the main text where these components are described, ensuring proper acknowledgment and improved clarity.

[1] Gao et al., Frequency-oriented efficient transformer for all-in-one weather-degraded image restoration, TCSVT 2023.

[2] Shi et al., Learning Frequency-Aware Dynamic Transformers for All-In-One Image Restoration, arXiv 2024.

R3.10:Generalization on unseen datasets:

Please note, as with previous works like AirNet and PromptIR, all-in-one models are designed to address unspecified degradation types rather than unseen corruptions. In this context, “multiple degradations” refers to datasets that contain a variety of degradation types, where each image exhibits only one specific type of degradation. Consequently, a model trained on a limited set of degradation types cannot be expected to perform optimally on all unseen complex degradations.

Achieving universal applicability to all possible degradation types, including unseen and mixed degradations, remains a highly challenging research goal. While it would be ideal to address such universal scenarios, it is beyond the current state-of-the-art capabilities of all-in-one restoration models, including AdaIR.

R3.11:Channel attention scores:

Given that low-frequency features offer a global perspective, we employ a channel attention mechanism in the L-H unit to transfer low-frequency features to the high-frequency branch. This approach helps high-frequency features avoid overemphasizing challenging regions. As illustrated in Figure 7, the channel attention unit assigns smaller weights to channels that overly focus on challenging regions, while allocating larger weights to other channels. This effectively reinforces the already recovered sharper regions, contributing to improved restoration performance.

R3.12:Clarification for Encoder+Decoder+AFLB:

The encoder is responsible for extracting deep features from the input, while the decoder focuses on reconstructing sharp, high-resolution features. During the encoding stage, the model may struggle to provide meaningful high- and low-frequency information for effective exchange, which can adversely affect overall performance. A similar observation has also been reported in PromptIR.

R3.13:Additional ablation studies of using separate attention:

Following the suggestion, we separately process low- and high-frequency features using channel attention mechanisms. The results presented in Table R1a and Table R1c demonstrate that our model achieves superior performance under this configuration. Additionally, we evaluated the use of separate spatial attention mechanisms in the two branches (Table R1b), which resulted in lower performance compared to our proposed approach.

R3.14:Citation for CBAM:

In the original paper, we cited this method in Appendix B (Design Choices of FMoM) and Figure 8. Following the suggestion, we have now ensured that this citation is included comprehensively throughout the manuscript wherever relevant.

R3.4:Rational behind different designs for H-L and L-H units:

High-frequency features are rich in spatial details, such as edges and textures. These features are transmitted to the low-frequency branch through a spatial attention mechanism, enabling it to focus on complex, localized regions. Conversely, low-frequency features provide global context, helping the high-frequency branch avoid overemphasizing challenging regions. To achieve this, we employ a channel attention mechanism that effectively integrates global information while assigning smaller weights to channels that overemphasize such regions, as demonstrated in Figure 7. The distinct designs are tailored to the characteristics of each frequency component. Spatial attention is well-suited for processing high-frequency features due to their localized nature, whereas channel attention is more effective for leveraging the global, structural information of low-frequency features. To demonstrate further the effectiveness of this design, we conducted additional experiments using alternative attention mechanisms for both units. The results, summarized in the table, show that our proposed design achieves the best performance, underscoring its effectiveness in facilitating cross-interaction between frequency components.

Table R1: Comparisons between different attention methods

R3.5:Explicit columns for PSNR/SSIM score:

Thank you for your suggestion. In response, we have explicitly added column headers to Tables 1, 2, and 12 to clearly indicate the metrics being reported, such as PSNR and SSIM.

R3.6:Comparisons with recent methods:

Thank you for your valuable suggestion. As recommended, we have incorporated several recent methods, including U-WADN [3], Art$_{\text{PromptIR}}$ [4], Gridformer [5], and InstructIR [6], into the three-task and five-task all-in-one experiments. The updated results, provided in Table R2 and Table R3, demonstrate that our method performs competitively and achieves state-of-the-art results against these approaches.

Table R2 Results for the three-task setting (Table 1 in the paper)

Table R3 Results for the five-task setting (Table 2 in the paper)

[3] Xu et al., Unified-width adaptive dynamic network for all-in-one image restoration, arXiv 2024.

[4] Wu et al., Harmony in diversity: Improving all-in-one image restoration via multi-task collaboration, ACMMM 2024.

[5] Wang et al., Gridformer: Residual dense transformer with grid structure for image restoration in adverse weather conditions, IJCV 2024.

[6] Conde et al., High-quality image restoration following human instructions, ECCV 2024.

R3.7:Visualizations:

We have included both the original and enlarged versions of the images in Figures 4–6 for better visualization. Our method is more effective in removing different degradations.

R3.8:Parameter comparisons:

Parameter comparisons have been incorporated into Tables 1 and 2. Our method performs favorably against state-of-the-art algorithms while maintaining a comparable parameter count.

R3.9:Clarification for fixed masks:

The term fixed indicates that a $10\times 10$ mask is utilized in our experiment, and this value remains constant throughout the entirety of the experiment.

We extend our sincere gratitude to the reviewer for their valuable time and insightful feedback. We value your constructive feedback and hope that our responses have appropriately addressed all the concerns.

We really appreciate the valuable time to respond to our feedback based on the reviewer's comments. Further, we are happy to address any remaining concerns.

I appreciate the authors' efforts in addressing the concerns raised. After reviewing the authors' responses and discussions with other reviewers, many issues have been resolved, and I have raised my score to 5. However, this does not indicate support for acceptance of this paper but reflects that the paper does not warrant rejection at a score of 3.

1. Lack of comparisons with frequency-based models

The authors did not provide the performance comparisons with frequency-based models [1-2] requested in the initial review. Although new experiments cannot be requested at this stage, I reiterate this point as the benchmark comparisons are crucial to validating the authors' stated motivation for this work.

The absence of experiments comparing the proposed method with frequency-based all-in-one restoration models would make it difficult to evaluate whether the proposed method is better than the previous frequency-based models, as the authors reply in R3.1.

I surveyed the performance of previous work [2] and compared it with AdaIR. AdaIR shows slightly lower performance (PSNR: -0.37, SSIM: -0.009) on BSD68 compared to previous work, suggesting the need for direct comparisons with similar approaches to validate the novelty.

2. Novelty and insufficient analysis of MGB

The MGB is said to adaptively mine frequency components, but no analysis shows how and why it is superior to existing methods like FFTformer [3], which uses a Quantization Matrix for frequency selection. Extracting important frequency components has been addressed in prior works, and aside from this, this is the only module that offers substantial novelty.

3. Citation concerns with the H-L unit

The authors renamed CBAM to the H-L unit without citation in the main text, which could mislead readers into thinking the H-L unit is newly proposed. While the authors added citations in the appendix, the main text must clarify this, e.g., "We use CBAM's spatial attention as the H-L unit." Proper attribution in the main text is essential to avoid misunderstanding.

[1] Wen Y, Zhang K, Zhang J, Chen T, Liu L, Luo W. Frequency-oriented efficient transformer for all-in-one weather-degraded image restoration. IEEE Transactions on Circuits and Systems for Video Technology. 2023 Jul 27.

[2] Shi, Zenglin, et al. "Learning Frequency-Aware Dynamic Transformers for All-In-One Image Restoration." arXiv preprint arXiv:2407.01636 (2024).

[3] Kong, Lingshun, et al. "Efficient frequency domain-based transformers for high-quality image deblurring." CVPR 2023.

R2.1:LPIPS evaluation:

As recommended, we assess the image reconstruction quality of our method against state-of-the-art approaches using the LPIPS metric. The results presented in the table indicate that our method surpasses the performance of existing approaches.

R2.2:Baseline performance:

As recommended, we have incorporated Restormer into our comparative analysis. Notably, our proposed method demonstrates a substantial performance improvement, achieving a PSNR gain of 1.94 dB over the baseline model.

R2.3:Comparisons with more recent methods:

Thank you for pointing out this paper. Compared with Art$_{\text{PromptIR}}$ [1], our method achieves an average performance improvement of 0.2 dB. We have included this work, along with several other recent methods, in our comparisons. Please refer to Tables 1 and 2 in the main paper for the updated results.

[1] Wu et al., Harmony in diversity: Improving all-in-one image restoration via multi-task collaboration, ACMMM 2024.

We extend our sincere gratitude to the reviewer for their valuable time and insightful feedback. We value your constructive feedback and hope that our responses have appropriately addressed all the concerns.

We really appreciate the valuable time to respond to our feedback based on the reviewer's comments. Further, we are happy to address any remaining concerns.

R1.1a:Out-of-distribution generalization experiments:

As suggested, we assess the generalization capability of our model on additional out-of-distribution degradation types: out-of-focus blurring and raindrops. Results show that our method demonstrates superior performance on these unseen degradation types compared to other competing approaches. (These results are also provided in Table 9 of the paper.)

R1.1b:Real-image visual example:

To provide qualitative results on real-world images, we use the UAVDT [3] dataset, which consists of images captured by UAVs at varying altitudes and exhibiting diverse levels of haze. Visual comparisons are provided in Figure 8 of the paper.

R1.2:Compare AdaIR with Specialized single-task approaches:

We appreciate your suggestion to compare AdaIR’s performance with specialized single-task models for a more comprehensive evaluation. However, we would like to clarify the following points. (1) SOTA single-task models are typically designed and optimized for handling a specific type of degradation. As a result, they inherently benefit from task-specific architectures, making them more capable of achieving superior performance compared to all-in-one architectures. (2) To ensure a fair and consistent evaluation, we followed the experimental protocols established by prior works such as PromptIR and AirNet. This allows us to avoid retraining all the competing methods and simply extend existing quantitative tables. (3) All-in-one methods (PromptIR, AirNet, AdaIR, etc.) use much smaller training datasets compared to single-task models. Comparing AdaIR directly with single-task SotA models would require retraining these specialized models on the same dataset used for AdaIR. Unfortunately, this is not feasible within the limited time and computing resources we have.

R1.3:Computational comparisons:

We have incorporated FLOPs comparisons in Table 17. Notably, despite the additional processing introduced by Fourier transformations, our model demonstrates competitive computational efficiency. Specifically, AdaIR operates 1.28x faster than the PromptIR approach, while achieving a significant performance improvement of 0.63 dB in PSNR and utilizing 19% fewer parameters.

We acknowledge the potential concerns regarding compatibility with edge platforms. However, this challenge is not unique to AdaIR and applies broadly to all the compared approaches. For deployment on edge devices, further optimization would be necessary to reduce AdaIR’s computational requirements, which could be achieved by reducing model capacity.

R1.4:Additional references:

We thank the reviewer for providing valuable references to related works. These references have now been included in the Related Work section for a more comprehensive discussion.

R1.5:Comparison with adaptive-filtering approach:

As suggested, we conduct experiments using an adaptive method for frequency decoupling [4]. This variant achieves a PSNR of 30.25 dB, which is 0.99 dB lower than the performance of our AdaIR algorithm.

R1.6:Masks for different tasks:

We have included visualizations for additional degradation types in Figure 11 of the paper. As seen, our model can adaptively decouple the images into different frequency bands guided by the generated masks.

R1.7:How the interaction contributes to restoration:

Figure 7 demonstrates the complementary relationship between different frequency bands. Specifically, high-frequency features aid the low-frequency branch by emphasizing spatial edges, while low-frequency features assist the high-frequency branch in avoiding excessive emphasis on challenging areas. Consequently, the combined features enhance the sharpness of the input features, facilitating high-quality reconstruction, as depicted in Figure 12.

R1.8:Frequency differences:

Figure 1 illustrates that the amplitude variations across tasks highlight how different degradations affect images at distinct frequency ranges.

[1] Abuolaim et al., Defocus deblurring using dual-pixel data, ECCV 2020.

[2] Qian et al., Attentive generative adversarial network for raindrop removal from a single image, CVPR 2018.

[3] Du et al., The unmanned aerial vehicle benchmark: Object detection and tracking, ECCV 2018.

[4] Zhou et al., Seeing the unseen: A frequency prompt guided transformer for image restoration, ECCV 2024.

We extend our sincere gratitude to the reviewer for their valuable time and insightful feedback. We value your constructive feedback and hope that our responses have appropriately addressed all the concerns.

We really appreciate the valuable time to respond to our feedback based on the reviewer's comments. Further, we are happy to address any remaining concerns.