FourierMamba: Fourier Learning Integration with State Space Models for Image Deraining

Q1: Although the motivation for zigzag scanning is well-discussed, the paper lacks details on its implementation. Considering the complexity involved, the implementation is non-trivial and deserves attention. The way zigzag scanning is executed, along with pixel organization strategies, can significantly impact inference speed. Including a comparison of running times would enhance the evaluation. (Implementation and efficiency of zigzag scanning.)

A1: In fact, our implementation of zigzag scanning involves two steps: obtaining the path and executing the scan. We obtain the scanning path by combining the zigzag scanning algorithm used in JPEG compression with the characteristics of the Fourier domain. For instance, given a Fourier spectrum, we take the high frequency in the upper left corner as the starting point, scan to the low frequency in the center according to the path of the ziazag algorithm, and then return from the low frequency to the high frequency in the upper right corner based on the symmetry of the Fourier spectrum. This process generates a complete scanning path. Then, we save the obtained path as a list, and then each time we run the scan, we only need to read the index in the list to scan according to the preset path, thereby significantly reducing running times. We compare the runtime of our method with several others using 512×512 images on an NVIDIA 4090 GPU. The results, shown below, indicate that the runtime of our method is comparable to that of other approaches. According to the reviewer’s suggestion, we have provided this comparison result in Appendix A.3.

Table 1：Runtime comparison between our method and other methods.

Q2: The visual comparison baselines could also be updated. It would be beneficial to include comparisons with more recent methods (post-2022) and provide additional zoomed-in patches or error maps to better highlight performance differences.

A2: According to the reviewer’s suggestion, we have updated the visual comparison baseline by including visual results of more recent methods and providing additional zoomed-in patches, as shown in Figures 5 and 6 of the revision.

Q3: There are several confusing typographical issues, such as "FCS" being used as the abbreviation for "Fourier Channel Evolution" and "C-FFT" lacking a clear explanation (possibly referring to "channel FFT").

A3: Thank you for the reviewer's corrections. We have addressed these issues in the revision. We have updated the abbreviation for "Fourier Channel Evolution" to "FCE." Additionally, "C-FFT" indeed refers to the channel-dimension Fourier transform.

Q4: The paper overlooks several closely related works.

A4: Thank you for the reviewer’s suggestion. Wavelet Approximation-Aware Residual Network for Single Image Deraining proposes a wavelet approximation-aware residual network, which efficiently removes rain from low-frequency structures and high-frequency details at each level separately. A Hybrid Transformer-Mamba Network for Single Image Deraining introduces a network combining Transformer and Mamba to capture long-range dependencies related to rain. Image Deraining with Frequency-Enhanced State Space Model achieves effective deraining by parallelizing frequency-domain processing branches with the Mamba branch. In contrast to these methods, our work explores how to integrate information across all frequency bands in the Fourier domain using the Mamba architecture, fully leveraging the complementarity between different frequency bands to enhance image deraining performance. According to the reviewer’s suggestion, we have included these closely related studies in the related work section of the revision to provide a more comprehensive overview of the research background and the connections between the methods.

The comparison results for image dehazing are presented in the following table.

Table 4: Comparison of methods on Dense-Haze and NH-HAZE datasets.

It can be seen that our method demonstrates good generalization and potential for other low-level vision tasks beyond rain removal. According to the reviewer’s suggestion, we have provided the results for these additional tasks in Appendix A.8 of the revised version.

[1] Focal Frequency Loss for Image Reconstruction and Synthesis, 2021.

[2] Fourmer: An Efficient Global Modeling Paradigm for Image Restoration, 2023.

[3] FreqMamba: Viewing Mamba from a Frequency Perspective for Image Deraining, 2024.

Q5: Applying the L1 norm directly in the frequency domain may be less accurate than alternative metrics, such as coherence-based distances. An ablation study on the impact of different frequency-domain loss functions would strengthen the analysis.

A5: According to the reviewer’s suggestion, we evaluate three additional frequency-domain loss functions—Phase Consistency Loss (PCL), Frequency Distribution Loss (PDL), and Focal Frequency Loss (FFL) [1]. The Phase Consistency Loss measures the mean squared error of the phase difference between two images in the frequency domain, used to assess the similarity of the phase information in the frequency domain between the two images. The Frequency Distribution Loss quantifies the difference in the frequency domain amplitude distributions of two images, comparing the amplitude distributions of the reconstructed image and the reference image in the frequency domain to measure their similarity. The Focal Frequency Loss adaptively focuses on frequency components that are hard to synthesize by down-weighting the easy ones. We perform ablation experiments on these loss functions as follows.

Table 2: Comparison results of different frequency-domain loss functions.

The results indicate that the performance achieved with these four loss functions is similar. The focus of this work is on the design of the network architecture, so we follow existing methods [2] to use the L1 norm in the frequency domain. We will explore more frequency-domain loss functions in future work. According to the reviewer’s suggestion, we have provided the description and results of the above ablation experiments in Appendix A.19 of the revision.

Q6: It would be valuable to evaluate the performance of the proposed method on other low-level vision tasks. Demonstrating its effectiveness beyond the deraining scenario would provide further evidence of its broader applicability.

A6: The reviewer’s suggestion is insightful. Following FreqMamba [3], we evaluate our method on low-light enhancement and image dehazing. We use the LOL-V1 and LOL-V2-synthetic datasets to evaluate the performance of our method on low-light enhancement, and the Dense-Haze and NH-HAZE datasets are used to evaluate the performance of our method on real-world image dehazing. The results for low-light enhancement are shown in the table below.

Tabel 3: Comparison of methods on LOL-V1 and LOL-V2-Syn datasets.

Dear Reviewer NHz8:

Thanks for your valuable suggestions and recognition. Your recognition means a great deal to us. We sincerely appreciate your effort and support once again!

Best Regards,

Authors of #8897

Q3: The quantitative comparison is conducted with respect to PSNR and SSIM. However, as more and more researchers realized, these two metrics can be often inconsistent with human perceptions and thus some newly developed metrics that can better reflect human perceptions should be considered.

A3: The reviewer's suggestion is very insightful. We use widely adopted perceptual metrics, including BRISQUE, NIQE, and SSEQ, to conduct quantitative evaluations, with some results shown in the table below. The complete evaluation results are provided in Appendix A.17. In future work, we plan to explore additional metrics that better reflect human perception.

Tabel 2：Performance comparison of different methods on Test2800 dataset.

Q4: In Section 4.1, the authors mentioned the "progressive training strategy". What does it mean? And why is it called "progressive"?

A4: The progressive learning strategy refers to training the network on smaller image patches during the initial stages and gradually increasing the patch size in later stages. This approach is referred to as "progressive" because the resolution increases incrementally. Many previous methods, such as Restormer and MambaIR, also adopt this training strategy.

Q5: There are several writing issues that I can find.

A5: Thank you to the reviewer for pointing these out. We have addressed and resolved these issues in the revised version.

Q1: It is unclear why Mamba is more effective than other architectures for processing the Fourier frequencies. Visualizing the intermediate features might be useful.

A1: First, Mamba utilizes sequence modeling to integrate information across all frequency bands, effectively leveraging the complementary relationships between different bands. In contrast, convolution, as a local operation, struggles to holistically model global features across all frequency bands when processing frequency information in the Fourier domain. This limitation significantly constrains its capacity in the Fourier space.

Second, Mamba's sequence modeling is orderly, which can help the network establish an orderly dependency relationship between different frequencies. This characteristic is critical for modeling image degradation information. Conversely, convolution is insufficient in capturing the dependencies between high and low frequencies in the Fourier domain, thereby weakening its ability to accurately represent degradation features.

In summary, based on these two advantages, Mamba achieves better coordination of high-frequency and low-frequency information in the Fourier domain during the image restoration process. On the one hand, it effectively captures rain streaks, and on the other, it enhances background reconstruction, significantly improving the quality of image restoration. According to the reviewer's suggestion, we have shown feature visualizations in Figure 8 of Appendix A.11.

Q2: It is better to discuss more about wavelet transformation and Fourier transformation in dealing with frequencies. Besides, as shown in Table 1, the performance of (Zhen et al., 2024) is close to the proposed method on several datasets, but the comparison with this method on SPA-Data is missing.

A2: FreqMamba (Zhen et al., 2024) employs Mamba scanning in the wavelet-transformed domain, which lacks the unique advantages of the Fourier domain. In image restoration tasks, the Fourier domain offers two critical characteristics: 1) Decoupling of degradation and background content: The Fourier transform can decouple the degradation components and content components of an image to some extent, effectively providing a prior related to degradation information. 2) Global characteristics: Each pixel in the Fourier domain is associated with all pixels in the spatial domain, offering a global perspective that is particularly important for capturing long-range correlations.

However, the design of FreqMamba is based on the wavelet transform, which divides the image into several patches and performs spatial domain scanning within each patch. Since the wavelet-transformed domain lacks the global characteristics of the Fourier domain, FreqMamba has relatively limited capability for modeling the correlations between frequencies.

To further evaluate the performance difference, we use FreqMamba's recently released open-source code to train and test on the SPA-Data dataset. The results are shown below.

Table 1: Quantitative comparison with FreMamba on the SPA-Data.

It can be seen that the performance of our method is better than FreqMamba. According to the reviewer’s suggestion, we have provided this comparison result in Appendix A.15.

Furthermore, the performance of FreqMamba shown in Table 1 of the manuscript is actually obtained by training and testing separately on each dataset. In contrast, like most other methods, we train on Rain13k and then test on individual datasets. This discrepancy may lead to an overestimation of FreqMamba's performance in Table 1 of the manuscript. Therefore, we provide the performance of FreqMamba using our experimental settings in Table 11 of Appendix A.9. Experiments show that under the same experimental conditions, our method can achieve better performance than FreqMamba.

The authors have addressed my concerns, and thus I decide to raise my rating.

Dear Reviewer xRKZ:

Thanks for your acknowledgment of our work and responses. We'll carefully revise our final paper. Your positive rating means a lot to us. We appreciate your constructive feedback that has helped refine our research.

Best Regards,

Authors of #8897

It can be seen that under the same experimental conditions, our method outperforms FreqMamba across all metrics. We provide a detailed description and results of this comparison in Appendix A.9. Additionally, to further validate our performance, we also train and test our method on the Test2800 dataset using the experimental setup of FreqMamba (see Appendix A.16). The results of the above experiments demonstrate that our method outperforms FreqMamba.

Q3: It is worth noting that FFT is a complex valued computation. Suggest the author to add a comparison of the actual inference time of the model.

A3: I understand the reviewer's concern regarding the use of the Fast Fourier Transform (FFT), which indeed involves complex-valued computations. However, the actual inference time of our model is not significantly higher than that of other methods. This is primarily due to two reasons. First, our network only uses half the number of channels and blocks of MambaIR, which inherently reduces runtime. Second, we leverage PyTorch's built-in torch.fft, which benefits from optimized FFT operations and hardware acceleration. The inference time comparison on 512×512 images using an NVIDIA 4090 GPU is presented below. According to the reviewer’s suggestion, we have provided a comparison of inference times in Appendix A.3.

Table 3: Runtime comparison between our method and other approaches.

Q1: The experiment is insufficient and lacks comparison with methods such as DRSformer and FADformer.

A1: According to the reviewer's suggestion, we add comparisons with methods such as DRSFormer and FADFormer. Following their experimental settings, we evaluate our method on Rain200L, Rain200H, DID-Data, DDN-Data, and SPA-Data. The comparison results are provided below.

Table 1: Performance comparison of methods across various datasets.

It can be seen that our method consistently achieves superior deraining performance on most datasets. We have provided the complete comparison of this experimental setup in Appendix A.10.

Q2: The difference between the results shown in Table 1 and FreqMamba is not significant. Please further explain the differences between the proposed method and FreqMamba, as they both use Fourier correlation strategies. It is recommended to add visualization results to supplement this.

A2: Our method focuses on customized design based on the characteristics of Fourier space, combining Fourier priors with state space models and exploring the potential of introducing Mamba directly in the Fourier domain. In contrast, FreqMamba operates in the Fourier space using only 1×1 convolutions, which fails to fully utilize the rich frequency information inherent to the Fourier domain.

Specifically, FreqMamba applies Mamba scanning in a wavelet-transformed domain. However, the wavelet-transformed domain lacks the notable advantages of the Fourier domain, such as the Fourier transform's ability to decouple degradations and its global representation properties. Additionally, after wavelet decomposition, FreqMamba divides the image into multiple patches and performs spatial scanning within each patch. This design limits FreqMamba's ability to effectively model frequency correlations.

In contrast, our method performs Mamba scanning directly in the Fourier domain, fully leveraging the global characteristics of the Fourier transform. This allows our approach to better capture rain streaks, which often exhibit high apparent repetitiveness. Consequently, from a visual perspective, our method demonstrates significantly better performance in removing rain streaks. We present the relevant visual comparisons in Figure 7 of the Appendix A.9, clearly illustrating this advantage.

Furthermore, the quantitative results of FreqMamba listed in Table 1 of the manuscript are taken directly from its original paper, where training and testing are performed on different datasets. This training approach may explain why the performance gap between our method and FreqMamba appears less pronounced. For a fairer comparison, we retrain FreqMamba using our experimental setup and evaluate it on various datasets. The results are shown in the table below.

Tabel 2: Performance comparison with FreqMamba.

Dear Reviewer 1tV5:

Thanks for your valuable comments! This has greatly encouraged us and made our paper more complete and coherent. It would be very appreciated if you raise your rating since we have generally addressed your concerns. Your support is really important for our work. Thanks again!

Best Regards,

Authors of #8897

Table 6: Performance comparison of methods across various datasets.

It can be observed that the combined effect of Mamba and the Fourier prior enables us to achieve superior performance. We provide the complete comparison results for this experimental setup in Appendix A.10.

Q5: Regarding the experimental results, the paper only provides results on four datasets in Tab.1, lacking the test100 dataset. Moreover, the competitors in Fig.6 are a little outdated. From my side, the experiments are insufficient. Specifically, compared to other mamba-based methods, the evaluation data/tasks are fewer. Compared to other deraining methods like drsformer, the authors adopt only a rain13 dataset plus spad for evaluation, which is also insufficient.

A5: We add comparisons on the Test100 dataset, as presented below. According to the reviewer’s suggestion, we have added results on the test100 dataset in Appendix A.4.

Tabel 3: Performance comparison on Test100.

In response to the reviewer's comment that the comparison methods in Figure 6 are outdated, we have added visual results of several recent methods, as shown in Figure 6 of the revised version.

In response to the reviewer's comment regarding the limited number of evaluation tasks, we follow FreMamba and conduct experiments on low-light enhancement and image dehazing tasks. For low-light enhancement, we use the LOL-V1 and LOL-V2-Synthetic datasets for evaluation, as shown in the table below.

Tabel 4: Comparison of methods on LOL-V1 and LOL-V2-Syn datasets.

For real-world dehazing, we evaluate on the Dense-Haze and NH-HAZE datasets, with results presented in the table below.

Table 5: Comparison of methods on Dense-Haze and NH-HAZE datasets.

It can be demonstrated that our method is effective for other image restoration tasks. We have added the results of the evaluation on these two tasks in Appendix A.8.

According to the reviewer's suggestion, we conduct evaluations on Rain200L, Rain200H, DID-Data, DDN-Data, and SPA-Data based on the experimental settings of DRSFormer. A subset of the comparison results is presented below.

The 1×1 convolution merely serves as a linear combination of input channels, limiting its interactions to a single frequency band. This is inadequate for handling the complex, periodic nature of rain streak noise. In contrast, our proposed channel Mamba architecture employs sequential modeling to fully integrate information across all frequency bands. This allows it to utilize non-rain frequency bands to effectively suppress rain streak noise. This deep interaction of all frequency bands not only achieves accurate removal of rain streaks but also avoids damaging the image content.

Q3: Moreover, the idea of interactions between frequencies is already implemented in Fourmer via convolutions.

A3: Fourmer processes information in the Fourier space solely through convolution, which limits its utilization of Fourier priors in the following ways:

1. Locality limits global modeling capabilities: Convolution is inherently a local modeling operation, while rain streak noise exhibits multi-scale characteristics and is often distributed across the entire image. Relying only on convolution to process frequency information makes it difficult to sufficiently model and interact with rain streak noise across all scales, thereby restricting the ability to capture global features. In contrast, our approach, based on Mamba, integrates global information across different frequency bands through sequential modeling. This method is better suited to the characteristics of rain image restoration, as rain streak noise has a multi-scale distribution, strong repetitiveness, and involves both high- and low-frequency degradation information.

2. Lack of ordered dependency modeling between frequencies: Convolution cannot establish ordered dependencies between different frequencies in the Fourier space, which are critical for modeling image degradation. Our method, utilizing Mamba, effectively captures the ordered relationships between high and low frequencies, enabling a more comprehensive representation of rain streak degradation features.

3. Insufficient interaction in the channel dimension: Fourmer concatenates multiple channels along the channel dimension and processes them with a simple 1×1 convolution, resulting in outputs that are merely linear combinations of each channel. This approach does not fully exploit the interaction between channels, thereby limiting the expressiveness of the features. In contrast, our method introduces a channel Fourier Transform (CFT) in the channel dimension, followed by Mamba to scan and integrate information in the frequency domain. This allows for comprehensive integration of all frequency bands along the channel dimension, enabling different bands to complement and compensate for each other, resulting in more effective and precise rain streak removal.

In conclusion, our design not only addresses the limitations of convolution in the Fourier space but also aligns better with the specific requirements of rain image restoration. It significantly enhances global modeling capability and improves the effectiveness of information interaction.

Q4: The authors replace mamba with 1x1 conv for the first ablation studies. However, the flops or parameter comparisons are not provided. Is it possible that the improvement of mamba versions is derived from the additional computation overhead compared to the 1x1 conv version?

A4: Following the reviewer's suggestions, we perform a revised ablation study on the FSI-SSM module in Table 3 of the manuscript. By further reducing the number of channels and blocks, we compress our model to match the computational cost of the model without FSI-SSM and compare the results, as shown in the table below. The results show that Mamba still achieves better performance.

Table 1: The computational overhead of the ablation study on FSI-SSM.

In addition, the ablation study on w/o FCE-SSM in Table 3 of the manuscript already considers the impact of computational cost. The variant without FCE-SSM (w/o FCE-SSM) stacks several cascaded 1x1 convolutions with residual connections to achieve a computational cost similar to that of Mamba. The computational costs of w/o FCE-SSM and our proposed method are provided below. We have added these results in Appendix A.5.

Tabel 2: The computational overhead of the ablation study on FCE-SSM.

Q1: The paper lacks the theory analyses to support the importance of frequency interactions for image restoration.

A1: The goal of the image deraining task is to restore image information damaged or obscured by rain streaks. Introducing a frequency interactions mechanism enables complementary integration of information across different frequency bands, effectively reducing the risk of loss or bias caused by relying on a single frequency band. This is because rain streaks impact image information in varying ways across different frequency bands. By modeling these bands separately and fusing them through interactions, the mechanism allows for more efficient and comprehensive image restoration. From a theoretical perspective, frequency interaction is essentially equivalent to a multi-scale processing method, enabling the model to simultaneously capture both global and local features. In the frequency domain, images often exhibit sparsity, with most energy concentrated in the low-frequency region and relatively little in the high-frequency region. This sparsity inherently provides an implicit prior constraint for frequency band interaction, functioning similarly to a regularization mechanism. When high-frequency and low-frequency information are not well-aligned, artifacts resembling rain streaks may appear in the restored image. However, the frequency band interaction mechanism can effectively smooth these artifacts while enhancing authentic high-frequency details, thereby improving image restoration quality. The theoretical values of frequency interactions can be summarized as follows:

1. Multi-scale modeling: Through frequency division, the model can simultaneously process global and local characteristics, capturing both structural and detailed features of the image comprehensively.

2. Information complementarity: Information from different frequency bands can complement each other, mitigating potential deficiencies that arise from processing a single frequency band.

3. Noise suppression: In the frequency domain, frequency interaction effectively filters out interference, smoothing high-frequency artifacts.

4. Implicit regularization: Through sparsity constraints, frequency interaction provides prior knowledge for image restoration, enhancing the generalization ability of the model.

5. Fusion of global and local features: The frequency interactions mechanism organically integrates characteristics at different scales, improving the model’s robustness and accuracy.

In conclusion, the frequency interactions mechanism provides multifaceted theoretical support and practical advantages for the image deraining task. It significantly enhances the robustness and quality of image restoration, establishing a solid basis for high-quality image draining.

Q2: The channel fft has already been proposed in REVITALIZING CHANNEL-DIMENSION FOURIER TRANSFORM FOR IMAGE ENHANCEMENT. Why does the channel mamba outperform the 1x1 channel version? The reviewer thinks a 1x1 convolution can also model interactions between channel spectra.

A2: REVITALIZING CHANNEL-DIMENSION FOURIER TRANSFORM FOR IMAGE ENHANCEMENT introduces the channel-dimension Fourier Transform (FFT), but our study differs from this work in several key aspects.

First, in terms of research purpose, the referenced paper focuses on image enhancement tasks, leveraging channel-dimension FFT primarily to enhance the discriminative ability of global features and thereby improve image expressiveness. In contrast, our study focuses on image deraining, which belongs to the field of image restoration. The core objective of this task is to accurately extract and represent degraded information. Therefore, we utilize channel-dimension FFT to enrich the global representation of degradation features. This highlights a fundamental difference in motivation between our work and the referenced paper.

Second, in terms of implementation, the core operation in the referenced paper is a 1×1 convolution, which facilitates relatively simple interactions across different frequency bands. However, for image deraining, especially in addressing high-frequency rain streak noise, this approach has two major limitations: (1) Rain streak noise often gets confused with the structural edges of objects, making it difficult for simple interactions to effectively distinguish between the two; (2) Rain streaks exhibit multi-scale characteristics, and relying solely on 1×1 convolution fails to capture the complex relationships between frequency bands. To address these challenges, we design the Mamba architecture for image deraining, incorporating channel-dimension FFT to specifically target the modeling of degraded information in the frequency domain.

Dear Reviewer yyRc:

Thanks for your valuable suggestions! Your feedback is incredibly meaningful to us. If we have addressed your concerns, we earnestly look forward to you reconsidering your score. Your support is really important to us. Once again, thanks for your time and effort!

Best Regards,

Authors of #8897