Improving the Learning Capability of Small-size Image Restoration Network by Deep Fourier Shifting

1,generalization.

Fourier shifting exhibits enhanced generalization due to its operation in the frequency domain, which captures global image information and maintains feature integrity while minimizing domain-specific artifacts. This approach is particularly advantageous for managing low-light degradation, a global phenomenon effectively addressed in the Fourier domain. By incorporating this domain-specific knowledge, our method benefits from a natural prior that significantly improves adaptability across diverse datasets. Moreover, Fourier shifting is more parameter-efficient and reduces artifacts compared to spatial shifting, which often suffers from issues like information loss and frequency aliasing. This leads to more robust feature extraction and consistent performance across varied scenarios. Our revised paper will thoroughly discuss these advantages, including the Fourier domain's role in addressing global degradations and how it contributes to the superior generalization of our method, ensuring better results across different real-world applications.

2,variants.

Like the spatial convolution techniques, our proposed “Fourier Up-Sampling” is a drop-in alternative for convolution operator, also the general strategy and each variant thus does not prefer a specified vision task. The two Fourier shifting variants each have specific strengths and potential drawbacks. The Amplitude-Phase Variant processes amplitude and phase components separately, which is beneficial for tasks requiring precise phase preservation. However, this variant involves trigonometric functions, which can introduce numerical instability due to the amplification of small deviations in angle, affecting precision-sensitive applications. On the other hand, the Real-Imaginary Variant processes the real and imaginary parts separately, offering greater numerical stability and being suitable for tasks involving complex textures or fine-grained restoration. In our revised paper, we will provide detailed guidance on choosing between these variants based on the task requirements, baseline characteristics, and potential numerical stability issues.

1, types.

We appreciate the suggestion and will expand our experiments to include various types of images with different textures, structures, and noise characteristics. This will provide a more comprehensive assessment of how Deep Fourier Shifting performs under a wider array of conditions, enhancing the manuscript’s applicability and robustness.

2, Complexity.

We appreciate the feedback on the practical implementation of our proposed method on edge devices. To address potential challenges, we will discuss key aspects such as memory usage, real-time processing capabilities, and adaptability to different hardware platforms. This will include information on memory requirements, real-time processing metrics like latency and throughput, and evaluations on various hardware configurations, including CPUs, GPUs, and specialized edge processors. By addressing these aspects, we aim to enhance the manuscript's practical relevance and demonstrate the feasibility of deploying our method in real-world edge device applications. Due to time constraints, we tested our method on the DnCNN model. In Table 5, we compared our Fourier shifting operator with the spatial shift operator and the baseline 3x3 convolution. Our Fourier shifting approach not only halved the number of parameters but also improved network performance, whereas the spatial shift operator caused a significant 14 dB performance drop. Additionally, we assessed runtime and FLOPs to evaluate suitability for edge devices. Our results show that the DnCNN-18 model with Fourier shifting achieved an average runtime of 7 milliseconds and 1.5 GFLOPs per 256x256 image on an NVIDIA GTX 1080 GPU, compared to 15-20 milliseconds and 3.9 GFLOPs for the baseline. This substantial reduction in both runtime and FLOPs highlights the efficiency of our method for resource-constrained hardware.

3, onwards.

Thank you for your observation. We will update the manuscript to include recent studies and advancements published in 2023 and onwards. Due to time constraints, we have conducted additional experiments using the recent work LFormer [1] presented at ACM-MM 2024. Specifically, we replaced the convolution operator in feature extraction with Fourier shifting to evaluate its impact on guided image super-resolution on world-view-II and Gaofen-2 satellites

Incorporating our proposed Fourier shifting design into the LFormer framework has demonstrated significant improvements in performance. The results indicate that our method not only enhances the accuracy of guided image super-resolution but also improves the efficiency of the feature extraction process. This advancement is evident from the substantial gains in quantitative metrics, underscoring the effectiveness of our approach in optimizing and refining the image restoration capabilities of the model. Moving forward, we will incorporate additional methodologies to further validate and enhance the efficiency of our approach.

[1] Linearly-evolved Transformer for Pan-sharpening, ACM-MM 2024.

1, Fig.6.

Thank you for your feedback. Firstly, the Fourier transform is an efficient tool that amplifies image degradations in the Fourier domain, and our learnable parameters act as filters to eliminate these artifacts. Secondly, the artifacts arise from two aspects: (1) features are extracted from low-light degraded inputs, which naturally reflect these degradations, and (2) the testing baseline network inherently uses down-sampling operators to achieve multi-scale features, where the down-sampling inherently causes frequency truncation, leading to frequency aliasing and ringing effects. Additionally, to verify the robustness of our experiments, we analyzed both shallow and deep features in the network, finding that similar artifacts persist and even accumulate, whereas our Fourier shifting can eliminate these, resulting in cleaner features. Our frequency domain approach inherently handles these degradations. The input discrepancies are due to the network being trained from scratch after incorporating our operator, resulting in natural feature changes. Our network effectively reduces degradations both before and after the application of shift-sa, while the degradation persists with shift-sa. Finally, we used mutual information to measure information loss caused by shifting, which validates the effectiveness of our method.

2, MI.

In Figure 6 (left), mutual information is calculated by first extracting feature maps before and after applying the shift operator and averaging the channel dimensions of these feature maps. These averaged values are then used as the variables for mutual information computation. Joint and marginal histograms of the pixel intensities are converted into probability distributions. This measure assesses the amount of shared information between the feature maps, indicating how well the shift operator preserves feature information.In terms of the index representing the network depth, we have tested multiple stages of the network, ranging from shallow to deep layers, to demonstrate the completeness of information retention. Relevant experiments have also confirmed that our proposed shifting method effectively preserves information throughout the network. Our operator exhibits significantly higher mutual information than Shift-sa, showcasing its efficacy in information preservation.

3, real-world.

Thank you for your feedback. Due to time constraints, we initially conducted experiments on real-world remote sensing satellite scenes, which involve more complex degradations compared to the simulated scenarios with simple down-sampling and blurring. The results were as follows:

• PANNET:
  • D_{\lambda} = 0.0737
  • D_s = 0.1224
  • QNR = 0.8143

However, with our newly designed network, the updated metrics are:

• New Network:
  • D_{\lambda} = 0.0716
  • D_s = 0.1215
  • QNR = 0.8156

Here, lower values of D_s and D_{\lambda}, along with a higher QNR, indicate better performance. These improvements demonstrate that our approach continues to perform effectively in real-world scenarios. We will update the paper to reflect these results and emphasize the effectiveness of our method in practical applications.

4, comlexity.

We understand that the paper’s content might not fully align with the title. To address this, we will revise the paper to better demonstrate how our method enhances small-size restoration networks. This revision will include a thorough discussion of its impact on parameters, FLOPs, and inference time, particularly for tasks such as low-light image enhancement using networks like SID. We will ensure that the paper clearly explains the application and evaluation of our proposed concept in this context. We also acknowledge that the implementation details might be unclear. To clarify, we will explicitly define the baseline or original model used for comparison, including a detailed description of the model architecture, configurations, and any modifications. Due to time constraints, we tested our method on the DnCNN model. In Table 5, we compared our Fourier shifting operator with the spatial shift operator and the baseline 3x3 convolution. Our Fourier shifting approach not only halved the number of parameters but also improved network performance, whereas the spatial shift operator caused a significant 14 dB performance drop. Additionally, we assessed runtime and FLOPs to evaluate suitability for edge devices. Our results show that the DnCNN-18 model with Fourier shifting achieved an average runtime of 7 milliseconds and 1.5 GFLOPs per 256x256 image on an NVIDIA GTX 1080 GPU, compared to 15-20 milliseconds and 3.9 GFLOPs for the baseline. This substantial reduction in both runtime and FLOPs highlights the efficiency of our method for resource-constrained hardware.

1, Complexity.

Thank you for your feedback. In Table 5, we compared the performance of our proposed Fourier shifting operator against the spatial shift operator and the baseline 3x3 convolution. Replacing convolutions with Fourier shifting not only reduced the number of parameters by half but also improved network performance, whereas the spatial shift operator led to a significant 14 dB performance drop. Additionally, we evaluated runtime and FLOPs to assess suitability for edge devices. Our experiments show that the DnCNN-18 model with our Fourier shifting configuration achieves an average runtime of 7 milliseconds and 1.5 GFLOPs per 256x256 image, compared to 15-20 milliseconds and 3.9 GFLOPs for the baseline on an NVIDIA GTX 1080 GPU. This indicates a substantial reduction in both runtime and FLOPs, highlighting the efficiency of our method for deployment on resource-constrained hardware. We will update Table 5 to include these metrics for a comprehensive evaluation. The lower portion of Table 5 uses a controlled variable approach to ensure consistent parameters while evaluating the network's robustness to shifting size. It is important to note that the spatial-shifting configurations were not aligned with our method's parameters throughout. Our results demonstrate that our network exhibits high robustness to varying shifting sizes.

2, artifacts.

Thank you for your feedback. Firstly, the Fourier transform is an efficient tool that amplifies image degradations in the Fourier domain, and our learnable parameters act as filters to eliminate these artifacts. Secondly, the artifacts arise from two aspects: (1) features are extracted from low-light degraded inputs, which naturally reflect these degradations, and (2) the testing baseline network inherently uses down-sampling operators to achieve multi-scale features, where the down-sampling inherently causes frequency truncation, leading to frequency aliasing and ringing effects. Additionally, to verify the robustness of our experiments, we analyzed both shallow and deep features in the network, finding that similar artifacts persist and even accumulate, whereas our Fourier shifting can eliminate these, resulting in cleaner features. Our frequency domain approach inherently handles these degradations. The input discrepancies are due to the network being trained from scratch after incorporating our operator, resulting in natural feature changes. Our network effectively reduces degradations both before and after the application of shift-sa, while the degradation persists with shift-sa. Finally, we used mutual information to measure information loss caused by shifting, which validates the effectiveness of our method.

3,minor issue.

Thank you for your detailed review and valuable feedback. We will carefully review the entire manuscript to correct any typos and grammar errors to improve the overall clarity and quality of our work.