Decomposition Ascribed Synergistic Learning for Unified Image Restoration

Thank you so much for your valuable comments. Actually, the well presentation of the paper has been widely appreciated by other three reviewers, and the attractive and compelling insight has also been broadly acknowledged. We suggest the dear reviewer to reconsider the contribution of this paper. Your kind suggestions have been carefully incorporated in the revised version. In the following, we first revisit the main contribution of our paper and then followed by point-by-point responses, which hope address your concerns well.

Recall the main contribution of the paper.

[1] The observed SVD-ascribed degradation analysis presents that the decomposed singular vectors and singular values of the image naturally undertake the different types of degradation information, ascribing various restoration tasks into two groups, i.e., singular vector dominated degradations, such as rain, noise, blur, and singular value dominated degradations, such as low-light, haze.

[2] The intention of the proposed DASL is to support the decomposed optimization of singular vectors and singular values of the degraded signal, through two developed operators, i.e., SVEO and SVAO. Note that both of them decently evade the explicit time-consuming SVD on signals. Therefore, the proposed DASL can achieve superior performance with even accelerated inference time, compared to baseline models (Tab. 5 of the paper).

[3] SVEO only optimize the singular vectors of the signal without touching singular values. We perform this by the fact that the orthogonal matrices multiplication only impacts singular vectors (Theorem 1 in sec. 3.2).

[4] SVAO only optimize the singular values of the signal which is decoupled from the optimization of singular vectors. We perform this by resort to the Fast fourier transform (FFT) as the transformed fourier coefficients undertake the same role as singular values in terms of signal formation principle, i.e., the combined coefficients for a set of basis (detailed in sec. 3.3). Note that FFT is substantially faster than SVD (516x), as shown in Tab. 1 of the paper.

Briefly, the proposed decomposed optimization in DASL built upon SVD-ascribed degradation analysis could greatly benefit the multiple degradation processing within a single model, where the potential degradation relationship within ascribed groups and the conflict between groups could be inherently utilized and alleviated.

About SVD entity and how SVD represents the image.

The proposed SVD-ascribed degradation analysis is performed in image space, namely, the decomposed singular vectors and singular values of the image naturally undertake the different types of degradation information. In fact, SVD operation provides us $X=U\Sigma V^T$ for signal decomposition. In the opposite, we can also represent the original signal $X$ with decomposed singular vectors $U$, $V$, and singular values $\Sigma$, which are two sides of SVD.

About Fourier analysis and how related to DASL.

The fourier transform has been widely applied in image processing for a long term [1] and is also prevailing in existing low-level vision investigation [2], which is supposed to be common sense. Basically, fourier analysis transforms the signal to the frequency space through Discrete Fourier Transform (DFT), and reverses back to the spatial space through Inverse Discrete Fourier Transform (IDFT). The most relevant form to our method, i.e., the Inverse Discrete Fourier Transform (IDFT) is presented in Eq. (3) of the paper, in comparison with the signal formation principle of SVD in Eq. 2, where both of them can be regarded as a weighted sum on a set of basis. And the transformed fourier coefficients $G(u, v)$ in Eq. (3) and decomposed singular values $\sigma_{i}$ in Eq. 2 undertake the same role, i.e., the combined coefficients for the set of basis. Therefore, the unattainable singular value optimization can be translated to the accessible and efficient fourier coefficients optimization, which is the main idea behind the SVAO.

[1] RC Gonzales, Digital image processing, Addison-Wesley Longman Publishing Co., Inc., 1987.

[2] Fourmer: An Efficient Global Modeling Paradigm for Image Restoration, ICML, 2023.

Thank you so much for your constructive comments and positive feedback, and we are really encouraged from that. Your suggestions have been carefully incorporated in the revised paper. In the following, we hope our point-by-point responses address your concerns well.

Performance on other benchmarks.

Thanks for pointing that out. Basically, considering the decomposed optimization of DASL for respective singular vector/value dominated degradations, the optimal intra-group optimization pattern will be favored in developed operators, i.e., SVEO and SVAO, for overall performance improvement, which may slightly decline the baseline in few cases. To further verify the effectiveness of DASL, we provide more comparison results of DASL integration and vanilla baselines below.

The metric is reported as PSNR, where DASL brings consistent performance gain on all tasks with reduced model complexity, further validating the effectiveness of the proposed method.

Concern about the form of the decomposition loss.

We present the comparison experiments below, where $L_{Tdec}$ denotes the above three terms formulation in [Q1], and $L_{dec}$ denotes the form of Eq. (4) in the paper.

Experimentally, $L_{dec}$ is more appealing than $L_{Tdec}$ for decomposition regularization. Particularly, considering that any signal can be regarded as a weighted sum on a set of basis from the SVD perspective, i.e., $X=U\Sigma V^T=\sum_{i=1}^{k}\sigma_{i}u_{i}v_{i}^T$. The SVD-ascribed degradation analysis proposed in DASL revealed that different types of degradation information primarily distribute in different portion of the signal, i.e., singular vectors (base components $\cup_{i=1}^k\\{u_{i}v_{i}^T\\}$) and singular values (combined coefficients $\cup_{i=1}^k\\{\sigma_{i}\\}$). Therefore, the regularization on base components in forms of $L_{dec}$ is sufficient, as we merely concentrate on whether the corrupted based components have been recovered regardless of how it is composed. While $L_{Tdec}$ further delve into the base components is unnecessary which may harm the performance with undesirable strengthened regularization.

Could DASL be applied to transformer-based restoration methods.

We present comparison results of DASL integration and baseline in transformer-based methods below, including SwinIR and Restormer.

It is worth noting that DASL is not only favorable for CNN-based models, but also applicable to transformer-based methods. The concerned architecture incompatibility problem is not come to be an obstacle. Note that we replace the projection layer at the end of the attention mechanism with developed decomposed operators for transformer-based methods. Thanks so much for pointing that out, which makes improvement to our paper.

How to get the results presented in Tabs.2, 3?

All comparison methods in Tab. 2 have been retrained on the same mixed dataset as DASL with their default settings for fair comparison. Results in Tab. 3 are derived from the models in Tab. 2 by simply testing on image denoising datasets across different noise ratio.

We hope our response address your concerns well, and we are delightful to receive feedbacks if there exist any questions.

Thank you so much again for your precious time and valuable comments.

Relating the proposed method with recent unified image restoration methods.

Thanks very much for pointing that out which makes improvement to our paper.

VRD-IR [1] concentrated on low-for-high problem and proposed to align various degradations in semantic space for consequent recognition-driven enhancement, which is originally different from our method in spirit. Methodologically, they handle diverse degradations through forced normalization for potential conflicts, compared to our natural SVD-based degradation analysis.

IDR [2] and ADMS [3] concentrated on unified image restoration with diametrically opposite perspective. IDR proposed to correlate various degradations through underlying degradation ingredients, while ADMS advocated to separate the diverse degradations processing with specific attributed discriminative filters. Note that multiple tasks-specific models have to be trained in ADMS for filter attribution, which is excessively inconvenient.

We present the comparison results with IDR below, which is the most relevant to our method. Compared to DASL, IDR correlates various degradations from microcosmic perspective with underlying degradation ingredients, such as directionality and unnatural image layering. While DASL performs the correlation from macroscopic perspective through SVD-based degradation analysis, ascribing various degradations into two groups. Despite the parallel motivation, the SVD-based analysis may benefit more other low-level vision problems for its generality, and the experimental comparisons are presented below.

The metric is reported as PSNR, where our DASL is integrated with Restormer baseline which is the same as IDR for fair comparison, and the superior performance verifying the effectiveness of the proposed method.

The above discussion and comparison results have been accordingly incorporated into the revised version.

Relating the proposed method with previous SVD-based image restoration methods.

Thanks so much for bringing our attention to previous SVD-based image restoration works.

Generally, SVD has been widely applied for a range of restoration tasks, such as image denoising, image compression, etc., attributing to the attractive rank properties [1] including truncated energy maximization and orthogonal subspaces projection. Particularly, the former takes the fact that SVD provides the optima low rank approximation of the signal in terms of dominant energy preservation, which greatly benefiting the signal compression. The latter exploits the fact that the separate order of SVD decomposed components are orthogonal, which inherently partition the signal into independent rank space, e.g., signal / noise space or range / null space for further manipulation, supporting the application of image denoising or even prevailing inverse problem solvers [2]. We acknowledge the great efforts the pioneer works made in cultivating the SVD potential for restoration problems.

Apart from above rank-based SVD methods, SVD-based degradation analysis proposed in DASL excavates another promising property that SVD possessed from the vector-value perspective, which is essentially different and has never been explored. Encouragingly, the above two SVD perspectives have the potential to collaborate well and the separate order properties are supposed to be incorporated into the DASL for sophisticated degradation relationship investigation in future works, as presented in sec. 4.4.

We hope our response address your concerns well, and we are delightful to receive feedbacks if there exist any questions.

Thank you again for your time and constructive comments.

[1] Yang et al. Visual Recognition-Driven Image Restoration for Multiple Degradation with Intrinsic Semantics Recovery, CVPR 2023

[2] Zhang et al. Ingredient-oriented Multi-Degradation Learning for Image Restoration, CVPR 2023

[3] Park et al. All-in-one Image Restoration for Unknown Degradations Using Adaptive Discriminative Filters for Specific Degradations, CVPR 2023

[4] R. Sadek. SVD Based Image Processing Applications: State of The Art, Contributions and Research Challenges, IJACSA, 2012.

[5] Wang et al. Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model, ICLR, 2023.

Thank you so much for your valuable comments, and we really appreciate that “SVD based analysis for multiple degradations looks neat and novel”. Your suggestions have been carefully incorporated in the revised paper. In the following, we hope our point-by-point responses address your concerns.

Computation complexity. The intension of the proposed DASL is to utilize the property of “SVD based degradations analysis” without explicitly performing the time-consuming SVD. We present the computation overhead of the proposed DASL below, which is also provided in Tab. 5 of the paper. Note that DASL achieves advanced performance compared to incorporated baselines with considerable inference acceleration, e.g.12.86% accelerated on MPRNet and 58.61% accelerated on AirNet.

(The metrics are reported as baseline / DASL integrated for comparison.)

Tips: Why faster and how to sidestep the time-consuming SVD?

DASL comprises two lightweight operators, i.e., SVEO and SVAO, to replace the basic calculation units of the baseline network for decomposed optimization of singular vectors and singular values, while both of them decently evade the time-consuming SVD. Conceptually, SVEO takes the fact that the simple orthogonal matrices multiplication can impact singular vectors of the signal without touching singular values (Theorem 1 in sec. 3.2). SVAO resorts to the Fast fourier transform (FFT) as the transformed fourier coefficients undertake the same role as singular values in terms of signal formation principle (detailed in sec. 3.3). Albeit efficient compared to SVD (516x faster as shown in Tab. 1 of the paper), the consequent overhead of FFT is also been considered, and we adopt the SVAO merely in bottleneck layers of the backbone considering the examined global statistical properties of singular values. Therefore, the decomposed optimization in DASL can be performed decently with efficiency.

Details about the integration layout of DASL is presented in sec. 3.1, and the unit reformulation strategy is provided in Appendix B.

Whether more degradations conform to the proposed SVD-based analysis, and the concern about dehazing.

We provide more degradation analysis in Appendix F to verify the generality of the proposed SVD-based analysis, including downsampling, compression, sharpness, underwater enhancement, and sandstorm enhancement. While the former three types are ascribed into singular vector dominated degradations and the latter two types are ascribed into singular value dominated degradations. Together with five deteriorations presented previously, SVD-based degradation analysis has been successfully applied to ten types of degradation!

Tips: A deeper understanding of the proposed SVD-based degradation analysis.

Experimentally, if we reexamine the two groups of degradation ascribed by SVD-based analysis, namely, rain, noise, blur, downsampling, compression, sharpness in singular vector dominated and hazy, low-light, underwater, sandstorm in singular value dominated, it can be concluded that the singular vectors responsible for the spatial content information, while the singular values represent the global statistical properties of the image. Note that degradations ascribed by such analysis, i.e., content corruption and global corruption, inevitably cover most of scenes.

Theoretically, the above conjecture can also be derived from the signal formation principle of SVD, where any signal can be regarded as a weighted sum on a set of basis, i.e., $X=U\\Sigma V^T=\\sum_{i=1}^{k}\\sigma_{i}u_{i}v_{i}^T$. From this perspective, the singular vectors $\cup_{i=1}^{k}\\{u_iv_i^T\\}$ represent the base components of the signal for content composition, and singular values $\cup_{i=1}^{k}\\{\sigma_i\\}$ represent the combined coefficients for statistical properties. We are confident that most degradations will conform to the proposed SVD-based analysis in virtue of the closed form of signal formation principle. The above discussion have been accordingly incorporated into the revised version.

Dehaze case: The MSE reconstruction error in Fig. 2 (a) may not suitable for the dehaze due to the sensitivity to global corruption, which may yield both large error for singular vectors and singular values regardless of visual effect. However, the disparate singular value distribution in Fig. 2 (b) and the lowest singular vector difference in Fig. 2 (c) indicate that the dehaze is definitely ascribed into singular value dominated degradation. The visual analysis of the dehazing in Fig. 1 and Appendix F are more intuitive and evident, validating that SVD-based degradation analysis capturing the dehaze well, and support the generality of the proposed method.

Thank you so much for your valuable comments and encouraging feedback, and we really appreciate that. Your suggestions have been carefully incorporated in the revised paper. In the following, we hope our point-by-point responses address your concerns well.

A Deeper understanding of SVEO and SVAO.

Recall that the SVD ascribed degradation analysis presents that the decomposed singular vectors and singular values naturally undertake the different types of degradation information. In this way, the intention of the proposed DASL is to support the decomposed optimization of singular vectors and singular values of the degraded signal, through developed operators, i.e., SVEO and SVAO. Particularly, SVEO merely optimizes singular vectors and is decoupled from the optimization of singular values, while SVAO operates in the opposite way. Without interference from the degradation-irrelevant portion of the signal, the decomposed operators, i.e., SVEO and SVAO, are capable of concentrating more on degradation processing, which interprets that SVEO is capable of boosting the performance on singular vector dominated degradations, including deblurring, and SVAO is capable of boosting the performance on singular value dominated degradations, including low-light image enhancement.

Tips: What the SVEO and SVAO truly focused on?

If we reexamine the singular vector dominated degradations, including rain, noise, blur, and singular value dominated degradations, including hazy, low-light, it can be conjectured that the singular vectors responsible for the spatial content information and texture details, and the singular values represent the global statistical properties of the image. The above conjecture can also be derived from the signal formation principle of SVD in theory, where singular vectors rerpesent the base components of the signal and singular values rerpesent the combined coefficients, as show in Eq. (2) of the paper. Therefore, SVEO is concentrated on the content corruption through the orthogonal regularized convolution for impacting singular vectors (Theorem 1 in sec. 3.2), while SVAO is concentrated on the global statistical corruption through approximated fourier coefficients for affecting singular values (homogeneous signal formation principle in sec. 3.3). From this point, the decomposed operators are developed precisely for respective singular vector/value dominated degradations.

Why DASL could reduce the computations of baseline model?

Thanks for your question. Actually, DASL replaces the basic calculation units of the baseline model with developed lightweight decomposed operators, instead of introducing extra modules into baseline model. Basically, we substitute half of the convolution layers in baseline model with SVEO, and only perform SVAO at the bottleneck layers of the backbone network, considering the global statistical properties of singular values and non-negligible overhead of Fast Fourier Transform (FFT). We present the integration layout of DASL in sec. 3.1, and the detailed unit reformulation strategy is provided in Appendix B.

Whether the proposed method sensitive to the hyper-parameters?

Faithfully, the proposed DASL is supposed to be sensitive to the involved two hyper-parameters, i.e., $\lambda_{orth}$ and $\lambda_{dec}$, due to the large number of $L_{orth}$ and $L_{dec}$. Concretely, $L_{orth}$ is related to tremendous network parameters for orthogonal regularization, and $L_{dec}$ is related to the decomposition space error, especially the huge discrepancy in singular values.

We present the ablation experiments for hyper-parameters $\lambda_{orth}$ and $\lambda_{dec}$ below. The metrics are reported on average of all tasks with MPRNet baseline.

Note that the default setting is marked in bold. We further present the ablation experiment of hyper-parameters with DGUNet baseline below.

Comfortingly, the hyper-parameters in DASL are agonistic to the baseline model, which exhibit the similar effect tendency as variation. Therefore, it is convenient to utilize the same hyper-parameter setting for various baseline models without cumbersome selection. And it is suggested that the DASL is best used with recommended hyper-parameters for promising performance.

Thanks so much for pointing this out.

The figure of the paper has been accordingly revised. We hope our response address your concerns well, and we are delightful to receive feedbacks if there exist any questions.

Thank you so much again for your time and constructive comments.