Rethinking RGB Color Representation for Image Restoration Models

We thank you for sparing your valuable time to review our work and raise important and intriguing questions. To provide comprehensive answers, we would like to categorize your concerns in four questions. Please feel free to notify us if we have misinterpreted your comments. Due of the length constraint, our comment is divided in two.

[Q1: Significance of the performance improvement]

Based on the experiments, compared to previous methods, the improvement brought by vggloss is quite limited, with an increase of 0.1dB (PSNR) in Table 1 and 0.02dB in Table 2.

• First, as a response to your concern in the performance gain of our $a$RGB representation space, we have conducted a new experiment on another deblurring method [4*]. The results in the table above show consistent improvement.

• While performance gains are sometimes marginal (e.g., 0.02 dB in Table 2), we would like to note that 0.1 dB difference in PSNR in Table 1 is significant, since PSNR is measured in log10 scale. Furthermore, including the table above, we have shown consistent performance improvement across various restoration tasks and models, with the same pre-trained $a$RGB autoencoder. We believe the results indicate that our $a$RGB space can serve as a promising alternative to the RGB space.

• We would like to highlight that our main focus in this work is to test whether a pixel-wise color difference is a necessary loss function in various image restoration tasks. Please note that the results in both tables are obtained from the models that do not use the RGB representation space for calculating the loss functions, and yet show comparable (often better) performance in terms of RGB PSNR. We believe these results raise research questions as to whether RGB color space should be must-use space for restoration tasks, hopefully introducing an interesting research direction.

[Q2: Dependence of the task in the optimal number of experts]

Is the selection of the number of "experts" highly dependent on experience? Will different tasks have significant variations? It seems that an inappropriate selection of the number of experts might lead to even lower performance than not using this loss function at all.

• In addition to our ablation study in Table 4 and Section 4.4 in the manuscript, we provide an additional ablation study with a deblurring model [4*]. The results are shown in the table below. Results demonstrate that the number of experts in our system is indeed critical for the final performance. However, the tendency of such influence is similar among different tasks, showing that the advantage of $a$RGB is maximized when using 20 experts.

(Continued from the last comment)

[Q3: Concerning the difference between the RGB and the $a$RGB spaces]

Although this paper claims to introduce a method that doesn't calculate loss in the RGB domain, the loss function used in training still falls within the category of pixel-based feature scale.

• To our understanding, your concern in this part is that how can we call our representation space not the RGB even if it does have the same spatial scale in its features. We believe your question highlights the key difference between the scale and the content of image features.

• As you have pointed out, we have designed our representation space to have the same spatial dimensions with the image. However, we have increased the channel dimensions from 3 (RGB) to 128. Overall, the feature dimension of our $a$RGB encoder is 128 X H X W, larger than that of the input image. Our aim was to introduce more information in our representation space for each individual pixel, rather than to remove unnecessary information from it. Therefore, we have dubbed our representation space augmented RGB. Our representation space includes the information of the RGB colors as its (linear) subspace. We believe that our $a$RGB space-based losses are different from the RGB space-based ones, not because our $a$RGB space represents completely different information from the colors, but because it extends the pixel-wise color information.

• We have revised the manuscript in Section 2.1 accordingly to better clarify our claim.

[Q4: On the performance of perceptual loss]

Moreover, it hasn't been compared with other perceptual methods, such as lpips or ssim loss.

• Please understand if we have misread your intention in this part. To our understanding, you have interpreted our $a$RGB encoder’s role similar to the VGG-Net in the monumental perceptual loss. However, we wish to point out that their roles are greatly different. The key is that our $a$RGB representation space replaces the RGB space for loss calculation, whereas perceptual losses, such as VGG loss, LPIPS loss, DISTS loss, differentiable SSIM loss, and the dual-space losses such as the Fourier space loss [2] all require RGB distances to guide the training. In other words, the $a$RGB space-based distance losses and perceptual losses do not compete with each other. Rather, they are complementary in their roles.

• Table 1 and Section 3.2 in our manuscript show one possible collaboration of these two types of losses in perceptual super-resolution task. Here, existence of the VGG loss helps our $a$RGB loss stabilize the training of ESRGAN [3].

[Concluding remark]

Finally, we thank you for your recognition of our main claim and contributions. Although we have tried our best providing answers to each of your concerns, we might have missed a point or two. Please understand if we have done so and feel free to raise any issue. We are always open to new discussions.

[Reference]

• [4*] Seungjun Nah et al., “Deep Multi-Scale Convolutional Neural Network for Dynamic Scene Deblurring,” In CVPR 2017.

(Continued from the last comment)

[Q1: Difference from Fourier loss]

The proposed loss is similar to Fourier loss, which is also decomposed the image upon pre-defined basis. Can authors discuss the difference between them and compare their performance?

• We find three key differences between the Fourier space loss and the per-pixel loss in our $a$RGB space.

  1. Fourier space loss is typically combined with its primary loss, i.e., the RGB space-based loss, without removing the necessity of the RGB loss. This is the common practice of the Fourier loss we can found in other notable works [6*, 7*], too. However, in our work, we suggest an alternative to the RGB space with our $a$RGB space.

  2. Fourier space loss calculates distances in the discrete dual of the image space by transforming the pair of images using FFT algorithm. FFT is known to be a linear operation, and therefore a Fourier space loss can be thought of as a linearly weighted distances between two image patches of fixed size. On the other hand, our $a$RGB encoder is nonlinear, and therefore the distance metrics are contracted and expanded based on the local patch structure. This unique feature of ours helps our $a$RGB-based loss functions adaptively weigh important structures. The evidence of the metric warping of our $a$RGB space is shown in Figure 5d.

  3. Although the Fourier space loss can handle long range nonlocal correlations via resonance in Fourier domain, their resonance frequency is strictly fixed because the linear weight of the FFT algorithm is a constant. In contrast, our $a$RGB encoder only cares for the local structure within its receptive fields, yet it can encode more complex structures adaptively thanks to its Mixture-of-Experts architecture. Figures 5b, 5c and 20 in our manuscript show that pixel assignments to each of $a$RGB encoder’s expert depends on the local structures of its neighborhood.

• To demonstrate the effectiveness of our $a$RGB loss over the Fourier space loss, we design an experiment to train a deblurring model [4*] using two types of losses: (1) the combination of the primary (the L1-RGB) and the dual (the L1-Fourier space loss) losses following recent works [6*, 7*] and (2) the L1-$a$RGB loss without using the L1-RGB loss. Please understand that the time and resources permitted to us is limited, so we are temporarily reporting the intermediate scores. However, the difference between the two method clearly demonstrates the superirority of our $a$RGB loss.

[Concluding remark]

We hope this answer fits you well. We would like to conclude our remark with our deepest thanks to your acknowledgement to our novelty in designing the representation space and the versatility of our demonstration.

[Reference]

• [4*] Seungjun Nah et al., “Deep Multi-Scale Convolutional Neural Network for Dynamic Scene Deblurring,” In CVPR 2017.
• [5*] Xintao Wang et al., “ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks,” In ECCVW 2018.
• [6*] Sung-Jin Cho et al., “Rethinking Coarse-to-Fine Approach in Single Image Deblurring,” In ICCV 2021.
• [7*] Yuning Cui et al., “Selective Frequency Network for Image Restoration,” In ICLR 2023.
• [8*] Xintao Wang et al., “Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data,” In ICCVW 2021.
• [9*] Cheng Ma et al., “Structure-Preserving Image Super-Resolution,” In T-PAMI 2021.
• [10*] Jie Liang et al., “Details or Artifacts: A Locally Discriminative Learning Approach to Realistic Image Super-Resolution,” In CVPR 2022.
• [11*] JoonKyu Park et al., “Content-Aware Local GAN for Photo-Realistic Super-Resolution,” In ICCV 2023.

We express our deepest gratitude for your constructive review and the time and effort you have invested for reviewing our work. We are excited to discuss about your considerate concerns. Due to the length constraint of this comment section, we split our response into two.

[W1: Regarding Table 1]

In the paper, the performance of the proposed loss are demonstrated on perceptual SR task. The results in table 1 are confusing. The PSNR and SSIM of RRDBNet are the highest among all the settings, but they are not bolded. The SSIM of the last setting is worse than most of settings for DIV2K-Val dataset, but it is bolded as better score.

• We appreciate your constructive feedback. We have changed Table 1 accordingly to clarify the main points we wanted to make with it.

• In Table 1 and Section 3.2 of the main manuscript, our goal was to excel in perceptual metrics in perceptual image super-resolution. In particular, we would like to note that we focus on the improvement brought by replacing the L1 loss in the RGB space with the L1 loss in our $a$RGB space.

• The first row stands for the PSNR-oriented reference model just for the reference. This is not trained under the perceptual super-resolution setting. The rows two to five represent methods to enhance perceptual metrics. Here, we intended to make comparisons between the second and third rows, while comparing the fourth and fifth rows.

• We hope this change removes the original confusion.

[W2-3: Significance of the denoising and deblurring results]

For perceptual SR and image deblur tasks, there are considerable baselines perform better than ESRGAN and MPRNet. For example, restormer and NAFNet could be used for deblurring evaluation. In this way, we can test whether the proposed loss could consistently boost performance and lead to a new SOTA. The performance gains are too small, which can hardly verify the effectiveness of the proposed loss.

• Regarding state-of-the-art methods, we agree with your concerns that the goal of designing a loss function is to work with the best models we have at that time. However, please understand that due to the strict resource constraint upon us, we had to find the optimal combination of experiments to best demonstrate the versatility of our method in various tasks and model types.

• Regarding our experiments for perceptual super-resolution in Section 3.2 of our manuscript, due to heavy computation of Restormer and NAFNet, many previous works (ESRGAN [5*], Real-ESRGAN [8*], SPSR [9*], LDL [10*], CAL-GAN [11*]) adopt the RRDB network as their baseline. Moreover, we would like to note that NAFNet-width64, the state-of-the-art model you have mentioned is used for the denoising task, yet not for the deblurring task, in Section 3.3 of our manuscript.

• From the experiments in Section 3 of the manuscript, we can observe consistent performance gain in various tasks and model architectures. The improvements were achieved using the same pre-trained $a$RGB autoencoder. Therefore, we would like to argue that our $a$RGB space suggests that there can be alternatives to the RGB space for the guidance of training image restoration models.

• As a final remark, though it is not a state-of-the-art, we hope we can justify our approach more clearly with an additional experiment. Here, as an answer to your regard on our small performance gain, we apply our method on another deblurring model [4*], as presented in the table below. Consistent with Table 2 in the main manuscript, our method demonstrates superior performance. The numbers are calculated from the author’s official code repository.

(Continued from the last comment)

[W1: Significance of the denoising and deblurring results]

The performance improvement of the proposed aRGB space in the denoising and debluring tasks seems insignificant. In Table 2, comparing the first two rows, and the last two rows, the PSNR gains are only 0.02 dB and 0.03 dB, respectively. In Table 3, the PSNR improvements between the last two rows are 0.07 dB on GoPro and 0.02 dB on HIDE dataset.

• We acknowledge your concerns. We have conducted additional experiment on another deblurring model [4*], changing its original L1-RGB loss with our L1-$a$RGB loss. The results in the table below shows greater improvement, consistent with Table 2 in the main manuscript.

[Concluding remark]

We hope our answers suits your great questions. If any more concern rises, please feel free to ask us at any time. We, again, thank you for your recognition of our contributions, i.e., our analysis and visualizations, the novelty, the versatility and the interpretability of our approach.

[Reference]

• [4*] Seungjun Nah et al., “Deep Multi-Scale Convolutional Neural Network for Dynamic Scene Deblurring,” In CVPR 2017.
• [12*] Manuel Fritsche et al., “Frequency Separation for Real-World Super-Resolution,” In ICCVW 2019.

We deeply appreciate your positive review with comprehensive acknowledgement on our key contributions. We will try our best clearing your remaining questions on our work. Since our comment exceeds the character limit, we are splitting it into two.

[Comment 1: Typo]

Equation 6, L_{pair} should be L_{pixel}

We thank for your correction. We have revised the equation in the updated pdf.

[Q1: Explanation of the suppression of artifacts originated from adversarial training]

The space aRGB is originally designed to encode structure information on a pixel basis. Why it can exhibit suppression of artifacts in SR tasks?

• We believe that there are at least two reasons the artifacts are well suppressed by our $a$RGB encoder.

  1. Local information propagation: Through the $a$RGB encoder, each pixel receives gradients from the pixel difference of its neighboring pixels. Visual artifacts can be seen as a disruption in high-frequency variation among adjacent pixels [12*]. The information propagation through our $a$RGB-based loss suppresses not only each pixel’s regression error, but each local region’s structural variations. This is not like in the conventional RGB space-based loss, which only cares about each pixel’s regression error. This analysis is further justified by our discussion in Appendix A. Here, we have shown that the amount of information propagation, i.e., the scale of gradients from adjacent pixels, is significant in our $a$RGB encoder.

  2. Edge encoding: We can also consider this phenomenon from the representation perspective. As visually demonstrated in Figures 5a, 17, and 18, the $a$RGB encoder appends local edge information to each pixel. This means that the $a$RGB embedding space effectively has channels responsible for edge information. This structural part is directly compared by a per-pixel $a$RGB loss, suppressing undesired structure presented in the restoration output.

• In short, we believe that the cause of artifact suppression you have mentioned is due to $a$RGB encoder’s stronger guidance to minimize errors in high-frequency local structures.

[Q2: Demonstration of the information encoded in the $a$RGB autoencoder]

The training process of aRGB auto-encoder does not involve any loss regarding local structure. Is it possible that the encoder also learns other information, e.g., texture, style?

• As you may have noticed, our $a$RGB autoencoder embeds simple local neighborhood structures such as edges at each pixel as visualized in Figures 5a, 17, and 18. We have also tried to clarify the sources of this structural information by decomposing the $a$RGB space in Section 4.1.

• To our understanding, your concern is beyond this simple neighboring structures. It depends on how people define textures and styles. Nevertheless, we doubt that our $a$RGB space encodes semantic level of textures, for example, a checkerboard or long hairy textures. We have two reasons to believe so.

  1. Small receptive fields: Our $a$RGB encoder has a receptive field size of 9x9. This size is sufficient to suppress high-frequency artifacts generated by other auxiliary losses as demonstrated in Table 1, and to enhance the edge structures in deblurring and denoising as suggested in results in Appendix D. However, this is not sufficient for the encoder to perceive semantic textures. Thus, we believe that our $a$RGB encoder encodes local structure by appending local edge information to each pixel within neighborhood of small area corresponding to its receptive field.

  2. Empirical reasons: Although we have not included in our manuscript, we have conducted a simple experiment on training a single 1x1 convolutional layer to return VGG features from the $a$RGB embedding. If this experiment succeeds, we can conclude that the textual information up to VGG features’ level of abstract is indeed embedded in our $a$RGB space. However, this quickly has failed when we try to mimic the features from after a second downscaling block of the VGGNet.

• To sum up, the advantage of our $a$RGB space is not based on the semantic embedding. In other words, the $a$RGB space-based loss functions and the ones derived from deeper networks, e.g., perceptual losses and adversarial losses, are complements rather than substitutes. We believe that experimental results in Table 1 of our manuscript supports this claim.

(Continued from the last comment)

[Q5: Autoencoder’s dependency on training datasets]

5. Does the proposed the aRGB architecture rely on the dataset trained on?

• Yes, as stated in Table 4 and elaborated in Section 4.4 of the main manuscript. However, we tried to minimize the dependency by training our $a$RGB autoencoder with a broad set of clean, natural images (DIV2K, Flickr2K, and DIV8K) in order to maximize the versatility of our $a$RGB loss. The results on various restoration tasks in Table 1, 2, and 3 are reported with the same $a$RGB autoencoder model throughout. The result clearly states that that this pre-training stage need not be done more than once. We only have to plug in the pre-trained $a$RGB encoder to the loss function.

[Q6: Significance of the result of image deblurring]

6. The reviewer thinks that the performance improvement on GoPro is minimal. For example, only a 0.05 dB PSNR gain is obtained for MPRNet on GoPro. What do the authors think about this?

• To further justify our approach, we apply our method on another deblurring method [4*], as presented in the table below. Consistent with Table 2 in the main manuscript, our method demonstrates superior performance. The numbers are calculated from the author’s official code repository.

[Concluding remark]

We hope our answers clear your concerns. Finally, we deeply thank you for your recognition of our work’s analysis and for your acknowledgement of our contribution to improve performance on various restoration tasks.

[Reference]

• [3*] Xiangtao Kong et al., “ClassSR: A General Framework to Accelerate Super-Resolution Networks by Data Characteristic,” In CVPR 2021.
• [4*] Seungjun Nah et al., “Deep Multi-Scale Convolutional Neural Network for Dynamic Scene Deblurring,” In CVPR 2017.
• [5*] Xintao Wang et al., “ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks,” In ECCVW 2018.
• [6*] Sung-Jin Cho et al., “Rethinking Coarse-to-Fine Approach in Single Image Deblurring,” In ICCV 2021.
• [7*] Yuning Cui et al., “Selective Frequency Network for Image Restoration,” In ICLR 2023.

First of all, we appreciate your in-depth review and intriguing questions regarding our work. Since we find that the possible weaknesses you have queried are repeating in the Questions section (Questions 4 and 6), we would like to answer each of your questions directly. Your thoughtful questions are invaluable to us for strengthening this work, and we are excited to discuss them with you. Due to the character limit, we split our response into two comments.

[Q1-2: Architecture design]

1. The authors design a nonlinear mixture-of-experts encoder and a linear decoder for aRGB representation. Can this design principle be applied to guide the architecture design of image restoration networks?
2. Is the additional en/decoder equivalent to adding an additional branch for learning? I doubt whether the improved performance is yielded by the additional computation overhead.

• Indeed exploiting experts in architecture may help, as suggested in [3*]. However, we would like to emphasize that our aRGB is not an additional branch for restoration models. Rather, it is similar to how perceptual loss does not count as an architectural augmentation. Thus, our aRGB space does not incur additional computational overhead during the test time, as described in our answer below.

[Q3: Extra computational overhead]

3. The widely used dual-domain loss in models, such as MIMOUNet (Cho et al, ICCV'21) and SFNet (Cui et al, ICLR'23), can introduce global information refinement. How does aRGB compare to this loss function? This function does not lead to much computation overhead.

• We clearly note that our method only applies to the training phase of a restoration model, and therefore does not cause any additional computational overhead in the test time.

• For training time, our model does introduce additional computation for the embeddings. The amount of this overhead is not significant compared to the image restoration model itself. For example, training RRDB [5*], a widely used super-resolution model, with our $a$RGB loss increases training time about 9%.

• Moreover, the model itself is very lightweight, compared to the massive volume of the current state-of-the-art restoration models. The table below compares the computational burden with other loss functions and restoration models.

*Our $a$RGB decoder has only 0.01M parameters. It is a linear layer. Furthermore, the decoder is not used in the training of image restoration models and therefore its computational cost does not affect the overall performance.

[Q4: Comparison with frequency domain loss]

4. Does aRGB lead to extra computation overhead during training and inference?

• In contrast to the Fourier space-based loss used in MIMO-UNet [6*] and SFNet[7*], our $a$RGB space-based loss has the following three unique characteristics.

  1. Whereas Fourier space loss is used in combination with the RGB space-based counterparts, our $a$RGB loss is designed as a new alternative for the RGB space.
  2. The dual representation in the Fourier space loss is obtained by applying the FFT algorithm to images. This is a constant linear operation, and hence the correlations it models only depend on the coordinate differences, and not the contents of the signal. In contrast, our $a$RGB encoder is nonlinear and adaptive to the input signal. The metrics in the $a$RGB space expands and contracts based on the images’ local structures as demonstrated in Figure 5d of our manuscript.
  3. Fourier space loss applies globally within the patch extent, yet its correlation model is fixed. In contrast, our $a$RGB encoder behaves based on the local image structure, yet it can encode more complex structures adaptively. Figures 5b, 5c and 20 in our manuscript demonstrates its adaptability on the local image structure.

• We design an experiment to further compare the two types of space modeling. Here, a deblurring model [4*] is trained using two types of losses: (1) the Fourier space loss with L1-RGB loss [6*, 7*] and (2) the $a$RGB loss only. Please understand that the time and resources permitted to us is limited, so we are temporarily reporting the intermediate scores. Still, the table clearly exhibits the advantage of our $a$RGB loss over the Fourier loss.