sRGB Real Noise Modeling via Noise-Aware Sampling with Normalizing Flows

Dear Reviewer VvGJ,

We would like to thank the reviewer for taking time to review our manuscript and acknowledging our contributions. Below is our response to reviewer’s concerns.

Comparison with C2N

We acknowledge that comparing C2N with other sRGB noise modeling methods is unfair. C2N differs from other models in that it learns noise and clean data in an unpaired manner, utilizing additional clean dataset (e.g., BSD500) and noisy dataset (e.g., DND) during training. However, due to the limited availability of comparable sRGB noise modeling methods, we have included C2N for comparison, as demonstrated by the previous method (NeCA). Nevertheless, as the reviewer pointed out, we have specified the training dataset of C2N in the revision (Sec 5.2).

Applications of noise synthesis and modeling method

The main purpose of noise modeling is to understand noise properties and to denoise a distorted image for aesthetic purposes or improving downstream tasks (e.g., object detection, segmentation). For RAW images, the prior works [5*, 6*] outperform the denoising results using only the synthesized noisy-clean pairs over the real noisy-clean pairs. However, sRGB noise is further distorted due to several non-linear operations during the RAW-sRGB conversion, making it more difficult for sRGB noise modeling to outperform denoising performance using real data. To alleviate the problem, prior studies blend synthetic noise datasets with the real noise datasets to enhance the denoiser’s performance [7*] or generating synthetic noise datasets using additional clean datasets [8*]. We believe that more improved results can be achieved through a better understanding of sRGB noise and improved technical contributions in future works.

Reference

[5*] Abdelhamed, Abdelrahman, et al. "Noise flow: Noise modeling with conditional normalizing flows." ICCV. 2019.

[6*] Maleky, Ali, et al. "Noise2noiseflow: realistic camera noise modeling without clean images." CVPR. 2022.

[7*] https://github.com/GarrickZ2/Image-Denoising

[8*] Yuanhao Cai, et al. “Learning to generate realistic noisy images via pixel-level noise-aware adversarial training.” NeurIPS. 2021.

We appreciate your valuable feedback and are pleased to hear that your concerns have been addressed. Thank you for your thoughtful review and support.

Additional noise quality metrics

While Table 3 in the manuscript illustrates the PSNR differences between denoisers trained with real noise and synthesize noise, as you suggested, we have rephrased PSNR Gap for clarity, and measure the noise quality in terms of MMD. As a result, NAFlow demonstrates robust performance in terms of MMD. For PNGAN, official weight parameters were not provided, so we report the results based on the results provided in the paper and the official GitHub repository of PNGAN [3*].

The reason for not using MMD

MMD was employed as a noise quality metric used in PNGAN. For the evaluation, PNGAN randomly sampled 100 images from both real noisy and synthetic noisy image sets and measured the difference between the two distributions using MMD. Instead of using this approach, we opt to directly compare the entire image set following the evaluation metrics employed by NeCA, the state-of-art sRGB noise modeling network. We made this decision to ensure a fair comparison with NeCA and other conventional approaches such as C2N and Flow-sRGB.

Response to (iv)-[3]

The adaptation now is conducted in a self-supervised manner.

Our method does not include adaptation process and self-supervised training.

Fully supervised result in Restormer

We did not include the comparisons against PNGAN due to fairness concerns. Because, in the process of learning and generation noise, PNGAN uses an external dataset and a pre-trained denoiser (RIDNet). Regardless, following the suggestions by reviewer, when our model is trained using the same approach, the results are as follows:

PNGAN does not provide pretrained weights on the official GitHub repository, so we have filled in the performance values listed on the official GitHub [3*, 4*] In the above table. We use our NAFlow to generate noisy images and trained Restormer by mixing SIDD real noisy datasets with synthetically generated noisy datasets, similar to PNGAN, and can see slightly better performance than the results from PNGAN.

Comparison between other SOTA noise generation algorithms

We conducted a focused comparison with state-of-the-art methods in noise modeling methods, NeCA. So, we use the same comparison methods as used in NeCA. As the reviewer suggested, we also compare the performance of the SOTA noise generation methods in terms of AKLD on the SIDD dataset.

Additionally, we provide device-specific KLD measurement results in our paper, as shown in Table 2. As mentioned above, PNGAN does not provide official pre-trained weights, and since neither the paper nor the official GitHub provides device-specific KLD values, we left these entries blank.

Response to (v)

The network and the method (Meta-train and Fast Adaptation) are pretty complicated.

Our paper does not include any meta-training and fast adaptation.

Reproducibility

The code will be available upon acceptance.

Dear Reviewer s3bG,

We would like to thank the reviewer for the thorough examination on our work. Below is our response to the reviewer's concerns.

Motivation of choosing Normalizing Flow

We have chosen the normalizing flow (NF) architecture based on three primary considerations: 1. precise likelihood estimation capabilities, 2. allowance for different noise distributions in latent space and 3. efficiency. Our proposed NAFlow demonstrates the capacity to manage multiple noise distributions using a single unified network and can synthesize noise without relying on noise metadata. This distinguishes it from other generative models, such as autoencoders or GANs, which face challenges in producing input-noise-aware results in the absence of metadata. And we also note the efficiency of our approach in comparison to diffusion-based models, which incur higher computational costs due to multiple denoising steps for generation. Furthermore, we would also like to point out that our NAFlow has x40 fewer model parameters compared to NeCA, the current GAN-based SOTA model, while maintaining competitive performance (Tab 2, 4).

Statistic of Normal distribution in latent space

In our framework, the mean and variance of normal distributions are also trainable parameters (Sec 4.1), and we can measure the likelihoods between the latent distribution of the input data and the learned normal distribution with the learned mean and variance parameters at test time (Sec 4.2). In addition, we perform ablation studies to evaluate the effectiveness of each parameter and demonstrate the expressiveness of Gaussian Mixture Models (GMMs) in modeling diverse noise distributions within the latent space (Tab 7). Specifically, we observe that the noise embedded in the normal distribution with trainable mean and variance achieves better noise quality, and it implies that the normal distribution with flexible statistic helps our network to handle diverse real-world noise distributions.

Response to (iii)-[2]

The proposed method is trained on SIDD, even the pre-trained denoiser.

We do not use pre-trained denoisers for denoising experiments. All denoisers are trained from scratch (Sec 5.5).

Overfitting problem on SIDD dataset

We acknowledge the limitations of the SIDD dataset (Sec A.5), which contains a limited number of captured scenes and smartphone device types, and it can lead to an overfitting problem. Consequently, achieving generalized performance is a challenging problem for all models trained on the SIDD dataset. To demonstrate a better generalization capability of NAFlow, we conducted additional experiments on the SIDD+ dataset, which is not included in the training dataset in Table 3 of the main manuscript. Note that the SIDD+ dataset consists of new smartphone device types (e.g., HTC U12+, LG G7) and different scenes that were not included in the original SIDD dataset. In Table 3, NAFlow shows better performance compared to the other SOTA models such as NeCA on the external dataset, owing to the proposed noise-aware sampling, which allows NAFlow to approximate the input noise distribution fully facilitating learned noise information without requiring any metadata. See the table below for more results.

Evaluation on different datasets

Following the reviewer's feedback, we evaluated the quality of the generated noise on additional datasets (e.g., DND, PolyU, Nam) in terms of KLD. For the DND dataset, it should be noted that due to the lack of clean images, measuring KLD is not feasible, so we provide results on the SIDD+, PolyU, and Nam datasets.

In the table below, we provide experimental results from the same setup as described in Sec 5.3 (Noise Generation with SIDD+). As the reviewer pointed out, Nam and PolyU, which are the noisy-clean paired datasets captured by DLSRs, contain noise distributions different from the SIDD dataset because DLSRs have different hardware specifications (e.g., image sensor type) and image processing pipelines. As a result, the noise quality is degraded on Nam and PolyU dataset due to the domain gap and improving real-world noise modeling on diverse environment is our future research direction.

Dear Reviewer EqxC,

We would like to thank the reviewer for the thorough examination on our work. Below is our response to the reviewer's concerns.

Multi-scale noise modeling performance on Flow-sRGB

We observe that modeling real-world noise with a multi-scale embedding, as presented in our main manuscript, also improves the performance of Flow-sRGB. We evaluate the similarity of the basic noise characteristics using the KLD metric on SIDD validation set, the experiments presented in the table above specifically show the impact of multi-scale embedding on Flow-SRGB. However, despite the fact that NAFlow uses a single unified model without the need for metadata, NAflow consistently produces better performance than Flow-sRGB across all scale levels.

Although modeling real-world noise with a multi-scale embedding may bring improvements to other models, such as Flow-sRGB, we argue that the major factor in the performance improvement in our work is caused by learning the multiple different noise distributions in the latent space and noise-aware sampling (Tab 5, 6, 7). For a more detailed comparison, please refer to the "Response to ALL reviewers" section, which highlights the key differences with NeCA and Flow-sRGB.

Typos

Thanks to your feedback, we have rectified the typos on the revision (Sec 5, Tab 2)

Thanks for detailed response. The authors have solved my concerns. Hence, the rating has been updated.

Thank you for your thorough feedback and positive reassessment. We appreciate your time and consideration in reviewing our work.

Dear Reviewer YHPu,

We would like to thank the reviewer for recognizing our contribution and for pointing out the specific strength of our method, as well as constructive concerns. We address the reviewer's concerns as follows.

Limitation of the SIDD dataset

We also acknowledge the limitations of the SIDD dataset as suggested by the reviewer and have reflected this in the revision (Sec A.5). We appreciate the valuable feedback.

Evaluation on external datasets

As the reviewer recommended, we have extended our experiment by training NAFlow on the SID[1*] dataset. We select 10 camera configurations (C=10) that have a sufficient number of images for training from the SID dataset, and since the SID dataset does not provide noise-free images, we adapt the denoiser [2*] pre-trained on the SIDD dataset to produce pseudo-ground truth images. In the table below, we measure the similarity between the real noise and our synthesized noise using KLD and AKLD, demonstrating that our method performs similarly on the Fuji dataset while showing a performance gap on the Sony dataset when compared to the SIDD dataset. The drop in performance can be attributed to the fact that the SID dataset was collected from more challenging environments, such as diverse low-light indoors and outdoors scenes, making real-world noise modeling more difficult than the SIDD dataset. Hence, our future research is to bridge the gap between varied environments. We highly appreciate your insightful suggestions to steer the future direction of this field.

Reference

[1*] Chen, Chen, et al. "Learning to see in the dark." CVPR. 2018.

[2*] Zamir, Syed Waqas, et al. "Restormer: Efficient transformer for high-resolution image restoration." CVPR. 2022.