Rethinking Nighttime Image Deraining via Learnable Color Space Transformation

We thank Reviewer ohMd for his\her efforts. We are encouraged by the reviewer’s positive comments on the paper presentation, dataset contribution, and superior performance. We answer the questions below and will incorporate all feedback in the revised version.

1. Details on architecture

We have provided the implementation details of CST-Net in Section I of the supplementary materials.

2. Effectiveness in daytime scenes

We would like to claim that we have evaluated our method on the RealRain1k and RainDS-real benchmark datasets, both of which contain rainy images captured in daytime scenes. As shown in the quantitative results in Table 2 of the main paper, our method still achieves good performance on both datasets. We have provided visual examples in Figures 24, 25, and 26 in Section O of the supplementary material, demonstrating that our model also achieves robust deraining performance in daytime rainy scenes.

3. Comparison with Mamba-based methods

According to the reviewer’s suggestion, we further compare our method with recent Mamba-based method, FreqMamba [1]. As shown in the table below, our method outperforms FreqMamba across all three subsets, including rain streaks (RS), raindrops (RD), and the mixture of rain streaks and raindrops (SD). We will add these results in the revised paper.

Ref:

[1] Zhen, Zou, et al. “FreqMamba: Viewing Mamba from a Frequency Perspective for Image Deraining.” ACM MM 2024.

4. Generation of rain streak and raindrop masks

We have provided detailed descriptions of the generation processes for rain streak masks and raindrop masks in Section C of the supplementary material.

5. Issues of the defocus blur operation

As stated in L135–138 of the main paper, we introduce defocus blur based on real-world observations to simulate more realistic nighttime rainy images. The defocus blur is applied to the rain-free nighttime background, without considering the overall nighttime illumination distribution of the image. Illumination distribution is only considered during the synthesis of the rain mask to generate more realistic, non-uniform masks.

We thank Reviewer bmcM for his\her efforts. We are encouraged by the reviewer’s positive comments on the dataset contribution and very helpful research. We answer the questions below and will incorporate all feedback in the revised version.

1. Novelty on color space transformation

Our novelty in color space transformation lies in introducing a learnable nonlinear coefficient matrix to replace the traditional fixed linear transformation used in previous methods (as in the reviewer-mentioned paper [1], which describes “using a linear transformation defined by standard conversion matrices”). Specifically, as shown in Equations (8) and (9) of the main paper, each transformation weight is replaced with a learnable variable and further processed through a multilayer perceptron (MLP) for nonlinear transformation. This approach not only retains the ability to handle luminance during the color space conversion but also enables adaptive parameter adjustment based on specific datasets and application scenarios. It adapts to various lighting conditions, scene types, and content characteristics, making the extracted features more robust to the complexity and randomness of nighttime rain effects. We will add a discussion with [1] in the revised paper.

Ref:

[1] Fang, Wenxuan, et al. “Guided real image dehazing using YCbCr color space.” AAAI 2025.

2. Innovations in network design and training process

We would like to claim that the main innovation of our method lies in formulating a new end-to-end trainable two-stage framework for better nighttime image deraining. We decompose the complex nighttime deraining task into a degradation removal stage and a color refinement stage for joint learning. Although we use some common modules and loss functions as the base backbone, our focus lies in developing two new components to better help the feature learning of these two stages. Firstly, we develop a learnable Color Space Converter (CSC) in the degradation removal stage to better facilitate nighttime rain removal in the Y channel. Second, instead of employing explicit illumination models for brightness estimation as in existing methods, we introduce a new Implicit Illumination Guidance (IIG) module in the color refinement stage, which leverages implicit neural representation technology to encode complex illumination information. We will clarify this in the revised paper.

3. Issues of t-SNE distribution

The t-SNE distribution figure shows that our dataset more closely aligns with the distribution of real-world nighttime rainy images. To prove this and avoid the influence of different background content on the results, as stated in the caption of Figure 3 in the main paper, we select samples with similar nighttime street backgrounds and use ResNet50 to extract features. After applying t-SNE dimensionality reduction, images from the same dataset exhibit clustering due to the similarity in rain streak features. By doing so, the distance between different data distributions more accurately represents the size of their rain distribution differences. Note that, due to the domain gap between the synthetic dataset and real-world data, there is low overlap between the different distribution clusters.

In this paper, our goal is to enhance the quality and realism of the synthetic nighttime rain images to reduce the domain gap with real-world data. As shown in the t-SNE distribution visualization, compared to other existing datasets, our proposed dataset successfully narrows the domain gap with real images, which further indicates that our dataset synthesis pipeline generates nighttime rain patterns that are closer to real-world conditions.

Thank you very much! The author's reply was very detailed and addressed my main concerns. The three references provided by the reviewer regarding the application of color space in rain removal further helped me understand the paper. Best of luck with the paper!

We thank Reviewer aW1g for his\her efforts. We are encouraged by the reviewer’s positive comments on our dataset contribution, model framework innovation, and excellent performance. We answer the questions below and will incorporate all feedback in the revised version.

1. Reliance on fixed hyperparameters

In our dataset construction, our thresholding strategy is based on observations from a large number of real nighttime rainy images, where rain streaks are nearly imperceptible in extremely dark or overly bright regions. To justify our choice of hyperparameters, we have provided a user study and validation results in Sections H and L of the supplementary material, respectively. As shown in Table 11 of the supplementary material, we tested random thresholds as well as various threshold combinations. The results demonstrate that our chosen thresholds yield the best performance. To further assess the robustness and generalization capability of different threshold settings, we trained models on datasets generated with different threshold values and evaluated them on the real-world RealRain1k-L dataset. Our fixed threshold setting achieved the best performance, supporting its effectiveness and rationality.

Furthermore, as illustrated in Figure 15 of the supplementary material, we have conducted an online user study to evaluate the perceptual quality of images generated with different threshold settings. The results show that our setting received the highest preference, indicating that our dataset is more consistent with human perception and better aligned with real-world visual appearance.

2. Simplifications in the synthesis pipeline

The core of our data synthesis pipeline lies in the incorporation of illumination information from background images. The resulting non-uniform rain masks align with nighttime imaging characteristics, significantly enhancing the realism of the dataset compared to existing synthesis methods. As shown in Figure 16 of the supplementary material, user study results indicate a competitive preference for our synthesized images, suggesting better alignment with human perception. In future work, we will continue to explore advanced synthesis strategies to better reflect real-world optical complexities and further improve the realism of our dataset.

3. Limited scope to single degradation

We would like to claim that our method not only solves the deraining task but also handles mixed adverse conditions. As shown in Table 6 of the main paper, we have provided experimental results on the Multi-Weather6k dataset, which includes rain, snow, and haze scenarios. The results indicate that our model has promising potential for restoring images degraded by mixed adverse weather conditions.

As for the robustness in more complex multi-degradation scenarios, we have evaluated on the RealRain1k dataset, which includes both daytime and nighttime rainy scenes, and the RainDS-real dataset, which contains rain streaks, raindrops, and their combinations, as shown in Table 2 of the main paper. The results demonstrate that our method achieves robust performance across these diverse real-world conditions. Qualitative results are presented in Figures 24, 25, and 26 of the supplementary material.

The effectiveness of our model in addressing the above issues stems from its core design: a learnable mapping from the RGB color space to a luminance-dominated domain, which enables more effective degradation modeling. Since degradations such as snowfall also significantly affect image brightness and contrast, their visual artifacts become more prominent in the Y channel. As a result, CST-Net is not limited to nighttime rain degradation but can also handle other degradation types that impact luminance structures. In addition, our proposed Implicit Illumination Guidance (IIG) helps disentangle illumination and structure at the feature level, allowing the model to capture degradation-specific patterns under diverse conditions and guide the restoration process accordingly. Finally, our training paradigm does not rely on degradation-specific structural priors, thus demonstrating strong transferability across various weather-related degradation scenarios.

For rain combined with fog or haze, we have discussed the limitations in Section 6 of the main paper. In future work, we will further consider expanding the extra dataset under low-light conditions to include veiling effects and explore the incorporation of physical models to address this problem.

4. Issues of the data synthesis pipeline

The traditional linear addition method adopts Equation (1) in the main paper, $R_s=B+S$, where rain streaks $S$ are simply added to the clean background $B$. This simpler addition ignores the interaction between rain streaks and the background, often resulting in a noticeable “sticker-like” effect. This is particularly evident around object boundaries, where rain streaks may unnaturally overlap with edges. As shown in the example image in Figure 3(a) of the main paper.

In our paper, we adopt Equation (2) in the main paper, $R_s=f[B,\sigma(S)]$, where a 3×3 convolution operation $f[\cdot]$ is applied to model the local interactions between rain streaks and the background. Rather than treating rain as pixel-wise additive noise, this approach introduces contextual perturbations to surrounding pixels, more accurately reflecting the physical process of rain formation in real-world imagery.

We further compare the impact of different fusion strategies on the final dataset's quality, and the results are shown in the table below. The quantitative results show that our merge strategy achieves the best performance in the NIQE [1] and NRQM [2] non-reference metrics, significantly improving the quality of the dataset. We will add these results in the revised paper.

Ref: [1] Anish Mittal, et al. “Making a completely blind image quality analyzer.” SPL 2012. [2] Ma, Chao, et al. “Learning a no-reference quality metric for single-image super-resolution.” CVIU 2017.

5. Effect of data synthesis pipeline components on model performance

According to the reviewer’s suggestion, we further add experiments to evaluate the effect of each component in the data synthesis pipeline on the final model’s performance. Specifically, we analyze our model performance on datasets generated using different synthesis pipeline configurations, and the results are shown in the table below. As observed, our data synthesis pipeline incorporates convolutional merge $f[\cdot]$, illumination merge $\sigma[\cdot]$, and defocus blur $\rho[\cdot]$, which better reflect real-world degradation conditions. We will add these results in the revised paper.

6. How the model performs in colored rain streak removal?

As shown in Figure 22 of the supplementary material, our method effectively removes colored rain streaks influenced by yellow light sources, demonstrating strong robustness in handling colored rain streaks. If the Cb/Cr channel features are directly passed to the color refinement stage, it would not cause color artifacts. The underlying reason is that we designed the Implicit Illumination Guidance (IIG) module, which guides the model’s attention toward complex rain streaks near light sources (including colored rain streaks) and extracts implicit degradation information to guide the second-stage color refinement process.

We thank Reviewer jK5Y for his\her efforts. We are encouraged by the reviewer’s positive comments on the dataset quality, effective method, paper presentation, and solid experiments. We answer the questions below and will incorporate all feedback in the revised version.

1. Extend to video deraining and snow removal

For snow removal task, we have provided the experimental results on the multi-weather dataset Multi-Weather6k (including rain, snow, and haze) in Table 6 of the main paper and elaborate on them in Section 6.5, demonstrating that our method is also effective for image desnowing.

For the video deraining task, we use the RainVID&SS [1] dataset, where the Cam-Vid subset consists of training frames extracted from 3 long clips and testing frames extracted from 2 long clips. We adopt this subset and train all models using patches of size 128×128 pixels. We compare our method with two video deraining approaches, S2VD [2] and MPEVNet [1], and the quantitative results are shown in the table below. The results demonstrate that our CST-Net still exhibits strong restoration potential in the video deraining task. We will add these results in the revised paper.

Ref:

[1] Sun, Shangquan, et al. “Event-aware video deraining via multi-patch progressive learning.” IEEE TIP 2023.

[2] Yue, Zongsheng, et al. “Semi-supervised video deraining with dynamical rain generator.” CVPR 2021.

2. Issues of Fig. 1

The histogram statistics in Fig. 1 are based on multiple paired images from the public RealRain1k dataset. Due to rebuttal format limitations, we are unable to provide corresponding image pair examples here. Note that we normalize the background luminance before analyzing the rain effects in the Y channel. By doing so, our approach ensures that the analysis focuses on the nighttime rain features themselves, rather than being dominated by lighting conditions in the scene. We will clarify this in the revised paper.

3. Missing references

Thanks for the suggestion. We will add the suggested references in the revised paper.