Night-to-Day Translation via Illumination Degradation Disentanglement

Q1: Omission of a key citation.

R1: We apologize for the omission of this key citation, which has caused confusion. We will ensure that the key citation is included in the revised version.

Q2: Why are four types of degradation considered?/What is the rationale behind categorizing degradation into four types?

R2: At nighttime, the intensity of illumination is the most important criterion for determining whether patterns are far from each other, which categorizes nighttime images into three non-overlapping regions: high-light, well-lit, and darkness. However, within well-lit regions, colored illumination still results in complex patterns with similar intensity levels, which require further subdivision. Therefore, we derive an invariant to extract features related to colored illumination and propose categorizing and disentangling patterns into darkness, well-lit regions, light effects, and high-light to address these challenges.

We hope our explanation can address your concerns.

Q3: The paper’s citation style is inconsistent.

R3: Thank you for your advice. We will revise the citation and ensure the style is consistent in the camera-ready version.

Q4: why not approach the night-to-day translation task using diffusion models?

R4: Thank you for your advice. We acknowledge that diffusion models have significant potential for addressing translation-based image tasks. However, applying diffusion models directly does not effectively solve the night-to-day translation problem. This task is fundamentally a restoration problem that requires recovering information rather than merely performing style transfer, as seen in day-to-night translation tasks. The lack of paired training data further complicates night-to-day translation, making current supervised diffusion backbones insufficient.

While we acknowledge the value and potential of diffusion models for night-to-day translation, further research into the mechanisms of nighttime imaging and the development of techniques for extracting significant information from such images are more crucial. These advancements will substantially influence the effective application of diffusion models in night-to-day translation tasks, particularly in selecting controllable information as conditions in the diffusion process.

Q5: Could degradation be categorized into more types? Could a segmentation model be trained for this purpose?

R5: Yes, the degradations can be categorized into more types based on varying levels of illumination density, different types of colored illumination, scattering, and reflective flare. However, apart from degradations with distinct edge structures, such as scattering flares, the efficacy of segmentation models in extracting additional types of degradation is limited.

In my opinion, research into the mechanisms of imaging and optical systems, combined with physically informed machine learning, represents a more promising approach for ultimately addressing night-to-day translation and related nighttime imaging tasks. We sincerely appreciate your advice.

Response to Academic Integrity

Thank you for your valuable comments.

First, we want to assure you that academic integrity is paramount to us. We have taken this feedback seriously and have implemented additional checks in our manuscript preparation process to prevent such oversights in the future. The omission was due to an unfortunate lapse during our final citation review process. Citation [1] was included in an earlier version of the manuscript but was inadvertently removed during subsequent revisions, and this omission went unnoticed. Additionally, in Sections 3.1 and 3.2, we mistakenly believed the paper was cited and retained simplified statements, which led to difficulties in understanding. We sincerely apologize for this oversight. We have thoroughly revised our manuscript and now explicitly cite [1] in all relevant sections, including Eq. (5) and Eq. (9).

Second, while our work builds on the invariant presented in [1], there are several key differences:

• The different photometric model. As the invariant in [1] are derived from the photometric model,$E(λ,x)=e(λ)i(x) R_{\\infty}(λ,x)$, which is tailored for the colored uneven illumination, our invariant are derived from the photometric model :E(\\lambda, x) = \\{ \\begin{array}{ll} e(\\lambda, x) & \text{if } x \\notin \\Omega, \\\\ e(\\lambda, x)R(\\lambda)C(x) & \\text{if } x \\in \\Omega \\end{array} , designed to describe complex illumination in nighttime environments.

• The different characteristics. The different characteristics. The Lemma 8 in [1] introduces the characteristic that extracts object reflectance, specifically edge-relevant features in images. In contrast, the invariant in our work, described in Eq (6), demonstrates the ability of extracting of light effects, which are degradation-related features specific to nighttime conditions

• The different computation process. Unlike the computation process in [1], which focuses on extracting high-frequency features such as edges, our work employs two additional normalization and activation functions to extract relatively low-frequency light effects. These steps are specially designed for nighttime disentanglement.

These differences uniquely demonstrate our invariant’s ability to extract light effects in nighttime images, both empirically and theoretically. This specific invariant is derived from our nighttime photometric model, which was developed through our original research.

Despite these distinctions, we fully understand the importance of giving proper credit to prior work. We commit to including citation [1] in the revised version and thank you again for your valuable comments.

[1] Color Invariance. J. M. Geusebroek, R. van den Boomgaard, A. W. M. Smeulders, H. Geerts. IEEE TPAMI, 2001.

Response to Technical Soundness

Thank you for your comments.

Enhancement-based methods prioritize preserving information from the night domain. However, night-to-day translation has higher requirements, prioritizing the conversion of images to the daytime domain first, while maintaining semantic consistency. We acknowledge that the GAN-based methods are not perfect and may result in the semantic changes. To address this, we developed a degradation-aware contrastive learning approach designed to maintain semantic consistency and ensure successful translation. Our strategy has proven effective, significantly outperforming earlier GAN-based night-to-day works [1, 2, 3].

Additionally, our method demonstrates significant benefits for downstream tasks, such as nighttime image localization and semantic segmentation, compared to enhancement-based methods. As shown in Table 1 of the main paper, our approach achieves a 5.26 mIoU improvement and a 9-point gain in SIFT scores compared to the most advanced enhancement-based methods, indicating greater potential for enhancing downstream performance.

Thanks for your comments and we wish these explanations can address your concerns.

[1] Night-to-day image translation for retrieval-based localization. A. Anoosheh, T. Sattler, R. Timofte, M. Pollefeys, and L. V. Gool. ICRA, 2019.

[2] Forkgan: Seeing into the rainy night. Z. Zheng, Y. Wu, X. Han, and J. Shi. ECCV, 2020.

[3] Adverse weather image translation with asymmetric and uncertainty-aware gan. J. Kwak, Y. Jin, Y. Li, D. Yoon, D. Kim, and H. Ko. BMVC, 2021.

Thank you for your valuable feedback.

First, in our photometric model, we assume that the reflectance function $R_{\\infty}(λ,x)$ can be decomposed into the product of two dependent functions: $R(λ)$ and $C(x)$. This decomposition is grounded in the following assumption and definition.

We assume that materials are uniform and homogeneous within a local area under normal conditions. Specifically, the optical properties of a material within a small region are described by the function $R(λ)$, which characterizes the material's properties as a function of wavelength and is independent of location. Similar assumption is also used in materials science, optics and computer vision [1, 2, 3]. Under this assumption, we can simplify the reflectivity function $R_{\\infty}(λ,x)$ in the local area to $cR(λ)$, where $c$ is coefficient that describe the material type.

However, this model is limited to describing photometric properties in a local area and does not capture global nighttime conditions. To address this limitation, we introduce the material spatial distribution function $C(x)$ is defined as: $C: \\mathbb{R}^n→{c_1,c_2,…,c_m}$. With $C(x)$, we can model more complex nighttime scenes with diverse material types at macro scales, as detailed in Eq (4) of our paper.

The function $C(x)$ does not influence the derivation of subsequent invariant and their properties. In the derivation of our invariant, the function $C(x)$ appears in both the numerator and denominator of the derivative fraction and cancels out, as shown in Eq (6) of our paper. Nevertheless, the introduction of $C(x)$ is necessary to ensure that our photometric model accurately represents and describes the nighttime environment.

We will emphasize these points in the revised version.

[1] Materials science and engineering: an introduction. W. D. Callister and D. G. Rethwisch New York: Wiley, 1999.

[2] Optical properties of solids. E.A. Moore and L.E Smart. CRC Press, 2020.

[3] Color invariance. J. M. Geusebroek, R. van den Boomgaard, A. W. M. Smeulders, H. Geerts. IEEE TPAMI, 2001.

Second, we present additional ablation studies on the four components, as detailed in the following tables. The studies reveal that while performance slightly improves with refined classification into four clusters, a more accurate segmentation based on our physical model significantly enhances performance and achieves optimal results. The challenge arises from the similarity in intensity between light effect regions and well-lit areas, making it difficult to differentiate them using a simple KNN. Our physical prior, which extracts features beyond intensity, enables better subdivision and contributes significantly to the final performance.

Third, in our experiments, $L$ refers to the setting where only three clusters—darkness, well-lit, and high light—based on illumination intensity are used for disentangling. In contrast, $N$ in Table 2 indicates that we further incorporate physical priors to extract light effects during disentangling, using four clusters—darkness, well-lit, light effect, and high light. We will provide additional clarification on this aspect to enhance understanding.

Thank you to the authors for their further clarification. My concern regarding academic integrity is mostly resolved, though I still find it unusual to use a first-person narrative like "we develop" when referring to results derived from other literature (e.g., Eq. (9)). This phrasing suggests that the person writing the paper may not be the one who proposed the method, conducted the experiments, or fully understood the related work. However, I would now like to focus on new questions that have emerged as I delve deeper into the technical details of this paper.

Firstly, I agree with the authors that their image formation model differs from the one used in [1]. Specifically, [1] assumes a matte, dull surface with Fresnel reflectance set to 1, while this paper employs a model where Fresnel reflectance can either approach 0 or 1. However, I find the derivation from Equation (2) to (3) problematic. Specifically, $R_\infty(\lambda, x)$ cannot be decomposed into two separate functions dependent on $\lambda$ and $x$, respectively.

The explanation following Eq. (2) states that for a given pixel location $x_0$, $R_\infty$ only depends on $\lambda$. While this is obvious since $R_\infty$ depends on two variables $x$ and $\lambda$ only, it does not logically lead to the conclusion that $R_\infty(\lambda, x)$ can be decomposed into $R(\lambda)C(x)$.

For instance, consider a simple model $R(\lambda, x) = \lambda x +1$ (it does not refer to a specific physical model, but it satisfies the statement "for any local pixels, the material reflectivity is determined if the material is given"). If functions $C$ and $R$ exist such that $R_\infty(\lambda, x) = R(\lambda)C(x)$, then setting $x = 0$ would imply $R(\lambda) = \frac{1}{C(0)}$ for all $\lambda$, which is incorrect since $R$ should not be a constant function.

Therefore, further assumptions are necessary for reaching Eq. (3).

Additionally, I have concerns regarding the experiments presented in this paper. Specifically, while the experiments show that using a four-component decomposition is better than a three-component one, they do not demonstrate that this four-component decomposition is optimal or superior to a heuristic approach, such as classifying the pixels into four clusters using a k-nearest algorithm. This ablation is necessary as the qualitative results in Figure (3) seem similar to a simple intensity (RGB) based separation of light-effect and well-lit areas.

Lastly, could the authors clarify what the invariant $L$ represents in Table 2 and Line 278?

Overall, I would like more clarification on Eq. (3) and will reconsider my rating after reviewing the authors’ additional response.

Thank you for clarifying the photometric model and notations and providing the additional experiments.

However, as I mentioned under Technical Soundness, I believe maintaining semantic consistency is crucial in night-to-day translation, an area where the proposed method falls short. A more straightforward alternative to the GAN-based approach could involve learning a per-pixel curve [1, 2] for different regions identified by N2D3, rather than applying a single curve across the entire image.

[1] Li et al., Learning to Enhance Low-Light Image via Zero-Reference Deep Curve Estimation, TPAMI, 2021.

[2] Wang et al., Unsupervised Face Detection in the Dark, TPAMI, 2023.

Additionally, the intuition behind the 4-category segmentation is not fully explained. While I acknowledge its superiority over 3/4-cluster KNN-based segmentation, I remain concerned about the scalability of ‘Degradation-Aware Contrastive Learning’ when faced with more degradation types, as each patch may contain pixels from multiple categories, and patches of different categories may not necessarily form a negative pair. This limitation raises doubts about the method’s foundation for future work.

Lastly, as acknowledged, the paper lacks essential explanations of the photometric model, notations, and important citations.

Given these points, I believe a thorough revision is necessary before acceptance, and I therefore raise my score to a borderline reject.

Thank you for your valuable feedback.

First, we recognize that maintaining semantic consistency is crucial in this field. Our methods successfully preserve global semantic consistency across most scenes, though slight artifacts may appear in finer details. Nonetheless, our approach significantly outperforms previous methods in this regard and shows advantages in downstream tasks.

Moreover, our methods are fundamentally different from Zero-DCE [1], as Zero-DCE can not translate a nighttime into the daytime domain (in terms of FID in Table 1), despite GANs are utilized. In downstream tasks, our approach also outperforms Zero-DCE as shown in Table 1.

However, we still acknowledge that introducing per-pixel curves for different degradation types could be a valuable direction for future research. We appreciate your suggestion and will consider it in our ongoing work.

Second, for the two misunderstandings that have caused the concerns about the scalability, we would like to make the following clarifications:

Regarding the misunderstanding about multiple categories, we clarify that each patch in our method originates from a single category, with no overlap between different degradation types. Categories such as darkness, well-lit, and high light are uniquely labeled based on illumination intensity using KNN. The light effect regions are decomposed from well-lit regions, as detailed in Eq (11) of our paper, and the refinement operation $M_n←M_n-M_le$ in line 158 ensures no overlap.

Regarding the misunderstanding that patches from different categories may not necessarily form negative pairs, we clarify that negative pairs are formed within the same category, not across different categories. This ensures that all negative examples are hard negative examples, meaning they are sufficiently similar to the query patch but not identical. Mainstream theoretical studies indicate that hard negative example mining can enhance the performance of contrastive learning, which is consistent with our empirical results. Our method demonstrates performance advantages compared to randomly selects patches as negative samples (CUT [2]), indicating that our approach effectively mines negative samples.

Moreover, to further mitigate potential disentanglement errors, we introduced a reweight matrix based on optimal transport, which reassesses the weights for the sampled pairs to ensure optimal negative sample mining. The quantitative comparison in Table 2 confirms that our disentanglement approach significantly enhances performance through effective hard negative sample mining, with additional improvements achieved through the reweighting operations.

Third, we have provided a detailed explanation of these aspects in our previous comments and appreciate your recognition of our earlier explanations. In the revised version, we will add the key citations in line 114, and include the assumptions and notations between Eq (2) and Eq (3).

[1] Learning to enhance low-light image via zero-reference deep curve estimation. Li et al., TPAMI, 2021.

[2] Contrastive learning for unpaired image-to-image translation. T. Park. A. A. Efros, R. Zhang, J. Y. Zhu. ECCV 2020.

Q1: Clarity of the method sections can be improved. For instance, including more rigorous definitions or visualizations of what well-lit and different light effects mean and provide a motivation why it is helpful to disentangle those illumination causes separately.

R1: Thanks for the advise. A key observation is that, when perform Night2Day translation, treating all regions equally leads to significant artifacts by mixing different patterns from various regions. For example, mixing light effects patterns with high-light patterns, as shown in the Figure 1 of the main paper. Intuitively, separating these patterns and regularizing the structure will help mitigate these artifacts.

Based on this intuition, we sought a disentanglement strategy to separate these patterns. At nighttime, the intensity of illumination is the most important criterion for determining whether patterns are far from each other, which categorizes nighttime images into three non-overlapping regions: high light, well-lit, and darkness. However, within well-lit regions, colored illumination still results in complex patterns with similar intensity levels, which require further subdivision. Therefore, we derive an invariant to extract features related to colored illumination and propose categorizing and disentangling patterns into darkness, well-lit, light effects, and high-light.

We hope our explanation can address your concerns.

Q2: The authors can also add proper citations to previous work when they mention “by common practice”, for instance in line 107, line 142 and line 196.

R2: Thanks for the advice. We will revise these problems in the camera-ready version.

Q3: Why is illuminance computed by the maximum of RGB channel (line 107)? I think this is indeed not a common practice.

R3: This estimation operation is broadly used in low-light image enhancement work for rough illumination estimation, such as in LIME (TIP 2016), URetinex-Net (CVPR 2022), and PairLIE (ICCV 2023). We hope these reference can solve your concerns.

Q4: Could you provide some ablation visualizations with and without applying the disentanglement method in 3.2?

R4: We provide such ablation visualization in the PDF file of the Author Rebuttal. Thanks for your advice again.

Q5: Can you provide some visual examples of the segmented regions of those previous methods and your method?

R5: We provide visual examples of the segmented regions comparison in the PDF file of the Author Rebuttal. Thanks for your advice again.

Q6: I wonder how the authors think the framework could benefit or inspire other types of adverse weather image restoration tasks.

R6: The core of this framework is utilizing physical priors to disentangle the mixture of complex patterns in degraded images, allowing the generator to learn these patterns more effectively. From a metric learning perspective, this approach acts as physical-informed hard negative example mining during contrastive learning. We believe that this disentanglement framework can also benefit other types of adverse weather image restoration with translation-based methods, as it leverages related physical priors to extract corresponding degradation patterns.

Q1: The writing is difficult to understand. The explanations and derivations for Eqs (1) to (5) lack logical coherence and necessary references, making them hard to follow. The derivations for Eqs (7) to (9) also lack supporting references, casting doubt on their validity.

R1: We apologize for the omission of a key citation, which has led to poor readability in this part, as explained in the author rebuttal. Moreover, Eq (1) is derived from [1], and Eqs (2)–(4) are the simplified photometric model tailored for nighttime environments proposed by us. Eq (5) presents the invariant from [1], and we derived new characteristics based on our simplified model in Eq (6). From Eqs (7)–(9), we follow [1] to compute the basic invariant and refine it for disentangling. We hope this explanation address your concerns.

[1] Color Invariance. J. M. Geusebroek, R. van den Boomgaard, A. W. M. Smeulders, H. Geerts. IEEE TPAMI, 2001.

Q2: The motivation is unclear. Please explain the motivation behind it.

R2: A key observation is that, when perform Night2Day translation, treating all regions equally leads to significant artifacts by mixing different patterns from various regions, as shown in the Figure 1 of the main paper. Intuitively, separating these patterns and regularizing the structure will help mitigate these artifacts. At nighttime, the intensity of illumination categorizes nighttime images into three non-overlapping regions: high light, well-lit, and darkness. However, within well-lit regions, colored illumination still results in complex patterns with similar intensity levels, which require further subdivision. Therefore, we derive an invariant to extract features related to colored illumination and propose categorizing and disentangling patterns into darkness, well-lit, light effects, and high-light.

We hope this explanation address your concerns.

Q3: There is no baseline network, making it difficult to determine the performance gain for the specific module.

R3: The baseline network is the CUT [2], which operates without any physical information guidance in contrastive image translation. This baseline is included in the performance comparison in the Table 1, showing an improvement of 13.8 in FID scores and 9.84 SIFT scores on the Alderley dataset. Additionally, it shows an improvement of 24 in FID scores and 12.25 in mIoU on the BDD100k dataset compared to the baseline. Our ablation study in Table 2, Section 4.4, identifies the effectiveness of each module. We promise to highlight these improvements in the main paper for better understanding.

[2] Contrastive learning for unpaired image-to-image translation. T. Park. A. A. Efros, R. Zhang, J. Y. Zhu. ECCV 2020.

Q4: For instance, there are no ablation experiments to verify the impact of introducing four regions for disentanglement versus three regions (e.g., excluding the light effects region).

R4: We have provided this analysis in Section 4.4, as shown in the second table of Table 2. When only $L$ is activated, it indicates that we are only incorporating darkness, well-lit, and high light for disentangling. When only $L$ and $N$ both are activated, it represents the full method. The full method demonstrates over a 10-point improvement in FID performance compared to using only three types of disentanglement across both datasets. We promise to clarify this expression for better understanding.

Q5: There is a need to compare with more recent methods for unpaired image-to-image translation, such as COCO-FUNIT, StegoGAN, GP-UNIT, etc.

R5: We provide a comparison on the two datasets with the most advanced method, StegoGAN (CVPR 2024). It is clear that our method consistently outperforms StegoGAN, demonstrating superior performance. We will include this in the camera-ready version.

Q6: Please check the reference [5], as it does not seem to be published in CVPR.

R6: This paper was published in the CVPR Workshop 2023. We will revise this in the main paper. Thank you for your reminder.

Q7: How is the reweighting matrix obtained through optimal transport? Are there any references that have implemented this approach?

R7: The reweighting matrix is obtained by solving the optimization problem in Eq (14). A similar approach is employed in MoNCE (CVPR 2022), which is also compared in our experiments. Unlike MoNCE, which computes the reweighting matrix across entire image patches, our method designs the reweighting matrix specifically for each degradation regions. This tailored approach results in over 20 FID score improvements on the Alderley dataset and nearly 10 FID score improvements on the BDD100K dataset compaired to MoNCE, as shown in Table 1.

Q8: Contrastive learning has been widely used in image translation. The paper lacks ablation experiments to support the performance gains from the reweighting operation.

R8: We discuss the performance gains from the reweighting operation in the first subtable of Table 2 in Section 4.4 of the main paper. The notation (b) represents the degradation-aware reweighting operation. This table shows nearly a 5 FID score improvement from the reweighting operation on both two datasets.

Dear Reviewer rJuz,

Thank you for taking the time to review our submission and for your constructive comments and favorable recommendation. We would like to confirm whether our responses have adequately addressed your earlier concerns. If you have any additional questions or suggestions, we would be happy to address them to further enhance the quality of our paper.

Best regards,

Authors

Q1: Although experimentally shown to be effective, the mechanism and the basic ideas behind leveraging the illuminance distribution as well as the physic priors for realizing disentanglement of four degradation categories (i.e. darkness, well-lit, light effects, and high-light) are not well explained, in which the physical meanings for Eq.1 to Eq.11 are hard to understand and follow.

R1: Thanks for your valuable feedback.

First, we will explain our basic ideas behind leveraging the illuminance distribution as well as the physic priors for realizing disentanglement of four degradation categories. At nighttime, the intensity of illumination is the most important criterion for determining whether patterns are far from each other, which categorizes nighttime images into three non-overlapping regions: high light, well-lit, and darkness. However, within well-lit regions, colored illumination still results in complex patterns with similar intensity levels, which require further subdivision. Therefore, we derive an invariant to extract features related to colored illumination and propose categorizing and disentangling patterns into darkness, well-lit , light effects, and high-light.

Second, we apologize for the omission of a key citation, which made it challenging to understand the derivations from Eq (1) to Eq (11). Specifically:

• Eq (1) is derived from [1].
• Eqs (2)–(4) present a simplified photometric model tailored for nighttime environments, proposed by us.
• Eq (5) introduces the invariant from [1], and we derive new characteristics based on our simplified model in Eq (6).
• Eqs (7)–(10) follow [1] for computing the basic invariant and refining it for disentangling.
• Eq (11) employs the estimated light effect to refine the mask and extract light effect regions from the original well-lit regions.

We will include these clarifications and the key citation in the revised version to improve understanding.

[1] Color Invariance. J. M. Geusebroek, R. van den Boomgaard, A. W. M. Smeulders, H. Geerts. IEEE TPAMI, 2001.

Q2: As the illumination degradation disentanglement plays a key role in the proposed method, it would be great to have the robustness analysis on such disentanglement if possible and how would the translation model learning be affected once there are erroneous disentanglement?

R2: We agree that a robustness analysis of disentanglement is important. Unfortunately, conducting such research is nearly impossible due to the lack of illumination-related annotations. Creating such a dataset is also challenging due to the absence of unified measurement criteria in this field. Despite this, we provide a task-oriented evaluation of different disentanglement strategies by comparing final performance in the second subtable of Table 2 in the main paper. This table demonstrates that the 4 types of degradation disentanglement leads to significant performance improvements compared to the initial 3 types, indicating that the proposed 4 types disentanglement is empirically more reasonable.

We are committed to advancing this area and will continue to work on robustness analysis, including proposing measurement standards to make the field more complete.

Dear Reviewer JHwd,

Thank you for taking the time to review our submission and for your constructive comments and favorable recommendation. We would like to confirm whether our responses have adequately addressed your earlier concerns. If you have any additional questions or suggestions, we would be happy to address them to further enhance the quality of our paper.

Best regards,

Authors