CLODE: Continuous Exposure Learning for Low-Light Image Enhancement using Neural ODEs

We are glad to hear that you found the paper to be well-written and structurally organized. We have diligently examined your comments and concerns as a reviewer, and have prepared responses addressing the raised concerns.

W1: Reference format

• Thanks to your thorough suggestion, we believe we can make the reference format consistent. We promise to make the revision in the final version.

Q1: Hyper-parameters

• The exposure level parameter $E$ in Eq.(11) guides the network to train the resulting final brightness of the output image.
• For the exposure level parameter, we followed the prior settings of previous curve-adjustment based methods ([6, 9]). To ensure fairness and eliminate any effects from changes in $E$, we consistently used the same value.
• For the setting of hyper-parameters in Eq.(17), is detailed in L459-461 of the Appendix. As we provided in the Appendix A.1, the weights for each loss function are set to balance the scale of losses. To reiterate, the weights for the loss function $w_{col}$, $w_{param}$, $w_{spa}$, $w_{exp}$ and $w_{noise}$ are set to 20, 200, 1, 10 and 1 respectively.

Q2: Challenge of applying ODE to continuous exposure learning.

There were three main challenges in applying ODE to continuous exposure learning: inference speed, tolerance setting, and time consumption in training.

[Inference speed]

• First challenge is inference speed as written in L330-332. Since Neural ODE requires iterative processes to find the optimal solution, can result in somewhat slower inference times. To tackle this problem, we provide CLODE-S which is a compact version composed of a 2-layer network (R-fig. 1(b)). Although CLODE-S takes 0.0004M parameters and takes 0.005 second for image inference, CLODE-S shows promising performance in Table 5. Improving the inference speed of CLODE is our future research goal.
• One possibility is to apply RectifiedFlow [1*] as mentioned in UK8A W2. RectifiedFlow[1*] transforms the solution paths of Neural ODEs into straight lines, enabling faster estimation of the Neural ODE system. We can first assume that the optimal solution found by CLODE is the expected optimal solution and then apply CLODE specifically to [1*].
• Given the potential to apply such cutting-edge Flow Matching methods, we believe that CLODE is highly promising and can achieve fast inference speeds.

[Tolerance setting]

• In addition, setting the tolerance is crucial in the NODE system because the ODE solver determines the state is optimal and has to be terminated, by the error of the current state within the allowable error rate. The error rate is defined by the following formula: $\Gamma_t = atol + rtol \times \text{norm}(Err_t)$
• Here, $Err_t$ is the current state error, and atol and rtol represent the absolute and relative tolerance, respectively.
• If $Err_t > \Gamma_t$, the ODE solver adjusts the step size to minimize the error.
• If $Err_t \leq \Gamma_t$, the current image is considered the optimal solution state, and the process terminates.
• Since $\Gamma_t$ is calculated by atol and rtol, setting these tolerance value is important. If atol and rtol are too small, training may fail, and if they are too large, effective brightness enhancement may not be achieved. As mentioned in the Appendix. L472, CLODE empirically sets atol and rtol to $1e^{-5}$. For additional details about CLODE, please kindly refer to our global rebuttal G1.

[Time consumption in training]

• Finally, as NODE involves simulation-based training, the model takes longer to train as it grows in size. To overcome this obstacle, we designed the architecture of CLODE to be as compact as possible while maximizing low-light image enhancement performance. For a better understanding of "simulation-based learning", we provide a schematic representation of CLODE (dopri5) in actual image enhancement at the bottom of R-Fig.2.

Q3: Figure 4

• We understand Reviewer Qjwa's curiosity about Fig.4. We can explain the reason why the background of Fig.4 is quite different from the ground truth in two aspects.
• One reason is the unsupervised methodology of CLODE. Since training is done without ground-truth images, CLODE improves the input images using only the given information from itself. The other reason comes from the lack of information in the input images. In some overexposed images in the SICE dataset, the overexposed regions do not contain the same details as the ground truth images.
• In particular, the lack of information in the input image causes the same problem with supervised methods too, as shown in Fig.4. CLODE’s enhancement results may not match the ground truth perfectly, but despite this, our method performs better than other unsupervised methods and competes well with supervised methods. To address issues with over-exposed images, we will need to use a generative model, which we plan to explore in future research.

Thank you very much for taking the time to review our work. If there are any additional questions or points, we would be delighted to address them.

References

[1*] Liu, Xingchao, and Chengyue Gong. "Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow." In ICLR, 2023

We would like to thank the reviewer for recognizing our contribution and pointing out constructive concerns. We have made efforts to address the reviewer’s concerns as follows.

Q1, W7: More explanation of CLODE

Due to character limitation we placed explanation of this question in global rebuttal. Please refer to global rebuttal G1 to address reviewer's concern. We have prepared explanations along with additional analysis (R-fig. 2). In brief, CLODE includes an early stop mechanism.

W1, W4: User Control

• First, we would like to cautiously mention that CLODE, even without user-control, is already state-of-the-art.
• In Table 1 and 2, CLODE+ was determined based on the average values of images selected by two experts in the field.
• To address Reviewer Z6DS’s curiosity, we conducted an extra user study on the LOL dataset with 21 participants who had no prior knowledge of the ground-truth images. The results are in R-Table 6. From the optimal state provided by CLODE, five images were generated for the participants by shifting the time steps by -0.5, -1, +0.5, and +1, respectively.
• The results of the user study fall between the values for CLODE and CLODE+. Since CLODE already produces high-quality images through optimal solutions and empirically achieves more appealing images near the optimal state, finding user-preferred images is not challenging. Furthermore, the reason for the lower results in the user study is due to exceptionally dark reference images, even though the results from CLODE+ look better. Therefore, there is no significant difference in terms of visual quality (PI).

[Image retouching]

• CLODE leverages NODE to offer user controllability as a kind of ‘free bonus feature', and this can be understood in terms of image retouching as Reviewer Z6DS's comment. However, the ability to perform image retouching through unsupervised learning is a clear advantage that makes CLODE more practical for use in diverse environments. We would also like to assert that our method demonstrates superior performance compared to various methods without user control.

W2: Noise Problem

• Before the explanation, we would like to clarify that $I_T$ contains noise.​
• In Section 3.2.1, the input to the ODE function is $I_t$, which is passed through the Noise Removal module to obtain the denoised image $\tilde{I_t}$ as shown in Eq.(7). The denoised $\tilde{I_t}$ is used as input to the Curve Parameter Estimation module and is provided by Eq.(8). The reason for using the denoised $\tilde{I_t}$ as an input of Curve Parameter Estimation module is to obtain a fine-grained curve parameter map $\mathcal{A}_t$. The final output for the ODE function is expressed as $\mathcal{A}_t \otimes I_t \otimes (1-I_t)$ (Eq.(9)), where we utilize $I_t$ rather than the denoised image $\tilde{I_t}$. The reason for not using $\tilde{I_t}$ is that it is difficult to preserve the details of the input image when image enhancement is performed by repeatedly denoising the image. Thus, the mentioned $I_T$ is the result of the improvement using the fine-grained curve parameter map, and since no direct denoising is performed in the iteration process, $I_T$ contains some noise. Lastly, to obtain a noise-free image $\tilde{I}_T$, we apply the Noise Removal module to $I_T$ as described in L196-197.

W3: Experimental step

• The reason for using a single training dataset for each task is to ensure a fair comparison with previous methods that include both supervised and unsupervised approaches.
• Since most supervised methods’ official weight parameters are trained on a single dataset, we adopted the same approach.
• We agree that training with diverse datasets is beneficial and provide results from training on all LOL and SICE datasets in R-Table 7. The overall performance is comparable to that achieved with a single training dataset.

W5: Losses

• The four losses, excluding $\mathcal{L}_{noise}$, were used in the same way as in the previous curve-adjustment methods. We acknowledge that some models do not account for noise, and although we mentioned the case without the noise module in Table 4, we kindly present a comparison again in R-Table 3. As the image becomes brighter, noise becomes more amplified, leading to a slightly lower SSIM. However, our method still outperforms other methods.

W6: Noise Module, Color Casts

• Our noise removal module consists of very few parameters (Model size: 0.085MB, 22,275). While this may result in lower performance compared to existing denoising models, it effectively learns during the image enhancement process in CLODE.
• For more explanation of the denoiser, we recommend referring to global rebuttal G2. Additionally, compared to low-light enhancement models that include noise removal ([10, 13]), the unsupervised denoising performance is competitive. (R-Table 3)
• CLODE enhances the image based on the color statistics of the input image in an unsupervised manner, which can lead to the occurrence of color casts. To elaborate, while curve adjustment methods preserve the details of the input image and enhance it to a naturalness, the color loss follows the Gray-World hypothesis (L224), leading to these issues. We use the same color constancy loss as in previous curve-adjustment method [6]. Nevertheless, in comparison to existing methods, CLODE exhibits superior performance in terms of naturalness image quality metrics and color matching histogram loss in R-Table 4.
• For further explanation on color casts, please refer to our response to 9pkv’s W1, Q1. We apologize for the lack of detailed explanation due to space limitations but assure you that we will address any additional questions thoroughly during the discussion period.

Thank you for providing a thoughtful review. For enhancing our paper, we have diligently reviewed the weaknesses and questions raised regarding our paper and have prepared additional experiments and answers.

W1: Novelty

• We cautiously wish to assert the novelty of our approach. In contrast to previous curve-adjustment methods that use discrete updates for gradual image enhancement, CLODE addresses the limitations of existing methods by reformulating them into NODE, which facilitates solving for the optimal solution in continuous space.
• In addition, we would like to refer to Reviewer UK8A's mention, our motivation is strong and solid, and we address existing shortcomings in a straightforward and effective manner.
• The proposed method shows optimal training results by reorganizing the curve-adjustment equation to NODE, which optimally handles the various exposure images in the inference time, different from a limitation of the existing curve-adjustment method. In addition, we designed a suitable network consisting of Noise Removal module and Curve Parameter Estimation module for this purpose (R-fig. (a)), as shown in Section 3.2.1 ODE function, and showed excellent performance compared with the existing unsupervised method.
• Furthermore, by offering a user controllability that utilizes the features of NODE as shown in Fig.3 in the main paper, the potential of the model has been enhanced. User controllability features are among the most effective and practical aspects of applying NODE, as they allow for customized brightness outcomes to suit individual preferences.
• In the context of unsupervised methods in low-light image enhancement, we firmly believe that the first attempt of transporting discrete curve-adjustment method problem to continuous space by neural ordinary differential equations, designing adequate architecture for the NODE method, and providing high visual performance, user-controllability, constitute meaningful contributions. Accordingly, we kindly ask for a reconsideration of the novelty of our work.

W2: References (strong supervised methods)

• In the main paper, Retinexformer is referred, a transformer-based state-of-the-art method and the second-best performer in the NTIRE 2024 low-light challenge [1*], as a strong supervised comparison method.
• We are thankful for Reviewer 9uXV’s concern and we are going to add additional strong supervised diffusion-based baselines GSAD [2*] and PyDiff [3*]. Even when compared to diffusion-based supervised methods, CLODE demonstrates competitive performance.​ We will include performances of two methods in our final revised version.

W3: Perceptual metrics

• In response to this question, we provided perceptual metrics (NIQE, BRISQUE, PI, Entropy) in Table 2 (SICE dataset) and Table 7 (LOL and LSRW datasets) in the Appendix.
• Additionally, following the reviewer’s suggestion, we provide LPIPS (Learned Perceptual Image Patch Similarity) [4*] performance results. CLODE exhibits superior average LPIPS performance compared to other methods.

W4: The writing and the presentation need improvement.

• Lastly, as Reviewer 9uXV advised, we are willing to improve writing and presentation of the paper based on comments. In the final revision, we will incorporate the reviewer’s suggestions and additional results and improve the quality of the manuscript.

Reference

[1*] Liu, Xiaoning, et al. "NTIRE 2024 challenge on low light image enhancement: Methods and results." In CVPRW, 2024.

[2*] Hou, Jinhui, et al. "Global structure-aware diffusion process for low-light image enhancement." In NeurIPS, 2023.

[3*] Zhou, Dewei, Zongxin Yang, and Yi Yang. "Pyramid diffusion models for low-light image enhancement." In IJCAI, 2023.

[4*] Zhang, Richard, et al. "The unreasonable effectiveness of deep features as a perceptual metric." In CVPR, 2018.

We are glad to hear that you found the paper to be strong and solid. We have diligently examined your comments and concerns as a reviewer, and have prepared responses addressing the raised concerns.

W1: Highlight region

• We understand the reviewer’s concern in Fig. 4. As a reminder, CLODE is trained in an unsupervised manner without ground truth images and loss functions are aimed to enhance regions of the image to the desired exposure values (e.g., $E$=0.6) while preserving spatial and color constancy based on the input image statistics.
• Although we employ spatial consistency loss and color constancy loss, because our method relies solely on the input image statistics, and the inferred color may appear grayish in enhanced images due to the Gray-World hypothesis (L224). This is a significant issue that needs to be addressed in unsupervised methods. We have spent a lot of effort searching for an appropriate color loss to apply to unsupervised methods, but we could not find a more suitable loss for environments that use only a single input image.
• However, we would like to emphasize that our model is more robust to over-exposed images compared to other models. The third row in Fig.4 compares the results of each model for a given over-exposed image, and our model is the most robust to the over-exposure situation. The tonal adjustment result is closer to the ground truth than the other methods.
• Over-exposed images partially contain pixel values outside the color range, which do not provide sufficient information for image enhancement. As can be seen in Fig.4, this issue is also present in supervised methods. However, we would like to underline that CLODE can provide visually reasonable results even for over-exposed images. Please also have a look at Fig.15 in the Appendix for additional results. Our method shows visually impressive results on various exposure conditions. Additionally, resolving issues with over-exposed images will require the use of a generative model, which we plan to explore in future research.

W2: Processing speed

• In Sec. 5 Limitations, as Reviewer UK8A concerned, we reported less inference speed compared to existing methods. As Reviewer UK8A mentioned, the inference speed of CLODE is comparable to that of RetinexFormer. However, in Table 5 and L332, we also reported CLODE-S which is a compact version of the proposed method. CLODE-S consists of 0.0004M parameters and takes 0.005 second to infer, and its architectural details are shown in R-Fig.1(b). In addition, we remain a distillation approach for shortening inference time as a future work.
• Another possibility is to apply RectifiedFlow [1*] in an unsupervised manner, which transforms the solution paths of Neural ODEs into straight lines, facilitating faster estimation of the Neural ODE system. We can initially assume that the optimal solution found by CLODE aligns with the expected optimal solution in [1*], and then apply CLODE specifically to [1*].
• Given the potential to implement advanced Flow Matching methods [1*, 2*], we believe CLODE holds great promise and could achieve rapid inference speeds.

References

[1*] Liu, Xingchao, and Chengyue Gong. "Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow." In ICLR, 2023

[2*] Tong, A., Malkin, N., Huguet, G., Zhang, Y., Rector-Brooks, J., Fatras, K., ... & Bengio, Y. "Improving and generalizing flow-based generative models with minibatch optimal transport." In Transactions on Machine Learning Research, 2024

Thank you very much for thoroughly reviewing our paper. We appreciate your feedback. To address the concerns you raised, we are providing several experimental results and our perspectives.

W1, Q1: Color cast

[What leads to the color cast?]

• CLODE enhances the image based on the color statistics of the input image in an unsupervised manner, which can lead to color casts. To elaborate, while curve adjustment methods preserve image details and enhance naturalness, color loss following the Gray-World hypothesis (L224) can lead to color cast issues as the exposure level changes.
• CLODE follows the same color constancy loss Eq. (12) as previous zero-reference based method [6] for color constancy. Nevertheless, compared to existing methods, CLODE using NODE scenario exhibits superior performance in terms of naturalness image quality metrics and color matching histogram loss. (R-Table 4)

[Color-matching histogram loss function]

• To address Reviewer 9pkv’s color casts concern more precisely, we prepare additional comparison results on color-matching histogram loss function in HistoGAN [1*] which is designed for controlling color of input image in [1*] by matching color histograms of target image and input image. We utilized color-matching histogram loss to measure color casting degrees from ground-truth images to output images.
• In the R-Table 4, we compare non-reference metrics and color-matching histogram loss results with existing unsupervised methods. The color-matching histogram loss of CLODE+ and CLODE ranked first and second-best in SICE dataset and first and third-best in LOL dataset. This indicates that CLODE exhibits fewer color casts compared to existing unsupervised methods, despite achieving high-quality enhancement results.

W2, Q2: Generalizing the method

• As suggested, CLODE can be generalized to existing curve adjustment methods. While it is possible to apply NODE to existing architectures, accurate curve estimation is crucial for high quality. Therefore, we developed a new compact and efficient architecture that can effectively apply on NODE and estimate the fine curves as depicted in R-Fig.1 (a).
• Regarding the conversion of previous methods to NODE, our experiments with the existing network [6] demonstrated that while applying NODE is feasible, it is not effective. ([6]_large* is a version of the network from [6] with increased parameters for performance improvement.) Method|#params (M)|PSNR/SSIM :---|:---:|:---: [6]+NODE|0.0794|17.17/0.571 [6]_large*+NODE|0.3593|19.26/0.637 CLODE|0.2167|19.61/0.718
• We can also set the maximum allowable steps for CLODE, but instead of setting a fixed optimal step, using optimal steps is effective for diverse image datasets. (Please refer to R-Fig.2 and global rebuttal G1 for further explanation of CLODE).
• Thus, we developed our own network (CLODE) using an adaptive solver with optimal steps that must vary for each image.
• For additional explanations regarding the optimal step, please refer to the following Q3.

Q3: Optimal step statistics

• We believe this concern is very constructive. As previously mentioned in L465, the maximum allowable step for the ODE solver is empirically set to 30, considering speed. The ODE solver terminates early if it finds the optimal solution within the maximum steps.
• Furthermore, in order to provide the statistical analysis proposed by 9pkv, the average number of steps for the ODE solver was calculated across the SICE, BSD100, DIV2K, and LOL datasets. The SICE dataset comprises five to seven images per sample, with exposure levels ranging from under-exposed to over-exposed. Additionally, BSD100 and DIV2K were used to provide additional statistics for normal-exposed conditions. The results based on the exposure conditions are in R-Table 5.
• The number of calculation steps increases in the order of under-exposed, over-exposed, and normal-exposed images. This is because improving a normal-exposed image is considered a stiff problem. For normal-exposed images, where minimal improvement is required, the dynamics are more stiff than for other images. Specifically, for dynamically stiff ODE problems, the step size taken by the solver is forced down to a small level even in a region where the solution curve is smooth, and these decreased step sizes may require more evaluation steps.
• For a better understanding, we plotted R-Fig.3. In the inference time, CLODE aims to find the optimal solution by minimizing the loss functions, so in R-Fig.3 the y-axis represents the non-reference loss value, the x-axis represents time, and each point indicates a step.
• The dopri5 solver we primarily use is a nonstiff solver. Although a stiff solver (e.g., ode15s) could be employed to shorten inference steps for normal-exposed images, it is not efficient for improving under- or over-exposed images that are close to nonstiff problems. Thanks to the quality suggestion, future research may explore ode solver algorithms that are dynamically adapted based on input image conditions.

W3, Q4: Weakness of denoiser

• First of all, as mentioned in Q4, our method can be improved by adopting existing denoising modules. However by considering the inference speed issue of the model, we designed a compact size of denoiser. For 9pkv’s worry of the effectiveness of denoiser, we cautiously want to assert Table 4 in the main paper. From comparison of case (2c) and (2d), we could find the Noise Removal module (denoiser) raises PSNR 0.94dB and SSIM 0.141. To address the concern clearly, we prepared experiments on diverse aspects (+using existing denoiser) in global rebuttal. Please refer G2 in global rebuttal.

Reference

[1*] Mahmoud Afifi, Marcus A. Brubaker, and Michael S. Brown. "HistoGAN: Controlling Colors of GAN-Generated and Real Images via Color Histograms." In CVPR, 2021.

Thank you for your feedback. My concerns are well addressed, and I will raise my rating to borderline accept.